Commit Graph
43 Commits
Author SHA1 Message Date
mathiasandClaude Opus 4.8 3df0459fed feat(store): order video list by published_at, NULLS LAST
ListVideos now sorts summarized-first, then published_at DESC with undated
videos last, then seen_at DESC as a tiebreak (was seen_at only). Aligns the
list with the recency framing — newest content surfaces first — so the
recency-bounded feed reads coherently (UX review B2).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-08 13:50:30 +02:00
mathiasandClaude Opus 4.8 e7c2e575d3 refactor: remove Dex local-password invite provisioning (ADR-019)
Authentik owns invites now (infra ADR-0001). Delete adapters/dex, the
/invite set-password UI, the tapir invite CLI, the InvitationStore/
DexPasswordCreator ports + App wiring, the invite Templ pages, and the
invite Taskfile target. New users are invited via Authentik, log in via
OIDC, and hit the existing /register gate. invitations table (mig 009)
left in place (append-only; harmless). task check green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 23:01:14 +02:00
mathias 9cd3f7e934 fix(dex): passwordName must match Dex's internal passwordID() — maps non-[a-z0-9-] to '-'
CI / Lint / Test / Vet (push) Failing after 12s
CI / Build & Import (push) Has been skipped
Tapir used human-readable substitutions ('@' -> '-at-', '.' -> '-dot-') when
deriving the Password CR name from an email. Dex's internal passwordID() maps
every non-[a-z0-9-] character to plain '-'. This caused a name mismatch:
Tapir wrote the CR as 'mathias-at-d-ma-dot-be', Dex looked it up as
'mathias-d-ma-be', got not-found, and returned 'Invalid credentials' on every
invite login — while static configmap passwords (a different code path) worked
fine. Diagnosed by adding the email to staticPasswords and confirming login
succeeded, proving the kubernetes CR lookup was the failure point.
2026-06-07 11:37:24 +02:00
mathias c812c71ecc fix(dex): store raw bcrypt hash in Password CR, not base64-encoded
CI / Lint / Test / Vet (push) Successful in 26s
CI / Build & Import (push) Successful in 12s
The original NOTE claimed Dex's kubernetes storage types Hash as []byte,
requiring the bcrypt string to be base64-encoded before storage. This was
wrong: Dex v2.41 stores and compares the hash field as a plain string. The
base64-encoding caused every invite login to fail with 'Invalid credentials'
because Dex passed the base64 bytes (starting with 'J' not '$') directly to
bcrypt. Static passwords in the configmap always used raw bcrypt strings and
worked fine — confirming the dynamic CR encoding was the bug.
2026-06-07 09:28:11 +02:00
mathias 57e29ca06c feat(web): pipeline stats bar, summarized-first sort, Try now button for rate-limited videos
CI / Lint / Test / Vet (push) Successful in 15s
CI / Build & Import (push) Successful in 10s
Three UX improvements for the pending-transcript state:
1. Summarized videos sort to top (ORDER BY (s.id IS NOT NULL) DESC, seen_at DESC)
   so completed summaries are always immediately visible without filtering.
   ListVideos default limit raised from 50 to 500 to show the full backlog.
2. Pipeline stats bar above the video list: '2 summarized · 256 fetching soon · 12
   no captions' — computed from the unfiltered row set, hidden when everything is
   summarized.
3. 'Try now' button on rate-limited cards replaces the passive 'Retrying later'
   chip. POST /v/{id}/retry-now clears rate_limited_at then calls ProcessVideo
   through the shared globalFetchGate — same rate limiting as the scheduler, safe
   under concurrent use.
2026-06-06 19:20:20 +02:00
mathias f1e9739900 feat(store,runner,web): channel unavailability notice (migration 013)
CI / Lint / Test / Vet (push) Successful in 26s
CI / Build & Import (push) Successful in 11s
YouTube channels that 404 on playlist discovery (deleted/private) are now:
1. Wrapped in domain.ErrChannelUnavailable by the YouTube adapter (instead of
   a generic error), so the runner can identify them without string-matching.
2. Stored per-user in channel_errors (migration 013, RLS-guarded) via runner's
   new UpsertChannelError path — removed from the generic Errors counter,
   counted separately as ChannelUnavailable.
3. Shown on the account page under "Unavailable channels" with name, chip-warn
   badge, and first-seen date, so users know why some subscribed channels
   produce no videos.
2026-06-06 10:09:52 +02:00
mathias 940f80899a fix(store): migration 012 — back-fill auto_summarize via RLS bypass
CI / Lint / Test / Vet (push) Successful in 12s
CI / Build & Import (push) Successful in 11s
Migration 011's UPDATE ran without tapir.current_user_id set, so FORCE RLS
blocked all rows and 0 users were updated (skipped_manual=607 in scheduler).
Migration 012 temporarily drops FORCE so the table owner can run the UPDATE,
then restores it.
2026-06-06 10:01:13 +02:00
mathiasandClaude Opus 4.8 5d029a2823 feat(store): default auto_summarize ON for new users (ADR-018)
Migration 011 flips the auto_summarize column default to TRUE and brings
existing rows (maintainer + current registrations) along. Onboarded friends
now get zero-friction discovery: scheduled discovery (ADR-018) both discovers
AND summarizes new videos, so a user's list fills and summarizes itself
instead of presenting an empty list of manual Summarize buttons.

Safe only because the process-wide caption-fetch rate gate (ADR-014 item 2,
prior commit) now exists — auto + scheduled + multi-user would otherwise
self-inflict 429s every cycle. The down migration reverts the default but
intentionally leaves existing rows as-is (no surprise manual regression on
rollback). RegisterUser already lets the column default drive the value, so
no app change is needed; the account-page manual toggle still works.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 23:42:23 +02:00
mathiasandClaude Opus 4.8 149ec2adae feat(store): ListAllUsers for scheduler user enumeration
The in-process scheduler (ADR-018) needs to enumerate every user to run a
discovery pass each. user_identities is the un-RLS'd map; add ListAllUsers as
a plain pool query (no withUser) — the same enumerate-then-act pattern
UserBySubject and the login_events gate query established. Scoping it to a
single user would defeat the point; user_identities carries no RLS by design.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 23:36:44 +02:00
mathiasandClaude Opus 4.8 f5021a8436 feat(youtube): process-wide caption-fetch rate gate (ADR-014 item 2)
ADR-014 item 2 — a single per-egress-IP rate gate shared by every caption
fetch — was specced but only per-video backoff (rate_limited_at) shipped.
Build the real gate now: it is load-bearing once ADR-018 puts auto-summarize
on an in-process schedule across multiple users (all fetches leave one pod's
egress IP, concurrently with live "Summarize" clicks — without a shared gate
that self-inflicts 429s every cycle).

globalFetchGate (golang.org/x/time/rate, default 2s/req burst 1) is consulted
in httpDo before every live outbound fetch — player, watch-page, timedtext —
so the scheduler runners and the web click-path serialise through one limiter
regardless of how many users/goroutines are upstream. The test seam
(a.transport != nil) skips the gate so fakes are not throttled.

TAPIR_FETCH_RATE (Go duration, default 2s, 0 = unlimited) wires SetFetchRate in
cmdRun; the existing per-video backoff stays as the complementary 429 handler.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 23:36:17 +02:00
mathiasandClaude Opus 4.8 561ba79360 feat(cli): tapir report — Stage-0 usage gate query
CI / Lint / Test / Vet (push) Successful in 12s
CI / Build & Import (push) Successful in 10s
ActiveWeeks computes per-user distinct active weeks (reads login_events UNION
acts summary_actions) for the gate (VISION/ADR-016: usage in >=2 distinct
weeks). Because the user-owned tables are FORCE RLS under a non-superuser owner,
a single cross-user query is deny-all; instead it enumerates users from the
un-RLS'd identity map and counts each inside withUser — no privilege escalation,
no policy change.

`tapir report` prints the per-user table and the pass/fail verdict (needs only
TAPIR_DB_DSN). Pure formatter + store query are unit-tested, including the
cross-table shared-week dedup.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 23:46:03 +02:00
mathiasandClaude Opus 4.8 de54cd33b2 feat(store): throttled per-day StampLogin + login_events delete cascade
StampLogin appends one login_events row per user per day via an atomic
INSERT ... SELECT ... WHERE NOT EXISTS, run through withUser so the throttle
probe is itself RLS-scoped to the caller. DeleteUser now deletes login_events
explicitly (no FK = no cascade — the summary_actions footgun, repeated).

Extends the two-user RLS isolation proof and the delete-account proof to cover
login_events, and adds throttle / new-day / user-scoping tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 23:46:03 +02:00
mathiasandClaude Opus 4.8 b070347597 feat(store): add append-only login_events table with forced RLS
Stage-0 usage measurement (VISION/ADR-016): summary_actions captures acts
(watch/skip/save) but not reads. A reader who logs in weekly and clicks
nothing is invisible — for a reading product that return is the signal the
gate ("usage in >=2 distinct weeks") is defined on. login_events records
THAT a user was active, append-only, one row per user per active day.

Per-user isolation via the same GUC-keyed FORCE RLS policy as migration 003.
No FK to users (mirrors summary_actions) — the cascade footgun is handled by
DeleteUser in a later commit. Adds an up/down reversibility test and registers
the table in both truncate helpers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 23:46:03 +02:00
mathiasandClaude Opus 4.8 e44485df16 feat(dex): in-cluster Password CR client for local-password accounts
Writes passwords.dex.coreos.com CRs against the in-cluster Kubernetes
API using the pod's service-account token + cluster CA (no kubectl /
client-go dependency). NewPasswordClient returns ErrNotInCluster off
cluster so the web layer degrades gracefully in dev.

Load-bearing: Dex's kubernetes storage types Password.Hash as []byte,
which k8s JSON-marshals as base64 — so the `hash` field carries the
base64 of the bcrypt string, not the raw string. Storing the raw string
makes Dex's base64-decode-on-login produce garbage and every login fail.

409 -> ErrPasswordExists, 401/403 -> ErrForbidden (RBAC missing) so the
handler can give precise messages. Tested against an httptest TLS server.

bcrypt cost-12 hashing lives in the web handler; golang.org/x/crypto was
already a transitive dep (now promoted in go.sum).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 23:14:08 +02:00
mathiasandClaude Opus 4.8 8b7ef07ba3 feat(store): invitations table + create/peek/claim methods
Stage-1 email onboarding: Mathias mints an invite, the recipient claims
it to set a Dex password. Invitations exist before their user, so the
table carries no user_id FK and is deliberately outside RLS — the
32-byte crypto-random token is the capability (single-use, time-boxed).

ClaimInvitation consumes atomically (UPDATE ... WHERE used_at IS NULL
... RETURNING) so concurrent claims of one token can't both succeed.
PeekInvitation validates the link for the form without consuming it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 23:12:40 +02:00
mathiasandClaude Opus 4.8 1e81965519 feat(store): TranscriptStatus read field + status setter/loader
Surfaces videos.transcript_status (migration 007) on SummaryRow and adds
SetTranscriptStatus / GetTranscriptStatus / RateLimitedVideoIDs.

SetTranscriptStatus is the single choke point for the rate-limit lifecycle:
"rate_limited" stamps rate_limited_at = NOW(), every other status clears it,
so the runner's backoff window and the UI badge read one consistent source.
RateLimitedVideoIDs is the per-pass loader (mirrors SeenVideoIDs) the runner
uses to skip still-throttled videos without re-hitting the caption endpoint.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 22:52:48 +02:00
mathiasandClaude Opus 4.8 689500c85e feat(store): migration 007 — per-video transcript status
CI / Lint / Test / Vet (push) Successful in 20s
CI / Build & Import (push) Successful in 10s
Adds videos.transcript_status (NULL|none|fetched|rate_limited) and
videos.rate_limited_at, so the runner can record a 429 and skip re-fetching a
still-throttled video until a backoff window elapses. Columns inherit the
existing videos RLS policy (migration 003); no policy change needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 22:45:29 +02:00
mathiasandClaude Opus 4.8 4678d473b8 feat(youtube): map caption 429 to SourceRateLimited
The baseUrl fetch mapped every non-200 to SourceNone, recording a 429 as a
permanent "no captions". 429 is the IP being rate-limited, not an absent
transcript. Return SourceRateLimited (still a graceful degrade, no error) so
the runner can retry after a backoff window. Other non-200s (403/404/5xx)
stay SourceNone.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 22:45:29 +02:00
mathiasandClaude Opus 4.8 bdbdce7de1 feat(store): summarization-mode methods + all-videos read
Add the store surface for summarization mode:
- SetAutoSummarize / GetAutoSummarize — per-user auto/manual toggle; an absent
  user reads as manual (the safe default).
- RequestSummarize — queue one video (summarize_requested=TRUE); ErrNotFound
  when the video is absent or not owned (RLS hides another user's row).
- RequestedVideoIDs / ClearSummarizeRequested — the run-loop side: load the
  queued set per pass (mirrors SeenVideoIDs), clear after summarizing.
- ListVideos / GetVideoRow — drive from the videos table LEFT JOIN summaries so
  discovered-but-unsummarized videos appear with empty summary fields. SummaryRow
  gains additive Summarized + SummarizeRequested fields; the summary-only reads
  are untouched.

RLS proof: rls_test.go gains a cross-user "queue B's video" write asserting it
touches zero rows (store-level scoping can't prove this — the test pool is a
superuser that bypasses RLS, same caveat documented there).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 19:26:52 +02:00
mathiasandClaude Opus 4.8 748d5eb0bd feat(store): migration 006 — per-user auto_summarize + per-video summarize_requested
Add the schema for the summarization-mode feature: a per-user auto/manual
toggle (users.auto_summarize, default FALSE = manual) and a per-video manual
queue flag (videos.summarize_requested, default FALSE).

Both columns land on tables that already have ENABLE + FORCE ROW LEVEL
SECURITY (migration 003), so they inherit per-user isolation automatically —
no policy changes needed. auto_summarize is per-user, not global, per ADR-012.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 19:26:43 +02:00
mathiasandClaude Opus 4.8 17d5e8c393 feat(secrets): FileStore.Delete to purge a user's OAuth tokens
Account disconnect/delete needs to remove the per-user YouTube refresh
token from the SecretStore. Add Delete(ref) on the file-backed store,
mirroring Put: atomic temp-file+rename, 0600, no-op on an absent ref.

Kept off the read-only ports.SecretStore (Get) — write/delete follow the
existing auth.TokenWriter convention of narrow capability interfaces, so
the youtube adapter's read-only dependency is unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 16:50:20 +02:00
mathiasandClaude Opus 4.8 c7624d97fe feat(store): DeleteUser + DisplayName for account management
DeleteUser permanently removes a user and all owned data, scoped via
withUser. The users-row ON DELETE CASCADE reaches videos, transcripts,
summaries → sink_deliveries, video_connections, and the user_identities
map (cascades bypass RLS, so a scoped connection still wipes child rows).
summary_actions carries user_id but has NO FK to users (migration 002),
so it is deleted explicitly in the same scoped transaction. Idempotent.

Tapir-side only (decision 2026-06-03): Dex identity is left untouched;
secrets live in the SecretStore and are removed by the account handler.

DisplayName returns the registered name for the account page.

Test proves deletion removes every row for the target user across all
isolated tables (incl. video_connections AND user_identities) and leaves
another user's rows fully intact.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 16:49:50 +02:00
mathiasandClaude Opus 4.8 0c9531a9b8 feat(store): video_connections table + RLS + connection store methods
Add the video_connections table (data-model VIDEO_CONNECTION) with FORCE
row-level security keyed off tapir.current_user_id, identical to migration
003's per-user isolation pattern — connections are user-owned data and must
be isolated at the DB layer, not only by application WHERE clauses.

Store methods (UpsertConnection / ConnectionsForUser / DeleteConnection) all
route through withUser so RLS scopes every access. UpsertConnection is
idempotent on (user_id, provider). The OAuth refresh token never lives here;
token_ref is the opaque SecretStore reference.

Extend the RLS isolation proof to cover video_connections: seeded per user,
included in the deny-all + scoped-read assertions, and added to the
cross-user write-invisibility and survivor checks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 16:12:34 +02:00
mathiasandClaude Opus 4.8 f396e01243 feat(store): user_identities map + UserBySubject/RegisterUser (ADR-012)
Add the Dex-subject -> tapir-user_id bridge for multi-user Stage 1.

Migration 004 creates user_identities (dex_subject PK, user_id UNIQUE FK
ON DELETE CASCADE). It is intentionally NOT RLS-enabled: it holds no user
data and must be readable BEFORE a user_id is known (the lookup is what
yields the id used to set tapir.current_user_id). RLS here would be a
chicken-and-egg deadlock; data isolation stays on the user-owned tables.

UserBySubject resolves subject -> user_id as a plain pool query (pre-scope,
no withUser). RegisterUser generates the UUID app-side (stdlib crypto/rand,
no new dep) so the forced-RLS WITH CHECK (id = GUC) passes, then inserts the
users row via withUser(newID) and the identity row in the same transaction.
Re-registration of a subject errors with ErrSubjectRegistered.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 15:53:34 +02:00
mathias 9bff59037f merge: video embed on detail page (Worker R2)
CI / Lint / Test / Vet (push) Successful in 13s
CI / Build & Import (push) Successful in 11s
CI / Mirror to GitHub (push) Failing after 2s
# Conflicts:
#	internal/web/views_templ.go
2026-06-03 15:38:31 +02:00
mathiasandClaude Opus 4.8 f28fdc0292 test(store): prove RLS isolation as a non-superuser role
The isolation proof for ADR-012. embedded-postgres's default user is a
SUPERUSER, which bypasses RLS regardless of FORCE — a test run as it would be
fake-green. So this test creates a dedicated non-superuser role ("app", mirroring
the prod owner tapir), grants it DML, asserts rolsuper is false, and runs every
scoped query as that role.

Assertions: (1) deny-all — with no GUC set, every isolated table returns zero
rows, proving the enforcement path is live, not bypassed; (2) a connection scoped
to user A sees exactly its own one row in every table (and B likewise); (3)
cross-user UPDATE/DELETE aimed at B's rows touches zero rows; (4) B's rows survive
unchanged, verified via the superuser pool.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 15:15:58 +02:00
mathiasandClaude Opus 4.8 7b139c2cd7 feat(store): route all DB access through withUser for structural scoping
Add Store.withUser(ctx, userID, fn) — a single choke point that BEGINs a tx,
sets the transaction-local GUC tapir.current_user_id via
set_config(..., true), runs fn, and commits. set_config is used over SET LOCAL
because it is parameterizable; the local flag means the value auto-resets on
commit/rollback so a pooled connection never leaks one request's user into the
next.

Route all 9 DB-touching methods through it (Deliver, HasSummary, SeenVideoIDs,
ListSummaries, GetSummaryByVideo, SetAction, ClearAction, ActionsFor, and
UpsertVideo; attachActions flows via ActionsFor). Scoping is now structural —
not a per-query opt-in someone can forget — and arms the migration-003 RLS
policies. Method signatures and existing WHERE clauses are unchanged (defence in
depth; superuser DSNs in existing tests bypass RLS so behaviour is preserved).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 15:15:52 +02:00
mathiasandClaude Opus 4.8 6775e5f53d feat(store): migration 003 — enforce per-user isolation via forced RLS
Enable AND FORCE row-level security on every user-owned table (users, videos,
transcripts, summaries, summary_actions, sink_deliveries) per ADR-012. Each
policy keys off the per-request GUC tapir.current_user_id; an unset GUC yields
NULL → deny-all (the safe default).

FORCE is load-bearing: the app connects as the table owner (tapir), and owners
bypass RLS unless forced. Without FORCE the policies are dead for the prod user.

sink_deliveries has no user_id; its policy derives ownership from the summary it
belongs to via EXISTS against the GUC, so it is self-contained rather than
silently depending on summaries' own RLS being applied to a subquery.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 15:15:44 +02:00
mathias 897a21a1d6 feat(store): expose provider_video_id on SummaryRow
Read-only addition (column + field + scan) so the web layer can build a
video embed URL. No write-path or restructuring.
2026-06-03 15:10:29 +02:00
mathias 7589c09201 fix(youtube): use a browser UA for the watch-page caption scrape
CI / Lint / Test / Vet (push) Successful in 9s
CI / Build & Import (push) Successful in 9s
CI / Mirror to GitHub (push) Failing after 3s
The ANDROID InnerTube player now returns 0 captionTracks (PoToken-gated), so the
watch-page scrape is the real path — but it sent an Android *app* User-Agent,
which makes YouTube serve a page WITHOUT ytInitialPlayerResponse, so captions
came back empty. Result: 87% of a live run skipped as 'no transcript' despite
the videos having captions. A desktop-browser UA returns the player JSON with
captionTracks. Confirmed live: app-UA=0 tracks, browser-UA=1.
2026-06-03 09:35:36 +02:00
mathiasandClaude Opus 4.8 a032c324f6 feat(store): persist summary actions (watch/skip/save) — Stage-0 metric
CI / Lint / Test / Vet (push) Successful in 6s
CI / Build & Import (push) Failing after 1s
CI / Mirror to GitHub (push) Has been skipped
The summary_actions table records the maintainer's act on a summary, the
column that makes the Stage-0 headline test ("acts on >=1 summary") queryable
(ui-spec.md §5, ADR-011). This is the gate lanes B/C build on.

- Migration 002: summary_actions (id, user_id, video_id TEXT, action, acted_at)
  with a CHECK on action IN ('watched','skipped','saved') and a UNIQUE
  (user_id, video_id, action). Per-user isolation: every row carries user_id.
- New actions.go: SetAction (idempotent, atomic watched<->skipped mutual
  exclusion in one tx; saved independent), ClearAction, ActionsFor for the list
  view, plus Go-side action validation.
- reads.go: additive SummaryRow.Actions, populated by composing ActionsFor
  (Go-side, not a SQL join — summaries.video_id is UUID, actions.video_id TEXT).
- embedded-postgres tests: set/clear, mutual exclusion, saved coexistence,
  idempotency, invalid rejection, user scoping, read-view surfacing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 23:36:55 +02:00
mathiasandClaude Opus 4.8 af1c163a11 fix(youtube): discover via uploads playlist, not search.list (100x cheaper)
CI / Lint / Test / Vet (push) Successful in 5s
CI / Build & Import (push) Failing after 1s
CI / Mirror to GitHub (push) Has been skipped
NewVideos called search.list at 100 quota units/call. With ~143 channels one
discovery pass = 14,300 units > the 10,000/day YouTube Data API cap, exhausting
the whole day in a single loop (live quotaExceeded).

Switch to playlistItems.list (1 unit/call) against the channel's uploads
playlist. For a standard channel id UCxxxx the uploads playlist is UUxxxx,
derived at zero API cost (uploadsPlaylistID). Non-standard ids fall back to
channels.list (1 unit) to read contentDetails.relatedPlaylists.uploads. Newest-
first ordering and MaxVideosPerSubscription cap preserved.

Side effect: removing search.list also removes the accountDelegationForbidden
error that endpoint threw for one channel — no separate hardening needed.

New per-pass quota: /subscriptions (1) + ~1/channel discovery (143) + any
channels.list fallbacks ≈ 145 units/day, well under 10k. Caption fetch (ADR-010
timedtext) uses no Data API quota.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 23:15:27 +02:00
mathiasandClaude Opus 4.8 8b14ef4add feat(youtube): acquire captions via player/timedtext baseUrl (ADR-010)
CI / Lint / Test / Vet (push) Successful in 6s
CI / Build & Import (push) Failing after 1s
CI / Mirror to GitHub (push) Has been skipped
The Data API captions.download endpoint is owner-only: every subscription
video the user does not own returned HTTP 403, producing 0 summaries and a
~150-line error spew in the first live Stage-0 run. Captions-first (ADR-007)
is sound; only the acquisition mechanism was wrong.

FetchTranscript now resolves caption tracks from the InnerTube player
response (ANDROID client, unauthenticated) and GETs the chosen track's
timedtext baseUrl with a plain http.Client — no OAuth token, which can break
the endpoint. The srv3 XML, json3, and legacy <transcript> formats all parse;
non-asr tracks in a preferred language win. Watch-page ytInitialPlayerResponse
scrape is the fallback when InnerTube returns no tracks.

Degrade, don't error (explicit quick-fix): no captionTracks, empty baseUrl, a
non-200 fetch, or an unparseable body yield Source=none, not an error. Only
genuine transport faults error — this kills the spew. OAuth stays on
ListSubscriptions/NewVideos (Data API); only transcript fetch goes unauthed.

Validated live from koala: the ANDROID client returned working baseUrls and
real transcript text for public videos the run identity does not own.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 23:08:13 +02:00
mathias a38002df80 merge: demo wiring — tapir auth/run + config (Worker F, agent/demo-wiring)
CI / Lint / Test / Vet (push) Successful in 6s
CI / Build & Import (push) Failing after 1s
CI / Mirror to GitHub (push) Has been skipped
# Conflicts:
#	cmd/tapir/main.go
2026-06-02 21:29:48 +02:00
mathiasandClaude Opus 4.8 6beb779df6 feat(store): UpsertVideo to persist video metadata
The Sink port carries only a Summary, so video title/url/published_at would
never reach the store. UpsertVideo (new file, store.go untouched) persists them
and returns the durable videos.id UUID, idempotent on
(user_id, provider, provider_video_id). The run loop uses that id as v.ID, so
it equals summaries.video_id and SeenVideoIDs dedup survives restarts.
subscription_id stays NULL: the YouTube resource id is not a UUID (Stage 0).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 21:00:04 +02:00
mathiasandClaude Opus 4.8 c153ff35ce feat(store): add read methods for stored summaries
ListSummaries (recent-first, user-scoped, limit) and GetSummaryByVideo
LEFT JOIN videos for title/url/published_at, null-safe when no videos
row exists. Channel mirrors provider for now — channel_title lives on
the not-yet-migrated subscriptions table (data-model.md). New file so it
does not collide with Worker F's concurrent edits to store.go.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 20:58:04 +02:00
mathiasandClaude Opus 4.8 c93b433aaf feat(secrets): file-backed SecretStore for Stage-0
Implements ports.SecretStore over a 0600 JSON file as a stand-in for op/ESO so
the demo runs without live op. Put persists atomically (temp + rename) and
merges; Get returns ErrNotFound for unknown refs so a missing token fails loud.
Behind the port, so swapping to op/ESO later is wiring, not code (ADR-002).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 20:57:44 +02:00
mathiasandClaude Opus 4.8 c40b46b661 fix(llm): send generous max_tokens on every request
The copied OpenAI-compatible client sent no max_tokens. Thinking models
(qwen3, deepseek-r1) spend their budget on the reasoning trace and return
EMPTY content when max_tokens is unset, which the summarizer treats as an
error. ADR-004 says change Tapir's copy rather than the hyperguild upstream,
so set a generous default (8192) leaving room for both reasoning and output.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 20:55:49 +02:00
mathiasandClaude Opus 4.8 2695b5d91e feat(adapters): add Postgres store sink with durable dedup
CI / Lint / Test / Vet (push) Successful in 10s
CI / Build & Import (push) Failing after 1s
CI / Mirror to GitHub (push) Has been skipped
Implements ports.Sink over Postgres (pgx/v5 + pgxpool, DSN from env per
estate convention). This is the primary sink (ADR-003) and the source of
the engine's durable, cross-restart dedup — the in-engine processed map is
process-lifetime only.

- Migrations (golang-migrate, NNN_name.up/down.sql per estate convention,
  applied from an embedded FS on New): users, videos, transcripts,
  summaries, sink_deliveries. Every user-owned table carries user_id
  (Stage-0 per-user isolation promise, data-model.md). summaries has
  UNIQUE(user_id, video_id) — at most one summary per video; highlights /
  takeaways are jsonb.
- Deliver upserts the summary idempotently on (user_id, video_id)
  (ON CONFLICT DO UPDATE) inside one tx with its sink_delivery row. Re-
  delivering the same summary updates in place, never duplicates or errors.
- Dedup reads (store methods, not a new port): HasSummary(ctx,userID,
  videoID) and SeenVideoIDs(ctx,userID) — both user_id-scoped, so one
  user never sees another's videos.

summaries.video_id is intentionally not FK-constrained to videos at Stage 0:
the sink receives only a Summary, so the dedup key stands alone; video-row
persistence is the engine/source's concern, deferred.

Tested against a real in-process Postgres via embedded-postgres (real SQL:
constraints, ON CONFLICT, jsonb, user_id scoping) — no docker, no live
cluster, no creds, fully offline.

Deps: golang-migrate/migrate/v4 and jackc/pgx/v5 (runtime),
fergusstrange/embedded-postgres + stretchr/testify (test-only). go mod tidy
raised the go directive to 1.25.0 (minimum required by the dep graph;
estate elsewhere already runs 1.26.1).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 20:04:04 +02:00
mathias 0af2991d79 merge: captions-first YouTube VideoSource (Worker C, agent/youtube-source)
CI / Lint / Test / Vet (push) Successful in 12s
CI / Build & Import (push) Failing after 0s
CI / Mirror to GitHub (push) Has been skipped
2026-06-02 17:22:51 +02:00
mathiasandClaude Opus 4.8 9a7ba3a346 feat(adapters): add captions-first YouTube VideoSource
Implements ports.VideoSource against the YouTube Data API v3:
ListSubscriptions (paginated), NewVideos (recent per channel), and
captions-first FetchTranscript — an absent caption track yields
domain.SourceNone (not an error) per ADR-007, with no audio download
or speech-to-text.

OAuth is written fresh on golang.org/x/oauth2 (ADR-006, distinct from
ingestion's inbound MCP auth); the Google token endpoint is inlined to
avoid the heavy x/oauth2/google dep. The per-connection refresh token is
resolved through the SecretStore port from an opaque TokenSecretRef and
is never stored on the adapter or logged.

Unit-tested against an httptest server + fake SecretStore (no live
googleapis egress): subscriptions list/pagination, new-video detection,
captions present -> Source set, captions absent -> SourceNone no error,
and secret-ref resolution failure surfacing as an error.

oauth2 pinned to v0.30.0 to keep the go directive at 1.23.x (koala
runner), not the v0.36 line that requires a newer toolchain.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 17:06:40 +02:00
mathiasandClaude Opus 4.8 6da9d61e63 feat(adapters): add Summarizer backed by local-first llm routing
Implement ports.Summarizer in internal/adapters/summarizer. It routes through a
local Primary endpoint first and an optional BYO Fallback, owning the routing
itself (not delegating to llm.Router) so it can record AIProvider, AIModel, and
FallbackUsed on domain.Summary. Prompt asks for JSON {summary, highlights,
takeaways}; the parser tolerates thinking-model fences/reasoning and rejects an
empty summary.

The summarizer is the single egress point for content toward an AI model, so it
enforces the local-first guarantee from ai_routing.feature: with no BYO
configured (nil fallback) there is no external endpoint, so content reaches the
local stack and nowhere else. Tests assert all four scenarios via a fake client.

Model alias is config (TAPIR_SUMMARIZER_MODEL, host/name) — not hardcoded;
docs/homelab-integration.md notes it stays `confirm` and that thinking models
need an explicit max_tokens or they return empty content.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 17:04:29 +02:00
mathiasandClaude Opus 4.8 0aeb3aa99e feat(adapters): copy stdlib llm package (Client + Router) per ADR-004
Copy hyperguild/ingestion/internal/llm into internal/adapters/llm and own it.
Tapir owes that repo nothing at the dependency level — no module dep added.
Router gives the local-Primary -> BYO-Fallback path needed for ai_routing.feature.

Copied tests rewritten from testify to stdlib testing to keep go.mod
dependency-free (repo has zero deps; acceptance tests are stdlib too).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 17:01:56 +02:00