Account disconnect/delete needs to remove the per-user YouTube refresh
token from the SecretStore. Add Delete(ref) on the file-backed store,
mirroring Put: atomic temp-file+rename, 0600, no-op on an absent ref.
Kept off the read-only ports.SecretStore (Get) — write/delete follow the
existing auth.TokenWriter convention of narrow capability interfaces, so
the youtube adapter's read-only dependency is unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DeleteUser permanently removes a user and all owned data, scoped via
withUser. The users-row ON DELETE CASCADE reaches videos, transcripts,
summaries → sink_deliveries, video_connections, and the user_identities
map (cascades bypass RLS, so a scoped connection still wipes child rows).
summary_actions carries user_id but has NO FK to users (migration 002),
so it is deleted explicitly in the same scoped transaction. Idempotent.
Tapir-side only (decision 2026-06-03): Dex identity is left untouched;
secrets live in the SecretStore and are removed by the account handler.
DisplayName returns the registered name for the account page.
Test proves deletion removes every row for the target user across all
isolated tables (incl. video_connections AND user_identities) and leaves
another user's rows fully intact.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the video_connections table (data-model VIDEO_CONNECTION) with FORCE
row-level security keyed off tapir.current_user_id, identical to migration
003's per-user isolation pattern — connections are user-owned data and must
be isolated at the DB layer, not only by application WHERE clauses.
Store methods (UpsertConnection / ConnectionsForUser / DeleteConnection) all
route through withUser so RLS scopes every access. UpsertConnection is
idempotent on (user_id, provider). The OAuth refresh token never lives here;
token_ref is the opaque SecretStore reference.
Extend the RLS isolation proof to cover video_connections: seeded per user,
included in the deny-all + scoped-read assertions, and added to the
cross-user write-invisibility and survivor checks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the Dex-subject -> tapir-user_id bridge for multi-user Stage 1.
Migration 004 creates user_identities (dex_subject PK, user_id UNIQUE FK
ON DELETE CASCADE). It is intentionally NOT RLS-enabled: it holds no user
data and must be readable BEFORE a user_id is known (the lookup is what
yields the id used to set tapir.current_user_id). RLS here would be a
chicken-and-egg deadlock; data isolation stays on the user-owned tables.
UserBySubject resolves subject -> user_id as a plain pool query (pre-scope,
no withUser). RegisterUser generates the UUID app-side (stdlib crypto/rand,
no new dep) so the forced-RLS WITH CHECK (id = GUC) passes, then inserts the
users row via withUser(newID) and the identity row in the same transaction.
Re-registration of a subject errors with ErrSubjectRegistered.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The isolation proof for ADR-012. embedded-postgres's default user is a
SUPERUSER, which bypasses RLS regardless of FORCE — a test run as it would be
fake-green. So this test creates a dedicated non-superuser role ("app", mirroring
the prod owner tapir), grants it DML, asserts rolsuper is false, and runs every
scoped query as that role.
Assertions: (1) deny-all — with no GUC set, every isolated table returns zero
rows, proving the enforcement path is live, not bypassed; (2) a connection scoped
to user A sees exactly its own one row in every table (and B likewise); (3)
cross-user UPDATE/DELETE aimed at B's rows touches zero rows; (4) B's rows survive
unchanged, verified via the superuser pool.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add Store.withUser(ctx, userID, fn) — a single choke point that BEGINs a tx,
sets the transaction-local GUC tapir.current_user_id via
set_config(..., true), runs fn, and commits. set_config is used over SET LOCAL
because it is parameterizable; the local flag means the value auto-resets on
commit/rollback so a pooled connection never leaks one request's user into the
next.
Route all 9 DB-touching methods through it (Deliver, HasSummary, SeenVideoIDs,
ListSummaries, GetSummaryByVideo, SetAction, ClearAction, ActionsFor, and
UpsertVideo; attachActions flows via ActionsFor). Scoping is now structural —
not a per-query opt-in someone can forget — and arms the migration-003 RLS
policies. Method signatures and existing WHERE clauses are unchanged (defence in
depth; superuser DSNs in existing tests bypass RLS so behaviour is preserved).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Enable AND FORCE row-level security on every user-owned table (users, videos,
transcripts, summaries, summary_actions, sink_deliveries) per ADR-012. Each
policy keys off the per-request GUC tapir.current_user_id; an unset GUC yields
NULL → deny-all (the safe default).
FORCE is load-bearing: the app connects as the table owner (tapir), and owners
bypass RLS unless forced. Without FORCE the policies are dead for the prod user.
sink_deliveries has no user_id; its policy derives ownership from the summary it
belongs to via EXISTS against the GUC, so it is self-contained rather than
silently depending on summaries' own RLS being applied to a subquery.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ANDROID InnerTube player now returns 0 captionTracks (PoToken-gated), so the
watch-page scrape is the real path — but it sent an Android *app* User-Agent,
which makes YouTube serve a page WITHOUT ytInitialPlayerResponse, so captions
came back empty. Result: 87% of a live run skipped as 'no transcript' despite
the videos having captions. A desktop-browser UA returns the player JSON with
captionTracks. Confirmed live: app-UA=0 tracks, browser-UA=1.
The summary_actions table records the maintainer's act on a summary, the
column that makes the Stage-0 headline test ("acts on >=1 summary") queryable
(ui-spec.md §5, ADR-011). This is the gate lanes B/C build on.
- Migration 002: summary_actions (id, user_id, video_id TEXT, action, acted_at)
with a CHECK on action IN ('watched','skipped','saved') and a UNIQUE
(user_id, video_id, action). Per-user isolation: every row carries user_id.
- New actions.go: SetAction (idempotent, atomic watched<->skipped mutual
exclusion in one tx; saved independent), ClearAction, ActionsFor for the list
view, plus Go-side action validation.
- reads.go: additive SummaryRow.Actions, populated by composing ActionsFor
(Go-side, not a SQL join — summaries.video_id is UUID, actions.video_id TEXT).
- embedded-postgres tests: set/clear, mutual exclusion, saved coexistence,
idempotency, invalid rejection, user scoping, read-view surfacing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
NewVideos called search.list at 100 quota units/call. With ~143 channels one
discovery pass = 14,300 units > the 10,000/day YouTube Data API cap, exhausting
the whole day in a single loop (live quotaExceeded).
Switch to playlistItems.list (1 unit/call) against the channel's uploads
playlist. For a standard channel id UCxxxx the uploads playlist is UUxxxx,
derived at zero API cost (uploadsPlaylistID). Non-standard ids fall back to
channels.list (1 unit) to read contentDetails.relatedPlaylists.uploads. Newest-
first ordering and MaxVideosPerSubscription cap preserved.
Side effect: removing search.list also removes the accountDelegationForbidden
error that endpoint threw for one channel — no separate hardening needed.
New per-pass quota: /subscriptions (1) + ~1/channel discovery (143) + any
channels.list fallbacks ≈ 145 units/day, well under 10k. Caption fetch (ADR-010
timedtext) uses no Data API quota.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Data API captions.download endpoint is owner-only: every subscription
video the user does not own returned HTTP 403, producing 0 summaries and a
~150-line error spew in the first live Stage-0 run. Captions-first (ADR-007)
is sound; only the acquisition mechanism was wrong.
FetchTranscript now resolves caption tracks from the InnerTube player
response (ANDROID client, unauthenticated) and GETs the chosen track's
timedtext baseUrl with a plain http.Client — no OAuth token, which can break
the endpoint. The srv3 XML, json3, and legacy <transcript> formats all parse;
non-asr tracks in a preferred language win. Watch-page ytInitialPlayerResponse
scrape is the fallback when InnerTube returns no tracks.
Degrade, don't error (explicit quick-fix): no captionTracks, empty baseUrl, a
non-200 fetch, or an unparseable body yield Source=none, not an error. Only
genuine transport faults error — this kills the spew. OAuth stays on
ListSubscriptions/NewVideos (Data API); only transcript fetch goes unauthed.
Validated live from koala: the ANDROID client returned working baseUrls and
real transcript text for public videos the run identity does not own.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Sink port carries only a Summary, so video title/url/published_at would
never reach the store. UpsertVideo (new file, store.go untouched) persists them
and returns the durable videos.id UUID, idempotent on
(user_id, provider, provider_video_id). The run loop uses that id as v.ID, so
it equals summaries.video_id and SeenVideoIDs dedup survives restarts.
subscription_id stays NULL: the YouTube resource id is not a UUID (Stage 0).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ListSummaries (recent-first, user-scoped, limit) and GetSummaryByVideo
LEFT JOIN videos for title/url/published_at, null-safe when no videos
row exists. Channel mirrors provider for now — channel_title lives on
the not-yet-migrated subscriptions table (data-model.md). New file so it
does not collide with Worker F's concurrent edits to store.go.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements ports.SecretStore over a 0600 JSON file as a stand-in for op/ESO so
the demo runs without live op. Put persists atomically (temp + rename) and
merges; Get returns ErrNotFound for unknown refs so a missing token fails loud.
Behind the port, so swapping to op/ESO later is wiring, not code (ADR-002).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The copied OpenAI-compatible client sent no max_tokens. Thinking models
(qwen3, deepseek-r1) spend their budget on the reasoning trace and return
EMPTY content when max_tokens is unset, which the summarizer treats as an
error. ADR-004 says change Tapir's copy rather than the hyperguild upstream,
so set a generous default (8192) leaving room for both reasoning and output.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements ports.Sink over Postgres (pgx/v5 + pgxpool, DSN from env per
estate convention). This is the primary sink (ADR-003) and the source of
the engine's durable, cross-restart dedup — the in-engine processed map is
process-lifetime only.
- Migrations (golang-migrate, NNN_name.up/down.sql per estate convention,
applied from an embedded FS on New): users, videos, transcripts,
summaries, sink_deliveries. Every user-owned table carries user_id
(Stage-0 per-user isolation promise, data-model.md). summaries has
UNIQUE(user_id, video_id) — at most one summary per video; highlights /
takeaways are jsonb.
- Deliver upserts the summary idempotently on (user_id, video_id)
(ON CONFLICT DO UPDATE) inside one tx with its sink_delivery row. Re-
delivering the same summary updates in place, never duplicates or errors.
- Dedup reads (store methods, not a new port): HasSummary(ctx,userID,
videoID) and SeenVideoIDs(ctx,userID) — both user_id-scoped, so one
user never sees another's videos.
summaries.video_id is intentionally not FK-constrained to videos at Stage 0:
the sink receives only a Summary, so the dedup key stands alone; video-row
persistence is the engine/source's concern, deferred.
Tested against a real in-process Postgres via embedded-postgres (real SQL:
constraints, ON CONFLICT, jsonb, user_id scoping) — no docker, no live
cluster, no creds, fully offline.
Deps: golang-migrate/migrate/v4 and jackc/pgx/v5 (runtime),
fergusstrange/embedded-postgres + stretchr/testify (test-only). go mod tidy
raised the go directive to 1.25.0 (minimum required by the dep graph;
estate elsewhere already runs 1.26.1).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements ports.VideoSource against the YouTube Data API v3:
ListSubscriptions (paginated), NewVideos (recent per channel), and
captions-first FetchTranscript — an absent caption track yields
domain.SourceNone (not an error) per ADR-007, with no audio download
or speech-to-text.
OAuth is written fresh on golang.org/x/oauth2 (ADR-006, distinct from
ingestion's inbound MCP auth); the Google token endpoint is inlined to
avoid the heavy x/oauth2/google dep. The per-connection refresh token is
resolved through the SecretStore port from an opaque TokenSecretRef and
is never stored on the adapter or logged.
Unit-tested against an httptest server + fake SecretStore (no live
googleapis egress): subscriptions list/pagination, new-video detection,
captions present -> Source set, captions absent -> SourceNone no error,
and secret-ref resolution failure surfacing as an error.
oauth2 pinned to v0.30.0 to keep the go directive at 1.23.x (koala
runner), not the v0.36 line that requires a newer toolchain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implement ports.Summarizer in internal/adapters/summarizer. It routes through a
local Primary endpoint first and an optional BYO Fallback, owning the routing
itself (not delegating to llm.Router) so it can record AIProvider, AIModel, and
FallbackUsed on domain.Summary. Prompt asks for JSON {summary, highlights,
takeaways}; the parser tolerates thinking-model fences/reasoning and rejects an
empty summary.
The summarizer is the single egress point for content toward an AI model, so it
enforces the local-first guarantee from ai_routing.feature: with no BYO
configured (nil fallback) there is no external endpoint, so content reaches the
local stack and nowhere else. Tests assert all four scenarios via a fake client.
Model alias is config (TAPIR_SUMMARIZER_MODEL, host/name) — not hardcoded;
docs/homelab-integration.md notes it stays `confirm` and that thinking models
need an explicit max_tokens or they return empty content.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copy hyperguild/ingestion/internal/llm into internal/adapters/llm and own it.
Tapir owes that repo nothing at the dependency level — no module dep added.
Router gives the local-Primary -> BYO-Fallback path needed for ai_routing.feature.
Copied tests rewritten from testify to stdlib testing to keep go.mod
dependency-free (repo has zero deps; acceptance tests are stdlib too).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mathiasandClaude Opus 4.8