From 099b2d4c68d29a183fcfad7d0f9dc92d8c3cd23a Mon Sep 17 00:00:00 2001 From: Mathias Date: Tue, 9 Jun 2026 23:24:11 +0200 Subject: [PATCH] =?UTF-8?q?docs(decisions):=20ADR-021=20=E2=80=94=20shared?= =?UTF-8?q?,=20video-keyed=20transcript=20persistence?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Persist transcripts in a single shared table keyed by (provider, provider_video_id) — public caption content, NOT RLS-scoped — so re-analysis (re-summarize, paste of an already-seen video, a second user with overlapping subs) never re-fetches from YouTube. The avoided cost is the rate-gated, reputation-risky caption fetch (ADR-010/014), not LLM re-summarization, which is why this reopens the transcripts half of the "no global cross-tenant table" rejection while videos stay per-user. Summaries remain RLS-scoped (ADR-012 unchanged). The gate is neither bypassed nor weakened — persistence reduces fetch frequency, not pacing. Annotate the rejected-alternatives row to record the partial reopen. Co-Authored-By: Claude Opus 4.8 (1M context) --- DECISIONS.md | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 61 insertions(+), 1 deletion(-) diff --git a/DECISIONS.md b/DECISIONS.md index 4218f4c..4eeda4b 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -743,6 +743,66 @@ collapse keys off the same window (`App.RecencyWindow=0` → everything inline). --- +## ADR-021 — Persist transcripts as shared, video-keyed public content (re-analysis never re-fetches) + +**Status:** Accepted (2026-06-09). **Reopens the transcripts half of** the "Global cross-tenant +`videos`/`transcripts` table" rejection (data-model.md). **Builds on ADR-007** (captions-first), +**ADR-010/ADR-014** (the per-IP caption rate gate), and **ADR-012** (per-user RLS isolation). + +**Context.** Every summarization fetches the transcript fresh through the caption path, even when +the exact same transcript was fetched moments ago — for the same user re-summarizing, or for a +second user who happens to watch the same video. The caption fetch is the one genuinely scarce, +genuinely risky operation in the system: YouTube's timedtext endpoint is unofficial and per-IP +rate-limited (ADR-010), and tripping it risks the maintainer's Google standing (ADR-014). So the +operation we most want to *avoid repeating* is the one we currently repeat unconditionally. A +transcript is **public content** — the same words YouTube serves to anyone — and carries nothing +user-identifying. The per-user isolation that protects summaries, feeds, and tokens (ADR-012) is +the wrong shape for it: it forces a re-fetch per user for data that is identical across users. + +The original rejection ("Global cross-tenant `videos`/`transcripts` table") bundled videos and +transcripts together and rejected both on the grounds that "at 1–5 users, re-summarizing is +cheaper than the coupling." That reasoning holds for **videos** (per-user feed rows, genuinely +user-scoped) but not for **transcripts**: the cost being avoided is not LLM re-summarization, it +is a *rate-gated, reputation-risky network fetch*, and that cost is paid per re-fetch regardless +of user count. One re-fetch avoided is strictly worth more than the coupling it removes. + +**Decision.** +1. **A single shared `transcripts` table, keyed by the cross-user dedup key + `(provider, provider_video_id)`** — the stable public identity of the video, not Tapir's + internal per-user `videos.id`. Columns: the key, `source` (`captions`/`none`), `language`, + `content`, `fetched_at`. It holds **only public caption content + the video's public id** — + nothing user-identifying — and is therefore **NOT RLS-scoped**: no `user_id`, no policy, no + `FORCE ROW LEVEL SECURITY`. This is the deliberate, single exception to the ADR-012 isolation + boundary, and the only one. +2. **Summarize path becomes read-stored-first.** Have a stored transcript for this video? → + summarize from the stored text, **no caption fetch**. No stored transcript? → fetch *through + the unchanged gate* (ADR-014) → store it → summarize. The gate is neither bypassed nor + weakened; persistence reduces how *often* we reach it, never how *fast*. +3. **De-facto cross-user dedup is the intended behaviour, not a feature with a switch.** Two + users who share a video share the one transcript row. A permanent `source = 'none'` (no + captions) is stored too, so a known-caption-less video is not re-fetched by anyone. A + transient 429 (`SourceRateLimited`) is **never** stored as terminal — it stays a per-user + retry via the existing `transcript_status` backoff (ADR-014), so persistence cannot mask a + rate-limit into a false "no transcript." +4. **Per-user `summaries` stay RLS-scoped (ADR-012 unchanged)** and reference the transcript by + video id. Videos stay per-user. Only transcripts go shared. + +**Consequences.** Re-analysis (re-summarize, different model, paste of an already-seen video, +onboarding of a second user with overlapping subscriptions) never re-touches YouTube — the +primary win, and it *reduces* aggregate caption-gate pressure, reinforcing ADR-010/ADR-014 rather +than straining them. The isolation surface gains exactly one non-RLS table; an isolation test +asserts the boundary is *exactly* there and has not leaked to any user-owned table (this is the +proof the public-content classification was implemented as designed). It also unblocks +multi-model / customizable analysis (re-run analysis on stored text for free) — enabling that is +this ADR's point; building it is separate. + +**Reversibility.** The read-stored-first check is the only behavioural coupling; removing it +restores fetch-every-time. The down-migration recreates the per-user RLS-scoped transcripts shape +(001/003). No user-facing surface depends on cross-user sharing — sharing is the *storage shape*, +never exposed in the UI. + +--- + ## Rejected alternatives Approaches considered during the 2026-06-02 planning + grill session and **deliberately not @@ -757,7 +817,7 @@ maps to the ADR that settles it. | Lifting shared packages into a `brain-common` module | Couples Tapir's release cycle to the monolith for negligible code savings | ADR-004 | | Importing/replicating the filesystem `brain` package | Assumes co-location with the brain git checkout; wrong for a standalone networked service | ADR-005 | | Reusing `ingestion`'s `oauth` package for YouTube/Vimeo | Same name, opposite direction — it's inbound MCP-server auth, not outbound provider OAuth | ADR-006 | -| Global cross-tenant `videos`/`transcripts` table (dedup) | Reintroduces the cross-domain DB coupling the homelab review is removing; at 1–5 users, re-summarizing is cheaper than the coupling | data-model.md | +| Global cross-tenant `videos`/`transcripts` table (dedup) | Reintroduces the cross-domain DB coupling the homelab review is removing; at 1–5 users, re-summarizing is cheaper than the coupling. **Transcripts half reopened by ADR-021** — the avoided cost there is a rate-gated, reputation-risky *caption fetch*, not LLM re-summarization, so it outweighs the coupling; **videos stay per-user.** | data-model.md, **ADR-021** (transcripts only) | | Audio-download + Whisper STT in the core path | ToS-grey, breakage-prone (yt-dlp), contends for koala GPU with the JEPA PoC; captions alone test the core hypothesis | ADR-007 | | Building multi-tenant SaaS / Google OAuth verification now | "Real users soon" was lowered to Future B; SaaS machinery before the Stage 0 self-use gate is the primary documented anti-goal | ADR-008, VISION | | Delegating the S5 reuse spike to an agent swarm | A 1-hour sequential read-and-judge with a single coupled conclusion; orchestration overhead exceeds the work, and it's Diamond-1 judgment the maintainer wanted to own | (process note) |