1825 lines
125 KiB
Markdown
1825 lines
125 KiB
Markdown
# Architecture Decision Records — Tapir
|
||
|
||
Lightweight ADRs. Each records a decision, its context, and its consequences at the time
|
||
it was made. Decisions are **append-only**: to reverse one, add a new ADR that supersedes
|
||
it (note the supersession in both). This is the reversibility record the VISION guardrails
|
||
refer to.
|
||
|
||
Status values: `Accepted` · `Superseded by ADR-NNN` · `Proposed`.
|
||
|
||
---
|
||
|
||
## ADR-001 — Go, not Python
|
||
|
||
**Status:** Accepted (2026-06-02)
|
||
|
||
**Context.** The first design draft assumed a Python/FastAPI/arq stack. The homelab is a
|
||
Go estate (`template-go-web`, `template-go-agent`, `mcp-chassis`, `hyperguild/ingestion`).
|
||
A Python service would be the only one of its kind, outside every shared convention, chassis,
|
||
and CI pattern.
|
||
|
||
**Decision.** Tapir is written in Go. Python is used only where it makes no sense not to
|
||
(e.g. if an ML component later genuinely requires it — none does today).
|
||
|
||
**Consequences.** Reuses homelab Go conventions and the `mcp-chassis` auth pattern if/when
|
||
Tapir exposes an MCP surface. Closes the door on the Python async ecosystem (acceptable —
|
||
Go's concurrency model fits the watcher/worker shape).
|
||
|
||
---
|
||
|
||
## ADR-002 — No Supabase; reuse existing Dex / ESO / Postgres conventions
|
||
|
||
**Status:** Accepted (2026-06-02). The "Tapir does not write to the shared identity provider"
|
||
implication is **partially superseded by ADR-017** (invite flow writes Dex Password CRs); the
|
||
no-Supabase / reuse-existing-primitives decision stands.
|
||
|
||
**Context.** The draft proposed self-hosted Supabase for auth + RLS + secrets (Vault). The
|
||
homelab already runs **Dex** (OIDC), **ESO + 1Password** (secrets), and a **postgres18**
|
||
instance that the May-2026 architecture review is actively *de*-coupling by blast radius.
|
||
Supabase would bundle a second auth system, a second secrets store, and its own Postgres —
|
||
three duplications of things just consolidated.
|
||
|
||
**Decision.** No Supabase. Auth, secrets, and persistence reuse the homelab's existing
|
||
primitives: Dex for identity (when external identity is needed), ESO + 1Password for secret
|
||
custody, Postgres with per-tenant role/grant and the `tenant=` namespace label scheme for
|
||
isolation when Future B/C arrives.
|
||
|
||
**Consequences.** No new infra to operate or back up. Multi-tenancy uses the architecture
|
||
review's SC7/P6 primitives rather than Supabase RLS. For Stage 0 (single user) most of this
|
||
is dormant; it activates at Stage 1.
|
||
|
||
---
|
||
|
||
## ADR-003 — Standalone-first; brain is one sink behind an interface
|
||
|
||
**Status:** Accepted (2026-06-02)
|
||
|
||
**Context.** Tapir was initially framed as both a standalone service and a brain-ingestion
|
||
pipeline. The maintainer clarified that **standalone is the more important** of the two.
|
||
|
||
**Decision.** Tapir is a standalone service. Delivery of summaries is via a **Sink
|
||
interface** with multiple implementations. The user's own store is the primary sink; the
|
||
brain is *one optional sink among others*. The engine does not know or care which sinks are
|
||
attached.
|
||
|
||
**Consequences.** Brain ingestion can never dictate the core architecture (a named drift
|
||
signal in VISION). Homelab mode is "the brain sink is enabled"; it is not a separate build.
|
||
|
||
---
|
||
|
||
## ADR-004 — Copy the `llm` package from `hyperguild/ingestion`; do not lift to a shared lib
|
||
|
||
**Status:** Accepted (2026-06-02)
|
||
|
||
**Context.** Spike S5 (`infra/docs/superpowers/handoffs/2026-06-02-video-adapter-placement.md`)
|
||
examined what Tapir could reuse from `hyperguild/ingestion`. The `internal/llm` package
|
||
(`Client` + `Router`) is stdlib-only and its `Router` implements exactly the
|
||
Primary→Fallback pattern Tapir needs for local-first → BYO-AI. It is ~3.5 KB and stable.
|
||
|
||
**Decision.** **Copy** the `llm` package into Tapir and own it. Do not lift it into a shared
|
||
module. Lifting would couple Tapir's release cycle to the monolith to save ~100 lines —
|
||
a bad trade against the "standalone owes nothing" bias.
|
||
|
||
**Consequences.** Minor duplication. Tapir owes nothing to `hyperguild/ingestion` at the
|
||
dependency level. Primary = local stack via LiteLLM/piguard alias; Fallback = user BYO key.
|
||
|
||
---
|
||
|
||
## ADR-005 — Brain sink is an HTTP brain-mcp client, not the filesystem `brain` package
|
||
|
||
**Status:** Accepted (2026-06-02)
|
||
|
||
**Context.** `hyperguild/ingestion`'s `internal/brain` package manipulates the brain git
|
||
checkout **on the local filesystem** (`os.WriteFile` into `brain/wiki/...`). That assumes
|
||
co-location with the brain repo — fine for the monolith, wrong for a standalone service
|
||
where brain is one network-reached sink.
|
||
|
||
**Decision.** Tapir's brain sink is a thin **HTTP adapter calling brain-mcp's `brain_ingest`
|
||
tool**. It does not import or replicate the filesystem `brain` package.
|
||
|
||
**Consequences.** The brain sink works regardless of where Tapir runs. Adds a dependency on
|
||
brain-mcp availability when that sink is enabled (acceptable; sinks fail independently).
|
||
|
||
---
|
||
|
||
## ADR-006 — Outbound OAuth (YouTube/Vimeo) written fresh
|
||
|
||
**Status:** Accepted (2026-06-02)
|
||
|
||
**Context.** `hyperguild/ingestion`'s `internal/oauth` package is the MCP **server's
|
||
inbound** auth (client_credentials, authenticating claude.ai). Tapir needs **outbound**
|
||
authorization-code OAuth to YouTube/Vimeo with per-user token custody — a different concern
|
||
that happens to share a name.
|
||
|
||
**Decision.** Write the YouTube/Vimeo OAuth client fresh using `golang.org/x/oauth2`. Store
|
||
per-user refresh tokens via the homelab's ESO/1Password convention (ADR-002), not a new
|
||
secrets system.
|
||
|
||
**Consequences.** No reuse from the monolith here. Token custody is the highest-value
|
||
secret surface (see VISION Stage 2); it gets the existing, vetted secret path.
|
||
|
||
---
|
||
|
||
## ADR-007 — Captions-first; audio-download + speech-to-text deferred
|
||
|
||
**Status:** Accepted (2026-06-02). Acquisition mechanism superseded by ADR-010 (Data API
|
||
`captions.download` → player/timedtext baseUrl); captions-first stance and STT deferral stand
|
||
**for YouTube**. STT deferral superseded **for user-supplied audio only** by ADR-033; the
|
||
YouTube audio-download rejection is unchanged.
|
||
|
||
**Context.** YouTube's Data API does not expose transcripts. Options: official captions
|
||
(clean, limited coverage) vs. audio download + local Whisper (broad coverage, ToS-grey,
|
||
GPU-contending with the JEPA PoC on koala, breakage-prone via yt-dlp).
|
||
|
||
**Decision.** The core path is **captions-only**. When a video has no usable transcript,
|
||
Tapir records "no transcript" and moves on. Audio-download + speech-to-text is a deferred,
|
||
clearly-bounded *optional* component, not part of the core path or Stage 0.
|
||
|
||
**Consequences.** Some videos won't be summarized at Stage 0 — accepted. The "is this
|
||
useful to me?" hypothesis is testable on captioned videos alone. Avoids GPU contention and
|
||
ToS risk in the validated path.
|
||
|
||
---
|
||
|
||
## ADR-008 — Future C (public multi-tenant SaaS) deferred behind the Stage 0 gate
|
||
|
||
**Status:** Accepted (2026-06-02)
|
||
|
||
**Context.** "Real users soon" was initially asserted, which would load Google OAuth
|
||
verification, billing, and hardened multi-tenant custody early. Lowered to Future B (1–5
|
||
trusted users), with the maintainer as the real first customer.
|
||
|
||
**Decision.** Build for Stage 0 then Stage 1 (Future B). Public SaaS machinery (sign-up,
|
||
billing, Google OAuth app verification at scale) is **not built** until the Stage 0
|
||
self-use gate passes **and** real external demand appears.
|
||
|
||
**Consequences.** Smaller, faster first build. The staged Definition of Success in VISION
|
||
governs advancement. Reversible: if demand appears, a new ADR opens the Future C scope.
|
||
|
||
---
|
||
|
||
## ADR-009 — Trunk-Based Development
|
||
|
||
**Status:** Accepted (2026-06-02)
|
||
|
||
**Context.** Platform-wide convention (homelab architecture review invariant; gitea-mcp #27):
|
||
commit directly to `main`, one logical change per commit, every commit deployable.
|
||
|
||
**Decision.** Tapir follows TBD. Commit directly to `main`. No feature branches or PRs for
|
||
solo/agent work; short-lived `agent/<desc>` branches only when parallel agents are active on
|
||
the repo simultaneously. CI is the quality gate, not branch protection.
|
||
|
||
**Consequences.** Consistent with the rest of the estate. Depends on the direct-to-main write
|
||
path tracked in gitea-mcp #35 (item #1).
|
||
|
||
---
|
||
|
||
## ADR-010 — Third-party caption acquisition via the timedtext/player baseUrl
|
||
|
||
**Status:** Accepted (2026-06-02)
|
||
|
||
**Context.** ADR-007 settled *captions-first*. The first build used the YouTube Data API
|
||
`captions.list` + `captions.download` endpoints to acquire them. A live Stage-0 run proved that
|
||
`captions.download` is **owner-only**: it requires the OAuth identity to own the video, so every
|
||
subscription video the user does *not* own returns HTTP 403. Result: 0 summaries produced and a
|
||
~150-line error spew. The captions-first decision is sound; only the *acquisition mechanism* was
|
||
wrong.
|
||
|
||
**Decision.** Acquire captions from the **player response + timedtext baseUrl**, not the Data API
|
||
`captions` endpoints:
|
||
|
||
1. `POST https://www.youtube.com/youtubei/v1/player` with an **InnerTube `ANDROID` client
|
||
context** (no API key, no OAuth). Read
|
||
`captions.playerCaptionsTracklistRenderer.captionTracks[]`. Each track carries `baseUrl`,
|
||
`languageCode`, and `kind` (`"asr"` = auto-generated).
|
||
2. Select by `PreferredLanguages`, preferring non-`asr` when both exist.
|
||
3. **GET the track's `baseUrl` unauthenticated** (plain `http.Client`, no OAuth token attached —
|
||
the token can break the timedtext endpoint). The ANDROID `baseUrl` is pinned to `fmt=srv3`
|
||
(timedtext XML); the parser also accepts `json3` and the legacy `<transcript>` XML.
|
||
|
||
A **watch-page scrape** of `ytInitialPlayerResponse` is the documented fallback if InnerTube
|
||
returns no `captionTracks`.
|
||
|
||
**Live validation (from koala, 2026-06-02):** the `ANDROID` InnerTube client returned 6
|
||
`captionTracks` with working `baseUrl`s for a public video the run identity does not own, and the
|
||
unauthenticated `baseUrl` GET returned real transcript text. `ANDROID` is the client of record
|
||
(historically returns baseUrls without a PoToken). The `WEB` client and watch-page scrape are
|
||
fallbacks.
|
||
|
||
**Consequences.**
|
||
|
||
- Works for **any public captioned video**, not just owned ones — this is the fix for the 403 wall.
|
||
- **ToS-grey:** `youtubei`/`timedtext` are unofficial endpoints. They can break when Google shifts
|
||
InnerTube client requirements or introduces PoToken gating. Mitigation: degrade, never error — a
|
||
missing/empty/403/unparseable caption yields `domain.Transcript{Source: SourceNone}`, so a future
|
||
breakage produces "no transcript" rather than a crash or error spew. Only genuine transport
|
||
(network) faults error.
|
||
- **No OAuth** is needed for the transcript fetch. OAuth is still required for `ListSubscriptions`
|
||
and `NewVideos` (Data API) — only the transcript path goes unauthenticated.
|
||
- **Still no Whisper.** Speech-to-text stays deferred (ADR-007 unchanged).
|
||
|
||
**Supersedes:** the *acquisition mechanism* of ADR-007 (Data API `captions.download` →
|
||
player/timedtext baseUrl) only. ADR-007's captions-first stance and the STT deferral stand.
|
||
|
||
---
|
||
|
||
## ADR-011 — Web read-surface at Stage 0: Dex authn (single-user authz), action signal, public ingress + GitOps
|
||
|
||
**Status:** Accepted (2026-06-02)
|
||
|
||
**Context.** The Stage-0 CLI (`list`/`show`) reads summaries but doesn't capture the Stage-0
|
||
*headline test* — whether the maintainer **acts** on a summary (watches/skips/saves because of
|
||
it). A browser surface is wanted, and the maintainer chose to deploy it properly:
|
||
**`tapir.d-ma.be` via the homelab ingress, full k3s/Flux GitOps**, with **Dex login from the
|
||
start** rather than a Tailscale-only no-auth dev page. The data-model note says "auth is
|
||
dormant at Stage 0", so logging in early is a deliberate deviation worth recording.
|
||
|
||
**Decision.**
|
||
1. Add a Stage-0 web reader (`tapir serve`, HTMX+Templ) over the existing `store` — a new
|
||
transport, not a core change (ADR-003). Pages: summary list + full view + **watch/skip/save**
|
||
actions recorded in a new `summary_actions` table. The action signal instruments the
|
||
Stage-0 success metric directly.
|
||
2. **Authentication** via Dex OIDC (ADR-002 already names Dex for identity). **Authorization
|
||
stays trivial**: an allowlist of one subject (the maintainer). NO user CRUD, NO per-tenant
|
||
isolation — that authz/isolation work is the real Stage-1/2 line and stays deferred. The
|
||
distinction (authn now, authz/tenancy later) is what keeps this honest with VISION.
|
||
3. **Deploy** at `tapir.d-ma.be` via the existing homelab pattern: gitea CI (buildah) →
|
||
registry → **Flux** reconciling manifests in `mathias/infra` `k3s/apps/tapir/`; secrets via
|
||
**ESO + 1Password**; `postgres18` reached in-cluster. This is application of existing
|
||
convention, not a new infra decision.
|
||
|
||
**Consequences.** The maintainer gets a real, authenticated reading surface and the Stage-0
|
||
metric becomes queryable. A browser session-login path now exists (distinct from
|
||
`mcp-chassis`'s inbound Bearer-JWT validation — not the same code). Wiring login early adds a
|
||
Dex static-client registration + ingress/TLS as prerequisites. If multiple users ever arrive,
|
||
authorization/isolation is a new ADR (Stage 1) — this one deliberately does not build it.
|
||
Full spec: `docs/ui-spec.md`.
|
||
|
||
---
|
||
|
||
## ADR-012 — Open Stage 1: multi-user with enforced isolation (RLS) in the same slice
|
||
|
||
**Status:** Accepted (2026-06-03)
|
||
|
||
**Context.** ADR-011 shipped a single-user web reader with an allowlist of one, deferring
|
||
multi-user authz/isolation to "a new ADR (Stage 1)". The maintainer has chosen to **open
|
||
Stage 1 now** — multi-user registration, per-user YouTube connect, and account management —
|
||
ahead of the formal Stage-0 self-use gate. VISION's hard invariant ("data isolation is a
|
||
promise, not a feature flag … holds from the first user") and the drift signal ("building
|
||
Stage 1+ machinery before the gate") make one thing non-negotiable: **multi-user features
|
||
must not ship before isolation is enforced.**
|
||
|
||
**Decision.**
|
||
1. **Open Stage 1.** Build registration (explicit, not just-in-time: a Dex-authenticated
|
||
subject with no `users` row completes a registration step that creates it), per-user
|
||
web-initiated YouTube OAuth connect (distinct from the CLI `tapir auth`), and account
|
||
management (view / disconnect / delete).
|
||
2. **Isolation is DB-enforced via Postgres Row-Level Security**, not application-layer
|
||
filtering — realising ADR-002's per-tenant-role intent. RLS is `FORCE`d on every
|
||
user-owned table (the app connects as the non-superuser table-owner `tapir` role, which
|
||
would otherwise bypass RLS); every request scopes rows via `tapir.current_user_id`
|
||
(`SET LOCAL` inside a transaction), routed through a single structural helper so scoping
|
||
is not per-query opt-in.
|
||
3. **The Stage-2 isolation bar is pulled forward into THIS slice, not deferred.** A real
|
||
isolation test (two users, disposable/embedded Postgres) — user A reads/writes zero of
|
||
user B's rows across every table — ships green **with** the multi-user features. No
|
||
multi-user feature merges ahead of that test passing.
|
||
|
||
**Consequences.** Stage 1 + the Stage-2 isolation guarantee land together; isolation is
|
||
structural (DB), so it cannot be forgotten per-query. Adds `video_connections` and
|
||
`subscriptions` tables, a web OAuth callback, and a registration surface. The single-user
|
||
allowlist (ADR-011) is replaced by per-subject `users` rows. Reversible only by a superseding
|
||
ADR. This deliberately advances ahead of the Stage-0 gate — recorded as the maintainer's
|
||
explicit call, with isolation as the guardrail that keeps it safe.
|
||
|
||
---
|
||
|
||
## ADR-013 — Account deletion is Tapir-side only; the Dex identity is left intact
|
||
|
||
**Status:** Accepted (2026-06-03). The "Tapir never holds write access to the shared identity
|
||
provider" rationale is **partially superseded by ADR-017** (the invite flow now *creates* Dex
|
||
Password CRs). The deletion-behaviour decision itself — delete Tapir-side state, leave the Dex
|
||
identity intact — still stands; ADR-017 only changes the create side, not delete. See ADR-017
|
||
for the now-asymmetric posture (Tapir can create Dex accounts but still does not delete them).
|
||
|
||
**Context.** Stage 1 (ADR-012) added account deletion. A registered user is two things: a
|
||
`users` row (plus all their data, cascade-linked) in Tapir's Postgres, and a subject identity
|
||
in **Dex** (the homelab OIDC provider, shared across the estate — Tapir does not own it).
|
||
"Delete my account" could mean (a) erase all Tapir-side data and secrets, or (b) that plus
|
||
deprovision the Dex identity. The maintainer chose (a).
|
||
|
||
**Decision.** Deleting a Tapir account removes **only Tapir-side state**:
|
||
- The `users` row, cascading to all user-owned tables (`videos`, `transcripts`, `summaries`,
|
||
`sink_deliveries`, `video_connections`, and — via an **explicit delete**, because it has no
|
||
FK — `summary_actions`). The delete test asserts the cascade reaches every table and leaves
|
||
other users' rows untouched.
|
||
- All of that user's secrets in the SecretStore (the per-user YouTube refresh-token refs).
|
||
|
||
The **Dex identity is deliberately left intact.** Tapir does not deprovision, disable, or
|
||
modify the shared Dex directory **on deletion**. (Note per ADR-017: Tapir *does* now create Dex
|
||
Password CRs on invite — so the create and delete sides are deliberately asymmetric, and a
|
||
deleted user's Dex Password CR persists. See ADR-017 consequences.)
|
||
|
||
**Consequences.**
|
||
- **Clean re-registration:** a deleted user who logs in again arrives as a Dex-authenticated
|
||
subject with no `users` row, so they hit the registration gate as a "new" user — no special
|
||
resurrection path needed. This is a feature of the choice, not an accident.
|
||
- **Right-to-erasure is partial.** The user's *identity* still exists in Dex after deletion.
|
||
For Future B (trusted friends) this is acceptable: Dex is the maintainer's own directory and
|
||
the identity carries no Tapir content. **But if Tapir ever moves toward Future C (real
|
||
external/public users), this is a GDPR-shaped gap** — a true "delete my account" there must
|
||
also deprovision or anonymise the Dex identity, which is a new ADR and likely a Dex-admin
|
||
integration Tapir does not currently have. **ADR-017 widens this gap:** Tapir now *creates*
|
||
the Dex Password CR but does not delete it, so a deleted Tapir user leaves an orphaned Dex
|
||
local-password account. Tracked as a known gap in ADR-017.
|
||
- **Blast radius:** ADR-013 originally claimed "Tapir never holds write access to the shared
|
||
identity provider." **ADR-017 changes this** — Tapir now holds scoped `create`+`get` on
|
||
`passwords.dex.coreos.com` in the `auth` namespace. The blast radius is no longer zero; it is
|
||
bounded by that RBAC. See ADR-017 for the security analysis.
|
||
|
||
**Reversibility.** Adding Dex deprovisioning later is a superseding ADR; nothing about the
|
||
current choice blocks it. Recorded now because "deletion is partial by design" is a deliberate
|
||
semantic that future-Tapir (and any compliance review) must know was chosen, not overlooked.
|
||
|
||
---
|
||
|
||
## ADR-014 — Timedtext 429 handling: per-host backoff + honest in-flight UX, before any Whisper reconsideration
|
||
|
||
**Status:** Accepted (2026-06-03). **Partially implemented as of v0.6.0** — see the
|
||
implementation note at the end; the shared per-egress-IP rate gate (decision item 2) may not be
|
||
fully realised. Verify against `internal/runner` + the youtube adapter before treating as done.
|
||
**ADR-018 (in-process scheduled discovery + auto-summarize) makes decision item 2 load-bearing
|
||
and folds its confirm/finish into that build.**
|
||
|
||
**Context.** ADR-010 acquires captions from the unauthenticated `timedtext` baseUrl. Live runs
|
||
show that endpoint **rate-limits per source IP (HTTP 429) under volume** — many videos fetched
|
||
in one pass from one egress IP. Stage 1 (ADR-012) made this sharper in two ways: multiple users
|
||
now drive fetches from the *same cluster egress IP*, and the v0.4.0 "Summarize" button fires an
|
||
**immediate, synchronous-feeling** fetch on click (HTMX polls `/v/{videoId}/status`), so a 429
|
||
now surfaces as a *user-facing stall* rather than a background batch hiccup. A throttle
|
||
(`TAPIR_FETCH_DELAY`) exists but is a fixed inter-fetch delay, not 429-aware, and does not
|
||
coordinate across the concurrent click-path and the `tapir run` batch path.
|
||
|
||
This ADR is **not** a decision to build Whisper. ADR-007/010 keep STT deferred *pending
|
||
measurement of the sustainable caption rate* — and that rate cannot be measured while the
|
||
client reacts badly to the 429s it already provokes. Fix the backoff and the UX first; the
|
||
clean data then tells you whether Whisper is warranted.
|
||
|
||
**Decision.**
|
||
|
||
1. **429-aware backoff at the fetch layer.** On a 429 from the timedtext/InnerTube fetch,
|
||
respect `Retry-After` when present; otherwise exponential backoff with jitter. This replaces
|
||
reliance on a fixed `TAPIR_FETCH_DELAY` alone (which stays as a floor/politeness delay).
|
||
2. **A single per-egress-IP rate gate** shared by *both* the `tapir run` batch path and the
|
||
web click path, so they cannot collectively exceed the sustainable rate. Concurrency into
|
||
the timedtext endpoint is serialised/limited at this gate regardless of how many users or
|
||
goroutines are upstream. (The 429 is per *IP*, not per user — so the gate is process-/
|
||
cluster-egress-wide, not per-`withUser`.)
|
||
3. **Honest in-flight UX (the product-shaping part).** The status poll distinguishes states
|
||
the user can understand instead of a spinner that silently stalls:
|
||
- *summarizing* — actively processing (the existing tapir spinner).
|
||
- *queued / waiting for rate limit* — fetch deferred behind the rate gate; show a calm
|
||
"queued, this can take a few minutes when busy" state, not a stuck spinner.
|
||
- *no transcript* — terminal, per ADR-010's degrade-never-error (a 429 that exhausts retries
|
||
resolves to `SourceNone`, same as any unavailable caption — it must not present as a hard
|
||
error to the user).
|
||
The spinner promising imminence is the wrong signal under rate-limiting; the UX must be able
|
||
to say "waiting" truthfully.
|
||
4. **Measurement before Whisper.** Only once (1)-(3) are in and a real sustainable
|
||
per-IP rate is observed do we revisit whether caption coverage is good enough or whether the
|
||
deferred Whisper fallback (ADR-007) is finally warranted. That reconsideration is a future
|
||
ADR, gated on this data.
|
||
|
||
**Consequences.**
|
||
- Caption fetching becomes well-behaved under multi-user load instead of self-inflicting 429s;
|
||
the endpoint is treated as the shared, rate-limited resource it is.
|
||
- The click-path UX stays honest: "waiting" reads as waiting, failure degrades to "no
|
||
transcript", never a stuck spinner or error spew.
|
||
- A future per-IP cooldown / second egress IP / proxy becomes an option the rate gate can sit
|
||
in front of without UX changes.
|
||
- **Still no Whisper** — and now there's a clean path to the *data* that decides whether it's
|
||
ever needed (`docs/homelab-integration.md` and a future ADR own that measurement).
|
||
|
||
**Open (tracked, not in this ADR's scope):** the actual sustainable rate number; whether a
|
||
dedicated egress IP / outbound proxy is worth it; CronJob-driven `tapir run` interaction with
|
||
the rate gate (the batch path moves into k3s per the deferred CronJob item).
|
||
|
||
**Implementation note (2026-06-03, v0.6.0 — added during reconciliation).** A v0.6.0 release
|
||
shipped 429 handling: `domain.SourceRateLimited` (429 no longer collapsed to `SourceNone`),
|
||
youtube adapter maps 429 → `SourceRateLimited`, **migration 007 added `videos.transcript_status`
|
||
+ `videos.rate_limited_at`** (durable per-video retry state), `TAPIR_FETCH_BACKOFF` config,
|
||
runner records rate-limited + skips within the backoff window, and a "⏳ Retrying later" badge.
|
||
This realises decision items 1, 3, and 4 well. **Item 2 (a single shared per-egress-IP rate
|
||
gate) appears to be realised as per-*video* backoff state, NOT a process-wide IP gate** —
|
||
multiple videos can still each hit the endpoint and collectively trip the per-IP 429. Treat item
|
||
2 as **not yet confirmed done**; verify in `internal/runner`/youtube adapter and, if absent, it
|
||
remains open. (Resolved during reconciliation: the v0.6.0 report's reference to "migration 008
|
||
videos.rate_limited_at" was a **mislabel** — both columns shipped in migration **007**, not a
|
||
missing 008. No migration was lost; the sequence legitimately skips 008. `rate_limited_at`
|
||
exists and the runner's column reads are sound.) **ADR-018 makes item 2 a hard requirement:
|
||
in-process scheduled discovery + auto-summarize drives all users' fetches through one pod egress,
|
||
so the process-wide gate is confirmed/built as part of that slice.**
|
||
|
||
---
|
||
|
||
## ADR-015 — Per-user credentials: envelope-encrypted in PG18, not vault-stored
|
||
|
||
**Status:** Accepted (2026-06-03)
|
||
|
||
**Context.** The Stage-0/1 SecretStore (`internal/adapters/secrets/file.go`) holds per-user
|
||
YouTube OAuth refresh tokens as a flat key-value JSON map on a PVC — explicitly a stand-in for
|
||
"op/ESO later" (ADR-002, ADR-006). infra#86 proposed migrating it to an ESO-backed store. The
|
||
decision spike (infra#88) found that framing subtly wrong: **ESO syncs vault→cluster at
|
||
deploy/refresh time; it is not a runtime write API.** Per-user tokens are written *at runtime,
|
||
per end-user* (every YouTube connect; on token rotation) — they are application state, not
|
||
configuration. The homelab 1Password SA is also read-only, so a vault-write path would require
|
||
a new write-capable SA, widening Tapir's blast radius to shared estate infra to store what is
|
||
fundamentally Tapir's own row-data. Reading the actual SecretStore confirmed the shape: a
|
||
3-method port (`Get`/`Put`/`Delete`) over opaque refs, written interactively per user.
|
||
|
||
**Decision.** Per-user credentials are stored **envelope-encrypted in PG18**, not in the vault:
|
||
|
||
1. Tokens are encrypted with a **single app-level envelope key** and stored as ciphertext in
|
||
PG18, under the Row-Level Security already enforced and tested (ADR-012). Reads/writes go
|
||
through the existing `withUser` RLS-scoped seam.
|
||
2. The **envelope key** is the only secret in 1Password — fetched via the **existing read-only
|
||
SA** (confirmed working). No new write-capable SA; no per-user vault items.
|
||
3. The `ports.SecretStore` port is unchanged (`Get`/`Put`/`Delete`). The implementation swaps
|
||
`FileStore` (PVC JSON) for a `PGStore` (encrypted rows). Every consumer — connect,
|
||
disconnect, delete-account — is untouched (the port abstraction holds, ADR-003 spirit).
|
||
4. **Infra/operator credentials** (Dex client secret, MCP-auth tokens, service tokens) stay an
|
||
**ESO/1Password** concern. This ADR governs *per-user runtime* credentials only. The two
|
||
classes use two mechanisms deliberately — because they are two different things (runtime
|
||
app-state vs deploy-time config), not as a compromise. The "one mechanism" question
|
||
(maintainer's initial preference) was answered in #88 by correctly *classifying* the
|
||
secrets rather than unifying their storage.
|
||
|
||
**Consequences.**
|
||
- Runtime credential writes are normal RLS'd DB writes — no ESO sync latency, no indirection,
|
||
no write-SA blast radius. The interactive connect→store→use flow works without a vault
|
||
round-trip.
|
||
- Keeps PG18 and keeps ADR-002 intact (Supabase was considered and rejected again in #88 —
|
||
adding a datastore to hold a few encrypted strings PG18 already holds).
|
||
- Adds an encrypt/decrypt seam and an **envelope-key rotation** responsibility (re-encrypt the
|
||
per-user rows under a new key). infra#89 (build) must implement and test rotation, not assume
|
||
it — this is the real engineering cost of the choice.
|
||
- The vault's involvement shrinks to one static key via the SA already trusted for reads.
|
||
- **Supersedes** the "PVC stand-in for op/ESO" intent recorded in `secrets/file.go` and
|
||
`docs/homelab-integration.md` for the *per-user* secret path (the ESO/1Password reference in
|
||
ADR-006 stands for the *infra-cred* path).
|
||
|
||
**Reversibility / falsification (from infra#88).** Revisit if: per-user tokens need
|
||
high-frequency rotation writes (weak — PG18 handles it); an estate compliance policy requires
|
||
all credentials in 1P for a single audit surface (maintainer-knowable, not currently believed
|
||
to hold — would favour the vault-write path on policy grounds); or envelope-key rotation proves
|
||
operationally worse than per-secret vault rotation (the real cost #89 must prove). If none hold,
|
||
this stands. Full reasoning + rejected candidates (write-capable SA; Supabase): the infra#88
|
||
decision doc (`infra/docs/superpowers/handoffs/`). Build + reboot-validation: infra#89.
|
||
|
||
---
|
||
|
||
## ADR-016 — Stage 0 gate revised: "useful to me OR a friend", behavioural not feedback
|
||
|
||
**Status:** Accepted (2026-06-03). Revises the Stage 0 definition in VISION.md (supersedes the
|
||
original "useful to me, specifically" gate and folds in the old Stage 1 "a trusted user returns"
|
||
test). **The clock-start is further revised by ADR-018** (starts when scheduled discovery ships,
|
||
since unprompted use was not possible before then).
|
||
|
||
**Context.** The original Stage 0 gate was "the maintainer reads summaries weekly for four weeks
|
||
and acts on one." The maintainer chose to change it to include friendly users, reasoning that
|
||
early signal from friendly users is valuable. Two sub-decisions shaped the final form:
|
||
- *Me OR a friend* (not AND): either the maintainer or an onboarded friend showing use clears it.
|
||
- *Behavioural, not feedback*: the test is **return usage**, not stated approval.
|
||
|
||
**Decision.** Stage 0 passes when, over a 3–4 week window, **either the maintainer or at least
|
||
one onboarded friend returns to Tapir unprompted and reads/acts on summaries in ≥2 separate
|
||
weeks.** Friend feedback is gathered and valued but is **not** the gate.
|
||
|
||
**Why behavioural, not feedback (the load-bearing part).** Asked-for feedback from friendly
|
||
users is the least reliable signal in product development — politeness bias means a friend you
|
||
onboarded will tend to say encouraging things regardless of real value. The thing actually worth
|
||
knowing is whether they *come back on their own*. So the gate measures returns, not nice words.
|
||
This deliberately resists the most common way a principled gate dies: being declared "passed" on
|
||
the strength of a polite reaction.
|
||
|
||
**Honest note on what this change does.** This is a *guardrail edit made while the original gate
|
||
was unmet* (Stage 0 had barely started; build had run well ahead of use-evidence). That is
|
||
precisely the pattern that warrants scrutiny — redrawing a gate around work already done. It was
|
||
examined on that basis and proceeds because: (a) the new gate is **not softer in kind** — it
|
||
stays behavioural and sustained, merely broadening *who* can supply the signal; (b) friendly-user
|
||
signal is genuinely valuable; (c) the politeness-bias guard keeps it from collapsing into
|
||
"someone said it's nice." It is *not* a licence to treat the already-shipped Stage-1 machinery as
|
||
evidence the gate passed — use-evidence remains open.
|
||
|
||
**Consequences.**
|
||
- VISION.md Stage 0 rewritten; old Stage 1 ("a trusted user returns") folded in (it was
|
||
near-identical to the new test); hardening renumbered to Stage 1.
|
||
- New drift signal added: declaring the gate passed on polite feedback rather than return-usage.
|
||
- The 2026-07-01 check-in now asks "is anyone (me or a friend) coming back unprompted?", not
|
||
"am I using it weekly?". (Check-in date itself revised by ADR-018.)
|
||
|
||
**Reversibility.** A superseding ADR could tighten it back to maintainer-only or raise it to
|
||
require multiple returning users. Recorded with the full rationale (including the self-scrutiny
|
||
about editing a gate while it's unmet) so the reasoning survives, not just the new wording.
|
||
|
||
---
|
||
|
||
## ADR-017 — Invite flow: Tapir creates Dex local-password accounts (write access to the shared identity provider)
|
||
|
||
**Status:** ~~Accepted~~ **SUPERSEDED by [ADR-019](#adr-019--authentik-owns-invites-tapir-stops-provisioning-accounts) (2026-06-07).** The Dex local-password
|
||
invite provisioning was removed when the homelab IdP migrated Dex→Authentik
|
||
(infra ADR-0001); Authentik now owns invites. Original record below.
|
||
|
||
Accepted (2026-06-03), **recorded retroactively during reconciliation, then
|
||
deliberately ratified KEEP (2026-06-03).** This capability **shipped in v0.7.0 without an ADR** —
|
||
code, RBAC, and a deployed ServiceAccount landed before any decision record existed. This ADR
|
||
documents what shipped and honestly records that the decision-before-code discipline was not
|
||
followed here (see "Process note"). **Partially supersedes ADR-002 and ADR-013** (the "Tapir
|
||
holds no write access to the shared identity provider" posture).
|
||
|
||
**Keep-or-reverse, decided (2026-06-03).** After the retroactive recording, the maintainer
|
||
weighed keep vs. reverse (drop the RBAC + invite flow, rely on Google OIDC only). **Decision:
|
||
KEEP.** Deciding fact: not all intended Future-B users have / will use Google accounts, so Google
|
||
OIDC alone *cannot* onboard them — the invite flow is therefore **load-bearing, not a redundant
|
||
convenience**, and reversing it would leave some intended users with no onboarding path. The
|
||
trust-surface cost (scoped `create` on `passwords` in `auth`) is accepted deliberately in
|
||
exchange. The opposing argument (this contradicts ADR-015's "no write credential to shared infra"
|
||
logic, applied there to secrets) was considered and outweighed *only* because the capability is
|
||
genuinely necessary for real users here — it is **not** a precedent for widening the surface
|
||
further. **Explicitly NOT chosen:** adding `delete` to the RBAC to close the orphan gap (that
|
||
widens the surface in the wrong direction; accept the orphan at Future B instead — see open
|
||
items).
|
||
|
||
**Context.** Stage 1 onboarding needs a way for a friend to get a login. Two paths shipped:
|
||
Google OIDC via Dex (no Tapir code — Dex handles it), and an **invite flow** where Tapir itself
|
||
provisions a Dex **local-password** account. The invite flow is what this ADR is about.
|
||
|
||
**What shipped (reconstructed from `internal/adapters/dex/dex.go`, `internal/web/invite.go`,
|
||
migration `009_invitations`).**
|
||
1. `tapir invite <email>` (host CLI) writes an `invitations` row: a 32-byte crypto-random,
|
||
single-use, 7-day-expiry `token` (the token is the capability), `email`, `used_at`.
|
||
The `invitations` table is deliberately **not** RLS/`user_id`-scoped — the invitee has no
|
||
user yet and the token itself is the secret. (Sound; documented in the migration.)
|
||
2. Recipient visits `/invite/{token}` (public, no session), sets a password (validated *before*
|
||
the token is consumed, so a typo doesn't burn it), the token is claimed exactly once.
|
||
3. Tapir bcrypt-hashes (cost 12) and **POSTs a `passwords.dex.coreos.com` Custom Resource into
|
||
the `auth` namespace** via the in-cluster Kubernetes API, authenticating as the `tapir`
|
||
ServiceAccount. TLS validated against the mounted cluster CA. Off-cluster (dev) it returns
|
||
`ErrNotInCluster` and degrades without consuming the invite.
|
||
4. RBAC applied this deploy: the `tapir` ServiceAccount has **`create`+`get` on
|
||
`passwords.dex.coreos.com` in the `auth` namespace** (and only that).
|
||
5. Dex (kubernetes storage) then serves local-password login for that email; Tapir's
|
||
registration gate creates the profile on first login.
|
||
|
||
**Decision (as ratified now).** Accept the invite flow as built: Tapir *may* hold scoped write
|
||
access to Dex's `passwords` resource in the `auth` namespace, for the purpose of invite-based
|
||
local-account creation. This is a deliberate, bounded reversal of the prior "no write access to
|
||
the shared identity provider" posture (ADR-002/013).
|
||
|
||
**Why this is acceptable (the case for keeping it).**
|
||
- The RBAC is **minimally scoped**: `create`+`get` on one resource type in one namespace, not
|
||
broad Dex/cluster write. Blast radius is bounded and inspectable.
|
||
- It enables friend onboarding **without Google OAuth app verification** (ADR-008 deferred that),
|
||
which is genuinely useful for Future B — and necessary for users who won't use Google (the
|
||
deciding fact in the keep decision above).
|
||
- The code is careful: validate-before-consume, single-use tokens, graceful off-cluster degrade,
|
||
sentinel errors mapped to clear messages, the base64-bcrypt storage gotcha handled.
|
||
|
||
**Consequences / known gaps (the case to watch).**
|
||
- **Reverses a load-bearing principle.** ADR-002 and ADR-013 leaned on "Tapir never writes to the
|
||
shared identity provider" for blast-radius minimisation. That is no longer true. Anyone reading
|
||
those ADRs without this one would be misled — hence the cross-references added to both.
|
||
- **Create/delete asymmetry → orphaned Dex accounts.** Tapir now *creates* Dex Password CRs but
|
||
(per ADR-013) does not *delete* them on account deletion. A deleted Tapir user leaves an
|
||
orphaned Dex local-password account that can still authenticate (though it would hit the
|
||
registration gate with no profile). This widens the ADR-013 right-to-erasure gap. **Open —
|
||
accepted for Future B, revisit before Future C (do NOT add `delete` RBAC just to fix this).**
|
||
- **`create` on a shared-namespace identity resource** is a meaningfully larger trust surface than
|
||
the rest of Tapir. If the `tapir` pod is compromised, the attacker can mint Dex local accounts.
|
||
Bounded by the namespace/resource scope, but real — worth a deliberate look before Future C.
|
||
- Token custody for invites is in PG (`invitations.token`) in plaintext; single-use + 7-day TTL
|
||
bound the exposure, but a DB read yields live invite tokens until claimed/expired.
|
||
|
||
**Process note (why this ADR is retroactive).** This capability was built and deployed by
|
||
parallel agent sessions while the main planning thread was elsewhere, and shipped with no ADR —
|
||
the first time in this project a guardrail-reversing change skipped the decision-before-code
|
||
discipline. Recorded here not to rubber-stamp it but to restore the audit trail: the decision is
|
||
now visible, its principle-reversal is named, and its open gaps (orphaned accounts, the larger
|
||
trust surface) are tracked rather than buried in a release note. The maintainer ratified KEEP
|
||
after the fact (see status block) on the deciding fact that some intended users cannot use Google
|
||
OIDC.
|
||
|
||
**Open items this ADR creates (tracked in infra):**
|
||
- Confirm the RBAC really is `create`+`get` only (not broader) against the deployed manifest in
|
||
`infra` — a security claim currently resting on a sibling report, not a verified manifest read.
|
||
- Accept the orphaned-Dex-account gap for Future B; revisit (orphan cleanup + the whole Dex-write
|
||
surface) before any Future C move. Do **not** add `delete` RBAC solely to fix the orphan.
|
||
- Review the larger trust surface before any Future C move.
|
||
|
||
---
|
||
|
||
## ADR-018 — Make Tapir usable unprompted: in-process scheduled discovery, auto-summarize default, gate-clock reset
|
||
|
||
**Status:** Accepted (2026-06-03)
|
||
|
||
**Context.** The Stage-0 gate (ADR-016) measures whether the maintainer or a friend *returns and
|
||
reads/acts* over weeks. But the system could not actually be used that way: discovery (`tapir
|
||
run`) was **host-side manual**, so a newly onboarded user saw an empty list and had no reason to
|
||
return. The gate was structurally unmeetable — not because the product failed, but because the
|
||
*workflow* depended on the maintainer SSHing in to trigger each pass. The runner already has a
|
||
`Loop(ctx, interval)` (run-on-start, then every tick) and per-user mode/dedup/backoff; what was
|
||
missing was *invocation* — nothing called it for the deployed users on a schedule.
|
||
|
||
**Decision.**
|
||
1. **In-process scheduled discovery.** `tapir serve` launches a background goroutine that runs
|
||
discovery for **all users** on an interval (`TAPIR_DISCOVERY_INTERVAL`; unset/0 = off). It
|
||
enumerates users (un-RLS'd `user_identities`) and runs each user's pass inside `withUser`,
|
||
reusing the existing `runner` — not a new scheduler. Stateless timing; ctx-cancellable;
|
||
per-user failures isolated.
|
||
2. **Auto-summarize default ON** for Future-B users (new registrations default `true`; existing
|
||
rows updated), so discovery both populates *and* summarizes — the list fills itself. The
|
||
per-user manual toggle remains.
|
||
3. **Gate-clock reset.** The Stage-0 3–4 week window (ADR-016) **starts when this ships**, because
|
||
unprompted use was impossible before it. This is starting the clock when the experiment can
|
||
actually run, **not** a reset to dodge a failing gate (the prior window measured nothing —
|
||
there was no way to use the system unprompted). The 2026-07-01 check-in moves accordingly to
|
||
~3–4 weeks after this deploys.
|
||
|
||
**Why in-process and not a k8s CronJob.** Maintainer's call: simpler deploy (no second deployable),
|
||
acceptable at Future-B scale (1–3 users). The CronJob's advantage — failure isolation between
|
||
discovery and the web/read path — was weighed and traded away knowingly.
|
||
|
||
**Consequences / constraints.**
|
||
- **Discovery shares the web process's lifetime and egress.** A wedged discovery pass can degrade
|
||
the reading UI (the coupling a CronJob would have avoided). Accepted at this scale.
|
||
- **SINGLE-REPLICA ASSUMPTION (load-bearing).** If `tapir serve` ever runs >1 replica, *every*
|
||
replica runs the discovery loop → every user fetched in parallel (429s + duplicate work). Tapir
|
||
must stay single-replica while in-process scheduling is enabled, or this is revisited (move to
|
||
CronJob, or add leader-election). Recorded so a future scale-up doesn't silently double-run.
|
||
- **Makes ADR-014 item 2 (shared per-egress-IP rate gate) load-bearing.** In-process + auto +
|
||
multi-user drives all caption fetches through one pod egress, concurrent with click-path
|
||
Summarize. The build confirms/finishes the process-wide rate gate in the same slice; without it
|
||
the system self-inflicts 429s every cycle. **Auto-summarize ON is gated on the rate gate
|
||
existing** (fallback: ship discovery with auto OFF until it does).
|
||
|
||
**Reversibility.** Disable via `TAPIR_DISCOVERY_INTERVAL=0` (reverts to manual `tapir run`).
|
||
Moving to a CronJob later is a superseding ADR; the per-user `runner` is unchanged either way.
|
||
Spec: `docs/specs/scheduled-discovery.md`.
|
||
|
||
---
|
||
|
||
## ADR-019 — Authentik owns invites; Tapir stops provisioning accounts
|
||
|
||
**Status:** Accepted (2026-06-07). **Supersedes ADR-017** (Dex local-password invite
|
||
provisioning).
|
||
|
||
**Context:** infra ADR-0001 migrated the homelab IdP Dex→Authentik. Authentik provides
|
||
first-class invite flows; the Dex local-password connector never consulted the Password
|
||
CRs Tapir wrote (the defect that triggered the migration). Tapir-web's OIDC issuer now
|
||
points at Authentik (infra ADR-0001 step 3).
|
||
|
||
**Decision:** Tapir no longer provisions accounts. The Dex-password invite path is removed:
|
||
`internal/adapters/dex`, the public `/invite/{token}` set-password UI (`internal/web/invite.go`),
|
||
the `tapir invite` CLI (`cmd/tapir/invite.go`), the `InvitationStore`/`DexPasswordCreator`
|
||
ports + `App.Invitations`/`App.Dex` wiring, the invite Templ pages, and the
|
||
`tapir invite` Taskfile target. New users are invited via Authentik's flow, log into Tapir
|
||
via OIDC, and are captured by Tapir's existing provider-agnostic `/register` (display name).
|
||
Login + Google moved by config only (Authentik per-app issuer); the OIDC adapter is unchanged.
|
||
|
||
**Consequences:** smaller Tapir blast surface — no writes to the shared identity provider, no
|
||
configmap/CR access, the dedicated `passwords.dex.coreos.com` RBAC + ServiceAccount are
|
||
removed (infra side, coupled change). The `invitations` table (migration 009) is left in
|
||
place — migrations are append-only and the unused table is harmless; a future migration may
|
||
drop it. The `*_DEX_*` config/identity names (`TAPIR_DEX_CLIENT_*`, `dex_subject`, the
|
||
`DexAuth`/`oidc` package) are now misnomers; renaming is deferred (cosmetic, not behavioural).
|
||
|
||
**Rejected:** keeping Tapir's `/invite` UI but calling Authentik's API on claim — couples
|
||
Tapir to Authentik's admin API + a token for no real gain; Authentik's own invite flow is
|
||
the supported path.
|
||
|
||
---
|
||
|
||
## ADR-020 — Recency-bounded auto-summarize + honest sparse-state surface
|
||
|
||
**Status:** Accepted (2026-06-08). **Refines ADR-018** (auto-summarize) and **ADR-014**
|
||
(per-IP caption rate gate).
|
||
|
||
**Context.** Tapir is operational but sparse: at real subscription volume the maintainer's
|
||
account holds ~283 discovered videos, ~15 summarized, ~256 behind the respected per-IP caption
|
||
rate gate, ~12 no-captions. Two problems follow. (1) **Load:** ADR-018 auto-summarizes *every*
|
||
unseen video, so a large back-catalogue re-drives the whole queue through the gate every cycle —
|
||
self-inflicted 429s with no user value (nobody is waiting on a 6-month-old video). (2) **First
|
||
contact:** a new user sees a mostly-empty feed with no moving parts and a UI that implied
|
||
abundance/imminence ("fetching soon" ×256, "Run `tapir run`", "Summarize now"); the Stage-0 gate
|
||
is *return usage*, and the experience died at the first visit. Source: a UX heuristic review
|
||
(`docs/ux-review/UX-REVIEW-stage0-recency.md`).
|
||
|
||
**Decision.**
|
||
1. **Recency bound on auto-summarize.** In automatic mode the scheduler only summarizes videos
|
||
published within `TAPIR_AUTO_SUMMARIZE_WINDOW` (default ~7d). Older videos are still discovered
|
||
and listed but not auto-processed — they keep the manual "Summarize" affordance. An explicit
|
||
manual request bypasses the bound. `0` disables it (pre-recency behaviour). This bounds auto
|
||
*load*; it does not fetch harder — the gate (ADR-014) is untouched and the manual path still
|
||
serialises through it.
|
||
2. **Honest sparse-state surface.** Copy is reframed to surface scarcity truthfully, never to
|
||
look fuller: "N ready · M in queue · K no captions" (not "fetching soon"); a one-line "captions
|
||
are fetched slowly on purpose" note; "Summarize" (not "Summarize now"); the empty-connected
|
||
state stops printing an impossible CLI command.
|
||
3. **Feed IA = one list, noise-collapsed.** Summarized + recent un-summarized cards lead inline;
|
||
the older un-summarized back-catalogue collapses behind a single "Show N older videos"
|
||
disclosure; caption-less videos collapse to a one-line count instead of N dead cards. List is
|
||
ordered summarized-first, then `published_at DESC NULLS LAST`.
|
||
|
||
**Consequences.** The auto path's per-cycle fetch volume is bounded by recent uploads, not the
|
||
whole back-catalogue, so steady-state 429 pressure drops sharply. Older videos become explicitly
|
||
on-demand — a deliberate honesty trade (the user chooses to spend a scarce fetch on old content).
|
||
The single-replica assumption (ADR-018) is unchanged.
|
||
|
||
**Deliberately NOT done (premature until the Stage-0 loop is validated).** Return-nudges
|
||
(digest email / push) — a nudge contaminates the *unprompted*-return signal the gate measures
|
||
(ADR-016); building it now poisons the experiment. Also deferred: full-text search, channel
|
||
facets, read/unread, saved views — all need summary abundance to matter.
|
||
|
||
**Reversibility.** `TAPIR_AUTO_SUMMARIZE_WINDOW=0` restores summarize-every-unseen; the feed
|
||
collapse keys off the same window (`App.RecencyWindow=0` → everything inline).
|
||
|
||
---
|
||
|
||
## ADR-021 — Persist transcripts as shared, video-keyed public content (re-analysis never re-fetches)
|
||
|
||
**Status:** Accepted (2026-06-09). **Reopens the transcripts half of** the "Global cross-tenant
|
||
`videos`/`transcripts` table" rejection (data-model.md). **Builds on ADR-007** (captions-first),
|
||
**ADR-010/ADR-014** (the per-IP caption rate gate), and **ADR-012** (per-user RLS isolation).
|
||
|
||
**Context.** Every summarization fetches the transcript fresh through the caption path, even when
|
||
the exact same transcript was fetched moments ago — for the same user re-summarizing, or for a
|
||
second user who happens to watch the same video. The caption fetch is the one genuinely scarce,
|
||
genuinely risky operation in the system: YouTube's timedtext endpoint is unofficial and per-IP
|
||
rate-limited (ADR-010), and tripping it risks the maintainer's Google standing (ADR-014). So the
|
||
operation we most want to *avoid repeating* is the one we currently repeat unconditionally. A
|
||
transcript is **public content** — the same words YouTube serves to anyone — and carries nothing
|
||
user-identifying. The per-user isolation that protects summaries, feeds, and tokens (ADR-012) is
|
||
the wrong shape for it: it forces a re-fetch per user for data that is identical across users.
|
||
|
||
The original rejection ("Global cross-tenant `videos`/`transcripts` table") bundled videos and
|
||
transcripts together and rejected both on the grounds that "at 1–5 users, re-summarizing is
|
||
cheaper than the coupling." That reasoning holds for **videos** (per-user feed rows, genuinely
|
||
user-scoped) but not for **transcripts**: the cost being avoided is not LLM re-summarization, it
|
||
is a *rate-gated, reputation-risky network fetch*, and that cost is paid per re-fetch regardless
|
||
of user count. One re-fetch avoided is strictly worth more than the coupling it removes.
|
||
|
||
**Decision.**
|
||
1. **A single shared `transcripts` table, keyed by the cross-user dedup key
|
||
`(provider, provider_video_id)`** — the stable public identity of the video, not Tapir's
|
||
internal per-user `videos.id`. Columns: the key, `source` (`captions`/`none`), `language`,
|
||
`content`, `fetched_at`. It holds **only public caption content + the video's public id** —
|
||
nothing user-identifying — and is therefore **NOT RLS-scoped**: no `user_id`, no policy, no
|
||
`FORCE ROW LEVEL SECURITY`. This is the deliberate, single exception to the ADR-012 isolation
|
||
boundary, and the only one.
|
||
2. **Summarize path becomes read-stored-first.** Have a stored transcript for this video? →
|
||
summarize from the stored text, **no caption fetch**. No stored transcript? → fetch *through
|
||
the unchanged gate* (ADR-014) → store it → summarize. The gate is neither bypassed nor
|
||
weakened; persistence reduces how *often* we reach it, never how *fast*.
|
||
3. **De-facto cross-user dedup is the intended behaviour, not a feature with a switch.** Two
|
||
users who share a video share the one transcript row. A permanent `source = 'none'` (no
|
||
captions) is stored too, so a known-caption-less video is not re-fetched by anyone. A
|
||
transient 429 (`SourceRateLimited`) is **never** stored as terminal — it stays a per-user
|
||
retry via the existing `transcript_status` backoff (ADR-014), so persistence cannot mask a
|
||
rate-limit into a false "no transcript."
|
||
4. **Per-user `summaries` stay RLS-scoped (ADR-012 unchanged)** and reference the transcript by
|
||
video id. Videos stay per-user. Only transcripts go shared.
|
||
|
||
**Consequences.** Re-analysis (re-summarize, different model, paste of an already-seen video,
|
||
onboarding of a second user with overlapping subscriptions) never re-touches YouTube — the
|
||
primary win, and it *reduces* aggregate caption-gate pressure, reinforcing ADR-010/ADR-014 rather
|
||
than straining them. The isolation surface gains exactly one non-RLS table; an isolation test
|
||
asserts the boundary is *exactly* there and has not leaked to any user-owned table (this is the
|
||
proof the public-content classification was implemented as designed). It also unblocks
|
||
multi-model / customizable analysis (re-run analysis on stored text for free) — enabling that is
|
||
this ADR's point; building it is separate.
|
||
|
||
**Reversibility.** The read-stored-first check is the only behavioural coupling; removing it
|
||
restores fetch-every-time. The down-migration recreates the per-user RLS-scoped transcripts shape
|
||
(001/003). No user-facing surface depends on cross-user sharing — sharing is the *storage shape*,
|
||
never exposed in the UI.
|
||
|
||
---
|
||
|
||
## ADR-022 — Summarizer is a resilient endpoint chain, not a single model
|
||
|
||
**Status:** Accepted (2026-06-10). **Extends ADR-004** (the copied `llm` Primary→Fallback
|
||
routing). Triggered by the first friendly-pilot live run, where a connected user got **zero**
|
||
summaries after 12h.
|
||
|
||
**Context.** Stage-0 ran a single summarizer model (`koala/phi4-mini`) with no fallback wired
|
||
(`summarizer.New(primary, nil)`). The live run exposed three independent failure modes, each of
|
||
which silently produced no summary:
|
||
|
||
1. **Context overflow.** `phi4-mini` has an 8k context. Real transcripts (one was 11,602 tokens)
|
||
exceed it and the gateway returns HTTP 400 — and the request also sent `max_tokens=8192`, so
|
||
even a short transcript plus the completion budget could overflow the window.
|
||
2. **Malformed model output.** `phi4-mini` intermittently emits `highlights` as a bare string
|
||
instead of an array, producing `cannot unmarshal string into []string`. The old code returned
|
||
the parse error **without** trying any other model — a 200-with-bad-JSON short-circuited.
|
||
3. **No fallback existed at all** — any primary failure was terminal for that video.
|
||
|
||
`phi4-mini` is kept as primary deliberately: it is fast and, on transcripts that fit, correct.
|
||
The fix is resilience around it, not replacing it.
|
||
|
||
**Decision.**
|
||
1. **Ordered endpoint chain (`summarizer.NewChain`).** Endpoints are tried in order; the first to
|
||
return a *parseable* summary wins. Default chain:
|
||
`koala/phi4-mini` (primary, local) → `iguana/gemma4-26b` (fallback, local on a
|
||
*different host*) → `berget/mistral-small` (worst-case, external). All three are reached through
|
||
the **one** LiteLLM gateway by alias — the gateway already fronts both llama-swap and berget — so
|
||
a fallback is a different alias, not a second client config.
|
||
|
||
**Update 2026-06-11:** the local fallback moved from `koala/phi4-14b` to `iguana/gemma4-26b`.
|
||
koala now carries other GPU loads, so keeping the fallback on koala competed with them; iguana
|
||
(M2 Ultra) has the headroom, and a different host is also a different egress IP for the rare
|
||
fallback fetch. `gemma4-26b` is the brain-validated homelab general-purpose model (agentsquad
|
||
H2/H3 executor) and returned valid summary JSON on the real prompt in a smoke test
|
||
(~37s incl. cold-load — fine for a path hit only when the fast primary fails). Pure config:
|
||
`TAPIR_FALLBACK_MODEL`.
|
||
2. **A parse failure advances the chain, same as a transport error.** "Reliably summarized" means
|
||
*parseable summary returned*, not *HTTP 200*. This is the behaviour the old Primary→Fallback
|
||
shape missed.
|
||
3. **Tolerant parse.** `highlights`/`takeaways` coerce from a bare string (or a mixed scalar
|
||
array) to `[]string`, so the most common small-model quirk is absorbed **without** spending a
|
||
fallback round-trip — keeping the fast path fast.
|
||
4. **Transcript truncation (`TAPIR_MAX_TRANSCRIPT_CHARS`, default 18000).** Input is bounded
|
||
up-front to fit a small-context primary, so overflow is prevented rather than recovered-from.
|
||
5. **Bounded completion budget (`TAPIR_SUMMARY_MAX_TOKENS`, default 1500).** A summary needs few
|
||
hundred tokens; the old 8192 budget itself contributed to 8k-window overflow.
|
||
|
||
**Local-first guarantee preserved.** The chain ordering *is* the guarantee: locals are tried
|
||
first, so content reaches the external endpoint only after every local endpoint has failed.
|
||
`TAPIR_CLOUD_FALLBACK_MODEL=""` removes the external endpoint entirely — the lever a
|
||
**client/NDA deployment** pulls so content never leaves the local stack. With no external endpoint
|
||
configured the `ai_routing.feature` "content only local" scenarios hold unchanged.
|
||
|
||
**Reversibility.** Pure wiring + config. Setting `TAPIR_FALLBACK_MODEL` and
|
||
`TAPIR_CLOUD_FALLBACK_MODEL` empty collapses the chain back to single-primary behaviour; the
|
||
tolerant parse and truncation are strict supersets of the old behaviour (a previously-parseable
|
||
reply still parses; a transcript within budget is unchanged).
|
||
|
||
---
|
||
|
||
## ADR-023 — Drop Shorts/livestreams at discovery to protect the caption budget
|
||
|
||
**Status:** Accepted (2026-06-10). **Builds on ADR-014** (per-IP caption rate limit is the
|
||
binding constraint) and the ADR-022 live-run findings.
|
||
|
||
**Context.** The scarce resource is the unofficial timedtext caption fetch (per-egress-IP
|
||
429, ~3 successful/pass). The first multi-user run showed the candidate set was mostly noise —
|
||
Shorts, sub-minute clips, and live broadcasts — each of which still consumes a caption-fetch
|
||
attempt (and a "none" result is a *completed* fetch, so it costs budget even when it yields
|
||
nothing). Spending the rate-limited budget on content the user will not read is the waste to
|
||
cut first; it is cheaper and lower-risk than raising the ceiling (multi-IP, Whisper).
|
||
|
||
Duration and live status are NOT in the playlistItems discovery response, but they ARE in the
|
||
Data API `videos.list` (contentDetails.duration + snippet.liveBroadcastContent) — the official
|
||
**quota-based** API (1 unit/call, 50 ids/call), which is a *different* limit from the timedtext
|
||
429. So one cheap quota call buys a filter that saves many expensive throttled fetches.
|
||
|
||
**Decision.**
|
||
1. `NewVideos` enriches its candidates with a single `videos.list` call and drops, before
|
||
returning: videos shorter than `TAPIR_MIN_VIDEO_SECONDS` (default 60) and any `live`/
|
||
`upcoming` broadcast. Dropped videos are never persisted, so they also declutter the list.
|
||
2. The filter is **degrade-open**: `MinVideoSeconds=0` disables it (no quota call), and a
|
||
`videos.list` error returns the candidates unfiltered — discovery must never break because a
|
||
metadata call hiccuped (worst case = pre-ADR-023 behaviour).
|
||
3. The paste-a-URL path (`VideoByID`) is **not** filtered — an explicit user request for a
|
||
specific video (even a Short) is honoured.
|
||
|
||
**Reversibility.** Pure discovery-time filter + config. `TAPIR_MIN_VIDEO_SECONDS=0` restores
|
||
the old behaviour. No schema change, no effect on already-stored videos.
|
||
|
||
**Quota note.** Per-channel enrichment adds ~1 unit/channel/pass. At pilot scale (≤3 users)
|
||
this is well under the 10k/day cap; at larger scale, batch `videos.list` across channels
|
||
(50 ids/call) by collecting all discovered ids per pass before enriching.
|
||
|
||
---
|
||
|
||
## ADR-024 — Per-channel caption-availability memory
|
||
|
||
**Status:** Accepted (2026-06-10). **Builds on ADR-014** (per-IP caption budget), **ADR-021**
|
||
(shared transcript cache), **ADR-023** (Shorts filter).
|
||
|
||
**Context.** After ADR-021 caches transcripts and ADR-023 drops Shorts, the remaining caption
|
||
waste is the *first* fetch on every new video of a channel that never publishes English captions
|
||
(foreign-language news, music, etc.). Each costs one rate-limited fetch to resolve to "none" —
|
||
and on a throttled IP that fetch may 429 and churn the backoff machinery before it ever gets a
|
||
verdict. A pilot user's feed had several such channels.
|
||
|
||
**Decision.** Remember, per `(user, channel)`, a streak of consecutive no-caption outcomes
|
||
(`channel_caption_state`, migration 016, RLS-scoped like the rest of the user-owned schema).
|
||
Once the streak reaches `TAPIR_CHANNEL_CAPTIONLESS_THRESHOLD` (default 5) the channel is
|
||
suppressed — its videos are discovered/listed but not caption-fetched — for
|
||
`TAPIR_CHANNEL_CAPTIONLESS_WINDOW` (default 14d), after which one video is re-probed
|
||
(auto-recovery for a channel that starts adding captions). A successful fetch resets the streak;
|
||
a fresh 429 does NOT count (transient, not a caption verdict). An explicit manual request
|
||
bypasses suppression. `threshold = 0` disables the feature.
|
||
|
||
**Why per-user, not global.** Caption availability is really a channel property (public), so a
|
||
global table would let users share the learning. But subscriptions are per-user (ADR-012) and at
|
||
pilot scale users' channel sets barely overlap, so per-user + RLS keeps it consistent with the
|
||
existing isolation model with no new non-RLS exception to justify. Promoting to a shared table
|
||
(like transcripts, ADR-021) is a future optimisation if channel overlap grows.
|
||
|
||
**Reversibility.** Migration 016 is a clean drop; `threshold = 0` disables at runtime. The
|
||
memory only ever *suppresses fetches* — it never deletes content or affects already-stored
|
||
summaries.
|
||
|
||
---
|
||
|
||
## ADR-025 — Honest, state-aware foreground summarization status
|
||
|
||
**Status:** Accepted (2026-06-10). **Pillar B of the manual-mode UX work** (Pillar A, foreground
|
||
fetch priority, is a separate follow-up). Builds on ADR-014 (the rate limit the UX must make
|
||
legible).
|
||
|
||
**Context.** Clicking "Summarize" spawned a background goroutine and polled `/status`, which
|
||
returned only two states: the spinner (in-flight) or the normal card (done). But the web
|
||
`ProcessVideo` only recorded an outcome on *success* — a 429'd or caption-less click left
|
||
`transcript_status` unset, so the next poll silently reverted to the "Summarize" button. The
|
||
user saw either an endless spinner or a button that did nothing useful when clicked again. The
|
||
binding constraint (YouTube's caption rate limit) was completely invisible.
|
||
|
||
**Decision.**
|
||
1. **Record every outcome on the web path**, mirroring the runner: `ProcessVideo` stamps
|
||
`rate_limited` / `none` / `fetched`. A rate-limited video keeps its requested flag so the
|
||
background sweep retries it; `none` and `fetched` are terminal.
|
||
2. **`/status` is state-aware**: summarized → summary card; in-flight → working spinner;
|
||
`rate_limited` → a calm "waiting, will retry" card that keeps polling (every 30s) so the
|
||
summary appears on its own when the retry lands — the user never clicks again;
|
||
`none` → a terminal "no captions" card with no poll and no dead-end button.
|
||
3. **Charm status text** (Claude-Code / Crush inspired): the working spinner cycles playful,
|
||
tapir-themed gerunds ("Chewing the cud…", "Munching leaves…", "Distilling the gist…") via
|
||
CSS only — no JS, keeping the HTMX/no-JS ethos. Decorative (aria-hidden) with a stable
|
||
`role=status` line for assistive tech.
|
||
|
||
**Principle.** When the system cannot be fast (throttled IP), it is at least honest, and it
|
||
self-resolves without making the user retry. Honesty is the load-bearing half — Pillar A's
|
||
priority lane only improves the odds of a fast slot; it cannot beat an already-hot IP.
|
||
|
||
**Reversibility.** Pure transport-layer + view change over the unchanged engine/ports. No
|
||
schema change (reuses `transcript_status` from migration 007).
|
||
|
||
---
|
||
|
||
## ADR-026 — Foreground caption fetches take priority; the credentials probe is dead
|
||
|
||
**Status:** Accepted (2026-06-10). **Pillar A of the manual-mode UX work** (Pillar B was
|
||
ADR-025). Builds on ADR-014 (the shared per-IP gate).
|
||
|
||
**Context.** Every caption fetch — the background sweep and the web click-path — shared one
|
||
process-wide rate gate equally. So a user waiting on a "Summarize" click competed with the
|
||
firehose for both pacing and the scarce pre-429 window; on a busy IP the click was slow or
|
||
429'd while the background churned.
|
||
|
||
**Decision.** A context-marked priority lane. The web path
|
||
(`engineProcessor.ProcessVideo`) wraps its context with `ForegroundContext`; the gate gives
|
||
foreground fetches a token immediately, while **background fetches yield** — they wait until no
|
||
foreground fetch is pending before taking a token. Threaded via a context value (not new
|
||
signatures) and a process-wide `foregroundPending` counter. Clicks are rare and bursty, so the
|
||
background barely loses throughput; the waiting human gets the next (and cleanest) slot.
|
||
|
||
**Credentials probe — rejected, not built.** The idea was to fetch captions with the user's
|
||
auth in manual mode to dodge 429s. It is a dead end, already settled by ADR-010 and the code:
|
||
the caption path is *deliberately anonymous* because the InnerTube/timedtext endpoints **reject
|
||
or break on authenticated requests** (`captions.go`: "no OAuth token — it can break the
|
||
timedtext endpoint"). The user's OAuth (a Data API credential) does not authenticate InnerTube
|
||
at all, and the official `captions.download` is owner-only (403 on third-party). So auth cannot
|
||
help here and can actively hurt. No probe needed — building one would only re-confirm the ADR.
|
||
|
||
**Reversibility.** Context-marker + a yield loop in the gate; removing the marker collapses to
|
||
the prior equal-share behaviour. No schema or API change.
|
||
|
||
---
|
||
|
||
## ADR-027 — Chat with a video's stored transcript (deeper-dive, on an already-summarized video)
|
||
|
||
**Status:** Accepted (2026-06-11). **Consumes ADR-021** (the shared, video-keyed transcript
|
||
store) for the first time beyond summarization; **uses the ADR-022 chain models**; relates to
|
||
ADR-012 (isolation) and ADR-016 (the Stage-0 gate).
|
||
|
||
**Context — observed demand, not hypothetical.** The maintainer read 10+ real pilot summaries
|
||
and reported the reactions: *many good; some he wanted to dig deeper into; some less useful*
|
||
(the "less useful" split between weak-*model* output and uninteresting-*video* content). The
|
||
middle reaction is the signal: a good summary that makes the reader want *more* is the summary
|
||
succeeding at triage and then hitting a wall — there is nowhere to go deeper short of watching
|
||
the video. That want is the feature. It is also the cheapest possible feature to satisfy
|
||
honestly, because ADR-021 already persists the transcript: the deeper-dive runs entirely on
|
||
stored public-content text + local models, touching **no** caption fetch and **no** YouTube.
|
||
|
||
**Decision.** Add a per-video chat that lets the user ask questions against a video's stored
|
||
transcript.
|
||
1. **Entry from the summary view only.** A "dig deeper / ask" affordance on a summarized video —
|
||
the chat lives exactly where the "I want more" reaction happens. No standalone chat surface.
|
||
2. **Stored-transcript-only (load-bearing constraint).** Chat is available **only** for videos
|
||
that already have a stored transcript. It never triggers a caption fetch, so it cannot touch
|
||
the rate gate, the 429 surface, or YouTube at all — the entire account-safety constraint that
|
||
governs the rest of Tapir is satisfied *by construction* here, not by careful gating. (Entry
|
||
being "from a summary" guarantees the transcript exists.) On-demand fetch for un-stored videos
|
||
is explicitly deferred.
|
||
3. **Model = the summary's model by default; user-switchable among the ADR-022 chain models**
|
||
(`phi4-mini` / `gemma4-26b` / `mistral-small` to start). This is deliberate: it doubles as
|
||
live model-comparison instrumentation — ask the same question of the same transcript under two
|
||
models and the difference is directly felt. This is the mechanism by which the maintainer
|
||
learns *which* model is worth defaulting to, and it is the multi-model-analysis direction
|
||
ADR-021 anticipated, arriving as a user-facing capability.
|
||
- Chat is a **read-bounded retrieval/QA task** (the user supplies the focus), which is
|
||
*easier* than summarization (the model must decide what matters). So a model that summarizes
|
||
mediocrely may chat well — chat is plausibly a partial remedy for the weak-summary case, not
|
||
an inheritor of it.
|
||
4. **Ephemeral chat (v1).** No persisted history; chat is per-session. Persisting per-user,
|
||
RLS-scoped history is deferred until there is evidence anyone wants to revisit a conversation.
|
||
5. **Chat-only, trust-the-model (v1) — with a recorded limitation.** The chat does not expose the
|
||
raw transcript for verification in v1 (kept simple). **Known limitation:** because some
|
||
summaries were weak-model output, the user has reason not to fully trust a chat answer's
|
||
fidelity to the transcript, and v1 gives no in-UI way to check. The model-switcher partially
|
||
compensates (two models disagreeing on the same question is itself a signal). A
|
||
"show source / view transcript" verification path is the natural **v2** and is *not*
|
||
foreclosed — ADR-021's stored transcript already makes it cheap. Recorded so v2 is a known
|
||
next step, not a rediscovery.
|
||
|
||
**Why this is safe and in-scope.** It adds no caption-fetch surface (stored-only), no new
|
||
non-RLS table (transcripts already shared per ADR-021; ephemeral chat stores nothing), and no
|
||
auth change. It is additive to the read path. The one genuine product expansion — Tapir becomes
|
||
an interactive transcript-QA tool, not only a summarizer — is justified by *observed* demand from
|
||
real reading, which is exactly the kind of evidence the Stage-0 discipline asks for before
|
||
building.
|
||
|
||
**Relation to the Stage-0 gate.** This is **not** a return-nudge (those stay deferred, ADR-020) —
|
||
it adds nothing that prompts the user to return; it deepens the value *once they are already
|
||
reading*. It does not contaminate the unprompted-return signal. If anything it strengthens the
|
||
"useful to me" case the gate measures, by giving a good summary somewhere to lead.
|
||
|
||
**Reversibility.** Additive read-path feature over the unchanged engine + the ADR-021 store.
|
||
Removing the summary-view affordance removes the feature; nothing else depends on it. Ephemeral =
|
||
no migration, no stored state to unwind. Spec: `docs/specs/chat-with-transcript.md`.
|
||
|
||
---
|
||
|
||
## ADR-028 — Onboarding burst: pick likely-good videos, summarize them with a stronger model
|
||
|
||
**Status:** Accepted (2026-06-11). **Refines ADR-018** (the connect-time burst) and **ADR-020**
|
||
(recency-bounded auto-summarize). **Builds on ADR-022** (the endpoint chain), **ADR-023**
|
||
(the discovery-time `videos.list` enrichment), and **ADR-021** (the shared transcript cache).
|
||
Triggered by a Phase-1 investigation of the live pilot DB.
|
||
|
||
**Context.** A new user's first session decides whether they return (the Stage-0 gate, ADR-016).
|
||
The connect-time burst (ADR-018: summarize ≤`TAPIR_ONBOARD_SUMMARIZE_COUNT` newest videos so the
|
||
feed isn't empty) *fires* in production, but a live-DB investigation of the second pilot user
|
||
("Jonte") found it delivers a weak first impression for two reasons, and ruled out a third idea:
|
||
|
||
1. **Junk picks.** Selection was pure newest-first (`NewestUnsummarizedVideoIDs`,
|
||
`ORDER BY published_at DESC`) with **no quality signal**. Jonte's live burst-3 were a
|
||
stock-ticker **livestream** + two regional news clips — newest, not best. The cheap signals
|
||
that *could* gate this (duration, live status) are fetched by ADR-023's `videos.list`
|
||
enrichment at discovery and then **thrown away**: the `videos.duration_s` column (migration
|
||
001) was never written.
|
||
2. **Weakest model on the first impression.** All burst summaries ran on `koala/phi4-mini` — the
|
||
documented weak link (ADR-022 was born from its failures). The stronger, brain-validated
|
||
`iguana/gemma4-26b` was never used, even though the burst is only ~3 summaries.
|
||
3. **Cached-first instant summaries — REJECTED.** The idea: skip the fetch, summarize
|
||
already-cached transcripts (ADR-021) instantly. The pilot numbers kill it — only **11 videos**
|
||
overlap between the two users (~3% of each ~350–400-video library), **0** cached-and-
|
||
unsummarized, and a new user's newest-20 unsummarized are **20/20 NOT cached**. Newest-first
|
||
and cached-first are structurally incompatible: fresh uploads are exactly what nobody has
|
||
fetched. An empty lever at pilot scale.
|
||
|
||
**Decision.**
|
||
1. **Persist `duration_s` at discovery.** `filterLowValue` (ADR-023) already has each candidate's
|
||
duration in hand; carry it onto the kept `domain.Video` and have `UpsertVideo` write it,
|
||
COALESCE-preserving a known value (the channel-title backfill stance, migration 014). No new
|
||
migration — the column exists. The connect-triggered discovery pass runs *before* the burst,
|
||
so a fresh user's candidates are enriched in time.
|
||
2. **Junk-avoiding selection.** A new `OnboardBurstVideoIDs(userID, limit, minSeconds, maxSeconds)`
|
||
keeps the newest-first order but drops a video when its duration is *known* and outside
|
||
`[minSeconds, maxSeconds]` — `minSeconds` = `TAPIR_MIN_VIDEO_SECONDS` (60, the Shorts floor),
|
||
`maxSeconds` = new `TAPIR_ONBOARD_MAX_VIDEO_SECONDS` (default 14400 = 4h, to drop multi-hour
|
||
livestream VODs that pass the live filter once ended). A NULL duration is **unknown** — kept
|
||
(degrade-open) but ranked after known-good rows. **has-captions stays un-gateable pre-fetch**
|
||
(only knowable after a gate fetch or a ~0-probability cache hit); selection only *avoids
|
||
known-junk*, it does not *promise* captions.
|
||
3. **Stronger model for the burst only.** `TAPIR_ONBOARD_SUMMARIZER_MODEL` (default
|
||
`iguana/gemma4-26b`) leads a burst-specific summarizer chain (onboard model first, then the
|
||
standard ADR-022 chain as resilience, deduped), wrapped in a burst-specific processor over the
|
||
*same* store/cache/sink — a pure wiring choice; the engine and ports are unchanged (ADR-003).
|
||
Empty or equal-to-primary collapses the burst back onto the shared processor.
|
||
|
||
**Not a throughput change.** The caption rate gate (ADR-014) and the foreground priority lane
|
||
(ADR-026) are untouched — same pacing, same cap. This changes *which* ≤3 videos the burst spends
|
||
its fetches on and *which model* summarizes them, never how fast or how many. The engine's
|
||
existing read-stored-first (ADR-021) is unchanged and still yields a free instant summary on the
|
||
rare cache hit — we simply do not *select* for cache hits.
|
||
|
||
**Consequences.** Better odds of a strong first session: the burst avoids the obvious junk and
|
||
runs the better model on the one impression that decides return. The selection improvement is
|
||
forward-looking — existing rows have NULL `duration_s` until their next discovery pass backfills
|
||
it (lazy, like channel_title); a brand-new user benefits immediately because connect-discovery
|
||
runs first. `duration_s` becoming live also unblocks future length-aware features (feed sorting,
|
||
"long read" badges) for free.
|
||
|
||
**Reversibility.** Pure config + wiring + one column write + one query, no migration.
|
||
`TAPIR_ONBOARD_MAX_VIDEO_SECONDS=0` (and `TAPIR_MIN_VIDEO_SECONDS=0`) restores pure newest-first;
|
||
`TAPIR_ONBOARD_SUMMARIZER_MODEL=""` collapses the burst back to the shared processor.
|
||
Spec: `docs/specs/onboarding-wow-burst.md`.
|
||
|
||
---
|
||
|
||
## ADR-029 — Stateless session cookie (survives restarts, browser-close, idle)
|
||
|
||
**Status:** Accepted (2026-06-11). Triggered by pilot feedback: "lots of clicking to log in again
|
||
on iPhone." Supersedes the in-memory session store in the ADR-011 login.
|
||
|
||
**Context.** Three compounding causes made users re-login constantly:
|
||
1. **In-memory session store** (`sessionStore` map) — wiped on every pod restart, so each deploy
|
||
logged everyone out. During the active build period that was ~15 logouts.
|
||
2. **1-hour session TTL** — for a "check back tomorrow" reader, idle > 1h forced a re-login on
|
||
nearly every visit.
|
||
3. **No cookie Max-Age** — a session cookie (deleted on browser/app close); iPhone Safari closing
|
||
the tab dropped it.
|
||
Each re-login is the full Dex/Authentik redirect dance — many taps on mobile.
|
||
|
||
**Decision.** Make the session **stateless**: the identity (subject + email) and an absolute
|
||
expiry live INSIDE the existing HMAC-signed (HS256) cookie — no server-side table. Plus:
|
||
- **30-day sliding TTL** (was 1h), re-signed on each request so an active user never lapses.
|
||
- **Persistent cookie** (`Max-Age` set) so it survives browser/app close.
|
||
The cookie is HttpOnly + Secure + SameSite=Lax; the HMAC (keyed by the stable ESO
|
||
`tapir-session-secret`, which does NOT rotate per deploy) makes it tamper-proof. The payload is
|
||
identity, not secrets — the OIDC access/ID tokens are still discarded after callback.
|
||
|
||
**Consequences.** A deploy/restart no longer logs anyone out (proven by a test: a cookie issued by
|
||
one instance is accepted by a fresh instance with the same secret); works across replicas for
|
||
free. **Trade:** no server-side revocation — `logout` clears the cookie client-side, but a copied
|
||
cookie stays valid until expiry. Accepted for the Stage-0 reader pilot; revisit (server-side
|
||
revocation list, or shorter TTL + refresh) if it ever holds sensitive actions. Rotating
|
||
`tapir-session-secret` invalidates all sessions — the global logout lever.
|
||
|
||
**Not addressed here:** the tap-count of the IdP login page itself is Authentik's UX; with
|
||
re-login now rare (30-day idle or explicit logout), it matters far less.
|
||
|
||
---
|
||
|
||
## ADR-030 — Observability: slog timing + Prometheus metrics (AI-focused)
|
||
|
||
**Status:** Proposed (2026-06-11). Issue #15. **Draft for review — no code yet.**
|
||
|
||
**Context / requirements.** Nothing measures the activities that drive Tapir's performance and
|
||
UX, and the Stage-0 eval gate (ADR-016) needs a *performance* dimension to sit beside the
|
||
return-usage one. We need timing for: caption fetches (the scarce op), summarization (which model
|
||
won, how long, fallbacks), Q&A latency, LLM token spend, and basic session/usage (request rate,
|
||
latency by route, logins). Requirements:
|
||
- R1: structured `slog` timing at each AI call site (human-readable, already the logging stack).
|
||
- R2: Prometheus metrics for the same, scrapeable by the cluster's prometheus-operator.
|
||
- R3: **AI metrics are the priority** — summarize latency by `model`/`outcome`/`fallback`,
|
||
caption-fetch latency by `outcome`, chat latency by `model`, and LLM `tokens` by model+kind.
|
||
- R4: HTTP/session metrics via middleware — request count + latency by route, logins.
|
||
- R5: bounded label cardinality (no per-user, no raw-path labels).
|
||
- R6: `/metrics` must NOT be publicly exposed.
|
||
|
||
**Decision / architecture.**
|
||
1. **New package `internal/metrics`** owns all Prometheus collectors + a typed API
|
||
(`ObserveSummarize`, `ObserveCaptionFetch`, `ObserveChat`, `RecordTokens`, `IncLogin`,
|
||
`HTTPMiddleware`, `Handler`). Adapters call this API; they never import prometheus types.
|
||
2. **New dependency `github.com/prometheus/client_golang`.** Justification: it is *the* standard
|
||
Go Prometheus client and the cluster already runs prometheus-operator; hand-rolling exposition
|
||
is not worth it. (Needs the dep-justification note in the commit per repo rules.)
|
||
3. **The copied `llm` package stays stdlib-only (ADR-004).** It must not import `internal/metrics`.
|
||
Token usage is surfaced via an **optional callback** `llm.WithUsageHook(func(model string, prompt, completion int))`
|
||
set at wiring time (`buildSummarizer`/`buildChat`) to `metrics.RecordTokens`; `llm.Client` only
|
||
gains parsing of the response `usage` block. Our own adapters (`summarizer`, `youtube`, `chat`)
|
||
may import `internal/metrics` directly.
|
||
4. **HTTP middleware** reads `r.Pattern` AFTER routing (Go 1.22 sets it during ServeMux match), so
|
||
the `route` label is the bounded registered pattern (`GET /v/{videoId}`), satisfying R5;
|
||
unmatched → `other`.
|
||
5. **Dedicated metrics port** (`TAPIR_METRICS_ADDR`, default `:9090`) served by a second
|
||
`http.Server` in `cmdServe`; `/metrics` is never on the public app mux (R6). A **PodMonitor**
|
||
in `mathias/infra` scrapes it; the deployment exposes the port.
|
||
6. **slog** elapsed fields are emitted alongside each metric at the call sites (R1).
|
||
|
||
**Hook points (where the instrumentation lands).**
|
||
- `summarizer.Summarize` — per-endpoint timing + outcome (`success`/`parse_error`/`error`) + fallback flag.
|
||
- `youtube.FetchTranscript` — fetch timing + outcome from `domain.Transcript.Source`.
|
||
- `chat.Service` answer — timing by model.
|
||
- `llm.Client.Complete` — parse `usage`, fire the usage hook.
|
||
- `oidc.handleCallback` — `IncLogin`.
|
||
- `cmdServe` — wrap `Router()` in `metrics.HTTPMiddleware`; start the metrics server.
|
||
|
||
**Out of scope / later.** Persisting per-summary latency into Postgres for `tapir report`
|
||
(derive UX latency — publish/discovery → summary — from existing timestamps first; only persist
|
||
op-latency if the scrape proves insufficient). SPA view (#16) and visual refresh (#17).
|
||
|
||
**Reversibility.** Additive: a new package + middleware + a metrics port. Removing the PodMonitor
|
||
stops scraping; the app is unaffected. No schema change.
|
||
|
||
**Next steps (gated):** on approval of this ADR → BDD scenarios (`docs/use-cases/observability.feature`
|
||
+ scenario-coverage map) → TDD → implement → SemVer + docs + PodMonitor.
|
||
|
||
---
|
||
|
||
## ADR-031 — SPA-like reader: inline-expand summary + Q&A in the list (HTMX, no framework)
|
||
|
||
**Status:** Proposed (2026-06-12). Issue #16. **Draft for review — no code yet.**
|
||
|
||
**Context / requirements.** The reader is multi-page: a list of compact cards (`/`), then a
|
||
navigation to a separate detail page (`/v/{id}`) for the full summary + the docked chat (ADR-027).
|
||
It feels less fluid than a single integrated view. We want the full summary AND the per-video
|
||
Q&A to open **in place in the list**, no page hop. Requirements:
|
||
- R1: clicking a summarized card expands it in place to the full summary (summary/highlights/
|
||
takeaways) + the chat dock; a collapse returns it to the compact card.
|
||
- R2: **no SPA framework** — stay HTMX + Templ (ADR-003); reuse existing fragments, not a rewrite.
|
||
- R3: **progressive enhancement** — with JS off, the card link still navigates to `/v/{id}`
|
||
(the detail page stays as the no-JS + deep-link surface). Nothing becomes JS-only.
|
||
- R4: only **summarized** cards expand; pending/rate-limited/no-caption cards keep their current
|
||
footer behaviour (Summarize button, waiting/none states).
|
||
- R5: chat inside an expanded card works exactly as on the detail page (reuse `chatReveal`/
|
||
`chatSection` + the existing `/v/{id}/chat` endpoints, unchanged).
|
||
|
||
**Decision / architecture.**
|
||
1. **Reuse the existing fragments.** `summaryBody(r)` and `chatReveal(videoID)` already exist and
|
||
render the detail page; a new `expandedCard(r, chatEnabled)` composes the compact header + a
|
||
collapse control + `summaryBody` + `chatReveal`. `DetailPage` is refactored to also compose
|
||
`summaryBody` so the two never drift (DRY).
|
||
2. **Two fragment endpoints** (mirroring the existing list/status HTMX fragment pattern):
|
||
`GET /v/{videoId}/expand` → `expandedCard`; collapse reuses the existing compact `VideoCard`
|
||
via `GET /v/{videoId}/card`. Both are list-card `<li>` fragments with the SAME `id`
|
||
(`video-{id}`), swapped `outerHTML` — same mechanism as `processingCard`/`VideoCard` today.
|
||
3. **The compact card's title/"Read" affordance** becomes `hx-get=/v/{id}/expand`,
|
||
`hx-target=#video-{id}`, `hx-swap=outerHTML`, with `href=/v/{id}` as the no-JS fallback (R3).
|
||
The expanded card's collapse control is the inverse (`hx-get=/v/{id}/card`).
|
||
4. **Only when `r.Summarized`** does the expand affordance render (R4); the other states are
|
||
unchanged.
|
||
5. **v1 does NOT push the URL** (`hx-push-url`) — expand/collapse is ephemeral list UI state; the
|
||
detail page remains the deep-link/shareable URL. Deep-linking the open state via `hx-push-url`
|
||
is noted as a later option (needs list-state restore on back).
|
||
|
||
**Out of scope / later.** URL push / deep-linkable open state; the visual refresh (#17) — though
|
||
the expanded-card markup is where #17's TUI/charm styling will land, so they pair.
|
||
|
||
**Reversibility.** Additive: two fragment endpoints + one templ + an affordance swap on the
|
||
compact card. Removing the affordance reverts to plain list→detail navigation; the detail page is
|
||
untouched. No schema change.
|
||
|
||
**Next steps (gated):** on approval → BDD (`docs/use-cases/inline_expand.feature` + coverage
|
||
map) → TDD → implement → SemVer + docs.
|
||
|
||
---
|
||
|
||
## ADR-032 — Visual refresh: one charm-reader layout, light + dark themes
|
||
|
||
**Status:** Proposed (2026-06-12). Issue #17. **Draft for review — no code yet.** Follows the
|
||
sketch-first explore step (3 throwaway mockups in `docs/sketches/`, screenshotted for review).
|
||
|
||
**Context / decision.** The UI is flat. From the mockups, directions **B (light reader + charm)**
|
||
and **C (dark cozy terminal)** are the SAME layout — readable sans body, monospace meta, charm
|
||
palette accents, lipgloss-style bordered cards — in two palettes. Direction A (full-monospace TUI)
|
||
is dropped as too heavy to read long summaries. Decision: ship that one layout with **both a light
|
||
theme (B) and a dark theme (C)**, user-toggleable, defaulting to the OS preference.
|
||
|
||
**Requirements.**
|
||
- R1: one set of markup/structure; the two themes are pure palette (CSS variables), no duplicate templates.
|
||
- R2: a **theme toggle** persisted across visits; default to `prefers-color-scheme` when no choice stored.
|
||
- R3: charm language in both — mint/purple/pink accents, mono meta + section labels, lipgloss
|
||
bordered/gradient cards, the (fixed) ASCII tapir; readable sans body.
|
||
- R4: style the existing pieces — list, compact card, **expanded card incl. the already-present
|
||
video embed (ADR-031/summaryBody)**, detail page, chat dock, the queue note, the charm spinner.
|
||
- R5: WCAG-AA contrast for body text in BOTH themes; keep `prefers-reduced-motion` (already honoured).
|
||
- R6: stay HTMX+Templ; no CSS framework.
|
||
|
||
**Architecture.**
|
||
1. **Palette as CSS variables.** `:root` holds the light (B) tokens; `:root[data-theme="dark"]`
|
||
holds the dark (C) tokens; a `prefers-color-scheme: dark` media block sets the dark tokens when
|
||
no explicit `data-theme` is set. All component CSS references variables only (R1). The existing
|
||
`CharmMint/Purple/Pink/Cream/Dim` Go consts remain the source for the spinner's inline colours.
|
||
2. **Theme toggle** = a small inline script (a dozen lines, no framework) in `Layout`: on load,
|
||
apply stored theme (localStorage) or fall through to the media query; a header toggle button
|
||
flips `data-theme` on `<html>` and stores it. This is the one new bit of JS; everything else
|
||
stays server-rendered + HTMX. (Considered: cookie + server-render — rejected, a full round-trip
|
||
per toggle is clunky for a pure presentation flip.)
|
||
3. **Scope = the `styleTag` CSS in `view.go`** (the single style source) plus tiny class hooks in
|
||
the templ where needed; content/structure are unchanged, so existing view tests keep passing.
|
||
|
||
**Verification.** The deployed UI is behind auth (web-shot can't log in), so visual review is via
|
||
the mockups now + a styled full-set mockup screenshot before merge, then a live eyeball on device.
|
||
Automated tests stay structural/behavioural (theme tokens present, toggle persists, expanded card
|
||
embeds the video, no-JS still renders a readable default) — colours are not unit-tested.
|
||
|
||
**Reversibility.** A CSS theme swap + one small script + a few class hooks; revert `styleTag` to
|
||
roll back. No schema, no structural change.
|
||
|
||
**Next steps (gated):** on approval → BDD (`docs/use-cases/visual_theme.feature` + coverage map)
|
||
→ TDD → implement → SemVer + docs.
|
||
|
||
---
|
||
|
||
## ADR-033 — Upload audio, get a private Swedish transcript, discuss it (user-supplied STT)
|
||
|
||
**Status:** Proposed (2026-10-01). **Decider: Mathias — pending.** Issue #33 (audio-only subset of
|
||
#28). **Supersedes ADR-007's speech-to-text deferral for user-supplied audio only.** The
|
||
YouTube audio-download rejection in ADR-007 **stands untouched**. Builds on ADR-012 (isolation),
|
||
ADR-018 (single replica), ADR-021 (**the constraint**), ADR-022 (endpoint chain), ADR-025
|
||
(honest status), ADR-027 (chat). ADR-014 item 4 ("Whisper reconsideration is a future ADR,
|
||
gated on caption data") is a different question — whether to transcribe YouTube videos with no
|
||
captions — and this ADR does not answer it. Measurements: `docs/research/audio-length-ceiling-2026-10-01.md`.
|
||
|
||
**Context.** The maintainer has a concrete need: a Swedish Teams voice-only export, 30–90
|
||
minutes, to upload and then *discuss* (what was decided, what is open, who owes what). Every
|
||
piece exists — a Whisper endpoint on the gateway (infra#274), chat-with-transcript (ADR-027) —
|
||
and nothing connects them.
|
||
|
||
ADR-007 deferred STT for three reasons: yt-dlp fragility, ToS-grey scraping, and GPU contention
|
||
with the JEPA PoC. Upload-STT shares **none** of them: the user owns the file, nothing is
|
||
scraped, there is no caption rate gate, and the contention moved from koala to iguana. What it
|
||
does share is landing Whisper in the codebase, which ADR-007 deferred — so a new ADR is required,
|
||
and ADR-017's process note records the one time a guardrail-reversing change shipped without one.
|
||
|
||
**The constraint that shapes everything.** ADR-021 made `transcripts` a shared, **non-RLS**
|
||
table on the explicit grounds that it holds only public caption content and nothing
|
||
user-identifying. A recorded internal meeting is the opposite: private third-party content owned
|
||
by one user. The existing isolation test asserts `transcripts` *is* non-RLS, so a naive reuse
|
||
would leak across tenants **and pass CI green**.
|
||
|
||
**What was measured, not argued** (research doc above):
|
||
- **No transcription ceiling up to 90 minutes.** One 174 MB request through the LiteLLM NodePort
|
||
took 440 s (~12× realtime). The first failure is the public ingress `llm-api.d-ma.be`
|
||
returning `413` above roughly 8–10 MB (~5 min of WAV) — not whisper, and not a Metal OOM.
|
||
The 8:20 failure recorded in #33 is consistent with that cap.
|
||
- **`response_format=srt` survives LiteLLM, wrapped** in a JSON envelope (`{"text": "<srt>"}`).
|
||
- **The two backends differ.** `iguana/kb-whisper` returns whole-second segments;
|
||
`berget/kb-whisper` returns sub-second times and a different `language` token.
|
||
- **The existing chat path truncates at 18,000 chars (~22 min of speech), keeping the head.**
|
||
#33 assumed chat "retrieves rather than generates" and so has no length ceiling. It does
|
||
not retrieve; it stuffs the transcript into the prompt and cuts the tail. Measured against
|
||
the offered models: `phi4-mini` 400s at ~13k tokens; `gemma4-26b` is right to 90,000 chars and
|
||
silently drops the head at 120,000.
|
||
|
||
**Unknowns, stated.** Real-speech accuracy; compressed-input behaviour; concurrent long jobs on
|
||
iguana; and the Metal OOM, deliberately not provoked. (Whether the offered chat models can hold
|
||
an hour of Swedish *was* an unknown here; it is now measured, see decision 8.)
|
||
|
||
**Decision.**
|
||
|
||
1. **Audio is what is used; the transcript is the only thing kept.** An authenticated user
|
||
uploads a recording; Tapir transcribes its audio and stores the transcript. The audio itself
|
||
is deleted after transcription (temp file, removed in a `defer`, swept at start-up) — no
|
||
custody problem to solve, and nothing to hand to #31. **A file that also carries a video
|
||
track is accepted: its audio is extracted, the video is discarded, and the user is told so**
|
||
(see 4). *Amended 2026-10-01, the same day this ADR was drafted: the first draft rejected
|
||
video and named #28. The maintainer's first real recording — a Teams meeting with the camera
|
||
off — is still an `.mp4` with a video track, so rejecting video rejected the primary use
|
||
case.* This keeps no video and analyses none: playback, keyframes and what-is-shown analysis
|
||
remain #28. YouTube audio-download stays rejected. No public/anonymous upload (ADR-008).
|
||
|
||
2. **Private store: a separate RLS-scoped table, `uploaded_transcripts`** (resolves #28's
|
||
decision 1). `user_id NOT NULL REFERENCES users ON DELETE CASCADE`, `ENABLE` + `FORCE ROW LEVEL
|
||
SECURITY`, the same `tapir.current_user_id` policy as every other user-owned table. It carries
|
||
the transcript text and its timestamped segments (re-transcribing 90 minutes costs ~7
|
||
minutes; discarding the timestamps is irreversible), `status`, the `endpoint` that served
|
||
it, `audio_seconds`, and a typed `error_kind`. **Not** a `user_id` column on `transcripts`
|
||
with a partial policy, even though that would reuse more code: ADR-021's exception is
|
||
precisely that *that table holds no private row*, the isolation test asserts it has no
|
||
`user_id`, and a partial policy would turn "single exception" into "single exception, with a
|
||
private lane inside it" — the shape under which a future query drops the policy clause and
|
||
leaks. Two tables keep both boundaries independently assertable.
|
||
|
||
3. **The isolation test asserts both boundaries and the set.** (a) User A cannot read, update or
|
||
delete user B's uploaded transcript; no GUC set → deny-all. (b) `transcripts` is still
|
||
non-RLS with no `user_id`. (c) **The set of public-schema tables with row security off is
|
||
asserted exactly** — so a new table added without RLS fails CI instead of passing it. (c) is
|
||
new; the existing test only checks the named tables.
|
||
|
||
4. **Validate by probing, reject clearly, and say what was ignored.** Format, duration and
|
||
what a file contains cannot be read from a filename or MIME type — a Teams recording is an
|
||
`.mp4` whether or not the camera was on. The upload is probed (`ffprobe`): **accept any
|
||
container with an audio stream, whether or not it also has a video stream** (an mp3's cover
|
||
art counts as neither). Whether a real video track was present is recorded on the row
|
||
(`had_video`) and the transcript page says so plainly — *this file contained a video track;
|
||
only its audio was used and the video was discarded* — so the user is never left to wonder
|
||
what happened to the picture. Reject a file with **no** audio stream (including a screen
|
||
recording with no microphone) and an unreadable one. Limits, all configurable, all rejected
|
||
synchronously with a specific message rather than by timeout: **200 MB** upload and
|
||
**90 minutes** duration. 90 minutes is *the longest length measured*, not a ceiling that was
|
||
found; raising it is a measurement, not an edit. The size cap now matters more: a video file
|
||
is much larger than its audio and is spooled whole before the audio is extracted, so a long
|
||
recording with screen sharing can exceed 200 MB while its audio is a few tens of MB.
|
||
|
||
5. **Transcode server-side, always, to 16 kHz mono PCM WAV** — even when the input already is,
|
||
rather than assuming. whisper-server's `--convert` is rejected (its `--tmp-dir` is created at
|
||
process start and rots under macOS periodic cleanup). The transcoder sits behind a port; the
|
||
tests use a fake. **This puts `ffmpeg` and `ffprobe` in the runtime image**, which is
|
||
distroless-static today. **Provisional: copy static ffmpeg/ffprobe binaries into the existing
|
||
distroless image** (no shell, no package manager retained). This is a heavy new dependency
|
||
and is **the maintainer's call** — see Open items.
|
||
|
||
6. **Transcription adapter via the gateway, never the public hostname.** A `Transcriber` port;
|
||
the implementation calls `POST {TAPIR_GATEWAY_URL}/audio/transcriptions` where the gateway
|
||
URL is the in-cluster NodePort (`koala:30401/v1`). Primary `iguana/kb-whisper`, fallback
|
||
`berget/kb-whisper` (immune to iguana contention, which can wedge permanently — infra#273).
|
||
- **Fallback triggers on any primary failure** (transport error, timeout, non-2xx). It never
|
||
branches on the body: `{"error":"failed to process audio"}` has had at least five distinct
|
||
causes.
|
||
- **`response_format=verbose_json`**, normalised to one segment shape: times clamped to the
|
||
audio duration (the final segment can end past it), one language token. Not `srt`: the
|
||
envelope adds a parse step and iguana's segment timing is the same either way.
|
||
- **No chunking in v1.** Nothing failed to 90 minutes (measured). Chunk-and-stitch is the
|
||
answer *if* the cap is raised or smaller requests become necessary — see triggers.
|
||
- **The serving endpoint is recorded per row**, and surfaced, because the two backends
|
||
differ in timestamp granularity: which one served a transcript is a quality fact.
|
||
- **Berget is a cloud endpoint.** On fallback the audio of a private meeting leaves the
|
||
homelab. #33 asks for this fallback; it is configurable (`TAPIR_TRANSCRIBE_FALLBACK_MODEL`,
|
||
empty disables), recorded per row, and called out in Open items rather than defaulted
|
||
silently.
|
||
- Swedish only (`language=sv`). Language detection and routing (#28 decision 2) are not
|
||
touched.
|
||
- `-H 'Expect:'`, multipart field `file` — encoded in the adapter and its tests, not rediscovered.
|
||
|
||
7. **Jobs run in-process, one at a time (ADR-018 stands).** Transcription is minutes (~7.5 min
|
||
for 90) but a single goroutine behind a one-slot semaphore: two simultaneous 90-minute jobs on
|
||
iguana were not measured, so concurrency is 1 until they are. No worker, no new deployable,
|
||
no second replica. Honest status per ADR-025: `queued → transcribing → done | failed`, polled
|
||
by the same `/status` pattern, every terminal state a rendered card, never a stuck spinner.
|
||
Audio is not retained, so a restart mid-job **cannot resume**: at start-up any
|
||
`queued`/`transcribing` row is marked `failed` with `error_kind = interrupted`, and the user
|
||
re-uploads. The single-replica assumption gains a second dependent (the semaphore is
|
||
process-local) and is recorded here.
|
||
|
||
8. **Chat reuses `chat.Service`; it is not forked.** The uploaded-transcript chat handlers
|
||
look the row up through the RLS-scoped store, so a non-owner gets 404 by the same mechanism
|
||
as ADR-027's summary gate, **before any model is called**. ADR-027's decisions (ephemeral,
|
||
switchable models, trust-the-model) carry over unchanged. **Two things must differ, and both
|
||
were measured** (`docs/research/chat-context-over-long-transcripts-2026-10-01.md`):
|
||
- **The transcript budget.** The 18,000-char caption default cuts a one-hour meeting to its
|
||
first third. Uploaded transcripts take their own `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS`,
|
||
default **90,000** — the measured ceiling of the best offered model (`iguana/gemma4-26b`),
|
||
which answers correctly there but **silently loses the head of a 120,000-char transcript**
|
||
with no error and no truncation flag. (An earlier draft of this ADR proposed 120,000 as a
|
||
guess; the measurement refuted it.) The mechanism is a per-request `chat.Request.MaxChars`.
|
||
- **The opening model.** The offered list leads with the summarizer default `koala/phi4-mini`,
|
||
whose 8,192-token context returns HTTP 400 on an hour of speech (~13,000 tokens). Chat over
|
||
an upload opens on `TAPIR_UPLOAD_CHAT_MODEL` (default `iguana/gemma4-26b`); the switcher
|
||
lists **local models only** (amended by ADR-035 after the adversarial review found it listed
|
||
the cloud model too: the whole private transcript goes to the one chosen), and picking
|
||
`phi4-mini` on a long recording fails loudly and inline. A per-model budget is not built.
|
||
Past ~90,000 chars (≈100 minutes) no offered model can answer by stuffing; see triggers.
|
||
|
||
9. **Not decided here.** Rich `AVSNITT`/`ALTERNATIV` analysis (gated on #30 — `analyze_srt.py`
|
||
hardcodes `max_tokens: 6000`); video, keyframes, synced playback (#28); language detection;
|
||
per-transcript deletion UX (account deletion cascades; see Open items); retention of audio.
|
||
|
||
**On the sequencing question #28 asks of itself** — *is this the thing that best serves the
|
||
Stage-0 gate, or the most interesting thing to build?* Recorded answer: it proceeds **ahead of
|
||
#32** (the demand test) as the maintainer's explicit, knowing call (#28, comment of 2026-10-01),
|
||
on a concrete need rather than speculation. It is not a return-nudge (ADR-020/027): it adds
|
||
nothing that prompts a user to come back, so it does not contaminate the unprompted-return signal
|
||
ADR-016 measures. The honest risk is the one ADR-016 names: it is larger than anything ADR-020
|
||
deferred. The scope cut — chat first, no rich analysis, no video — is what keeps it bounded.
|
||
|
||
**Consequences.**
|
||
- **Positive.** The user's request is met end to end with existing parts. ADR-021's exception stays
|
||
singular and now provably so. The measured ceiling removes chunking from v1. No audio is stored.
|
||
- **Negative / watch.**
|
||
- Tapir's runtime image gains `ffmpeg`: a larger image and a new CVE surface to patch.
|
||
- iguana becomes load-bearing for a user-facing feature (#28 decision 3; infra#273 is still the
|
||
dependency that matters) — a Metal OOM now means *every upload fails over to berget*, i.e.
|
||
private audio leaving the homelab, silently unless the per-row endpoint is surfaced.
|
||
- A 90-minute job holds the single slot for ~7.5 minutes; a second user's upload queues behind it.
|
||
- Chat over a long transcript is bounded by model context, not by anything in Tapir, and the
|
||
failure at the edge is *silent* for gemma (the head is dropped, no error). The 90,000 default
|
||
sits just inside the measured limit and is only as good as that single measurement.
|
||
- **The default chat model for uploads is a large ollama model on iguana — the same host as
|
||
`iguana/kb-whisper`.** infra#273 is exactly that a large resident ollama model can Metal-OOM
|
||
whisper-server permanently. Chatting with a transcript can therefore load the model that
|
||
wedges the transcriber. Not solved here; the berget fallback limits the blast radius.
|
||
- Tapir's own ingress body limit for a 200 MB upload is **unverified** (it lives in `infra`).
|
||
- Accepting video means large files are spooled before their audio is extracted, and ffmpeg
|
||
demuxes (but, with `-vn`, does not decode) video it then throws away. A client-side audio
|
||
extraction would avoid the upload cost; it is not built.
|
||
|
||
**Validation / resolution triggers** (each observable by someone other than the author):
|
||
- *Chunking*: build it when the duration cap is raised above 90 min, or when a measured
|
||
request at the current cap fails with whisper-side (not ingress) errors.
|
||
- *Berget-first*: if more than a quarter of uploads in any week are served by the fallback,
|
||
swap the primary/fallback order and revisit infra#273.
|
||
- *Retrieval chat*: build chunked retrieval when a recording's transcript exceeds the measured
|
||
90,000-char ceiling (≈100 minutes) — i.e. the moment the duration cap is raised above that, or a
|
||
real recording turns out denser than the fixture — or when the offered model list changes and
|
||
the ceiling is re-measured lower.
|
||
- *Worker*: if a transcription job ever degrades the reading UI, or Tapir runs more than one
|
||
replica, ADR-018 is revisited.
|
||
- *The decision itself*: if the maintainer has not used the feature unprompted within the
|
||
ADR-016 window, it was the interesting thing, not the useful one — remove it.
|
||
|
||
**Open items this ADR creates.**
|
||
- **DECIDED (Mathias, 2026-10-02): option A.** Static ffmpeg and ffprobe 7.0.2 are copied into the
|
||
distroless image, pinned by version and sha256 (Dockerfile in this PR; #34). Measured at that time:
|
||
**about 160 MB** added, not the 70 MB first estimated (80 MB per binary); the build is **GPLv3**;
|
||
the checksum proves the bytes reviewed, not the publisher; upstream's `releases/` URL moves on, so
|
||
vendoring the verified tarball is the durable fix (not done). Uploads stay off until the deployment
|
||
sets `TAPIR_UPLOADS_ENABLED`. *The question as it was put:* *Options:* (A) static ffmpeg/ffprobe copied
|
||
into distroless **(recommended — the only option that satisfies "accept m4a/mp3/opus/mp4,
|
||
validate duration, reject video"; pure-Go decoders cover wav/mp3 only)**; (B) accept WAV
|
||
only, convert client-side (fails "I just want to upload it"); (C) a transcode sidecar
|
||
(second deployable, ADR-018's reasoning against). *Blocks:* the Dockerfile slice, not the
|
||
tests, which use the port.
|
||
- **DECIDED (Mathias, 2026-10-02): option A, enabled by default (#35).** Scope: the audio
|
||
transcription fallback only. Reconciliation text and chat over a recording never go to a cloud
|
||
model (ADR-034, ADR-035). *The question as it was put:* *Options:* (A) enabled by default as
|
||
#33 specifies, with the serving endpoint shown **(recommended, matches the issue)**; (B)
|
||
disabled by default, a failed iguana job fails honestly and the user retries. *Blocks:*
|
||
the production default only; the adapter ships either way.
|
||
- Verify Tapir's ingress `proxy-body-size` against the 200 MB cap (`infra`).
|
||
- Per-transcript deletion is not built; account deletion cascades. A private-content store
|
||
without a delete button is a gap worth a ticket, not a silent omission.
|
||
|
||
**Reversibility.** Additive: one table (down-migration drops it), one adapter, one upload
|
||
surface, one image change. Removing the upload route removes the feature; `transcripts`, `videos`
|
||
and ADR-021 are untouched.
|
||
|
||
---
|
||
|
||
## ADR-034 — Transcribe with two engines and let a local model reconcile their disagreements
|
||
|
||
**Status:** Proposed (2026-10-01). **Decider: Mathias — pending.** Issue #40 (decision B) and #33.
|
||
Builds on ADR-033 (private upload transcripts), ADR-022 (endpoint chain, a pattern reused), ADR-018
|
||
(single replica), ADR-025 (honest status). Evidence:
|
||
`docs/research/asr-kb-whisper-vs-pianissimo-2026-10-01.md` and
|
||
`docs/research/local-model-adjudicator-sweep-2026-10-01.md`.
|
||
|
||
**Context.** On the first real one-hour recording `iguana/kb-whisper` and `iguana/pianissimo` were
|
||
a near tie on accuracy (the speaker's blind verdicts on the 30 meaning-changing disagreements:
|
||
pianissimo 17, kb-whisper 13) and they **err in different places**: pianissimo is better on terms and
|
||
numbers, kb-whisper on ordinary words; kb-whisper's errors garble, pianissimo's read as plausible.
|
||
Pianissimo is ~6x faster (40 s vs 4m08s for 63.7 min) and runs through the same adapter. Two engines
|
||
that disagree in ~380 places per hour, with a ~92% word agreement, is information a single-engine
|
||
pipeline throws away. Separately, eleven local models were scored on adjudicating those disagreements
|
||
against the speaker's answers: **`gemma4-31b` 97% and `gemma4-26b` 92%** passed the pre-set bar
|
||
(>= 80% accuracy, >= 85% order-swap consistency, <= 10% unparseable), nothing else did.
|
||
|
||
**Unknowns, stated.** The adjudicator evidence is 30 items from one speaker and one domain, chosen by
|
||
Opus as decidable; the other ~350 disagreements (fillers, compounds, small words) have no ground
|
||
truth. Whether a *merged* transcript is better than the better single engine has not been measured.
|
||
Reconcile time per hour is estimated, not measured. `gemma4-31b`'s slowness was Low Power Mode on
|
||
iguana: with it off the 31B decodes at 22.3 tok/s, not 7.9 (2026-10-02, same-machine before/after on
|
||
a different synthetic prompt; see the research note). The reconciler default `gemma4-26b` was chosen
|
||
on the slow numbers and is reopened: 31B scored 97% against 92%.
|
||
|
||
**Decision.**
|
||
|
||
1. **Two engines, one WAV, both local, one after the other.** Each upload is transcribed by
|
||
`iguana/pianissimo` and `iguana/kb-whisper` from the same converted WAV, sequentially (concurrency
|
||
stays 1, ADR-033). Each engine fails independently; the berget fallback (ADR-033) applies only when
|
||
**both** fail. Off by default: with no second engine configured nothing changes.
|
||
|
||
2. **Align by word, with pianissimo as the base text.** Both texts are tokenised (lower case, letters
|
||
and digits only) and diffed word by word (Myers); neighbouring differences separated by a single
|
||
equal word are merged into one **dispute**. The base is pianissimo: sentence-level prose with
|
||
punctuation and millisecond timestamps, where kb-whisper's output is wrapped fragments. **This
|
||
choice is about readability and timing, not accuracy — accuracy was a tie — so it is the cheaper
|
||
mistake, not a measured winner.** A replaced span takes the alternative's own words.
|
||
|
||
3. **Resolve the trivial ones without a model.** A dispute that is identical once spaces are removed
|
||
(a compound split or joined) keeps the base; so does a dispute that is only a short filler word
|
||
on one side. Everything else is adjudicated. (Opus's classification of 380 disputes found ~46
|
||
compound and ~26 punctuation-only residue, ~65 filler.)
|
||
|
||
4. **A local model adjudicates the rest, bounded.** Each dispute goes to a local chat model (default
|
||
`iguana/gemma4-26b`; `gemma4-31b` is configurable and scored higher but is ~5x slower) with ~100
|
||
words of context either side and the two readings, in the wording that was measured. The reading
|
||
order alternates by dispute so position bias averages out. The verdict is *base*, *alternative* or
|
||
*unsure*; **unsure keeps the base and is flagged**. An optional order-check mode asks twice with the
|
||
order swapped and treats disagreement as unsure; it is **off by default** because it doubles the time
|
||
and the measured consistency of the passing models is 93%.
|
||
- **The model must be local.** Its alias must begin `koala/` or `iguana/`; a cloud alias is a
|
||
start-up configuration error. Private meeting content never goes to a cloud model for this step.
|
||
- **Bounded impact.** The reconciler can only choose between two readings the engines already
|
||
produced; it cannot write new text. Speech that says "answer 2" can at worst flip one dispute.
|
||
|
||
5. **Reconciliation never fails a job.** One engine fails → its text is used, recorded as
|
||
single-engine. The adjudicator is down (three consecutive errors) → the base text is used
|
||
unreconciled, recorded as skipped with the reason, and the remaining disputes are not retried one
|
||
timeout at a time. Both engines fail → ADR-033's fallback path.
|
||
|
||
6. **What is stored and shown.** `content` is the merged text. `segments` stay the base engine's timed
|
||
segments, untouched (**amended by ADR-035: they now carry the merged words, timing stays the
|
||
base's**): `content` is authoritative for chat, `segments` only for time. `endpoint`
|
||
lists every contributor and the reconciler. A new `reconciliation` column (migration 019, same
|
||
RLS-scoped table) records the status, counts, and the **uncertain and changed spots with a minute
|
||
mark and both readings**, so the user can check them against the audio. The transcript page says in
|
||
plain words what was done — "reconciled from two transcripts: N differences, K uncertain" — and
|
||
lists the uncertain spots. A reconciled transcript is never presented as simply "the" transcript.
|
||
|
||
7. **Configuration.** `TAPIR_RECONCILE_BASE_MODEL` (empty = off; the engine whose text is the base,
|
||
e.g. `iguana/pianissimo` — `TAPIR_TRANSCRIBE_MODEL` stays the other engine, e.g. kb-whisper),
|
||
`TAPIR_RECONCILE_MODEL` (the adjudicator; required when a base engine is set),
|
||
`TAPIR_RECONCILE_CHECK_ORDER` (off). All three models must be local aliases (`koala/` or `iguana/`):
|
||
a cloud alias is a start-up error, and the base engine must differ from the other one. The default
|
||
job timeout rises from 30 to 60 minutes: ~4 min of kb-whisper and ~40 s of pianissimo per hour of
|
||
audio, plus the adjudication.
|
||
|
||
8. **Not decided here.** A cloud adjudicator (rejected: private content); tuning the prompt; making
|
||
`gemma4-31b` the default; a "reconciling" progress state (the page says "transcribing" for the whole
|
||
job); feeding the uncertain spots to chat; whether pianissimo should also be a middle tier in the
|
||
fallback chain (#40, option A).
|
||
|
||
**Consequences.**
|
||
- **Positive.** Uses the finding that the engines err differently; stays local end to end; degrades
|
||
to today's behaviour at every failure; and is honest on the page about what it did.
|
||
- **Negative / watch.**
|
||
- Job time grows: roughly 300 adjudicated disputes x ~2 s on `gemma4-26b` ~ 10 minutes per hour of
|
||
audio — an estimate, not a measurement; double with order-check on, ~5x with `gemma4-31b`.
|
||
- Two gemma-class calls sit on iguana beside whisper. During the sweep a 17-20 GB model was resident
|
||
beside whisper-server without wedging it, far below the 52 GB that did (infra#273); the hazard is
|
||
reduced, not removed.
|
||
- A merged transcript can be wrong where both engines are wrong, and where the adjudicator picks
|
||
the wrong one (it was right 92% on a hand-picked set, ~97% for 31b).
|
||
- Reading the page takes more words: the reconciliation summary is new surface to keep honest.
|
||
|
||
**First live smoke (2026-10-01, synthetic audio, 3 disputes):** all 3 were changed to the
|
||
alternative's reading; against the known script 2 were right and **1 was wrong** (a name replaced by a
|
||
pronoun, a correct base reading lost). That is one run of three, not a measurement, but it is the failure
|
||
this design must expect: the reconciler can overwrite a correct reading, which is why changed spots, not
|
||
only uncertain ones, are listed on the page, and why the decision below is a blind check on a real
|
||
recording.
|
||
|
||
**Validation / resolution triggers.**
|
||
- **The test that decides it:** re-upload the same real recording after merge; the speaker judges, blind,
|
||
~30 disputes where the reconciled text differs from pianissimo alone. Keep the feature only if the
|
||
reconciled reading is right at least 85% of the time **and** no worse than the better single engine.
|
||
If not, switch it off (config) and keep the single-engine path.
|
||
**Decision (Mathias, 2026-10-02), on the first result below: keep it as an opt-in feature**, reading
|
||
the bar as 95% of decided items, with the second condition to be answered on the next real run.
|
||
For deployment the reconciler is `iguana/gemma4-31b` (97% against the speaker's answers, about 22
|
||
minutes for the 63.6-minute recording with Low Power Mode off). A cloud reconciler was raised and is
|
||
**not adopted**: it would send private transcript text to a cloud API on every upload; recorded as
|
||
an evaluation question in #45.
|
||
**First result, 2026-10-02, before merge** (`docs/research/reconcile-blind-check-2026-10-02.md`): 30
|
||
random replacement spots on the real 63.6-minute recording, judged blind by Opus from text (not by the
|
||
speaker, who delegated it): 21 right, 1 worse than pianissimo, 3 neither reading right, 5 undecidable.
|
||
That is 95% of decided items but 70% of all 30; the bar did not say how ties count, so it is **met on
|
||
one reading and missed on the other**, and the speaker has not ruled. The second condition, no worse
|
||
than the better single engine, is **not answered**: kept-base spots are not stored, so kb-whisper alone
|
||
cannot be compared. Left `Proposed`; the feature stays off by default.
|
||
- Reconcile time above 25 minutes per hour of audio: drop to a smaller adjudicator or skip more
|
||
disputes automatically.
|
||
- A different recording or speaker in which the engines agree far less (or far more): re-measure before
|
||
trusting the 92%/97% figures.
|
||
|
||
**Reversibility.** Off by default and additive: the second engine, the reconciler and the column are
|
||
unused when the settings are empty; the down-migration drops the column; the single-engine path is
|
||
unchanged.
|
||
|
||
---
|
||
|
||
## ADR-035 — Show an upload's progress as real phases, log the same transitions, and keep every judged spot
|
||
|
||
**Status:** Proposed (2026-10-02). Decider: Mathias (pending). Amends ADR-034 decision 6 (segments).
|
||
Closes #42, #43 and #44.
|
||
|
||
**Context (measured, not assumed).** The first real run (a 63.6-minute recording, 21 min 49 s) exposed
|
||
three gaps. (1) The page said only `queued`/`transcribing`/`done`; during 22 minutes nothing told the
|
||
user whether the job was working, hung or dead, and after a crash the row would say `transcribing`
|
||
until the next start. (2) The server log had no line about the job between start-up and the final
|
||
line, so an operator saw it only from outside (spool files, iguana CPU). (3) Evaluating the result
|
||
showed that spots the model *kept* as the base are counted but not stored (so ADR-034's "no worse than
|
||
the better single engine" cannot be measured after the fact, and the audio is not retained), insertions
|
||
and deletions cannot be placed without context, and the saved `segments` still carry the base engine's
|
||
wording while only `content` is merged.
|
||
|
||
**Decision.**
|
||
|
||
1. **Phases are data on the row.** A new `progress` JSONB column (migration 020, same RLS-scoped table)
|
||
holds an ordered list of steps `{key, label, state, started_at, finished_at, done, total}` with state
|
||
`pending|running|done|failed|skipped`. The plan is declared up front so the page shows what is coming:
|
||
received, preparing audio, then the transcriber's own steps (two-engine: engine A, engine B,
|
||
comparing, deciding n of N; single: transcribing), then saving.
|
||
2. **One source for the page and the log.** An `internal/progress` Tracker is the only writer. Every
|
||
transition is saved to the row and logged by the same call, so the UI and the log cannot disagree.
|
||
Engines and the reconciler report through a context-carried `Reporter` (no-op when absent), so the
|
||
`Transcriber` interface does not change.
|
||
3. **Alive is not the same as progressing, and the page says which it knows.** A heartbeat goroutine,
|
||
independent of any engine call, stamps `alive_at` every 10 s; `progress_at` moves only on a real
|
||
transition or advance. The page derives: *running*; *quiet* (the deciding step has not advanced for
|
||
5 minutes, so the model is probably waiting or slow); *lost* (`alive_at` older than 45 s, so the
|
||
service stopped reporting and may have restarted or crashed). The two engine calls are single blocking
|
||
requests, so inside them the page shows elapsed time only: **no percentage is ever shown that does not
|
||
come from real counts.** Only the deciding step has a bar.
|
||
4. **The page.** A server-rendered step list in tapir's existing tokens and mono font, swapped in place
|
||
by the existing HTMX poll (every 2 s while unfinished, none once done or failed). Finished steps
|
||
collapse to one line with their duration; the running step has a CSS-only spinner that respects
|
||
`prefers-reduced-motion`; a failed step and a lost service are visibly different from running.
|
||
No new JavaScript, framework or stylesheet.
|
||
5. **Logs.** One INFO line per phase transition and engine call (upload id, endpoint, elapsed), the
|
||
reconcile counts, `adjudicated n/N` at most once a minute, a WARN when an engine soft-fails or the
|
||
breaker trips, and a "still in phase X" line every 5 minutes. **Ids, counts, durations, endpoints and
|
||
error kinds only: never transcript text, disputed words or the filename**, asserted by a test.
|
||
6. **Every judged spot is stored.** `reconciliation.contested` keeps each model-decided spot with its
|
||
verdict (changed, kept base, uncertain) and a few words of left context, bounded at 2,000. The page
|
||
still lists only changed and uncertain spots. This makes ADR-034's second condition measurable on the
|
||
next real run.
|
||
7. **Segments carry the merged words.** The reconciled words replace the base's in the saved `segments`;
|
||
timing stays the base's. If the segments cannot be aligned to the merged text (a word wrapped across
|
||
a segment edge), they stay the base's and the result says so in the log. This replaces ADR-034
|
||
decision 6's "segments stay untouched".
|
||
|
||
8. **Hardening from the adversarial review** (`skills:pr-gatekeeper`, head `7029dd1`: BLOCK; recorded on
|
||
PR #39). Each finding was reproduced by a failing test first.
|
||
- **Chat over a recording offers and accepts local models only** (it listed `berget/mistral-small`);
|
||
`TAPIR_UPLOAD_CHAT_MODEL` must be local at start-up; with no local model, nothing is sent.
|
||
- **A model alias counts as local only if it is one plain name after `koala/` or `iguana/`**
|
||
(`koala/../berget/x`, `koala//x`, `koala/x?host=…` were accepted by a bare prefix check).
|
||
- **An error reaches a log line, the row or the page as a kind** (`timeout`, `canceled`, `http_502`,
|
||
`network`, `error`), never its text, which can hold a provider's response body, the spool path or
|
||
a word from the transcript (`internal/errkind`). The stored reconcile reason and an engine's
|
||
failure reason are kinds; the page no longer shows provider text.
|
||
- **Every local engine call has its own deadline** (`TAPIR_TRANSCRIBE_ENGINE_TIMEOUT`, default 20
|
||
minutes: two wedged engines in sequence then leave 20 of the job's 60 minutes for a fallback),
|
||
so a wedged engine fails while the job lives and the fallback can run; the job's own
|
||
deadline or a cancel still never reaches the cloud.
|
||
- **The duration limit is checked against what is converted**, not the file's header: ffmpeg is
|
||
capped at the limit plus a minute, a longer result fails the job (`audio_too_long`), and a result
|
||
under 80% of the claimed length and more than two minutes short is **transcribed and noted** on
|
||
the audio step ("the audio is 20 min long; the file claimed 1 h 19 min"), not refused. The second
|
||
adversarial review showed a complete variable-bitrate MP3 whose header over-estimated its length
|
||
by 4x was wrongly refused by the first version of this check; a note keeps a truncated file
|
||
visible without turning away real recordings.
|
||
- **The queue is bounded**, 20 waiting in total and 3 per user, and its room is checked before the
|
||
upload is spooled (not after 200 MB and an ffprobe run); a full queue is a 429 with a typed reason,
|
||
never a stuck row or a server error.
|
||
- **The configured upload chat model is callable.** It is added to the shared chat service's model
|
||
list when uploads are on and it is local, because the service re-maps a model it does not offer
|
||
(a valid `iguana/gemma4-31b` was being silently swapped for `koala/phi4-mini`). The video chat's own switcher
|
||
lists that local model too while uploads are on.
|
||
- **A client that hangs up mid-upload is not an error**: the handler logs INFO and writes nothing
|
||
(rendering the error page ran a database read on a cancelled context and logged a false ERROR). The
|
||
400 "cut off" answer is keyed on a marker for errors from *reading the client's body*
|
||
(`uploads.ErrClientRead`), not on `io.ErrUnexpectedEOF`, which is also what a database drop looks like.
|
||
- **A note is not hidden**: a finished upload's record opens by default when any step carries a note.
|
||
- **Bad input is a 4xx**: a non-UUID id is a 404, a malformed or cut-off multipart body is a 400
|
||
whose text is not logged, and a title with control bytes, bidirectional controls or invalid UTF-8 is
|
||
cleaned (emoji joiners, Swedish letters and variation selectors are kept), not a 500.
|
||
- **The image was built and inspected** (buildah on koala, 2026-10-02): the first build FAILED
|
||
(`tar: xz: Cannot exec`, the golang image has no xz); fixed, rebuilt at 180 MB, ffmpeg and ffprobe
|
||
run as `nonroot` in distroless. CI does not build the image on a pull request
|
||
(`if: github.event_name != 'pull_request'`), so the first CI build of it is the merge to main.
|
||
Not done, filed as follow-ups: a vendored ffmpeg tarball, CSRF cookie review, ffprobe CPU and
|
||
stream-count limits, mov external references, `alive_at` clock skew, a hung database blocking the
|
||
tracker, an unbounded gateway response body, a user-facing notice for a recording that is partial but
|
||
inside the tolerance.
|
||
|
||
**Alternatives rejected.** Server-sent events or a websocket: a long-lived connection through the
|
||
ingress, with buffering to get wrong, for a page that changes every few seconds at best; polling is what
|
||
the page already does. A percentage from the engine calls: there is no data behind it. A separate
|
||
progress table: progress belongs to the row and inherits its RLS. Logging only through the UI: an
|
||
operator has no page.
|
||
|
||
**Consequences.** Positive: running, quiet, lost and failed are distinguishable by the user and the
|
||
operator; the next blind check can answer ADR-034's second condition. Negative / watch: two more writes
|
||
per transition and one per 10 s per running job (one job at a time, so negligible); a heartbeat proves
|
||
the process is alive, not that an engine is making progress inside one request, which only the job
|
||
timeout (60 min) bounds. The stored spot list grows to a few hundred small objects per hour of audio.
|
||
|
||
**Validation.** A fake engine held open past the thresholds must change the page (quiet, lost); a log
|
||
capture on a synthetic transcript must contain none of its words; a screenshot review of the running,
|
||
quiet, lost, failed and done states against the existing pages (the visual claims tests cannot assert).
|
||
Revisit if the heartbeat write ever shows in the database's load, or if a longer recording cap makes
|
||
five-minute quiet detection too coarse.
|
||
|
||
---
|
||
|
||
## Rejected alternatives
|
||
|
||
Approaches considered during the 2026-06-02 planning + grill session and **deliberately not
|
||
taken**. Recorded so a later session doesn't re-propose them as if they were fresh ideas. Each
|
||
maps to the ADR that settles it.
|
||
|
||
| Rejected | Why rejected | Settled by |
|
||
|----------|-------------|-----------|
|
||
| Python / FastAPI / arq stack | Would be the only non-Go service in the estate, outside every shared chassis/convention/CI pattern | ADR-001 |
|
||
| Self-hosted Supabase (auth + RLS + Vault) | Duplicates Dex, ESO+1Password, and the postgres instance being actively de-coupled; three new things to run and back up | ADR-002 |
|
||
| Living inside the `hyperguild/ingestion` monolith | The only worthwhile reuse is ~100 lines (`llm`) that get copied anyway; standalone-first wants the service to owe the monolith nothing | ADR-003, ADR-004 |
|
||
| Lifting shared packages into a `brain-common` module | Couples Tapir's release cycle to the monolith for negligible code savings | ADR-004 |
|
||
| Importing/replicating the filesystem `brain` package | Assumes co-location with the brain git checkout; wrong for a standalone networked service | ADR-005 |
|
||
| Reusing `ingestion`'s `oauth` package for YouTube/Vimeo | Same name, opposite direction — it's inbound MCP-server auth, not outbound provider OAuth | ADR-006 |
|
||
| Global cross-tenant `videos`/`transcripts` table (dedup) | Reintroduces the cross-domain DB coupling the homelab review is removing; at 1–5 users, re-summarizing is cheaper than the coupling. **Transcripts half reopened by ADR-021** — the avoided cost there is a rate-gated, reputation-risky *caption fetch*, not LLM re-summarization, so it outweighs the coupling; **videos stay per-user.** | data-model.md, **ADR-021** (transcripts only) |
|
||
| Audio-download + Whisper STT in the core path | ToS-grey, breakage-prone (yt-dlp), contends for koala GPU with the JEPA PoC; captions alone test the core hypothesis | ADR-007 |
|
||
| Building multi-tenant SaaS / Google OAuth verification now | "Real users soon" was lowered to Future B; SaaS machinery before the Stage 0 self-use gate is the primary documented anti-goal | ADR-008, VISION |
|
||
| Delegating the S5 reuse spike to an agent swarm | A 1-hour sequential read-and-judge with a single coupled conclusion; orchestration overhead exceeds the work, and it's Diamond-1 judgment the maintainer wanted to own | (process note) |
|
||
| Vault-write SA for per-user OAuth tokens (ESO as runtime write path) | ESO syncs vault→cluster at deploy time, not a runtime write API; a write-SA widens blast radius to shared infra to store app row-data | ADR-015, infra#88 |
|
||
| Supabase for per-user credential storage | Adds a second datastore for a few encrypted strings PG18 already holds; reopens ADR-002 | ADR-015, infra#88 |
|
||
| Feedback-based Stage 0 gate (friends saying it's useful) | Politeness bias makes asked-for feedback the least reliable signal; return-usage is the real test | ADR-016 |
|
||
| Reverse the Dex-write invite flow (Google OIDC only) | Some intended Future-B users won't use Google; OIDC-only leaves them with no onboarding path — invite flow is load-bearing | ADR-017 |
|
||
| k8s CronJob for scheduled discovery (vs in-process) | At Future-B scale the in-process scheduler is simpler to deploy; CronJob's failure-isolation benefit was weighed and traded away knowingly (revisit if >1 replica or load grows) | ADR-018 |
|
||
| Cached-transcript-first onboarding burst (instant, zero-fetch picks) | Live pilot DB: ~3% cross-user video overlap, 0 cached-and-unsummarized, a new user's newest-20 are 20/20 uncached — newest-first and cached-first are structurally incompatible. Empty lever at pilot scale | ADR-028 |
|
||
| Uploaded transcripts as a `user_id` column + partial RLS policy on `transcripts` | Turns ADR-021's single exception into an exception with a private lane inside it; the isolation test asserts `transcripts` has no `user_id`. A separate RLS table keeps both boundaries independently assertable | ADR-033 |
|
||
| whisper-server `--convert` for server-side transcode | Its `--tmp-dir` is created at process start and rots under macOS periodic cleanup | ADR-033 |
|
||
| Calling the public `llm-api.d-ma.be` from in-cluster Tapir | Its ingress returns 413 above ~8–10 MB (~5 min of WAV); the NodePort has no such cap | ADR-033 |
|
||
| A cloud model (e.g. Opus) as the adjudicator | It scored 30/30 but would send private meeting content to a cloud API; local gemma4-26b/31b passed the same bar | ADR-034 |
|
||
| Pick one engine globally (pianissimo only, or kb-whisper only) | A near tie on accuracy with different error profiles; choosing one discards the information in their disagreement | ADR-034 |
|
||
|
||
If a future case genuinely reopens one of these, that's a new ADR superseding the relevant one —
|
||
not a silent reversal.
|