Compare commits

...
183 Commits
Author SHA1 Message Date
mathiasandClaude Sonnet 5.5 7ae888a468 docs(claude): two-engine reconciliation, and that task check does not gate lint (#41)
CI / Lint / Test / Vet (pull_request) Successful in 21s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:47 +02:00
mathiasandClaude Sonnet 5.5 b9e0dfa1c9 docs(homelab): two-engine mode, its settings and why every model must be local
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:47 +02:00
mathiasandClaude Sonnet 5.5 8dbcb9a9eb docs(data-model): reconciliation column
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:47 +02:00
mathiasandClaude Sonnet 5.5 26cc00c1cc test(acceptance): map the two-engine scenarios to their covering tests
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:37 +02:00
mathiasandClaude Sonnet 5.5 36a80ac4f5 docs(use-cases): scenarios for two-engine reconciliation (ADR-034)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:22 +02:00
mathiasandClaude Sonnet 5.5 2497554683 docs(adr): ADR-034 records the first live smoke, including the wrong call
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:22 +02:00
mathiasandClaude Sonnet 5.5 c3c9b1e48a test(e2e): two engines + local reconciler, live
TAPIR_E2E_RECONCILE=1 runs pianissimo + kb-whisper + gemma4-26b on the audio and
asserts the reconciliation is recorded, every contributor is named in the endpoint, and
the page says what was done. Run 2026-10-01 on 82 s of synthetic Swedish: 3 disputes, all
3 changed; against the known script 2 were right (virket, a missing 'procent') and 1 was
WRONG (Maria -> Vi replaced a correct base reading). A smoke test, not a measurement.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:22 +02:00
mathiasandClaude Sonnet 5.5 ed8ca54674 feat(cmd): buildTranscriber -- the base-engine setting turns on the two-engine ensemble
Default is the single-engine chain exactly as before. With TAPIR_RECONCILE_BASE_MODEL set,
the base engine and the primary run on the same WAV and a LOCAL model (config has already
refused any non-local alias) reconciles them, at temperature 0 with a short reply -- the
conditions its accuracy was measured under. The cloud fallback is unchanged and runs
only when both local engines fail.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:26:02 +02:00
mathiasandClaude Sonnet 5.5 7338d9d603 test(e2e): discard the deferred Close error golangci-lint flags
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:26:02 +02:00
mathiasandClaude Sonnet 5.5 3d755fd57a fix(transcribe): check (discard) the deferred Close errors golangci-lint flags
Found because the project's lint step cannot fail: the Taskfile runs
'golangci-lint run ./... || echo "not installed -- skipping"', so a failing lint
prints that message and exits 0. CI's log showed these findings and still passed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:26:02 +02:00
mathiasandClaude Sonnet 5.5 b492a3144e test(cmd): the base-engine setting turns on the ensemble, and nothing else does (red)
Red: buildTranscriber is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:25:10 +02:00
mathiasandClaude Sonnet 5.5 505670c0e1 feat(llm): WithTemperature, default unchanged
Zero is sent as zero (the field has no omitempty). The reconciler's measured accuracy
was at temperature 0, so it runs there; every other caller keeps 0.2.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:25:10 +02:00
mathiasandClaude Sonnet 5.5 7891675393 test(llm): a temperature option; zero is sent as zero, not 'unset' (red)
The reconciler was measured at temperature 0 but the client hardcodes 0.2. Zero must
be sent explicitly, and the default must not change for existing callers.

Red: WithTemperature is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:54 +02:00
mathiasandClaude Sonnet 5.5 8110fe56ab style(config): gofumpt the new test rows
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:38 +02:00
mathiasandClaude Sonnet 5.5 61fb50ce99 feat(config): two-engine reconciliation settings with a local-only guard (ADR-034)
TAPIR_RECONCILE_BASE_MODEL turns it on and needs TAPIR_RECONCILE_MODEL; every model must
be a local alias (koala/ or iguana/, matched as a PREFIX): a cloud adjudicator is a
start-up error because private meeting content must not reach it; the two engines must
differ; default job timeout 60m. Off by default. Mutation-checked.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:34 +02:00
mathiasandClaude Sonnet 5.5 f25fdaf398 test(config): a cloud alias that merely contains a local host name is still cloud
Mutation-testing showed a substring match instead of a prefix match went unnoticed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:34 +02:00
mathiasandClaude Sonnet 5.5 ad071d44cb test(config): reconciliation settings -- off by default, local only, complete (red)
Off by default; a base engine needs an adjudicator; every model must be a local alias
(koala/ or iguana/): a cloud adjudicator is a start-up error because private meeting
content must not reach it; the base engine must differ from the other engine; default
job timeout 60m (two engines plus adjudication).

Red: the Config fields do not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:20 +02:00
mathiasandClaude Sonnet 5.5 59f2893434 docs(adr): ADR-034 names the settings for what they are
TAPIR_RECONCILE_BASE_MODEL, not 'second model': the engine added to turn the
feature on is the one whose TEXT is the base, and the old name hid that. All three
models must be local aliases; the base must differ from the other engine. ADR is
still Proposed, so this is an in-place correction.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:09 +02:00
mathiasandClaude Sonnet 5.5 6fb7a52980 feat(web): say what reconciliation did, and never claim a second opinion that was not there
CI / Lint / Test / Vet (pull_request) Successful in 21s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Reconciled: the two engines, the deciding model and the exact counts; the contested
spots (changed or uncertain) listed with a minute mark and both readings, the first
50 shown and the rest counted from the exact totals. Single-engine and
model-unavailable results say why. A nil reconciliation renders nothing. Readings are
escaped. Mutation-checked: the cap, the minute rounding and the 'more' count.

uploads.go and the templ pair are one commit: the helpers and the template need each other.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:23:39 +02:00
mathiasandClaude Sonnet 5.5 d6ba11d288 test(web): the page says what reconciliation did, plainly, and never claims more (red)
Reconciled: who the two engines and the deciding model were, and the counts
(differences, settled automatically, decided by the model, uncertain). The contested
spots are listed with a minute mark, both readings and which was used, uncertain ones
marked, readings escaped, and a very long list shortened with its true count. A
single-engine transcript makes no claim of a second opinion; one-engine and
model-unavailable cases say why.

Red: the page renders none of this yet.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:23:00 +02:00
mathiasandClaude Sonnet 5.5 ae4f9fd775 feat(store): persist and read back the reconciliation
CompleteUpload writes it (SQL NULL when there was none, never the JSON text 'null':
the page claims a second opinion only if the column is non-NULL); GetUpload reads it,
List does not. Mutation-checked.

domain.go and the store are one commit: the store tests need the new field.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:22:09 +02:00
mathiasandClaude Sonnet 5.5 e838f6bb6d feat(store): migration 019 -- uploaded_transcripts.reconciliation
JSONB, NULL for a single engine so the page cannot claim a second opinion. On the
already FORCE'd, RLS-scoped table, so the exact-set isolation test is unaffected.
Up and down are one commit: golang-migrate pairs them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:21:52 +02:00
mathiasandClaude Sonnet 5.5 5bd41077af test(store): the reconciliation survives the round trip, and its absence stays absent (red)
Counts and contested spots round-trip; a single-engine transcript has NO
reconciliation (nil, not empty -- the page must not claim a second opinion); a
skipped reconciliation keeps its reason.

Red: Transcription.Reconciliation is stored nowhere and UploadedTranscript has none.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:21:36 +02:00
mathiasandClaude Sonnet 5.5 34bc8dbdee feat(transcribe): Ensemble -- two local engines, a local reconciler, fail-soft at every step
Runs the fast engine then the other from one WAV and reconciles; one engine down
uses the other and says so; the model down gives the base text unreconciled; the
cloud fallback runs only when BOTH local engines fail and is recorded as such; a
cancelled job stops everything and never reaches the cloud. The endpoint lists every
contributor. Mutation-checked, including both cancellation points.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:21:17 +02:00
mathiasandClaude Sonnet 5.5 8778376aa6 test(transcribe): a shutdown in the middle of either engine stops the job
Mutation-testing showed each of the two cancellation checks was redundant with the
other in my first test. Now a cancel during the first engine must not start the second
or the cloud, and a cancel during the second must not be recorded as a single-engine
success ('kb-whisper failed').

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:21:17 +02:00
mathiasandClaude Sonnet 5.5 ad39b6154c test(transcribe): two engines, then a reconciler -- and every way it can fail softly (red)
Both run in order (the fast one first) and are reconciled; identical texts cost no
model call; one engine down uses the other and says so, with the cloud fallback
untouched; only when BOTH local engines fail does the fallback run, and the row says
berget served it; both failing with no fallback reports both errors; a cancelled
job never reaches the cloud; a model that is down yields the base text unreconciled,
not a failed job.

Red: transcribe.Ensemble is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:20:40 +02:00
mathiasandClaude Sonnet 5.5 f25acebe4c feat(reconcile): merge two transcripts with a model choosing between disagreements
CI / Lint / Test / Vet (pull_request) Successful in 20s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Base text stands except where the model picks the alternative's reading; trivial
disputes skip the model; the reading order alternates; the base's context goes to
the model. Fails soft: three consecutive model errors return the base text
unreconciled (none of the partial changes) with the reason; a verdict returned with
an error is ignored; a cancelled job is an error, not 'skipped'. Counts are exact,
the stored list of contested spots is capped at 200 and carries the second each
starts at. Mutation-checked on nine behaviours.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:20:12 +02:00
mathiasandClaude Sonnet 5.5 7f9e967ad0 test(reconcile): a verdict returned with an error is not used
Mutation-testing showed the forced-unsure line was untested because my fake
always returned unsure with its errors.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:20:12 +02:00
mathiasandClaude Sonnet 5.5 0fead2e102 feat(domain): Reconciliation -- how a merged transcript came about (ADR-034)
Status (reconciled / single_engine / skipped), exact counts, and a bounded list of
contested spots (changed or unsure) with a position and both readings.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:19:54 +02:00
mathiasandClaude Sonnet 5.5 a1e16fa658 test(reconcile): fixtures need two shared words between disputes
Disputes separated by a single shared word are merged into one decision by design,
so several of my fixtures were secretly one dispute. Corrected before any
implementation existed; the rule is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:19:22 +02:00
mathiasandClaude Sonnet 5.5 b20558ffa6 test(reconcile): merging two transcripts, and failing soft (red)
Identical texts and trivial disputes cost no model call; an alt verdict takes the
alternative's own words, base/unsure keep the base and only unsure is flagged;
insertions and deletions land in the right place; the reading order alternates; the
model gets the base's context. Three consecutive model errors stop the whole
reconciliation (base text, none of the partial changes, reason recorded) while a
success in between resets the breaker; a cancelled job is an error, not 'skipped'.
Spots carry the second they start at; the stored list is capped, the counts are not.

Red: Reconcile, Options, MaxContested and domain.Reconcile* are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:19:00 +02:00
mathiasandClaude Sonnet 5.5 5e54c9beff feat(reconcile): LLM adjudicator with the measured prompt, plus an order-checked wrapper
Sends the exact wording that scored 92% (gemma4-26b) and 97% (gemma4-31b); maps
1/2 back to base/alt under both orders; an unreadable reply is unsure, a transport
error is an error. OrderChecked asks twice with the order swapped and keeps only
agreement. Mutation-checked: answer mapping, unparseable-as-guess, the word
boundary and the order check each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:18:11 +02:00
mathiasandClaude Sonnet 5.5 2bfe84ee64 test(reconcile): a number or word that merely starts with 1 or 2 is not an answer
Mutation-testing the adjudicator showed dropping the word boundary went unnoticed:
'ANSWER: 12' would be read as 1.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:18:11 +02:00
mathiasandClaude Sonnet 5.5 481321a75c test(reconcile): the adjudicator sends the measured prompt and maps answers back (red)
Readings in the requested order; 1/2 mapped to base/alt under both orders;
UNSURE, an unparseable or out-of-range reply is unsure rather than a guess; an
empty reading is described, not blank; a model error is an error, distinguishable
from 'unsure'. The optional order-checked wrapper asks twice with the order swapped
and keeps only agreement.

Red: Request, Verdict, NewLLMAdjudicator and NewOrderChecked are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:17:42 +02:00
mathiasandClaude Sonnet 5.5 2600e6c45a feat(reconcile): find disputes, resolve trivial ones without a model, give context
Disputes carry both readings with their own spelling; differences separated by
one shared word merge into one decision. Compound splits/joins and one or two
filler particles are resolved without a model; 'nej', numbers and content words
never are. Mutation-checked (merge distance, filler limit, 'nej', compound rule).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:17:21 +02:00
mathiasandClaude Sonnet 5.5 4e43327792 test(reconcile): finding disputes, resolving the trivial ones, and their context (red)
A dispute carries both readings with their own spelling; differences separated by
a single shared word are one decision, two shared words keep them apart; compound
splits/joins and a short run of filler words are resolved without a model, while
numbers, content words, and 'nej' (an answer) are not.

Red: FindDisputes, AutoResolve and ContextAround are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:16:55 +02:00
mathiasandClaude Sonnet 5.5 92a39da45c feat(reconcile): minimal word-level diff (Myers) with a bounded edit distance
Prefix/suffix trimmed first; adjacent deletes and inserts read as one Replace;
beyond 3,000 edits (two texts sharing almost nothing) the middle is one Replace
instead of gigabytes of trace. Mutation-checked: a wrong tie-break, missing
replace coalescing, wrong suffix accounting and a punctuation-bearing key each
turn tests red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:16:31 +02:00
mathiasandClaude Sonnet 5.5 aa5607b002 feat(reconcile): tokenise for word-level comparison
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:16:31 +02:00
mathiasandClaude Sonnet 5.5 984d92e3a9 test(reconcile): word-level diff is minimal, tiles both inputs and is fast (red)
A random-input property test compares the equal runs against a dynamic-programming
LCS: a diff that is valid but not minimal would turn real agreement into disputes
the model is then asked about. A 6,500-word, 380-edit case mirrors a real hour.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:15:44 +02:00
mathiasandClaude Sonnet 5.5 00fd884c6c test(reconcile): tokenising for word-level comparison (red)
Keeps the word as written, keys it without case/punctuation/hyphens/plus, joins a
lone dash or full stop to its neighbour so it is not a word the engines can
disagree about. Red: package reconcile does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:15:44 +02:00
mathiasandClaude Sonnet 5.5 de9871c9c3 docs(adr): ADR-034 transcribe with two engines, reconcile with a local model
CI / Lint / Test / Vet (pull_request) Successful in 29s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Proposed, decider pending. Two local engines on one WAV; word-level diff with
pianissimo as the base (readability and timing, NOT accuracy -- that was a tie);
trivial disputes resolved without a model; the rest adjudicated by a LOCAL model
(alias must start koala/ or iguana/) whose only freedom is choosing between the
two readings. Reconciliation never fails a job; the page says what was done and
lists the uncertain spots. Off by default. Validation: re-upload the real
recording and keep the feature only if the reconciled reading is right >= 85% of
the time and no worse than the better single engine.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:15:12 +02:00
mathiasandClaude Sonnet 5.5 e182a6b07d docs(research): can a local model adjudicate two ASR transcripts?
Eleven local models, 120 calls each, scored against the speaker's blind answers
on 30 disagreements, with pass criteria fixed before the run. gemma4-31b (97%)
and gemma4-26b (92%) pass; nothing else clears accuracy and order-consistency.
Records why 31b is slow (MLX gives the same speed; Low Power Mode is the
unproven suspect) and the caveats: 30 items, one recording, thinking models
under-budgeted. Aggregate numbers only; this repo is public.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:14:27 +02:00
mathiasandClaude Sonnet 5.5 ebc6cc3ff8 docs(research): kb-whisper vs pianissimo on one real hour of Swedish speech
CI / Lint / Test / Vet (pull_request) Successful in 22s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
A near tie on accuracy (blind A/B by the speaker: 17 vs 13 on the 30 spans that
change meaning) and a 6x difference in speed (40 s vs 4m08s for 63.7 min).
Different error profiles: pianissimo better on terms and numbers, kb-whisper on
everyday words. Aggregate numbers only -- no transcript text or domain terms,
this repo is public. Also records the mid-word wrapping bug the comparison found
and its fix.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 19:45:56 +02:00
mathiasandClaude Sonnet 5.5 d11afb5ff1 fix(transcribe): rebuild the transcript from segments so split words are whole
CI / Lint / Test / Vet (pull_request) Successful in 22s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
whisper.cpp wraps at ~60 chars and can wrap inside a word; the response's text
field joins segments with newlines, so chat saw 'nyt' / 't godkännande'. Text
is now rebuilt from the raw segments: concatenated when any segment carries a
leading space (whisper.cpp), joined by a space when none do (berget's sentence
level), whitespace collapsed; the text field is only a fallback.

Checked against the first real one-hour transcript with pianissimo's vocabulary
as an independent yardstick: 36 spans changed, 31 now consist entirely of words
pianissimo also produced, words unknown to pianissimo fell 224 -> 177 and word
agreement rose 92.4% -> 93.1%. Five rejoined spans still contain an unknown
word (names/loanwords or an unlucky glue) -- a residual risk, small.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 19:35:14 +02:00
mathiasandClaude Sonnet 5.5 500e11962b test(transcribe): rebuild Text from segments so a split word is not broken (red)
Found by comparing the first real one-hour transcript against pianissimo:
whisper.cpp wraps at ~60 chars and can wrap INSIDE a word. The continuation
segment has no leading space, the text field joins segments with newlines, and
a segment can be a lone '.'. Measured 2026-10-01: 7% of segments (96/1332) in a
synthetic hour, 42/1302 lines in the real one. My earlier tests used short
sentences and could not see it. berget's segments are whole sentences with no
leading space, so the two backends need different joins.

Red: Text still joins by newline.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 19:34:54 +02:00
mathiasandClaude Sonnet 5.5 206fc2f5ca feat(web): log upload rejections by reason only
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 19:28:10 +02:00
mathiasandClaude Sonnet 5.5 5368d5f8c2 test(web): a rejected upload is logged with its reason, never the filename or title (red)
Rejections are synchronous and leave no row, so an operator could not tell 'the
user never retried' from 'the page refused it' (it happened during the first
real-recording trial). The reason is operational; the filename and title are
the user's private content.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 19:28:06 +02:00
mathiasandClaude Sonnet 5.5 365dff5c85 docs(data-model): had_video
CI / Lint / Test / Vet (pull_request) Successful in 26s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:55 +02:00
mathiasandClaude Sonnet 5.5 adaaa9918f test(acceptance): map the video-track scenarios to their covering tests
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:55 +02:00
mathiasandClaude Sonnet 5.5 364a28eecd docs(use-cases): a video track is accepted and the user is told; no-audio is rejected
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:55 +02:00
mathiasandClaude Sonnet 5.5 1fa7ca3a8c test(e2e): a video file (camera-off Teams meeting) is transcribed and the user is told
Run 2026-10-01 with 82 s of synthetic Swedish muxed with black video (H.264 +
AAC .mp4, built with the real ffmpeg): accepted, had_video=true, transcribed by
iguana/kb-whisper, the page said only its audio was used, and chat answered.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:48 +02:00
mathiasandClaude Sonnet 5.5 cca0a7461b feat(web): tell the user when a video track was ignored, in every state
A Teams meeting with the camera off is still a video file. It is accepted, only
its audio is transcribed, and the page says so in queued, transcribing, done and
failed -- not just at the end.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:14 +02:00
mathiasandClaude Sonnet 5.5 5175496809 test(web): match the notice's real casing
My assertion expected a lowercase 'only'; the sentence begins 'Only'. The user
facing wording is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:14 +02:00
mathiasandClaude Sonnet 5.5 f102199097 feat(uploads): Intake passes whether the file had a video track to the store
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:00 +02:00
mathiasandClaude Sonnet 5.5 01eab1ab6f feat(uploads): a video track is no longer a reason to reject (ADR-033)
Only no-audio, unreadable length and too-long are refused. ReasonVideo is gone.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:00 +02:00
mathiasandClaude Sonnet 5.5 85cf89c86c feat(store): CreateUpload records had_video; Get/List return it
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:50 +02:00
mathiasandClaude Sonnet 5.5 603d02e504 feat(domain): UploadedTranscript.HadVideo
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:40 +02:00
mathiasandClaude Sonnet 5.5 3c4d2880f3 feat(store): migration 018 -- uploaded_transcripts.had_video
Up and down are one commit: golang-migrate pairs them. RLS is unchanged: the
column is on the already-FORCE'd table, so the exact-set isolation test still
holds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:33 +02:00
mathiasandClaude Sonnet 5.5 1157854441 test(cmd): fake store follows the CreateUpload signature (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:33 +02:00
mathiasandClaude Sonnet 5.5 7aec2538be test(web): a video file is accepted and the user is told the video was ignored (red)
Video with no audio is rejected for having none. A file with a video track is
accepted and marked had_video; every state of its page (queued, transcribing,
done, failed) says only its audio was used and the video was discarded, and a
plain audio file shows no such notice.

Red: the had_video column does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:33 +02:00
mathiasandClaude Sonnet 5.5 eb55c7d9d1 test(store): had_video is recorded on creation and survives done/failed (red)
Red: CreateUpload takes no hadVideo and the row has no HadVideo.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:13 +02:00
mathiasandClaude Sonnet 5.5 ab62cdaf1c test(uploads): Accept records whether the file had a video track (red)
Red: Store.CreateUpload does not take hadVideo yet.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:13 +02:00
mathiasandClaude Sonnet 5.5 6cbb7b2df0 test(uploads): a recording with a video track is accepted (red)
A Teams meeting with the camera off is still an .mp4 with a video track.
Video with no audio is now rejected as having no audio.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:04 +02:00
mathiasandClaude Sonnet 5.5 1e192e53ef docs(adr): ADR-033 accepts files with a video track, uses the audio, says so
The first draft rejected video and named #28. The maintainer's first real
recording -- a Teams meeting with the camera off -- is still an .mp4 with a
video track, so that rejected the primary use case. Decision 1 and 4 now:
accept, extract audio only, discard the video, record had_video, and tell the
user. A file with no audio stream is still rejected. Video features stay #28.
In-place amendment: the ADR is still Proposed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:41:49 +02:00
mathiasandClaude Sonnet 5.5 6ebb4e49e9 test(audio): opt-in live ffprobe check on hostile playlists and real audio
CI / Lint / Test / Vet (pull_request) Successful in 40s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Skipped unless TAPIR_LIVE_FFPROBE is set. Synthetic/crafted files only.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:14:29 +02:00
mathiasandClaude Sonnet 5.5 291a360a83 feat(audio): restrict ffmpeg/ffprobe input to local files and audio demuxers
The upload is attacker-supplied and its CONTENT can steer ffmpeg to open other
files or URLs (HLS/concat playlists). Pins -protocol_whitelist file,pipe and
-format_whitelist to the containers real recordings use, before the input
(after -i it would apply to the output).

Verified with the real static ffmpeg 7.0.2 on koala: real AAC .m4a and WAV
still probe/convert; a crafted concat file is rejected with 'Format not on
whitelist'. Note ffmpeg 7.0.2 ALREADY refused the playlists tried without the
whitelist, so the live check cannot prove the whitelist -- the argv unit tests
do; this protects against a different ffmpeg version being put in the image.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:14:29 +02:00
mathiasandClaude Sonnet 5.5 4a0c51d877 test(audio): attacker-supplied input must not steer ffmpeg to other files or URLs (red)
A file's CONTENT can tell ffmpeg to open other paths or URLs (HLS, concat
playlists). ffmpeg 7.0.2 refuses the ones tried, but that is version luck and
the image's ffmpeg is still to be chosen (#34). Pin protocol_whitelist to
file,pipe and format_whitelist to the demuxers real audio uses, before the input.

Red: audio.AllowedDemuxers is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:13:54 +02:00
mathiasandClaude Sonnet 5.5 1c726a810e docs(homelab): upload chat budget is 90000 and the chat model is gemma4-26b
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:08:33 +02:00
mathiasandClaude Sonnet 5.5 77fe6954e5 docs(adr): ADR-033 decision 8 -- chat budget and model from measurement
The proposed 120,000-char budget was a guess; measured, gemma4-26b silently
loses the head at that size and phi4-mini (first in the offered list) 400s on
an hour of speech. Budget is now 90,000, uploads open on gemma4-26b, the
retrieval trigger is concrete, and the iguana contention hazard is recorded.
ADR-033 is still Proposed, so this is an in-place correction, not a supersession.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:08:28 +02:00
mathiasandClaude Sonnet 5.5 4d3e2a471a docs(research): chat over a long transcript -- what the offered models really hold
phi4-mini 400s at ~13k tokens; gemma4-26b is correct to 90,000 chars and
silently loses the HEAD at 120,000 (no error, no truncation flag). The
end-needle test alone passed at 120k -- only a head-needle exposed it.
Records the default-model trap and the iguana contention hazard.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:08:13 +02:00
mathiasandClaude Sonnet 5.5 548bfaee85 test(e2e): use the same wireUploads as serve
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:41 +02:00
mathiasandClaude Sonnet 5.5 ad2e3e0e8e fix(cmd): serve now actually sets the upload chat budget and model
cmdServe assigned Uploads and UploadLimits but not UploadChatMaxChars, so chat
over a long recording would have used the 18,000-char caption default. It now
calls wireUploads, which is tested.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:41 +02:00
mathiasandClaude Sonnet 5.5 e6738bcd3e feat(cmd): wireUploads -- limits, chat budget and chat model into the web App
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:40 +02:00
mathiasandClaude Sonnet 5.5 3bb4e30f4c test(cmd): wireUploads sets the budget and model, and never a typed-nil Intake (red)
I wired Uploads and UploadLimits in cmdServe but not UploadChatMaxChars: chat
over a long recording would have silently used the 18,000-char caption
default. A helper that main and the e2e test both call makes that a tested fact.

Red: wireUploads is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:33 +02:00
mathiasandClaude Sonnet 5.5 6c39ab86cf feat(web): chat over an upload opens with App.UploadChatModel
The Chatter's own default is the summarizer's (koala/phi4-mini, 8,192-token
context), which answers HTTP 400 on an hour of speech. Empty keeps the old
behaviour.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:33 +02:00
mathiasandClaude Sonnet 5.5 4b6ac2a213 test(web): chat over an upload opens with the upload chat model (red)
Red: App.UploadChatModel is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:20 +02:00
mathiasandClaude Sonnet 5.5 af66419a44 feat(config): measured chat budget (90,000) and upload chat model (gemma4-26b)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:20 +02:00
mathiasandClaude Sonnet 5.5 4b09dc8f36 test(config): chat defaults for uploads come from measurement, not a guess (red)
iguana/gemma4-26b answered correctly at 90,000 chars but SILENTLY dropped the
head of a 120,000-char transcript (prompt tokens fell to 16k; 'not in the
text'); koala/phi4-mini has an 8,192-token context and answers HTTP 400 at
~13k tokens. So the budget is 90,000 and the upload chat model defaults to
gemma4-26b. Red: UploadChatModel is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:09 +02:00
mathiasandClaude Sonnet 5.5 8f16ab671a docs(claude): audio upload supersedes the STT deferral for user-supplied audio only
The 'no audio-download + STT' bullet is narrowed to YouTube (ADR-007 stands);
adds the upload packages, the exact-set isolation test and the pending ffmpeg
decision to the build state.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:02:11 +02:00
mathiasandClaude Sonnet 5.5 39aeda7c69 docs(homelab): speech-to-text endpoints, the ingress 413, and the upload knobs
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:02:05 +02:00
mathiasandClaude Sonnet 5.5 0fc25a24ed docs(data-model): uploaded_transcripts, and fix the stale claim that transcripts is RLS'd
Adds the private uploaded_transcripts entity and the exact-set isolation rule
(ADR-033). Also corrects two lists that still named `transcripts` as an RLS'd
table: it has been shared non-RLS since ADR-021 / migration 015, and these
lists sit next to the very boundary that matters.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:02:05 +02:00
mathiasandClaude Sonnet 5.5 07375f75de test(acceptance): map audio_upload.feature scenarios to their covering tests
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:01:38 +02:00
mathiasandClaude Sonnet 5.5 f9d6df9f84 docs(use-cases): audio_upload.feature -- scenarios for #33 (ADR-033)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:01:20 +02:00
mathiasandClaude Sonnet 5.5 3464cdf876 test(e2e): opt-in live upload -> transcript -> grounded chat check
Real ffprobe/ffmpeg, the live gateway and a real chat model, a private
embedded Postgres. Skipped unless TAPIR_E2E_AUDIO is set. Run 2026-10-01 with
82 s of synthetic Swedish as AAC .m4a:
- primary: iguana/kb-whisper, 82.6 s audio, 1,186 chars; asked who owns the
  timber order, the answer named Anders and the 15 November deadline from the
  transcript; the spool directory was empty afterwards.
- primary forced to fail (TAPIR_TRANSCRIBE_MODEL=iguana/does-not-exist): the
  fallback served it and the row recorded endpoint=berget/kb-whisper.
Synthetic audio only; this repo is public.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:01:06 +02:00
mathiasandClaude Sonnet 5.5 4fdb212cef feat(web): chat with an uploaded transcript, reusing the ADR-027 path (ADR-033)
GET/POST /uploads/{id}/chat, mounted only when both an Uploader and a Chatter
are wired. The row comes from the user-scoped private store, so another user's
upload is a 404 before any model is involved. The owner's transcript goes to
the SAME Chatter with App.UploadChatMaxChars instead of the 18,000-char caption
default. The chat panel is shared, not forked: chatView gains PostURL/Subject
(empty = a YouTube video, so existing callers are unchanged). The done card
links to chat only when chat is wired.

Mutation-checked: dropping the upload budget, ignoring the done-status guard,
or scoping the lookup to the wrong user each turn tests red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:58:25 +02:00
mathiasandClaude Sonnet 5.5 a39f58b730 test(web): match the HTML-escaped tail of the inline chat error
templ escapes the apostrophe in "couldn't", so the assertion could never match
the rendered page. A test defect, not a loosening: the failure must still be
shown inline with the conversation kept.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:58:25 +02:00
mathiasandClaude Sonnet 5.5 10ca45fc0a test(web): chat over an uploaded transcript reuses the ADR-027 path (red)
Owner sees the panel posting to /uploads/{id}/chat; an unfinished upload
redirects to its status; ANOTHER USER'S upload is 404 on GET and POST and no
model is ever called with it; the owner's transcript is sent with the upload
budget (not the 18,000-char caption default); an unoffered model falls back; a
model failure is inline and keeps the conversation; routes are absent without a
chat backend; the done card links to chat only when chat is wired.

Red: App.UploadChatMaxChars is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:57:47 +02:00
mathiasandClaude Sonnet 5.5 c72d366935 feat(chat): per-request transcript budget so a long recording is not cut to its head (ADR-033)
chat.Request.MaxChars overrides the Service cap when positive. The 18,000-char
default (~22 min of speech) suits captions; an uploaded hour is ~50,000 chars
and truncation keeps only the head. This is reuse of the existing chat path,
not a fork.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:57:01 +02:00
mathiasandClaude Sonnet 5.5 ec29c5e761 style(chat): gofumpt the new test
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:57:01 +02:00
mathiasandClaude Sonnet 5.5 867afefb08 test(chat): a request may carry its own transcript budget (red)
The 18,000-char default is ~22 min of speech and keeps only the head; an
uploaded hour is ~50,000 chars. Red: Request.MaxChars is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:56:51 +02:00
mathiasandClaude Sonnet 5.5 1c5d85f5c6 feat(cmd): serve mounts audio upload when enabled (ADR-033)
Wires buildUploads into cmdServe: Intake and limits into the web App, the
worker on the server's ctx. Logs the two facts an operator must not miss: the
queue is process-local (single replica), and a configured fallback is a cloud
endpoint that private audio can leave the homelab through.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:56:22 +02:00
mathiasandClaude Sonnet 5.5 9b3175db09 feat(cmd): buildUploads -- spool, sweep, fail interrupted rows, wire the chain (ADR-033)
Returns nils when disabled (never a typed-nil Intake). When enabled: private
0700 spool dir, delete audio left by a crash, fail rows a restart left queued
or transcribing (the audio is not retained, so they cannot resume), then build
the iguana->berget chain, worker and intake. An empty
TAPIR_TRANSCRIBE_FALLBACK_MODEL keeps private audio on the homelab. Enabling
with no gateway fails at start-up, not at the first upload.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:56:21 +02:00
mathiasandClaude Sonnet 5.5 eb9357a5de test(cmd): start-up removes spooled audio left by a crash (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:56:11 +02:00
mathiasandClaude Sonnet 5.5 3e802adade test(cmd): buildUploads wiring -- off by default, sweep first, private spool (red)
Red: buildUploads is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:55:55 +02:00
mathiasandClaude Sonnet 5.5 b8798dbd5d feat(config): upload and transcription settings (ADR-033)
Embedded UploadConfig. Off by default until the image carries ffmpeg; 200 MB /
90 min; iguana primary with a berget fallback that an explicit empty
TAPIR_TRANSCRIBE_FALLBACK_MODEL disables (private audio then never leaves the
homelab); zero or negative limits fail at load instead of rejecting every
upload at runtime.

uploads.go and the two-line hook in config.go are one commit: the tests need both.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:55:32 +02:00
mathiasandClaude Sonnet 5.5 fb521f2b89 test(config): upload and transcription settings (red)
Off by default until the image has ffmpeg; 200 MB / 90 min (the longest length
measured); iguana primary, berget fallback that an explicit empty value
disables; zero or negative limits are refused rather than silently rejecting
every upload.

Red: the Config fields do not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:55:11 +02:00
mathiasandClaude Sonnet 5.5 a8dadb8ca4 feat(uploads): in-process transcription worker, one job at a time (ADR-033)
Queue -> StartTranscribing -> ffmpeg to 16 kHz WAV -> gateway chain ->
CompleteUpload, with a per-job timeout so a hung endpoint ends in failed, not
a spinner. The upload and the WAV are deleted on every path (audio is never
retained). A shutdown mid-job is not a failure: the row is left for the
start-up sweep. Cancellation wins over a non-empty queue, so queued jobs are
never started on shutdown; stop() deletes their audio and refuses new jobs.

Concurrency is 1 on purpose: two simultaneous long jobs on iguana were never
measured, and a Metal OOM there latches whisper (infra#273). The queue is
process-local, so ADR-018's single-replica assumption now has a second
dependent.

Mutation-checked, -race clean: cleanup, shutdown-vs-failure, timeout,
concurrency and stop-cleanup each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:54:52 +02:00
mathiasandClaude Sonnet 5.5 3cfc95640e test(uploads): a queued job at shutdown must never be started; timeout test fails fast
Mutation-testing the worker showed deleting stop()'s cleanup went unnoticed:
Process was incidentally deleting the audio of jobs it should never have
started. The stop test now asserts only the running job was converted (red on
the old code). The timeout test no longer hangs for the 10-minute go test
default when the timeout is removed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:54:52 +02:00
mathiasandClaude Sonnet 5.5 52417e6b47 test(uploads): in-process transcription worker (red)
Success path deletes both the upload and the WAV; failures are typed and still
delete the audio; a shutdown mid-job leaves the row for the start-up sweep
(it is not a transcription failure); a hung endpoint is failed by the job
timeout, never an eternal spinner; strictly one job at a time (two long jobs on
iguana were never measured); stopping deletes queued audio and refuses new jobs.

Red: uploads.Worker is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:47:22 +02:00
mathiasandClaude Sonnet 5.5 4352446f36 feat(audio): ffmpeg adapter -- always transcode to 16 kHz mono PCM (ADR-033)
Replaces whisper-server --convert. A zero exit that wrote nothing is an error.
Also fixes the red test's stub, which wrote an empty file and so (correctly)
tripped the no-output check.

Checked against real ffmpeg on iguana with the same argv: AAC .m4a in,
pcm_s16le 16000 Hz mono WAV out, duration preserved.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:46:34 +02:00
mathiasandClaude Sonnet 5.5 3bc33fc34c test(audio): ffmpeg transcode to 16 kHz mono PCM, always (red)
Red: audio.FFmpeg is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:46:14 +02:00
mathiasandClaude Sonnet 5.5 2227cfff7e test(transcribe): opt-in live check against the real gateway
The httptest fake proves only what we THINK the gateway accepts. Skipped
unless TAPIR_LIVE_WAV and TAPIR_LIVE_GATEWAY_URL are set. Run once on
2026-10-01 with 11 min of synthetic Swedish (21 MB, above the 8-10 MB ingress
cap, via the NodePort): both iguana/kb-whisper (52.8 s) and berget/kb-whisper
(35.4 s) accepted the streamed, computed-Content-Length body and returned
usable verbose_json. Synthetic audio only; this repo is public.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:46:14 +02:00
mathiasandClaude Sonnet 5.5 cae423750c feat(transcribe): gateway client with an iguana->berget fallback chain (ADR-033)
Streams the WAV from disk with a computed Content-Length (no buffering, no
chunking), posts verbose_json with field `file` and language=sv, clamps the
final segment to the audio duration, and treats an empty transcript as a
failure. The chain falls back on ANY primary error, records the serving
endpoint, and never falls back once the context is cancelled.

gateway.go and chain.go are one commit: the chain's tests use the gateway's
StatusError and ErrEmptyTranscript, so neither compiles alone.

Mutation-checked: field name, clamp, cancel-fallback, content length and the
recorded endpoint each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:44:34 +02:00
mathiasandClaude Sonnet 5.5 bb285d8757 test(transcribe): gateway client and iguana->berget chain (red)
Gateway: field is `file`, no Expect header, a computed Content-Length (not
chunked, not buffered -- a 90 min WAV is ~170 MB), verbose_json normalised
with the last segment clamped to the audio duration, errors independent of
the body, an empty transcript is a failure. Chain: ANY primary failure falls
back and the row records the fallback endpoint; a cancelled job never falls
back to the cloud.

Red: package transcribe does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:43:51 +02:00
mathiasandClaude Sonnet 5.5 e12a5e7e89 feat(web): upload routes -- form, multipart intake, honest status (ADR-033)
GET/POST /uploads, GET /uploads/{id}, GET /uploads/{id}/status, mounted only
when an Uploader is wired. The multipart body is streamed to Intake, never
buffered. Rejections are synchronous: 422 (video, too long, unreadable) or 413
with the user-facing message and no row. Another user's id is a 404 that does
not leak the title. Unfinished states poll; done/failed render without
hx-trigger so nothing polls forever (ADR-025). A done transcript names the
endpoint that served it.

handlers.go, uploads.go and the templ pair are one commit: the App fields and
Store methods are what make the handlers compile.

Mutation-checked: dropping the 413 mapping, the not-found mapping, or letting
a done card poll each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:42:58 +02:00
mathiasandClaude Sonnet 5.5 1bc98813fc test(web): read the response body once; match the message's real case
Two defects in my red tests, found when going green: body() drains the
recorder so a second call read empty, and two assertions expected a different
case than the user-facing message. The assertions were not weakened.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:42:40 +02:00
mathiasandClaude Sonnet 5.5 e1ed6ea6f4 test(web): upload routes -- form, multipart POST, rejections, owner-only, honest status (red)
Real Router + real Intake + real Postgres; only ffprobe and the worker are
faked. Covers: the form states the limits up front; accepted audio queues a
row and redirects; video / too-long / too-large are rejected synchronously
(422/422/413) with a message and NO row; a non-owner gets 404 without the
title leaking; unfinished states keep polling and terminal states stop (no
stuck spinner); a done transcript names the endpoint that served it.

Red: App.Uploads / App.UploadLimits are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:41:42 +02:00
mathiasandClaude Sonnet 5.5 f84389028d feat(audio): ffprobe adapter implementing uploads.Prober (ADR-033)
Cover art is not video; non-zero exit is the file's fault (ErrUnreadable),
a missing binary / cancelled ctx / bad output is ours. WaitDelay bounds a
cancelled probe whose child holds the pipe open.

Checked against real ffprobe on iguana: JSON shape matches the fixtures and
/etc/hosts exits 1 with 'Invalid data found when processing input'. The unit
tests drive a stub process; ffprobe is not installed on koala.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:40:51 +02:00
mathiasandClaude Sonnet 5.5 0ab067c510 test(audio): ffprobe adapter -- cover art is not video, exit code vs missing binary (red)
Red: package audio does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:40:29 +02:00
mathiasandClaude Sonnet 5.5 1910d3dfb5 feat(uploads): Intake -- spool, probe, validate, queue; clean up on every failure (ADR-033)
Bounded read (cap+1 byte tells 'at the limit' from 'over'); a file that fails
validation creates no row; a prober fault is our error, not the user's bad
file; a row whose job cannot be submitted is failed, never left queued.

Mutation-checked: cap off-by-one, missing cleanup and deleting the handed-over
file each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:40:13 +02:00
mathiasandClaude Sonnet 5.5 b98e9df462 test(uploads): Accept -- size cap, probe verdicts, cleanup on every failure path (red)
Red: uploads.Job, Intake and ErrUnreadable are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:40:03 +02:00
mathiasandClaude Sonnet 5.5 e3e1f17c40 feat(uploads): validation policy for probed audio uploads (ADR-033)
Video is checked first so it is always refused as video; limits and reasons
are typed so callers and tests branch on Reason, not message text.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:39:51 +02:00
mathiasandClaude Sonnet 5.5 f42b8e23fc test(uploads): validation policy -- video, no-audio, too-long, unreadable (red)
Probe-based, not extension-based: a Teams voice-only export is typically an
.mp4 with no video stream, indistinguishable from video by name (ADR-033).
Red: package uploads does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:39:20 +02:00
mathiasandClaude Sonnet 5.5 836a501576 feat(store): Store API for private uploaded transcripts (ADR-033)
Create/Start/Complete/Fail/Get/List plus the start-up sweep that fails
unfinished rows as interrupted (audio is not retained, so no resume). Every
call goes through withUser AND names user_id in its predicate, so isolation
does not rest on RLS alone and stays testable under the superuser test
connection; a non-owner gets ErrUploadNotFound, never a distinct forbidden.
List drops transcript text.

Mutation-checked: removing the user predicate from Get or from the UPDATEs
turns the cross-user tests red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:38:25 +02:00
mathiasandClaude Sonnet 5.5 c29c0a7127 feat(domain): types for private uploaded transcripts (ADR-033)
Segment, Transcription, UploadStatus, UploadErrorKind and UploadedTranscript.
Error kinds are typed, never a provider's string. UploadedTranscript is a
distinct type from the shared public-caption Transcript on purpose.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:37:39 +02:00
mathiasandClaude Sonnet 5.5 4b557209c8 test(store): Store API for uploaded transcripts (red)
Lifecycle queued -> transcribing -> done|failed, per-row serving endpoint,
segments JSONB round trip, owner-only get/write (not-found, not forbidden),
list without transcript text, and the start-up sweep that fails unfinished
rows as `interrupted` for every user (audio is not retained, so no resume).

Red: CreateUpload and friends are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:37:30 +02:00
mathiasandClaude Sonnet 5.5 e36c4c1e0c feat(store): migration 017 -- private RLS-scoped uploaded_transcripts (ADR-033)
Separate table, not a user_id on the shared transcripts (ADR-021). ENABLE +
FORCE ROW LEVEL SECURITY with the migration-003 policy, user_id NOT NULL
ON DELETE CASCADE, typed error_kind, per-row serving endpoint, JSONB segments.
CHECK constraints: done => content, failed => error_kind.

Up and down are one commit: golang-migrate pairs them.

Mutation-checked: dropping FORCE, the policy, or the cascade each turn a test
red. FORCE is only observable structurally -- the app role is not the table
owner, so no behavioural test can see it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:36:50 +02:00
mathiasandClaude Sonnet 5.5 066ae6966a test(store): isolation tests for the private uploaded_transcripts table (red)
Drives RLS directly as the non-superuser app role: deny-all with no scope,
own-rows-only scoped reads, zero-row cross-user UPDATE/DELETE, rejected
cross-user INSERT, FORCE + NOT NULL user_id, cascade on DeleteUser. Also pins
the EXACT set of public tables with row security off, so a new table added
without RLS fails CI rather than passing it (ADR-033). The existing test only
named transcripts; #33's "the only non-RLS table" is false -- invitations,
user_identities and schema_migrations are off too.

Red: relation "uploaded_transcripts" does not exist. Set test passes today
and guards the migration that follows.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:36:22 +02:00
mathiasandClaude Sonnet 5.5 38b3f1cf33 docs(adr): ADR-033 upload audio -> private Swedish transcript -> chat (#33)
Proposed, decider pending. Supersedes ADR-007's STT deferral for
user-supplied audio only; the YouTube audio-download rejection stands.
Settles #28 decision 1 as a separate RLS-scoped uploaded_transcripts table
and records the measured length ceiling, the ingress 413, and the 18k-char
chat truncation that #33 assumed did not exist. Two DECISION NEEDED items
(ffmpeg in the runtime image; berget fallback for private audio).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:35:04 +02:00
mathiasandClaude Sonnet 5.5 feb108def3 docs(research): measure the audio length ceiling before designing chunking (#33)
No transcription ceiling found up to 90 min: one 174 MB request through the
LiteLLM NodePort transcribed in 440 s. The first real failure is the public
gateway ingress rejecting bodies over ~8-10 MB with 413, not whisper or a
Metal OOM. Also records that srt arrives wrapped in a JSON envelope, that the
two backends differ in timestamp granularity, and that the existing chat path
truncates at 18k chars (~22 min of speech).

Synthetic Swedish TTS fixture only; no real transcript.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:33:34 +02:00
mathias 4bafd5f082 fix(web): views.templ imports the pre-rename gitea.d-ma.be host
CI / Build & Import (push) Successful in 11s
CI / Deploy via GitOps (push) Successful in 4s
CI / Lint / Test / Vet (push) Successful in 19s
internal/web/views.templ:7 still imports
  gitea.d-ma.be/mathias/tapir/internal/adapters/store
while go.mod declares
  git.d-ma.be/mathias/tapir

Left behind by the gitea rename (infra ADR-0004). The old hostname still
resolves in DNS but no longer serves — HTTPS to it fails outright — so the
import is genuinely broken, not merely outdated.

Why nothing caught it: the COMMITTED internal/web/views_templ.go is correct,
so builds and CI use a good generated file and stay green. The rot is only in
the source. It surfaces the moment anyone runs `templ generate` — which
regenerates the broken import over the good output, and then the build fails
with

  no required module provides package gitea.d-ma.be/mathias/tapir/internal/adapters/store

Found exactly that way while evaluating this repo for the estate-wide lint
config: regenerating templ as part of the check turned a green repo red, and
the cause was the source, not the check.

Source fixed; regenerating now reproduces the committed generated file
byte-for-byte, so views_templ.go is unchanged here. Verified: go build and
go vet pass.
2026-09-05 14:44:39 +02:00
mathias e3e78e042b style: apply gofumpt
CI / Build & Import (push) Successful in 10s
CI / Lint / Test / Vet (push) Successful in 21s
CI / Deploy via GitOps (push) Successful in 4s
Estate-wide consolidation pass. The canonical .golangci.yml
(infra ci-templates/golangci.yml, infra#309 pattern) enables the gofumpt
formatter; golangci-lint v2 default set enables NO formatter, which is why
this drift accumulated silently -- the same hole that let two files rot in
dma-cli until they were found by hand (dma-cli#12).

Mechanical and deterministic: `gofumpt -w .`, nothing hand-edited. Verified
per repo before commit -- gofumpt -l now reports 0, gofmt -l reports 0, and
go build + go vet both pass.

Clears this repo to adopt the canonical lint config in a later wave.
2026-09-05 12:39:31 +02:00
mathiasandClaude Opus 5 6f1e0be09b ci(deploy): refuse to patch a manifest with an empty IMAGE_TAG (infra#311)
CI / Lint / Test / Vet (push) Successful in 44s
CI / Build & Import (push) Successful in 9s
CI / Deploy via GitOps (push) Successful in 5s
On 2026-08-25 a deploy job ran with an empty IMAGE_TAG, wrote
`image: localhost:5000/gitea-mcp:` into the infra repo and pushed it.
That is invalid YAML, so Flux failed the ENTIRE apps Kustomization --
every app stopped reconciling for about an hour, including three
unrelated deploys queued behind it.

This job's existing check could not catch it. It greps for
`image: <repo>:${IMAGE_TAG}` after the sed, and with IMAGE_TAG empty that
pattern matches the corrupted line itself, so the verification passes
because of the bug rather than despite it.

Asserting the input up front is the fix: an empty tag now fails the job
before anything is written, let alone committed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011PL5XcG68SvQqz3vGVbWbf
2026-08-25 17:25:24 +02:00
mathiasandClaude Opus 5 d45348ad2e ci: call the shared build-image script instead of an inline copy (infra#309)
CI / Lint / Test / Vet (push) Successful in 27s
CI / Build & Import (push) Successful in 11s
CI / Deploy via GitOps (push) Successful in 5s
The buildctl/skopeo step here was one of nineteen near-identical copies
across the estate. That duplication is why moving them off rootless
buildah (infra#241) took eleven commits and then had to be rediscovered
in both repo templates and the gitea-ci skill.

The implementation now lives in .gitea/build-image.sh, byte-identical to
mathias/infra's ci-templates/build-image.sh and held there by a drift
gate. What used to be textual differences between copies -- labels, tags,
build-args, build target -- are environment inputs.

Behaviour is unchanged: same registry, same tags, same labels, same
buildctl-against-host-buildkitd path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011PL5XcG68SvQqz3vGVbWbf
2026-08-25 11:29:55 +02:00
mathiasandClaude Opus 5 1c2ad2990f ci: build via buildctl instead of rootless buildah (infra#241)
CI / Build & Import (push) Successful in 8s
CI / Deploy via GitOps (push) Successful in 1s
CI / Lint / Test / Vet (push) Successful in 34s
Main has been CI-red since 2026-08-13. The build job dies extracting a base
image layer inside the gitea-runner pod:

  ApplyLayer ... potentially insufficient UIDs or GIDs available in user
  namespace (requested 0:42 for /etc/gshadow)

Root cause is the runner pod's securityContext, not this repo: it drops ALL
capabilities and sets allowPrivilegeEscalation:false, so newuidmap/newgidmap
can't gain CAP_SETUID/CAP_SETGID and rootless buildah falls back to a single
non-remapped uid/gid mapping. Adding those capabilities was tried live in the
infra repo and reverted as a confirmed no-op (no_new_privs blocks the
setuid-exec gain regardless of the container's capability set).

buildctl talks to koala's host buildkitd over /run/buildkit/buildkitd.sock,
which the runner pod already mounts, so the privileged layer work happens in
buildkitd on the host and the pod's hardened posture is untouched. Same
migration agentsquad (infra#233) and segment-embedder (infra#243) already made.

Drops the buildah smoke test with it: `buildah from ${REF}` would pull the
image back into the pod's buildah store and hit the very wall this migration
avoids, failing the job for a reason unrelated to the image. The deploy job's
`Verify rollout` is the stronger check — it requires the real pod to reach
Ready in-cluster, rather than the entrypoint printing its own name for 5s.
Tag builds lose their only image-level check (deploy runs on main pushes only);
accepted, since a tag is always cut from a main commit that already deployed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011PL5XcG68SvQqz3vGVbWbf
2026-08-17 12:07:51 +02:00
mathiasandClaude Opus 5 489b4bb2a1 chore(spike): recover the /bygge toolchain from tmpfs (#28)
CI / Lint / Test / Vet (push) Successful in 37s
CI / Build & Import (push) Failing after 5s
CI / Deploy via GitOps (push) Skipped
analyze_srt.py, build_page.py and transcode.yaml produced the /bygge
prototype. They were sitting in a session scratchpad on tmpfs, one reboot
from gone, while spike issues #29-#31 were written as if they had to be
built from scratch.

Committed as found, defects documented rather than fixed: max_tokens=6000
truncates anything past ~6 minutes of audio, the page title is hardcoded,
and the schema still carries fields no consumer renders.

The four-check validator is the part worth keeping — the coverage-gap and
Swedish-number-grounding checks catch errors that timestamp and citation
checks structurally cannot.

Real transcripts and analyses stay out: this repo is public and the
recordings are a named person discussing a client's project. Only the
synthetic fixture is committed, which exercises all four checks with no
model call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
2026-08-13 11:37:03 +02:00
mathias 21e7d6c74b fix(ci): smoke test hung 14min — buildah run doesn't die under plain timeout
CI / Lint / Test / Vet (push) Successful in 15s
CI / Build & Import (push) Successful in 20s
CI / Deploy via GitOps (push) Successful in 5s
Bare `/tapir` runs the long-running server, same as the old ctr-based smoke
test. ctr's --rm reliably force-killed it; a plain `timeout N buildah run`
does not — it only signals the wrapper, and the container process can
survive that and keep the log pipe open, hanging the whole job (observed
live: run 155, 14min before failure). Backgrounds the run and tears it down
with `buildah rm -f`, which forcibly kills regardless of wrapper state, and
captures output via a file instead of a blocking pipe.

Refs infra#132.
2026-07-26 06:17:18 +00:00
mathias 8814ba6673 ci(smoke): use buildah run instead of sudo k3s ctr for smoke test
CI / Lint / Test / Vet (push) Successful in 12s
CI / Build & Import (push) Failing after 13m42s
CI / Deploy via GitOps (push) Has been skipped
Same fix as cobalt-dingo — removes the sudo/host-containerd dependency so
this still works once the act_runner is containerized (infra#132).

Refs infra#132.
2026-07-26 05:57:37 +00:00
mathias 4b557a4325 ci(deploy): add Flux GitOps deploy job, mirroring cobalt-dingo's pattern
CI / Lint / Test / Vet (push) Successful in 27s
CI / Build & Import (push) Successful in 18s
CI / Deploy via GitOps (push) Successful in 5s
Fixes silent stale-deploy gap: CI built+pushed images but nothing bumped
k3s/apps/tapir/deployment.yaml, so merged features sat CI-green with zero
production effect (infra#111, infra#168). Flux native image-automation
can't scan localhost:5000 from inside k3s pods, so this patches the infra
repo directly via the existing INFRA_DEPLOY_KEY org secret (same key
cobalt-dingo and brain-gardener already use) on every push to main.

Refs infra#111.
2026-07-25 21:21:02 +00:00
mathiasandClaude Opus 4.8 72cb25111f test(store): address migrations by version, not step count (#8)
CI / Lint / Test / Vet (push) Successful in 12s
CI / Build & Import (push) Successful in 12s
The up/down migration tests stepped a hard-coded number of Steps(-N)/Steps(+N)
down from HEAD and back. The counts assumed a specific latest migration, so
adding one shifted every count by one and unrelated tests (010/011/014) went
red with confusing off-by-one symptoms — a papercut on every new migration.

Drive the schema to an exact version with m.Migrate(version) via two helpers
(headVersion, migrateTo). Each test now steps to just below its target by
version, asserts the down effect, steps up to the target, asserts the up
effect, then restores to the captured HEAD. A migration added on top changes
HEAD but shifts no count, so no test needs editing.

Verified by adding a throwaway migration 017 on top: all four tests stayed
green with zero edits.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QbdxXWxLefS5AwLN5eyze
2026-07-02 14:49:02 +02:00
mathiasandClaude Sonnet 4.6 38f222c931 chore: rename Go module path gitea.d-ma.be → git.d-ma.be
CI / Lint / Test / Vet (push) Successful in 11s
CI / Build & Import (push) Successful in 12s
Infra ADR-0004 renamed the Gitea host. Bulk replace across go.mod and
all .go import paths. Build and tests pass unchanged.

Closes #20

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt6aHEDWRjkK14Voi6HnGh
2026-07-02 14:37:33 +02:00
mathias e042b6d26d chore: add .mcp.json (brain + gitea) — wire tapir for the single-harness workflow
CI / Lint / Test / Vet (push) Successful in 16s
CI / Build & Import (push) Successful in 12s
tapir had no MCP config at all — a hyperguild session here couldn't reach brain
or Gitea, breaking the report-back path before it could start. Matches the
shape now standard post-consolidation (hyperguild#75/#76, infra#178 template
refresh). Prerequisite for tapir#20 being run as the first real workflow test.
2026-07-02 07:00:53 +00:00
mathias c85a32e770 Merge pull request 'LANGUAGE.md: fix LiteLLM gateway never-say (stale endpoints, not piguard)' (#21) from language-md-litellm-fix into main
CI / Lint / Test / Vet (push) Successful in 12s
CI / Build & Import (push) Successful in 10s
2026-06-16 15:45:45 +00:00
mathias b246c0e688 fix(language): LiteLLM gateway never-say — stale endpoints, not piguard
CI / Lint / Test / Vet (pull_request) Successful in 12s
CI / Build & Import (pull_request) Has been skipped
piguard is legitimately in the request path as the reverse proxy; only the
stale endpoint forms (piguard:4000, koala:4000) are wrong. Per brain#6 review
amendment #3 / tapir#18 follow-on.
2026-06-16 15:35:31 +00:00
mathias c7896cb3cc Merge pull request 'LANGUAGE.md pilot: hand-written vocabulary + caveman rubric (Phase 1)' (#19) from language-md-pilot into main
CI / Lint / Test / Vet (push) Successful in 12s
CI / Build & Import (push) Successful in 10s
2026-06-16 15:22:14 +00:00
mathiasandClaude Opus 4.8 b79fb892c8 docs(claude): point to LANGUAGE.md + caveman rubric
CI / Lint / Test / Vet (pull_request) Successful in 13s
CI / Build & Import (pull_request) Has been skipped
Wires the Phase 1 vocabulary pilot into the canonical agent instructions
(tapir#18). CLAUDE.md is canonical here (no .context/ source, no generation
header).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 22:54:57 +02:00
mathias c315e3e003 feat(language): add hand-written vocabulary + caveman rubric pilot
Phase 1 of the ubiquitous-language system (tapir#18). Tapir-specific terms
verified against domain/ports code + README/VISION; homelab-core terms (brain
sink, LiteLLM gateway) derived from brain#5 glossary. Tooling-free pilot.
2026-06-12 20:52:23 +00:00
mathiasandClaude Opus 4.8 64e3368f5f feat(web): charm-reader visual refresh with light/dark theme toggle (ADR-032, #17)
CI / Lint / Test / Vet (push) Successful in 11s
CI / Build & Import (push) Successful in 10s
The UI read flat and boring. Reskin to one charm/TUI-inspired layout in two
palettes (CSS custom properties): a warm "reader" light theme (sketch B) and a
"cozy terminal" dark theme (sketch C).

- Palette chosen in cascade order: :root light default; an OS-preference dark
  block scoped to :root:not([data-theme]) so it applies only absent an explicit
  choice; and :root[data-theme="dark"|"light"] set by a header toggle that
  outranks the media query by specificity and persists in localStorage (guarded,
  degrades to OS default). A <head> init script applies the stored choice before
  paint, so no flash of the wrong palette.
- Charm touches via existing classes (no templ structure churn): monospace meta
  lines, accent uppercase section dividers with a trailing rule, pill buttons, a
  lifted/accent-edged expanded card.
- Error/danger shades become --err-* tokens so they follow the theme, replacing
  three per-block prefers-color-scheme dark overrides.
- Theme toggle wired into Layout and PublicLayout headers.

BDD: docs/use-cases/visual_theme.feature un-pended, mapped in scenarioCoverage.
TDD: internal/web/visual_theme_test.go (palettes, OS default, persisted toggle,
expanded-card embed). Verified light+dark on list/reader/welcome via web-shot.
Sketches kept as the design record. ui-spec.md as-built row added.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 12:54:06 +02:00
mathias b01f8d5783 docs(bdd): visual_theme.feature scenarios (@pending until TDD, #17) 2026-06-12 12:29:31 +02:00
mathias 8ca767c144 docs(adr): ADR-032 visual refresh — light(B)+dark(C) charm-reader themes + sketches (issue #17) 2026-06-12 11:54:54 +02:00
mathiasandClaude Opus 4.8 f98b640531 feat(web): inline-expand summary + Q&A in the list (ADR-031, #16)
CI / Lint / Test / Vet (push) Successful in 11s
CI / Build & Import (push) Successful in 11s
Click a summarized card → its full summary (summaryBody) + chat dock (chatReveal)
expand in place via HTMX (GET /v/{id}/expand → expandedCard), collapse back via
GET /v/{id}/card → compact VideoCard. Same <li id>, outerHTML swap — the existing
list-fragment pattern. The card title carries href=/v/{id} as the no-JS fallback
(detail page stays for no-JS + deep links); only summarized cards expand. Reuses
summaryBody + chatReveal so the expanded card never drifts from the detail page.

BDD: inline_expand.feature un-pended + mapped. TDD: 6 handler/fragment tests
(expand/collapse fragments, chat dock, summarized-only, no-JS href, shared body).
Minimal CSS only — the TUI/charm restyle is #17.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 10:47:47 +02:00
mathias 607a8cbe8d docs(bdd): inline_expand.feature scenarios (@pending until TDD, #16) 2026-06-12 10:38:40 +02:00
mathias 54d60e53b9 docs(adr): ADR-031 inline-expand summary + Q&A in the list (issue #16) 2026-06-12 10:37:55 +02:00
mathias 9298e0c686 docs(homelab): TAPIR_METRICS_ADDR + key metric series (ADR-030, #15)
CI / Lint / Test / Vet (push) Successful in 10s
CI / Build & Import (push) Successful in 10s
2026-06-12 08:50:36 +02:00
mathiasandClaude Opus 4.8 cd461b95f8 feat(observability): instrument AI + HTTP paths, serve /metrics on a side port (ADR-030, #15)
CI / Build & Import (push) Successful in 11s
CI / Lint / Test / Vet (push) Successful in 10s
Wire the metrics package into the live paths and serve it:
- summarizer: per-endpoint latency by model/outcome(success|error|parse_error)/fallback + slog.
- youtube.FetchTranscript: latency by outcome (captions|none|rate_limited) + slog.
- chat: answer latency by model + slog.
- llm usage hook → token counts (prompt|completion) per model, wired in buildSummarizer/buildChat.
- oidc callback: login counter.
- cmdServe: wrap Router in metrics.HTTPMiddleware (request count + latency by bounded
  route pattern) and serve /metrics on TAPIR_METRICS_ADDR (default :9090), a SEPARATE
  port — never on the public app mux.

BDD: observability.feature scenarios un-pended + mapped. TDD: summarizer wiring tested
black-box via the /metrics scrape; metrics-not-on-public-mux asserted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 08:49:51 +02:00
mathiasandClaude Opus 4.8 9c2a04406b feat(llm): usage hook to surface token counts (ADR-030, #15)
WithUsageHook callback fires with model + prompt/completion tokens parsed from the
response usage block. Keeps the copied stdlib-only llm package decoupled from
metrics (ADR-004) — the caller wires it to internal/metrics. TDD covered.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 08:36:02 +02:00
mathiasandClaude Opus 4.8 a5a8cf6f6d feat(metrics): Prometheus collectors + typed API + HTTP middleware (ADR-030, #15)
New internal/metrics package: AI metrics (summarize/caption/chat latency, LLM
tokens), session metrics (requests+latency by bounded route pattern, logins), and
a /metrics Handler. Adapters call a typed API; never touch prometheus types.
Dep: github.com/prometheus/client_golang (standard Go client; cluster runs
prometheus-operator). TDD: collectors + middleware route-pattern cardinality
covered.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 08:32:41 +02:00
mathias 64d11af9ef docs(bdd): observability.feature scenarios (@pending until TDD, #15) 2026-06-12 08:28:46 +02:00
mathias b590d2708d docs(adr): ADR-030 observability — slog + Prometheus metrics (issue #15) 2026-06-12 08:27:56 +02:00
mathiasandClaude Opus 4.8 7314895ec4 fix(auth): stateless session cookie — stop logging users out on deploy (ADR-029)
CI / Lint / Test / Vet (push) Successful in 10s
CI / Build & Import (push) Successful in 11s
Pilot feedback: lots of re-logging-in on iPhone. Three causes: sessions lived in
an in-memory map (wiped on every pod restart/deploy), a 1h TTL (idle >1h forced
re-login on a check-back-tomorrow reader), and a session cookie with no Max-Age
(dropped on Safari close). Each re-login is the full IdP redirect dance.

Make sessions stateless: identity + absolute expiry live inside the existing
HMAC-signed cookie (no server table), TTL 1h → 30 days sliding, cookie now
persistent (Max-Age). Survives restarts (test: a cookie from one instance is
accepted by a fresh instance with the same secret), browser-close, and idle.
Trade: no server-side revocation — logout clears the cookie client-side; rotating
tapir-session-secret is the global logout lever. Accepted for the Stage-0 reader.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 23:07:13 +02:00
mathiasandClaude Opus 4.8 36dd182fb5 feat(report): Stage-0 usage gate counts from a baseline date (default 2026-06-11)
CI / Build & Import (push) Successful in 11s
CI / Lint / Test / Vet (push) Successful in 10s
The return-usage gate (ADR-016) counted distinct active weeks over ALL history,
so pre-launch noise — testing churn and the period the pilot sat blocked on zero
summaries — would inflate the signal. Add a baseline: ActiveWeeks(ctx, since)
filters login_events + summary_actions to seen_at/acted_at >= since. The report
command sets it to TAPIR_USAGE_GATE_START (YYYY-MM-DD, default 2026-06-11 — the
morning the pilot was unblocked) and prints the baseline. The gate now measures
whether users RETURN once it genuinely works.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 22:52:15 +02:00
mathias 40808f2d4b docs(onboard): BDD scenarios + architecture for the ADR-028 burst
CI / Lint / Test / Vet (push) Successful in 10s
CI / Build & Import (push) Successful in 10s
- connect_account.feature: refine the burst scenario to "best recent" (likely-good
  selection, skips too-short/too-long) and add a scenario for the stronger model.
- scenario_coverage: remap to TestOnboardBurstVideoIDs + add the model scenario
  (the old NewestUnsummarizedVideoIDs test was removed).
- architecture.md: new "Connect-time onboarding burst" subsection — junk-avoiding
  selection over persisted duration, the burst-only stronger-model chain, and why
  has-captions/cached-first are not selection signals.
2026-06-11 19:54:29 +02:00
mathias 9232f49555 feat(onboard): burst picks likely-good videos, summarizes with stronger model (ADR-028)
Wire the onboarding burst to its quality-aware selection and a stronger model:
- main.go onboard uses OnboardBurstVideoIDs (junk-avoiding) instead of pure
  newest-first, bounded by MinVideoSeconds / OnboardMaxVideoSeconds.
- buildBurstProcessor builds a burst-only summarizer chain led by the onboard
  model (burstChainModels: onboard -> standard ADR-022 chain, deduped, NDA lever
  intact), over the same store/cache/sink. Collapses onto the shared Processor
  when the onboard model is empty/equal-to-primary or config is incomplete.
- summarizerEndpoint + newYouTubeSource extracted so standard and burst wiring
  share one definition.
- Remove now-superseded NewestUnsummarizedVideoIDs: OnboardBurstVideoIDs(.,0,0)
  is identical pure-newest behaviour and its test covers RLS + ordering.
2026-06-11 19:54:29 +02:00
mathias 582c1a2065 feat(store): OnboardBurstVideoIDs — junk-avoiding burst selection (ADR-028)
Newest-first but quality-aware: excludes a video when its duration is KNOWN and
outside [minSeconds, maxSeconds], dropping Shorts and multi-hour livestream VODs
that waste a scarce caption fetch on a poor first impression. NULL/unknown
duration is kept (degrade-open) but ranked after known-good rows. 0/0 bounds
disable the filter (pure newest-first, the reversibility lever). RLS-scoped.
NewestUnsummarizedVideoIDs is left in place for callers that want pure-newest.
2026-06-11 19:54:29 +02:00
mathias 9f0d8cf198 feat(discovery): persist video duration_s instead of discarding it (ADR-028)
ADR-023's filterLowValue already fetches each candidate's duration via the
cheap videos.list quota call to drop Shorts/live, then threw it away — the
videos.duration_s column (migration 001) was never written. Carry it onto the
kept domain.Video and have UpsertVideo persist it, COALESCE-preserving a known
value so an unknown (0) re-upsert never clobbers it (the channel_title backfill
stance, migration 014). This is the enabling change for length-aware burst
selection. No new migration — the column already exists.
2026-06-11 19:54:29 +02:00
mathias d21077303d feat(config): add OnboardSummarizerModel + OnboardMaxVideoSeconds (ADR-028)
Two knobs for the onboarding-burst quality work:
- TAPIR_ONBOARD_SUMMARIZER_MODEL (default iguana/gemma4-26b): the stronger model
  the burst leads its chain with; empty collapses the burst onto the shared
  processor (lookupOr, so explicit-empty disables).
- TAPIR_ONBOARD_MAX_VIDEO_SECONDS (default 14400/4h): upper duration bound for
  burst picks; 0 disables, negative clamps to 0.

Table-driven tests cover defaults, explicit, disable, and invalid input.
2026-06-11 19:54:29 +02:00
mathias 9b2ee2e765 docs: spec onboarding-wow-burst + ADR-028 (better picks, stronger model)
Phase-1 live-DB investigation found the connect-time burst (ADR-018) fires
but delivers a weak first impression: pure newest-first selection picks junk
(a livestream + regional news for pilot user Jonte), and all burst summaries
run on the weak koala/phi4-mini instead of the validated iguana/gemma4-26b.
Cached-first was investigated and rejected — ~3% cross-user overlap, 0
cached-and-unsummarized, newest-20 all uncached (newest-first and cached-first
are structurally incompatible).

ADR-028 records: persist duration_s at discovery (ADR-023 already fetches it,
just stops discarding), junk-avoiding burst selection (drop known too-short/
too-long), and a stronger burst-only summarizer chain. Not a throughput change.
2026-06-11 19:54:29 +02:00
mathias b8fbc5a805 docs: spec first-session wow — verify & improve onboarding summary burst (investigate-first)
CI / Lint / Test / Vet (push) Successful in 11s
CI / Build & Import (push) Successful in 10s
The concern is new-user first-contact: see some GOOD summaries fast or they
don't return. Reframed as curation/latency for ~3 videos, NOT a 429/throughput
problem (3 fetches is nowhere near the wall). Phase 1 (report-and-stop) verifies
whether the existing cap-3 onboarding burst even fires today, what it delivers,
and — critically — how much transcript-cache overlap exists between users (drives
the blend). Phase 2 levers: cached-transcript-first (instant, zero-fetch),
likely-good selection (has-captions/good-length, not just newest), and optionally
the stronger model for the burst's few summaries. Blend deferred to the
maintainer post-Phase-1. Explicitly NOT bulk-fetch, NOT credentials (ADR-010/026
dead end), NOT a client extension.
2026-06-11 15:36:09 +00:00
mathiasandClaude Opus 4.8 19ca4282a8 refactor(web): dock chat inline below the summary, integrated view (ADR-027)
CI / Lint / Test / Vet (push) Successful in 10s
CI / Build & Import (push) Successful in 10s
The chat was a separate page — opening it left the summary behind, the very
context you're asking about. Now the chat docks open IN PLACE below the summary:
"Dig deeper" reveals the chat section (HTMX, outerHTML over the closed dock) so
the summary stays on screen above it; no navigation. The no-JS fallback renders
the full summary AND the open chat on one page (the same integrated view), so
progressive enhancement holds.

Extracts a shared summaryBody templ so the detail page and the chat page render
one identical summary, not two divergent ones. GET /v/{id}/chat returns just the
open chat section as a fragment for the inline reveal, or the full summary+chat
page for a no-JS navigation; POST swaps the panel inline or re-renders the whole
page. Tests assert summary+chat coexist on the page and the reveal is a fragment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 14:02:39 +02:00
mathiasandClaude Opus 4.8 71df696448 feat(web): per-video chat over the stored transcript (ADR-027)
CI / Lint / Test / Vet (push) Successful in 10s
CI / Build & Import (push) Successful in 10s
A deeper-dive chat entered from the summary view: ask questions about a video
against its already-stored transcript (ADR-021), no caption fetch, ever.

Safety by construction — the load-bearing property. The chat handlers reach the
chat.Service only after reading the SHARED stored transcript via Store.GetTranscript
(a pure DB read); the service holds no VideoSource. So an enabled chat cannot
trigger a caption fetch, touch the rate gate, or reach YouTube. A video with no
stored transcript gets an honest "not available" — no fetch, no model call. The
key web test wires the summarize/fetch collaborators as tripwires that fail the
test if chat ever routes into them, and asserts the model answered from the
stored text.

Model defaults to the summary's own model and is switchable among the ADR-022
chain (phi4-mini → gemma4-26b → mistral-small); switching re-runs against the
same transcript — deliberate model-comparison instrumentation. The cloud model
is absent from the switcher when TAPIR_CLOUD_FALLBACK_MODEL="" (the local-first /
NDA lever), honoured the same way the summarizer honours it. Reuses the existing
LiteLLM gateway client (a chat is a different call, not a new integration) and
the TAPIR_MAX_TRANSCRIPT_CHARS truncation, surfacing an honest bounded-context
note when a long transcript is cut.

Ephemeral v1: the multi-turn conversation rides in hidden request fields; no
table, no migration, nothing persisted. Entry is RLS-scoped through
GetSummaryByVideo, so chat is reachable only from the user's own summary view.
Show-source verification and on-demand fetch are deferred (ADR-027).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 09:26:58 +02:00
mathiasandClaude Opus 4.8 cc3cda4ab8 feat(chat): stored-transcript QA service (ADR-027)
A read-only deeper-dive over a video's already-stored transcript (ADR-021).
Safe by construction: the Service has no VideoSource and no caption-fetch
dependency — only a Completer factory over the existing LiteLLM gateway — so it
cannot reach YouTube or the rate gate. Reuses the summarizer's truncation
discipline (TAPIR_MAX_TRANSCRIPT_CHARS), reporting the cut so the UI can be
honest about a bounded transcript. Model defaults to the summary's model and is
switchable among an offered, local-first list; an un-offered alias is forced
back to the default so chat can never call the gateway with an arbitrary model.
Ephemeral: history is carried per-request, nothing persisted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 09:16:45 +02:00
mathiasandClaude Opus 4.8 69a49bc603 docs(decisions): add ADR-027 — chat with a video's stored transcript
Records the deeper-dive chat decision before its build: stored-transcript-only
(safe by construction — no caption fetch, no rate gate, no YouTube), entered
from the summary view, model defaulting to the summary's model and switchable
among the ADR-022 chain. Ephemeral v1; show-source verification deferred to v2.
Inserted before "Rejected alternatives" so the decision precedes the feature.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 09:13:31 +02:00
mathias 137804b0b1 docs: spec chat-with-transcript (ADR-027)
CI / Lint / Test / Vet (push) Successful in 11s
CI / Build & Import (push) Successful in 10s
Per-video chat against the ADR-021 stored transcript, entered from the summary
view, born from observed demand (maintainer read real summaries, some made him
want to dig deeper). HARD constraint: stored-transcript-only — never fetches
captions, never touches the rate gate or YouTube, safe by construction. Default
model = the summary's model, user-switchable among the ADR-022 chain models
(doubles as model-comparison instrumentation). Ephemeral v1 (no persisted
history); chat-only/trust-the-model with show-source verification recorded as the
natural v2. ADR-027 to be appended to DECISIONS.md as the first commit.
2026-06-11 06:38:16 +00:00
mathiasandClaude Opus 4.8 a9be5f285b feat(summarizer): move local fallback off koala to iguana/gemma4-26b
CI / Build & Import (push) Successful in 10s
CI / Lint / Test / Vet (push) Successful in 10s
koala now carries other GPU loads, so the first fallback should not run there.
Change the default chain to koala/phi4-mini → iguana/gemma4-26b → berget/mistral-small:
the local fallback now runs on iguana (M2 Ultra headroom, different host = different
egress IP for the rare fallback fetch). gemma4-26b is the brain-validated homelab
general-purpose model (agentsquad H2/H3 executor) and returned valid summary JSON on
the real prompt in a smoke test (~37s incl. cold-load — fine for a fallback path).

Pure config default (TAPIR_FALLBACK_MODEL); chain mechanism (ADR-022) unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 07:54:56 +02:00
mathiasandClaude Opus 4.8 fe56e2fe01 test: make embedded-postgres per-process so concurrent CI runs don't collide
CI / Lint / Test / Vet (push) Successful in 9s
CI / Build & Import (push) Successful in 10s
A push to main and its version tag fire two CI runs for the same commit. Both ran
`go test ./...`, which starts embedded-postgres on a FIXED port (54329/54330) and
a shared data dir. -p 1 serialises packages WITHIN a run, not across two
concurrent runs — so when the two runs overlapped they collided on the port/data
dir and BOTH failed the Lint/Test job (no image built). Prior commits passed only
because their two runs happened not to overlap.

Derive the port and runtime/data dirs from the PID; share only CachePath so the
PG archive downloads once. Proven: two concurrent `go test` of the store package
now both pass. Unblocks the v0.21.0 (Pillar A) build.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 07:26:33 +02:00
mathiasandClaude Opus 4.8 beeb5bc31b feat(gate): foreground caption fetches take priority over the background sweep (ADR-026)
CI / Lint / Test / Vet (push) Failing after 9s
CI / Build & Import (push) Has been skipped
A user waiting on a Summarize click shared the per-IP caption gate equally with
the background firehose, so on a busy IP the click was slow or 429'd. Add a
context-marked priority lane: the web path (engineProcessor.ProcessVideo) marks
its context foreground; the gate serves foreground immediately while background
fetches yield until no foreground is pending. Threaded via a context value (no
new signatures) + a process-wide foregroundPending counter. Clicks are rare, so
the background barely loses throughput; the waiting human gets the cleaner slot.

Drops the credentials probe: ADR-010 and captions.go already settle it — the
timedtext/InnerTube path rejects authenticated requests and the OAuth token does
not authenticate it anyway, so auth cannot help and can hurt. Documented in
ADR-026 rather than built.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 22:10:15 +02:00
mathiasandClaude Opus 4.8 09eb31d1fe fix(scheduler): derive rotation offset from wall-clock, not a reset-on-restart counter
CI / Lint / Test / Vet (push) Successful in 11s
CI / Build & Import (push) Successful in 14s
The lead-user rotation used an in-memory pass counter reset to 0 on every pod
restart, so the first-listed user always re-took the lead after a restart — a
deploy-heavy session re-starved the last user (the pilot stalled at 2 summaries
because each deploy reset his every-other-pass lead before the 2h tick fired).
Derive the offset from wall-clock (floor(now/interval)) so it advances with real
time and is identical across restarts: rotation stays fair however often the pod
bounces.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 21:59:12 +02:00
mathiasandClaude Opus 4.8 1665a1e7c4 feat(web): honest, state-aware summarize status with charm spinner (ADR-025)
CI / Build & Import (push) Successful in 10s
CI / Lint / Test / Vet (push) Successful in 12s
Clicking Summarize polled /status, which knew only "spinning" or "done". The web
ProcessVideo recorded an outcome only on success, so a 429'd or caption-less
click left transcript_status unset and the poll silently reverted to the
Summarize button — the rate limit was invisible and the click felt broken.

- ProcessVideo now records rate_limited / none / fetched (mirrors the runner); a
  rate-limited video keeps its requested flag so the background sweep retries it.
- /status is state-aware: summary card (done), working spinner (in-flight),
  a calm "waiting on rate limit, will retry" card that keeps polling so the
  summary lands on its own (no re-click), and a terminal "no captions" card.
- Charm status text (Claude-Code / Crush inspired): the spinner cycles playful
  tapir-themed gerunds via CSS only (no JS), decorative + an sr-only stable
  status line for a11y.

Pillar A (foreground fetch priority lane) is a separate follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 21:52:33 +02:00
mathiasandClaude Opus 4.8 5219561a91 feat(discovery): per-channel caption-availability memory (ADR-024)
CI / Lint / Test / Vet (push) Successful in 30s
CI / Build & Import (push) Successful in 11s
After transcript caching (ADR-021) and the Shorts filter (ADR-023), the remaining
caption waste is the first fetch on every new video of a channel that never has
English captions — each costs one rate-limited fetch to resolve to "none", and on
a throttled IP churns the backoff machinery first.

Remember, per (user, channel), a streak of consecutive no-caption outcomes
(channel_caption_state, migration 016, RLS-scoped). Once it reaches
TAPIR_CHANNEL_CAPTIONLESS_THRESHOLD (default 5) the channel is suppressed — videos
discovered/listed but not caption-fetched — for TAPIR_CHANNEL_CAPTIONLESS_WINDOW
(default 14d), then one is re-probed (auto-recovery). A successful fetch resets
the streak; a 429 does not count; an explicit manual request bypasses suppression.
threshold=0 disables.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 20:27:25 +02:00
mathiasandClaude Opus 4.8 9db06d8a63 feat(discovery): drop Shorts and livestreams before the caption fetch (ADR-023)
CI / Lint / Test / Vet (push) Successful in 12s
CI / Build & Import (push) Successful in 10s
The scarce resource is the per-IP timedtext caption fetch (ADR-014); the pilot's
candidate set was mostly Shorts/clips/livestreams, each burning a fetch (a "none"
result is a completed fetch — it costs budget even when it yields nothing).

NewVideos now enriches candidates with one cheap Data API videos.list call
(contentDetails.duration + snippet.liveBroadcastContent — the quota API, a
DIFFERENT limit from the timedtext 429) and drops, before returning: videos
shorter than TAPIR_MIN_VIDEO_SECONDS (default 60) and any live/upcoming
broadcast. Dropped videos are never persisted, so the list declutters too.

Degrade-open: MinVideoSeconds=0 disables it (no quota call); a videos.list error
returns candidates unfiltered so discovery never breaks on a metadata hiccup. The
paste-a-URL path (VideoByID) is not filtered — an explicit request is honoured.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 19:34:45 +02:00
mathiasandClaude Opus 4.8 1aa8a97f95 fix(scheduler): rotate over connected users only for true fair share
CI / Lint / Test / Vet (push) Successful in 12s
CI / Build & Import (push) Successful in 10s
The lead-user rotation rotated the full ListAllUsers set, so a connectionless
orphan identity ate a rotation slot — collapsing onto the next real user and
skewing the lead share (two real users got 2/3 vs 1/3 instead of 50/50). Filter
to connected users BEFORE rotating so the rotation is over exactly the users that
consume the caption budget. A dead identity can no longer skew fairness.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 18:28:54 +02:00
mathiasandClaude Opus 4.8 cc69a912f4 fix(scheduler): rotate lead user each pass so caption budget is shared
CI / Lint / Test / Vet (push) Successful in 12s
CI / Build & Import (push) Successful in 10s
Caption fetches share one per-egress-IP rate budget; whoever runs first each pass
spends the pre-throttle window before YouTube starts 429ing. ListAllUsers order
is unspecified and was stable, so the last-listed user was permanently starved —
a friendly-pilot user got 0 fetches in 12h (all rate_limited) while the
first-listed user got every successful fetch. rotateUsers left-rotates the user
order by pass index so each user leads 1/N passes and the lead slot is shared.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 18:07:01 +02:00
mathiasandClaude Opus 4.8 f4a0544903 fix(scheduler): cache transcripts on the scheduled path (ADR-021 regression)
buildUserRunner built the engine without engine.Transcripts = st, so the
scheduler — unlike the web "Summarize now" path — never read or wrote the shared
transcript cache. Every discovery pass re-fetched transcripts it had already
fetched, burning the scarce per-egress-IP caption budget (ADR-014) on redundant
work and starving other users' first-time fetches. The transcripts table was
empty despite summaries existing. Wire the cache on this path too.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 18:07:01 +02:00
mathiasandClaude Opus 4.8 e2a52789b9 feat(web): mode-aware backlog banner — stop telling manual users summaries auto-arrive
CI / Lint / Test / Vet (push) Successful in 12s
CI / Build & Import (push) Successful in 10s
The first pilot user sat in Manual mode reading "new summaries land gradually,
check back tomorrow" — copy that only makes sense in Automatic mode. Manual mode
never auto-summarizes, so the banner promised delivery that would never come.

The list page now reads the user's summarize mode and shows mode-correct copy:
- Auto: unchanged "land gradually" backlog note.
- Manual: "new videos appear here but are not summarized automatically — use the
  Summarize button" plus a "Switch to Automatic" link to /account.
The connected-but-empty first-run state is likewise mode-aware.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 09:14:52 +02:00
mathias e696b6405b docs(build-state): v0.15.0, summarizer is now a resilient chain (ADR-022)
CI / Lint / Test / Vet (push) Successful in 12s
CI / Build & Import (push) Successful in 10s
2026-06-10 08:59:26 +02:00
mathiasandClaude Opus 4.8 e9b5a3f3e7 feat(summarizer): resilient endpoint chain with local→cloud fallback (ADR-022)
CI / Lint / Test / Vet (push) Successful in 11s
CI / Build & Import (push) Successful in 10s
The first friendly-pilot live run produced zero summaries: koala/phi4-mini hit
three silent failure modes — 8k context overflow on long transcripts (HTTP 400),
intermittent malformed JSON (highlights as a bare string), and no fallback wired
at all (summarizer.New(primary, nil)).

Keep phi4-mini as the fast primary and add resilience around it:

- Ordered endpoint chain (summarizer.NewChain): phi4-mini → koala/phi4-14b
  (local) → berget/mistral-small (worst-case external). All reached through the
  one LiteLLM gateway by alias.
- A parse failure now advances the chain like a transport error — the old
  Primary→Fallback shape returned the parse error without trying anyone else.
- Tolerant parse: highlights/takeaways coerce string→[]string, absorbing the
  common small-model quirk without spending a fallback round-trip.
- Transcript truncation (TAPIR_MAX_TRANSCRIPT_CHARS=18000) prevents the overflow
  rather than recovering from it; validated to fit phi4-mini's 8k window.
- Bounded completion budget (TAPIR_SUMMARY_MAX_TOKENS=1500) — the old 8192 budget
  itself contributed to the overflow.

Local-first guarantee preserved by ordering: external endpoint is tried only
after every local one fails. TAPIR_CLOUD_FALLBACK_MODEL="" disables it entirely
for client/NDA deployments.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 08:36:16 +02:00
mathiasandClaude Opus 4.8 0ba78e8868 docs: refresh build-state for transcript persistence (ADR-021)
CI / Lint / Test / Vet (push) Successful in 11s
CI / Build & Import (push) Successful in 10s
Update the CLAUDE.md orientation block: last tag v0.14.0, migrations
001–015, and a transcript-persistence bullet (shared non-RLS store,
engine reads stored-first). The stale "v0.9.0" reference is corrected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 23:39:27 +02:00
mathiasandClaude Opus 4.8 821d5f99cd docs(bdd): scenario for transcript reuse — re-analysis never re-fetches
Capture the ADR-021 promise as a mapped BDD scenario: re-analyzing a
stored video reads the stored transcript and does not fetch captions.
Since paste-a-URL and the onboarding burst summarize through the same
engine chokepoint (resolveTranscript, store-first), this one scenario
covers their reuse path too — there is exactly one gated caption entry
point (youtube.FetchTranscript → WaitFetchGate) and one engine caller in
front of it, so the dedup is structural, not per-feature.

Mapped to TestProcessNewVideo_SecondSummarizeDoesNotRefetch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 23:37:54 +02:00
mathiasandClaude Opus 4.8 5c70408e75 feat(usecase): read stored transcript before fetching (ADR-021)
The engine now resolves transcripts store-first: a stored transcript —
including a stored SourceNone — is summarized without touching YouTube,
so re-analysis never re-fetches. On a miss it fetches through the source
(caption call still gated, ADR-014) and persists the terminal outcome for
the next analysis by any user. A transient SourceRateLimited is surfaced
to the runner for per-user backoff but never cached, so persistence can
never mask a 429 as a permanent "no transcript".

The TranscriptStore is optional (nil → fetch every time), keeping the
pure-core and scaffold wiring valid. cmd/tapir wires the store as both
summary sink and transcript cache, so `tapir run` and the web summarize
path (incl. paste + onboarding) all share the dedup.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 23:35:04 +02:00
mathiasandClaude Opus 4.8 cb6917ca59 feat(store): shared, video-keyed transcript persistence (ADR-021)
Reshape the dead per-user transcripts table (PK videos.id, user_id,
RLS-FORCEd — never read or written by app code) into the shared public
caption store ADR-021 specifies: keyed by (provider, provider_video_id),
no user_id, NOT RLS-scoped. Migration 015 (reversible). Add
ports.TranscriptStore + Store.GetTranscript/SaveTranscript via the raw
pool (no withUser): public content, shared across users by construction.
SaveTranscript persists only terminal outcomes (captions/none) and
refuses SourceRateLimited so a transient 429 can never be stored as a
false permanent absence (ADR-014).

Flip the isolation proof: transcripts leaves the RLS-scoped set;
TestTranscriptsTableIsSharedNotRLS asserts it is the SINGLE non-RLS
surface (writable/readable with no user scope, no user_id column, RLS off
on it alone, still on every user-owned table) — the proof the
public-content classification was applied exactly here and leaked nowhere.
appPool made idempotent so two tests can build it. Adjust the 010/011/014
up-down migration tests for the new HEAD. account.go: user deletion no
longer strips shared transcripts. Reconcile data-model.md + CLAUDE.md.

Wiring the engine to read-stored-first is the next commit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 23:32:12 +02:00
mathiasandClaude Opus 4.8 099b2d4c68 docs(decisions): ADR-021 — shared, video-keyed transcript persistence
Persist transcripts in a single shared table keyed by
(provider, provider_video_id) — public caption content, NOT RLS-scoped —
so re-analysis (re-summarize, paste of an already-seen video, a second
user with overlapping subs) never re-fetches from YouTube. The avoided
cost is the rate-gated, reputation-risky caption fetch (ADR-010/014), not
LLM re-summarization, which is why this reopens the transcripts half of
the "no global cross-tenant table" rejection while videos stay per-user.
Summaries remain RLS-scoped (ADR-012 unchanged). The gate is neither
bypassed nor weakened — persistence reduces fetch frequency, not pacing.

Annotate the rejected-alternatives row to record the partial reopen.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 23:24:11 +02:00
mathiasandClaude Opus 4.8 f66c1bcdcc feat(web): real channel filter — multi-select of the user's channels
CI / Build & Import (push) Successful in 10s
CI / Lint / Test / Vet (push) Successful in 11s
The free-text 'channel' filter was dead: it exact-matched SummaryRow.Channel,
which is just the provider ('youtube'), because videos never stored their source
channel. Now they do.

- migration 014: videos.channel_title (nullable; existing rows backfill on the
  next discovery pass, pasted videos immediately).
- discovery (NewVideos) + paste (VideoByID) populate channel_title; UpsertVideo
  persists it, preserving an existing title when an update arrives empty.
- store.DistinctChannels lists a user's channels (RLS-scoped); SummaryRow carries
  ChannelTitle via the shared projection.
- Filter: single Channel -> Channels []string, matching on ChannelTitle; the feed
  renders a multi-select of DistinctChannels (hidden until channels exist).
- migrate tests: 014 reversibility + fixed the relative-step counts in the 010/011
  up/down tests (014 shifted the topology).

TDD throughout: channel persist + distinct, adapter channel wiring, multi-channel
filter match, handler channel filter, migration up/down.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 23:02:54 +02:00
mathiasandClaude Opus 4.8 1e65c3b413 fix(web): show paste box to any connected user, not only on an empty feed
CI / Lint / Test / Vet (push) Successful in 11s
CI / Build & Import (push) Successful in 10s
hasConnected was computed only inside the buckets.empty() branch (it was added
for the empty-state copy), so a connected user WITH videos got hasConnected=false
and never saw the paste box (#2-regression of the v0.12.0 paste UI). Compute it
on every list render. Test: connected user with a non-empty feed sees the box.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 22:17:00 +02:00
173 changed files with 17885 additions and 1202 deletions
+121
View File
@@ -0,0 +1,121 @@
#!/usr/bin/env bash
# Build an OCI image with buildctl and push it to the local registry.
#
# CANONICAL COPY: mathias/infra ci-templates/build-image.sh (infra#309).
# Every repo carries a byte-identical copy at .gitea/build-image.sh, written by
# `task ci:sync` and held in place by the drift gate. DO NOT EDIT THE COPIES --
# edit here, re-sync, and the gate will tell you which repos still disagree.
#
# Why this exists: this step was hand-copied into a dozen repos, so migrating it
# off rootless buildah (infra#241) meant eleven near-identical commits, eleven
# verifications, and three separate rediscoveries of the same fix in the two
# repo templates and the gitea-ci skill. The differences between the copies were
# incidental -- a label here, a build-arg there -- never intentional. Those are
# inputs now, not forks of the implementation.
#
# Why buildctl and not buildah: the gitea-runner pod runs with
# `capabilities: drop [ALL]` and `allowPrivilegeEscalation: false`, which
# unconditionally blocks the setuid-root exec that rootless buildah's uid/gid
# remapping needs, and the pod has no OCI runtime at all. buildctl is a thin
# client -- it hands the build to the host's buildkitd over a socket, so the
# privileged layer work happens on the host and this pod's securityContext is
# not in the picture. buildah is absent from the runner image as of baked-v14
# and its absence is asserted at pod start (infra#308).
#
# Inputs, all via environment:
#
# IMAGE (required) image name, e.g. cobalt-dingo
# REGISTRY default localhost:5000
# SHA_TAG default `git rev-parse --short HEAD`
# VERSION_TAG optional extra tag, e.g. v1.2.3 on a tag build
# PUSH_LATEST default 1; set 0 to skip the :latest tag
# CONTEXT default . -- build context directory
# DOCKERFILE_DIR default $CONTEXT -- directory holding the Dockerfile
# DOCKERFILE_NAME optional, when the file is not named `Dockerfile`
# TARGET optional, multi-stage build target (--opt target=)
# BUILD_ARGS optional, space-separated K=V pairs
# BUILD_SECRETS optional, space-separated buildkit secret specs, each
# `id=NAME,src=PATH`. Pass the SPEC only -- the caller
# writes the secret to PATH itself. A secret VALUE must
# never be passed here: it would land in this script's
# argv and therefore in the CI log.
# EXTRA_LABELS optional, space-separated K=V pairs
# GIT_SHA optional, stamped as org.opencontainers.image.revision
# IMAGE_SOURCE optional, stamped as org.opencontainers.image.source
#
# Note on shell style below: every conditional is a full `if` block, never
# `[ -n "$X" ] && cmd`. Under `set -e` the && form exits the script whenever the
# test is false, which is a real bug this estate has already shipped once.
set -euo pipefail
if [ -z "${IMAGE:-}" ]; then
echo "build-image: IMAGE is required" >&2
exit 2
fi
REGISTRY="${REGISTRY:-localhost:5000}"
BUILDKIT_HOST="${BUILDKIT_HOST:-unix:///run/buildkit/buildkitd.sock}"
CONTEXT="${CONTEXT:-.}"
DOCKERFILE_DIR="${DOCKERFILE_DIR:-$CONTEXT}"
PUSH_LATEST="${PUSH_LATEST:-1}"
VERSION_TAG="${VERSION_TAG:-}"
if [ -z "${SHA_TAG:-}" ]; then
SHA_TAG="$(git rev-parse --short HEAD)"
fi
REF="${REGISTRY}/${IMAGE}:${SHA_TAG}"
TAR_FILE="$(mktemp)"
trap 'rm -f "$TAR_FILE"' EXIT
args=(
--frontend dockerfile.v0
--local "context=${CONTEXT}"
--local "dockerfile=${DOCKERFILE_DIR}"
)
if [ -n "${DOCKERFILE_NAME:-}" ]; then
args+=(--opt "filename=${DOCKERFILE_NAME}")
fi
if [ -n "${TARGET:-}" ]; then
args+=(--opt "target=${TARGET}")
fi
for spec in ${BUILD_SECRETS:-}; do
# Opaque id=NAME,src=PATH -- buildkit reads the file, nothing is echoed.
args+=(--secret "${spec}")
done
if [ -n "${GIT_SHA:-}" ]; then
args+=(--opt "label:org.opencontainers.image.revision=${GIT_SHA}")
fi
if [ -n "${IMAGE_SOURCE:-}" ]; then
args+=(--opt "label:org.opencontainers.image.source=${IMAGE_SOURCE}")
fi
for kv in ${BUILD_ARGS:-}; do
args+=(--opt "build-arg:${kv}")
done
for kv in ${EXTRA_LABELS:-}; do
args+=(--opt "label:${kv}")
done
echo "build-image: building ${REF}"
buildctl --addr "$BUILDKIT_HOST" build "${args[@]}" --output "type=oci,dest=${TAR_FILE}"
# buildctl has no registry-push equivalent to `buildah push`, so skopeo copies
# the OCI archive up -- once per tag, from the same archive, so every tag points
# at bit-identical content.
push_tag() {
echo "build-image: pushing ${1}"
skopeo copy --dest-tls-verify=false "oci-archive:${TAR_FILE}" "docker://${1}"
}
push_tag "$REF"
if [ "$PUSH_LATEST" = "1" ]; then
push_tag "${REGISTRY}/${IMAGE}:latest"
fi
if [ -n "$VERSION_TAG" ]; then
push_tag "${REGISTRY}/${IMAGE}:${VERSION_TAG}"
fi
echo "✓ build-image: ${REF} pushed"
+157 -27
View File
@@ -61,33 +61,163 @@ jobs:
echo "version-tag=${REF#refs/tags/}" >> "$GITHUB_OUTPUT" echo "version-tag=${REF#refs/tags/}" >> "$GITHUB_OUTPUT"
fi fi
# infra#241: rootless buildah inside the gitea-runner pod can't remap the
# UID/GID range newuidmap/newgidmap need (the pod drops ALL capabilities
# and sets allowPrivilegeEscalation:false — both deliberate, it also holds
# CI secrets), so extracting a base-image layer that chowns outside the
# single available mapping fails with `ApplyLayer ... requested 0:42 for
# /etc/gshadow`. Main has been red on exactly this since 2026-08-13.
# buildctl instead talks to koala's host buildkitd over the socket the
# runner pod already mounts: the privileged extraction happens inside
# buildkitd on the host, never in this pod. Same migration agentsquad
# (infra#233) and segment-embedder (infra#243) already made — zero
# security tradeoff, unlike loosening the pod's capabilities.
#
# buildctl has no registry-push flag of its own: it writes an OCI archive
# and skopeo pushes that, which is also why every tag below is one copy of
# a single build rather than a rebuild.
- name: Build and push to local registry - name: Build and push to local registry
run: | # infra#309: implementation lives in .gitea/build-image.sh, a
REGISTRY="localhost:5000" # byte-identical copy of mathias/infra's ci-templates/build-image.sh.
REF="${REGISTRY}/${{ env.IMAGE }}:${{ steps.meta.outputs.sha-tag }}" # Do not edit it here -- edit the canonical file, run `task ci:sync`,
buildah build \ # and infra's drift gate names any repo still carrying an old copy.
--label "org.opencontainers.image.revision=${{ github.sha }}" \ env:
--label "org.opencontainers.image.source=${{ github.repositoryUrl }}" \ SHA_TAG: ${{ steps.meta.outputs.sha-tag }}
-t ${REF} \ VERSION_TAG: ${{ steps.meta.outputs.version-tag }}
-t ${REGISTRY}/${{ env.IMAGE }}:latest \ GIT_SHA: ${{ github.sha }}
. IMAGE_SOURCE: ${{ github.repositoryUrl }}
buildah push --tls-verify=false ${REF} run: bash .gitea/build-image.sh
buildah push --tls-verify=false ${REGISTRY}/${{ env.IMAGE }}:latest
[[ -n "${{ steps.meta.outputs.version-tag }}" ]] && \
buildah push --tls-verify=false \
${REGISTRY}/${{ env.IMAGE }}:${{ steps.meta.outputs.version-tag }} || true
echo "Image pushed to ${REF}"
- name: Smoke test # The buildah-based smoke test that used to sit here is gone with the
run: | # buildctl migration above: `buildah from ${REF}` would pull the image
REGISTRY="localhost:5000" # back into the pod's buildah store and hit the very uid/gid wall the
REF="${REGISTRY}/${{ env.IMAGE }}:${{ steps.meta.outputs.sha-tag }}" # migration exists to avoid — it would fail the job for a reason that has
CNAME="smoke-${{ steps.meta.outputs.sha-tag }}" # nothing to do with the image being good. The `Verify rollout` step in
sudo k3s ctr images pull --plain-http ${REF} # the deploy job below is the stronger check anyway: it requires the real
OUTPUT=$(timeout 5 sudo k3s ctr run --rm ${REF} ${CNAME} /tapir 2>&1 || true) # pod to reach Ready in-cluster, not just the entrypoint to print its own
sudo k3s ctr containers delete ${CNAME} 2>/dev/null || true # name for 5s. Tag builds lose their only image-level check (deploy is
echo "$OUTPUT" | grep -q "tapir" \ # main-push only) — accepted; a tag is always built from a main commit
&& echo "Smoke test passed" \ # that already deployed.
|| echo "Smoke test inconclusive: $OUTPUT"
# ── 3. Mirror to GitHub — skipped for now (SSH key rotation pending) ─ # ── 3. Deploy via infra repo + Flux ─────────────────────────────────────────
# Flux native image-automation can't scan localhost:5000 from inside k3s pods
# (mathias/infra k3s/flux/flux-system/image-automation.yaml) — this job
# mirrors cobalt-dingo's proven pattern instead: patch the infra repo's
# manifest directly on every push to main, then annotate Flux for a fast
# reconcile. Fixes infra#111 (image built+pushed but manifest bump was
# manual, so merged features silently didn't deploy).
deploy:
name: Deploy via GitOps
needs: build
runs-on: self-hosted
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
environment: staging
steps:
- name: Update image tag in infra repo
env:
IMAGE_TAG: ${{ needs.build.outputs.image-tag }}
DEPLOY_KEY: ${{ secrets.INFRA_DEPLOY_KEY }}
run: |
set -euo pipefail
# infra#311: fail on an empty tag BEFORE patching anything. The
# grep verification further down CANNOT catch it -- with IMAGE_TAG
# empty its pattern matches the very corruption it exists to reject.
# The result is `image: <repo>:` in the manifest, which is invalid
# YAML, so Flux fails the ENTIRE apps Kustomization and every app
# stops reconciling, not just this one. That happened on 2026-08-25.
[ -n "${IMAGE_TAG:-}" ] || { echo "IMAGE_TAG is empty - refusing to patch"; exit 1; }
# INFRA_DEPLOY_KEY is a Gitea org secret (mathias org), already
# configured per docs/cd-pipeline.md in the infra repo — same key
# cobalt-dingo and brain-gardener use, no new secret needed.
mkdir -p ~/.ssh
echo "$DEPLOY_KEY" > ~/.ssh/id_infra
chmod 600 ~/.ssh/id_infra
ssh-keyscan -p 30022 10.0.1.20 >> ~/.ssh/known_hosts 2>/dev/null
export GIT_SSH_COMMAND="ssh -i ~/.ssh/id_infra -o IdentitiesOnly=yes"
rm -rf /tmp/infra
git clone -b main ssh://git@10.0.1.20:30022/mathias/infra.git /tmp/infra
cd /tmp/infra
DEPLOYMENT="k3s/apps/tapir/deployment.yaml"
# In-place update of the image tag. sed (not yq) so we don't
# depend on additional tooling on the runner — same as cobalt-dingo.
sed -i "s|image: localhost:5000/tapir:.*|image: localhost:5000/tapir:${IMAGE_TAG}|" "$DEPLOYMENT"
# Verify the patch took effect.
grep -q "localhost:5000/tapir:${IMAGE_TAG}" "$DEPLOYMENT" \
|| { echo "✗ image tag patch failed"; exit 1; }
if git diff --quiet "$DEPLOYMENT"; then
echo "ℹ image tag unchanged — skipping push"
else
git -c user.name="tapir CI" \
-c user.email="ci@tapir.local" \
commit -m "chore(deploy): tapir → ${IMAGE_TAG}" "$DEPLOYMENT"
git push origin main
echo "✓ pushed to infra repo"
fi
shred -u ~/.ssh/id_infra
- name: Trigger Flux reconcile (immediate)
run: |
# Without these annotations, Flux would still pick up the change
# within 30s (the apps Kustomization interval). The annotations
# cut latency to ~1s.
kubectl -n flux-system annotate gitrepository flux-system \
reconcile.fluxcd.io/requestedAt="$(date +%s)" --overwrite
kubectl -n flux-system annotate kustomization apps \
reconcile.fluxcd.io/requestedAt="$(date +%s)" --overwrite
- name: Wait for Flux to apply new image
env:
IMAGE_TAG: ${{ needs.build.outputs.image-tag }}
run: |
# Poll the Deployment spec until it reflects the new tag.
# Bound to 60s so a stuck Flux doesn't hang CI.
EXPECTED="localhost:5000/tapir:${IMAGE_TAG}"
for i in $(seq 1 60); do
CURRENT=$(kubectl get deploy tapir -n tapir \
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || echo "")
if [ "$CURRENT" = "$EXPECTED" ]; then
echo "✓ Flux applied new image after ${i}s"
break
fi
sleep 1
done
# Final assertion (in case the loop exited without matching).
kubectl get deploy tapir -n tapir \
-o jsonpath='{.spec.template.spec.containers[0].image}' \
| grep -qx "$EXPECTED" \
|| { echo "✗ Flux did not apply new image within 60s"; exit 1; }
- name: Verify rollout
run: |
kubectl rollout status deployment/tapir \
--namespace tapir \
--timeout=120s \
|| {
echo "── pod status ──"
kubectl get pods -n tapir -o wide
echo "── events ──"
kubectl get events -n tapir --sort-by='.lastTimestamp' | tail -20
echo "── describe ──"
kubectl describe pods -n tapir -l app=tapir | tail -40
exit 1
}
- name: Confirm pod running new image
env:
IMAGE_TAG: ${{ needs.build.outputs.image-tag }}
run: |
kubectl get pods -n tapir \
-l app=tapir \
--field-selector=status.phase=Running \
-o jsonpath='{.items[*].spec.containers[0].image}' \
| grep -q "localhost:5000/tapir:${IMAGE_TAG}" \
&& echo "✓ pod running new image" \
|| { echo "✗ pod image mismatch"; exit 1; }
# ── 4. Mirror to GitHub — skipped for now (SSH key rotation pending) ─
+11
View File
@@ -11,3 +11,14 @@
.env.* .env.*
!.env.example !.env.example
*.local *.local
# Spike media (#28): real recordings, their transcripts and derived analyses are
# private third-party content and this repo is public. Only synthetic fixtures
# are committed — see scripts/spike-media/README.md.
/scripts/spike-media/*.mov
/scripts/spike-media/*.mp4
/scripts/spike-media/*.wav
/scripts/spike-media/*.srt
/scripts/spike-media/*.json
/scripts/spike-media/*.html
!/scripts/spike-media/fixtures/
+18
View File
@@ -0,0 +1,18 @@
{
"mcpServers": {
"brain": {
"type": "http",
"url": "https://brain-mcp.d-ma.be/mcp",
"headers": {
"Authorization": "Bearer ${BRAIN_MCP_TOKEN}"
}
},
"gitea": {
"type": "http",
"url": "https://git-mcp.d-ma.be/mcp",
"headers": {
"Authorization": "Bearer ${GITEA_MCP_TOKEN}"
}
}
}
}
+28 -6
View File
@@ -12,6 +12,7 @@ docs it indexes.
3. `DECISIONS.md` — the ADRs. Decisions are settled here; do not re-litigate without a new ADR. 3. `DECISIONS.md` — the ADRs. Decisions are settled here; do not re-litigate without a new ADR.
4. `docs/architecture/architecture.md`, `docs/data-model.md`, `docs/use-cases/*.feature`. 4. `docs/architecture/architecture.md`, `docs/data-model.md`, `docs/use-cases/*.feature`.
5. `docs/homelab-integration.md` — the concrete endpoints/conventions you'll need. 5. `docs/homelab-integration.md` — the concrete endpoints/conventions you'll need.
6. `LANGUAGE.md` — the project vocabulary. Apply the caveman rubric before destructive operations.
## How to work in this repo ## How to work in this repo
@@ -46,10 +47,14 @@ These caused real mistakes that were caught and corrected; the corrections are l
(See `DECISIONS.md` for full rationale. Listed here so you don't propose them.) (See `DECISIONS.md` for full rationale. Listed here so you don't propose them.)
- **No Supabase** — reuse Dex / ESO+1Password / Postgres (ADR-002). - **No Supabase** — reuse Dex / ESO+1Password / Postgres (ADR-002).
- **No global cross-tenant video/transcript table** — per-user isolation (data-model). Dedup - **No global cross-tenant *video* table** — videos stay per-user (data-model). Transcripts ARE
across users is a Future C concern, not a Stage 0/1 default. shared since ADR-021 (public caption content, keyed by `(provider, provider_video_id)`, non-RLS)
- **No audio-download + speech-to-text in the core path** — captions-first (ADR-007). STT is a so re-analysis never re-fetches; the *videos* half of cross-tenant dedup stays a Future C concern.
deferred, bounded optional component. - **No YouTube audio-download + speech-to-text in the core path** — captions-first (ADR-007).
That rejection stands. ADR-033 superseded the STT deferral **for user-supplied audio only**:
an authenticated user may upload a recording and get a *private* transcript (own RLS table,
never the shared `transcripts`). It is off by default (`TAPIR_UPLOADS_ENABLED`) and does not
open the door to scraping audio.
- **No public SaaS / sign-up / billing / Google OAuth verification at scale** — Future C, - **No public SaaS / sign-up / billing / Google OAuth verification at scale** — Future C,
deferred behind the Stage 0 gate (ADR-008). deferred behind the Stage 0 gate (ADR-008).
@@ -85,7 +90,7 @@ Skills live in the canonical library `mathias/skills` and are wired into this re
## Current build state (start here for the first task) ## Current build state (start here for the first task)
The repo is **green and shipping** — last tag `v0.9.0`. `task check` passes (fmt, vet, lint, The repo is **green and shipping** — last tag `v0.15.0`. `task check` passes (fmt, vet, lint,
`go test -p 1 ./...`). Go is `1.26.1` (see `go.mod`). `go test -p 1 ./...`). Go is `1.26.1` (see `go.mod`).
- Clean Architecture core is implemented: `internal/domain` (entities), `internal/ports` - Clean Architecture core is implemented: `internal/domain` (entities), `internal/ports`
@@ -94,13 +99,30 @@ The repo is **green and shipping** — last tag `v0.9.0`. `task check` passes (f
green against it. green against it.
- Adapters present under `internal/adapters/`: `youtube` (captions-first `VideoSource`, - Adapters present under `internal/adapters/`: `youtube` (captions-first `VideoSource`,
timedtext/InnerTube acquisition per ADR-010), `summarizer` + `llm` (the copied AI router, timedtext/InnerTube acquisition per ADR-010), `summarizer` + `llm` (the copied AI router,
Primary→Fallback per ADR-004), `store` (Postgres, golang-migrate migrations 001–006), now a resilient endpoint chain — local primary → local fallback → external worst-case,
parse-failure-aware, ADR-004 + ADR-022), `store` (Postgres, golang-migrate migrations 001–019),
`secrets` (file-backed `SecretStore`). The brain HTTP sink (ADR-005) is the remaining `secrets` (file-backed `SecretStore`). The brain HTTP sink (ADR-005) is the remaining
optional sink. optional sink.
- Stage 1 is open (ADR-012): multi-user with **DB-enforced** isolation — Postgres RLS `FORCE`d - Stage 1 is open (ADR-012): multi-user with **DB-enforced** isolation — Postgres RLS `FORCE`d
on all user-owned tables (migration 003), two-user isolation test in on all user-owned tables (migration 003), two-user isolation test in
`internal/adapters/store/rls_test.go`. Registration gate, per-user YouTube web connect, and `internal/adapters/store/rls_test.go`. Registration gate, per-user YouTube web connect, and
account management (disconnect / delete, ADR-013) all shipped. account management (disconnect / delete, ADR-013) all shipped.
- Transcript persistence (ADR-021, migration 015): transcripts are a **shared, non-RLS** store
keyed by `(provider, provider_video_id)` — the single exception to the isolation boundary
(`TestTranscriptsTableIsSharedNotRLS`). The engine reads stored transcripts before any caption
fetch (`usecase.resolveTranscript`), so re-analysis — re-summarize, paste-a-URL, onboarding
burst — never re-touches YouTube. Per-user summaries/videos stay RLS-scoped.
- Audio upload (ADR-033, migration 017): `internal/uploads` (validation by **probing**, intake,
one-at-a-time in-process worker), `internal/adapters/audio` (ffprobe/ffmpeg), and
`internal/adapters/transcribe` (gateway client + iguana→berget chain). Transcripts are private
in `uploaded_transcripts`; the audio is never retained; chat reuses `chat.Service`. The
isolation test pins the **exact set** of non-RLS tables (`TestRowSecurityOffTablesAreExactlyTheKnownSet`).
The runtime image does not yet contain ffmpeg — that is a pending decision recorded in ADR-033.
Two-engine reconciliation (ADR-034, migration 019, off unless `TAPIR_RECONCILE_BASE_MODEL`):
`internal/reconcile` (word diff, disputes, a local-model adjudicator) and
`transcribe.Ensemble`; every model must be a local alias, enforced at start-up.
**Lint caveat:** `task check` does not fail on lint (#41: the Taskfile `|| echo` swallows
findings) — run `golangci-lint run ./...` yourself until that is fixed.
- `cmd/tapir` subcommands: `list`, `show`, `auth` (interactive host-side OAuth), `run` (batch - `cmd/tapir` subcommands: `list`, `show`, `auth` (interactive host-side OAuth), `run` (batch
watch→summarize), `serve` (the HTMX+Templ web reader/writer under `internal/web`, a new watch→summarize), `serve` (the HTMX+Templ web reader/writer under `internal/web`, a new
transport over the unchanged engine/ports — ADR-003). `tapir env` prints config. transport over the unchanged engine/ports — ADR-003). `tapir env` prints config.
+915 -2
View File
@@ -123,7 +123,9 @@ secret surface (see VISION Stage 2); it gets the existing, vetted secret path.
## ADR-007 — Captions-first; audio-download + speech-to-text deferred ## ADR-007 — Captions-first; audio-download + speech-to-text deferred
**Status:** Accepted (2026-06-02). Acquisition mechanism superseded by ADR-010 (Data API **Status:** Accepted (2026-06-02). Acquisition mechanism superseded by ADR-010 (Data API
`captions.download` → player/timedtext baseUrl); captions-first stance and STT deferral stand. `captions.download` → player/timedtext baseUrl); captions-first stance and STT deferral stand
**for YouTube**. STT deferral superseded **for user-supplied audio only** by ADR-033; the
YouTube audio-download rejection is unchanged.
**Context.** YouTube's Data API does not expose transcripts. Options: official captions **Context.** YouTube's Data API does not expose transcripts. Options: official captions
(clean, limited coverage) vs. audio download + local Whisper (broad coverage, ToS-grey, (clean, limited coverage) vs. audio download + local Whisper (broad coverage, ToS-grey,
@@ -743,6 +745,911 @@ collapse keys off the same window (`App.RecencyWindow=0` → everything inline).
--- ---
## ADR-021 — Persist transcripts as shared, video-keyed public content (re-analysis never re-fetches)
**Status:** Accepted (2026-06-09). **Reopens the transcripts half of** the "Global cross-tenant
`videos`/`transcripts` table" rejection (data-model.md). **Builds on ADR-007** (captions-first),
**ADR-010/ADR-014** (the per-IP caption rate gate), and **ADR-012** (per-user RLS isolation).
**Context.** Every summarization fetches the transcript fresh through the caption path, even when
the exact same transcript was fetched moments ago — for the same user re-summarizing, or for a
second user who happens to watch the same video. The caption fetch is the one genuinely scarce,
genuinely risky operation in the system: YouTube's timedtext endpoint is unofficial and per-IP
rate-limited (ADR-010), and tripping it risks the maintainer's Google standing (ADR-014). So the
operation we most want to *avoid repeating* is the one we currently repeat unconditionally. A
transcript is **public content** — the same words YouTube serves to anyone — and carries nothing
user-identifying. The per-user isolation that protects summaries, feeds, and tokens (ADR-012) is
the wrong shape for it: it forces a re-fetch per user for data that is identical across users.
The original rejection ("Global cross-tenant `videos`/`transcripts` table") bundled videos and
transcripts together and rejected both on the grounds that "at 1–5 users, re-summarizing is
cheaper than the coupling." That reasoning holds for **videos** (per-user feed rows, genuinely
user-scoped) but not for **transcripts**: the cost being avoided is not LLM re-summarization, it
is a *rate-gated, reputation-risky network fetch*, and that cost is paid per re-fetch regardless
of user count. One re-fetch avoided is strictly worth more than the coupling it removes.
**Decision.**
1. **A single shared `transcripts` table, keyed by the cross-user dedup key
`(provider, provider_video_id)`** — the stable public identity of the video, not Tapir's
internal per-user `videos.id`. Columns: the key, `source` (`captions`/`none`), `language`,
`content`, `fetched_at`. It holds **only public caption content + the video's public id** —
nothing user-identifying — and is therefore **NOT RLS-scoped**: no `user_id`, no policy, no
`FORCE ROW LEVEL SECURITY`. This is the deliberate, single exception to the ADR-012 isolation
boundary, and the only one.
2. **Summarize path becomes read-stored-first.** Have a stored transcript for this video? →
summarize from the stored text, **no caption fetch**. No stored transcript? → fetch *through
the unchanged gate* (ADR-014) → store it → summarize. The gate is neither bypassed nor
weakened; persistence reduces how *often* we reach it, never how *fast*.
3. **De-facto cross-user dedup is the intended behaviour, not a feature with a switch.** Two
users who share a video share the one transcript row. A permanent `source = 'none'` (no
captions) is stored too, so a known-caption-less video is not re-fetched by anyone. A
transient 429 (`SourceRateLimited`) is **never** stored as terminal — it stays a per-user
retry via the existing `transcript_status` backoff (ADR-014), so persistence cannot mask a
rate-limit into a false "no transcript."
4. **Per-user `summaries` stay RLS-scoped (ADR-012 unchanged)** and reference the transcript by
video id. Videos stay per-user. Only transcripts go shared.
**Consequences.** Re-analysis (re-summarize, different model, paste of an already-seen video,
onboarding of a second user with overlapping subscriptions) never re-touches YouTube — the
primary win, and it *reduces* aggregate caption-gate pressure, reinforcing ADR-010/ADR-014 rather
than straining them. The isolation surface gains exactly one non-RLS table; an isolation test
asserts the boundary is *exactly* there and has not leaked to any user-owned table (this is the
proof the public-content classification was implemented as designed). It also unblocks
multi-model / customizable analysis (re-run analysis on stored text for free) — enabling that is
this ADR's point; building it is separate.
**Reversibility.** The read-stored-first check is the only behavioural coupling; removing it
restores fetch-every-time. The down-migration recreates the per-user RLS-scoped transcripts shape
(001/003). No user-facing surface depends on cross-user sharing — sharing is the *storage shape*,
never exposed in the UI.
---
## ADR-022 — Summarizer is a resilient endpoint chain, not a single model
**Status:** Accepted (2026-06-10). **Extends ADR-004** (the copied `llm` Primary→Fallback
routing). Triggered by the first friendly-pilot live run, where a connected user got **zero**
summaries after 12h.
**Context.** Stage-0 ran a single summarizer model (`koala/phi4-mini`) with no fallback wired
(`summarizer.New(primary, nil)`). The live run exposed three independent failure modes, each of
which silently produced no summary:
1. **Context overflow.** `phi4-mini` has an 8k context. Real transcripts (one was 11,602 tokens)
exceed it and the gateway returns HTTP 400 — and the request also sent `max_tokens=8192`, so
even a short transcript plus the completion budget could overflow the window.
2. **Malformed model output.** `phi4-mini` intermittently emits `highlights` as a bare string
instead of an array, producing `cannot unmarshal string into []string`. The old code returned
the parse error **without** trying any other model — a 200-with-bad-JSON short-circuited.
3. **No fallback existed at all** — any primary failure was terminal for that video.
`phi4-mini` is kept as primary deliberately: it is fast and, on transcripts that fit, correct.
The fix is resilience around it, not replacing it.
**Decision.**
1. **Ordered endpoint chain (`summarizer.NewChain`).** Endpoints are tried in order; the first to
return a *parseable* summary wins. Default chain:
`koala/phi4-mini` (primary, local) → `iguana/gemma4-26b` (fallback, local on a
*different host*) → `berget/mistral-small` (worst-case, external). All three are reached through
the **one** LiteLLM gateway by alias — the gateway already fronts both llama-swap and berget — so
a fallback is a different alias, not a second client config.
**Update 2026-06-11:** the local fallback moved from `koala/phi4-14b` to `iguana/gemma4-26b`.
koala now carries other GPU loads, so keeping the fallback on koala competed with them; iguana
(M2 Ultra) has the headroom, and a different host is also a different egress IP for the rare
fallback fetch. `gemma4-26b` is the brain-validated homelab general-purpose model (agentsquad
H2/H3 executor) and returned valid summary JSON on the real prompt in a smoke test
(~37s incl. cold-load — fine for a path hit only when the fast primary fails). Pure config:
`TAPIR_FALLBACK_MODEL`.
2. **A parse failure advances the chain, same as a transport error.** "Reliably summarized" means
*parseable summary returned*, not *HTTP 200*. This is the behaviour the old Primary→Fallback
shape missed.
3. **Tolerant parse.** `highlights`/`takeaways` coerce from a bare string (or a mixed scalar
array) to `[]string`, so the most common small-model quirk is absorbed **without** spending a
fallback round-trip — keeping the fast path fast.
4. **Transcript truncation (`TAPIR_MAX_TRANSCRIPT_CHARS`, default 18000).** Input is bounded
up-front to fit a small-context primary, so overflow is prevented rather than recovered-from.
5. **Bounded completion budget (`TAPIR_SUMMARY_MAX_TOKENS`, default 1500).** A summary needs few
hundred tokens; the old 8192 budget itself contributed to 8k-window overflow.
**Local-first guarantee preserved.** The chain ordering *is* the guarantee: locals are tried
first, so content reaches the external endpoint only after every local endpoint has failed.
`TAPIR_CLOUD_FALLBACK_MODEL=""` removes the external endpoint entirely — the lever a
**client/NDA deployment** pulls so content never leaves the local stack. With no external endpoint
configured the `ai_routing.feature` "content only local" scenarios hold unchanged.
**Reversibility.** Pure wiring + config. Setting `TAPIR_FALLBACK_MODEL` and
`TAPIR_CLOUD_FALLBACK_MODEL` empty collapses the chain back to single-primary behaviour; the
tolerant parse and truncation are strict supersets of the old behaviour (a previously-parseable
reply still parses; a transcript within budget is unchanged).
---
## ADR-023 — Drop Shorts/livestreams at discovery to protect the caption budget
**Status:** Accepted (2026-06-10). **Builds on ADR-014** (per-IP caption rate limit is the
binding constraint) and the ADR-022 live-run findings.
**Context.** The scarce resource is the unofficial timedtext caption fetch (per-egress-IP
429, ~3 successful/pass). The first multi-user run showed the candidate set was mostly noise —
Shorts, sub-minute clips, and live broadcasts — each of which still consumes a caption-fetch
attempt (and a "none" result is a *completed* fetch, so it costs budget even when it yields
nothing). Spending the rate-limited budget on content the user will not read is the waste to
cut first; it is cheaper and lower-risk than raising the ceiling (multi-IP, Whisper).
Duration and live status are NOT in the playlistItems discovery response, but they ARE in the
Data API `videos.list` (contentDetails.duration + snippet.liveBroadcastContent) — the official
**quota-based** API (1 unit/call, 50 ids/call), which is a *different* limit from the timedtext
429. So one cheap quota call buys a filter that saves many expensive throttled fetches.
**Decision.**
1. `NewVideos` enriches its candidates with a single `videos.list` call and drops, before
returning: videos shorter than `TAPIR_MIN_VIDEO_SECONDS` (default 60) and any `live`/
`upcoming` broadcast. Dropped videos are never persisted, so they also declutter the list.
2. The filter is **degrade-open**: `MinVideoSeconds=0` disables it (no quota call), and a
`videos.list` error returns the candidates unfiltered — discovery must never break because a
metadata call hiccuped (worst case = pre-ADR-023 behaviour).
3. The paste-a-URL path (`VideoByID`) is **not** filtered — an explicit user request for a
specific video (even a Short) is honoured.
**Reversibility.** Pure discovery-time filter + config. `TAPIR_MIN_VIDEO_SECONDS=0` restores
the old behaviour. No schema change, no effect on already-stored videos.
**Quota note.** Per-channel enrichment adds ~1 unit/channel/pass. At pilot scale (≤3 users)
this is well under the 10k/day cap; at larger scale, batch `videos.list` across channels
(50 ids/call) by collecting all discovered ids per pass before enriching.
---
## ADR-024 — Per-channel caption-availability memory
**Status:** Accepted (2026-06-10). **Builds on ADR-014** (per-IP caption budget), **ADR-021**
(shared transcript cache), **ADR-023** (Shorts filter).
**Context.** After ADR-021 caches transcripts and ADR-023 drops Shorts, the remaining caption
waste is the *first* fetch on every new video of a channel that never publishes English captions
(foreign-language news, music, etc.). Each costs one rate-limited fetch to resolve to "none" —
and on a throttled IP that fetch may 429 and churn the backoff machinery before it ever gets a
verdict. A pilot user's feed had several such channels.
**Decision.** Remember, per `(user, channel)`, a streak of consecutive no-caption outcomes
(`channel_caption_state`, migration 016, RLS-scoped like the rest of the user-owned schema).
Once the streak reaches `TAPIR_CHANNEL_CAPTIONLESS_THRESHOLD` (default 5) the channel is
suppressed — its videos are discovered/listed but not caption-fetched — for
`TAPIR_CHANNEL_CAPTIONLESS_WINDOW` (default 14d), after which one video is re-probed
(auto-recovery for a channel that starts adding captions). A successful fetch resets the streak;
a fresh 429 does NOT count (transient, not a caption verdict). An explicit manual request
bypasses suppression. `threshold = 0` disables the feature.
**Why per-user, not global.** Caption availability is really a channel property (public), so a
global table would let users share the learning. But subscriptions are per-user (ADR-012) and at
pilot scale users' channel sets barely overlap, so per-user + RLS keeps it consistent with the
existing isolation model with no new non-RLS exception to justify. Promoting to a shared table
(like transcripts, ADR-021) is a future optimisation if channel overlap grows.
**Reversibility.** Migration 016 is a clean drop; `threshold = 0` disables at runtime. The
memory only ever *suppresses fetches* — it never deletes content or affects already-stored
summaries.
---
## ADR-025 — Honest, state-aware foreground summarization status
**Status:** Accepted (2026-06-10). **Pillar B of the manual-mode UX work** (Pillar A, foreground
fetch priority, is a separate follow-up). Builds on ADR-014 (the rate limit the UX must make
legible).
**Context.** Clicking "Summarize" spawned a background goroutine and polled `/status`, which
returned only two states: the spinner (in-flight) or the normal card (done). But the web
`ProcessVideo` only recorded an outcome on *success* — a 429'd or caption-less click left
`transcript_status` unset, so the next poll silently reverted to the "Summarize" button. The
user saw either an endless spinner or a button that did nothing useful when clicked again. The
binding constraint (YouTube's caption rate limit) was completely invisible.
**Decision.**
1. **Record every outcome on the web path**, mirroring the runner: `ProcessVideo` stamps
`rate_limited` / `none` / `fetched`. A rate-limited video keeps its requested flag so the
background sweep retries it; `none` and `fetched` are terminal.
2. **`/status` is state-aware**: summarized → summary card; in-flight → working spinner;
`rate_limited` → a calm "waiting, will retry" card that keeps polling (every 30s) so the
summary appears on its own when the retry lands — the user never clicks again;
`none` → a terminal "no captions" card with no poll and no dead-end button.
3. **Charm status text** (Claude-Code / Crush inspired): the working spinner cycles playful,
tapir-themed gerunds ("Chewing the cud…", "Munching leaves…", "Distilling the gist…") via
CSS only — no JS, keeping the HTMX/no-JS ethos. Decorative (aria-hidden) with a stable
`role=status` line for assistive tech.
**Principle.** When the system cannot be fast (throttled IP), it is at least honest, and it
self-resolves without making the user retry. Honesty is the load-bearing half — Pillar A's
priority lane only improves the odds of a fast slot; it cannot beat an already-hot IP.
**Reversibility.** Pure transport-layer + view change over the unchanged engine/ports. No
schema change (reuses `transcript_status` from migration 007).
---
## ADR-026 — Foreground caption fetches take priority; the credentials probe is dead
**Status:** Accepted (2026-06-10). **Pillar A of the manual-mode UX work** (Pillar B was
ADR-025). Builds on ADR-014 (the shared per-IP gate).
**Context.** Every caption fetch — the background sweep and the web click-path — shared one
process-wide rate gate equally. So a user waiting on a "Summarize" click competed with the
firehose for both pacing and the scarce pre-429 window; on a busy IP the click was slow or
429'd while the background churned.
**Decision.** A context-marked priority lane. The web path
(`engineProcessor.ProcessVideo`) wraps its context with `ForegroundContext`; the gate gives
foreground fetches a token immediately, while **background fetches yield** — they wait until no
foreground fetch is pending before taking a token. Threaded via a context value (not new
signatures) and a process-wide `foregroundPending` counter. Clicks are rare and bursty, so the
background barely loses throughput; the waiting human gets the next (and cleanest) slot.
**Credentials probe — rejected, not built.** The idea was to fetch captions with the user's
auth in manual mode to dodge 429s. It is a dead end, already settled by ADR-010 and the code:
the caption path is *deliberately anonymous* because the InnerTube/timedtext endpoints **reject
or break on authenticated requests** (`captions.go`: "no OAuth token — it can break the
timedtext endpoint"). The user's OAuth (a Data API credential) does not authenticate InnerTube
at all, and the official `captions.download` is owner-only (403 on third-party). So auth cannot
help here and can actively hurt. No probe needed — building one would only re-confirm the ADR.
**Reversibility.** Context-marker + a yield loop in the gate; removing the marker collapses to
the prior equal-share behaviour. No schema or API change.
---
## ADR-027 — Chat with a video's stored transcript (deeper-dive, on an already-summarized video)
**Status:** Accepted (2026-06-11). **Consumes ADR-021** (the shared, video-keyed transcript
store) for the first time beyond summarization; **uses the ADR-022 chain models**; relates to
ADR-012 (isolation) and ADR-016 (the Stage-0 gate).
**Context — observed demand, not hypothetical.** The maintainer read 10+ real pilot summaries
and reported the reactions: *many good; some he wanted to dig deeper into; some less useful*
(the "less useful" split between weak-*model* output and uninteresting-*video* content). The
middle reaction is the signal: a good summary that makes the reader want *more* is the summary
succeeding at triage and then hitting a wall — there is nowhere to go deeper short of watching
the video. That want is the feature. It is also the cheapest possible feature to satisfy
honestly, because ADR-021 already persists the transcript: the deeper-dive runs entirely on
stored public-content text + local models, touching **no** caption fetch and **no** YouTube.
**Decision.** Add a per-video chat that lets the user ask questions against a video's stored
transcript.
1. **Entry from the summary view only.** A "dig deeper / ask" affordance on a summarized video —
the chat lives exactly where the "I want more" reaction happens. No standalone chat surface.
2. **Stored-transcript-only (load-bearing constraint).** Chat is available **only** for videos
that already have a stored transcript. It never triggers a caption fetch, so it cannot touch
the rate gate, the 429 surface, or YouTube at all — the entire account-safety constraint that
governs the rest of Tapir is satisfied *by construction* here, not by careful gating. (Entry
being "from a summary" guarantees the transcript exists.) On-demand fetch for un-stored videos
is explicitly deferred.
3. **Model = the summary's model by default; user-switchable among the ADR-022 chain models**
(`phi4-mini` / `gemma4-26b` / `mistral-small` to start). This is deliberate: it doubles as
live model-comparison instrumentation — ask the same question of the same transcript under two
models and the difference is directly felt. This is the mechanism by which the maintainer
learns *which* model is worth defaulting to, and it is the multi-model-analysis direction
ADR-021 anticipated, arriving as a user-facing capability.
- Chat is a **read-bounded retrieval/QA task** (the user supplies the focus), which is
*easier* than summarization (the model must decide what matters). So a model that summarizes
mediocrely may chat well — chat is plausibly a partial remedy for the weak-summary case, not
an inheritor of it.
4. **Ephemeral chat (v1).** No persisted history; chat is per-session. Persisting per-user,
RLS-scoped history is deferred until there is evidence anyone wants to revisit a conversation.
5. **Chat-only, trust-the-model (v1) — with a recorded limitation.** The chat does not expose the
raw transcript for verification in v1 (kept simple). **Known limitation:** because some
summaries were weak-model output, the user has reason not to fully trust a chat answer's
fidelity to the transcript, and v1 gives no in-UI way to check. The model-switcher partially
compensates (two models disagreeing on the same question is itself a signal). A
"show source / view transcript" verification path is the natural **v2** and is *not*
foreclosed — ADR-021's stored transcript already makes it cheap. Recorded so v2 is a known
next step, not a rediscovery.
**Why this is safe and in-scope.** It adds no caption-fetch surface (stored-only), no new
non-RLS table (transcripts already shared per ADR-021; ephemeral chat stores nothing), and no
auth change. It is additive to the read path. The one genuine product expansion — Tapir becomes
an interactive transcript-QA tool, not only a summarizer — is justified by *observed* demand from
real reading, which is exactly the kind of evidence the Stage-0 discipline asks for before
building.
**Relation to the Stage-0 gate.** This is **not** a return-nudge (those stay deferred, ADR-020) —
it adds nothing that prompts the user to return; it deepens the value *once they are already
reading*. It does not contaminate the unprompted-return signal. If anything it strengthens the
"useful to me" case the gate measures, by giving a good summary somewhere to lead.
**Reversibility.** Additive read-path feature over the unchanged engine + the ADR-021 store.
Removing the summary-view affordance removes the feature; nothing else depends on it. Ephemeral =
no migration, no stored state to unwind. Spec: `docs/specs/chat-with-transcript.md`.
---
## ADR-028 — Onboarding burst: pick likely-good videos, summarize them with a stronger model
**Status:** Accepted (2026-06-11). **Refines ADR-018** (the connect-time burst) and **ADR-020**
(recency-bounded auto-summarize). **Builds on ADR-022** (the endpoint chain), **ADR-023**
(the discovery-time `videos.list` enrichment), and **ADR-021** (the shared transcript cache).
Triggered by a Phase-1 investigation of the live pilot DB.
**Context.** A new user's first session decides whether they return (the Stage-0 gate, ADR-016).
The connect-time burst (ADR-018: summarize ≤`TAPIR_ONBOARD_SUMMARIZE_COUNT` newest videos so the
feed isn't empty) *fires* in production, but a live-DB investigation of the second pilot user
("Jonte") found it delivers a weak first impression for two reasons, and ruled out a third idea:
1. **Junk picks.** Selection was pure newest-first (`NewestUnsummarizedVideoIDs`,
`ORDER BY published_at DESC`) with **no quality signal**. Jonte's live burst-3 were a
stock-ticker **livestream** + two regional news clips — newest, not best. The cheap signals
that *could* gate this (duration, live status) are fetched by ADR-023's `videos.list`
enrichment at discovery and then **thrown away**: the `videos.duration_s` column (migration
001) was never written.
2. **Weakest model on the first impression.** All burst summaries ran on `koala/phi4-mini` — the
documented weak link (ADR-022 was born from its failures). The stronger, brain-validated
`iguana/gemma4-26b` was never used, even though the burst is only ~3 summaries.
3. **Cached-first instant summaries — REJECTED.** The idea: skip the fetch, summarize
already-cached transcripts (ADR-021) instantly. The pilot numbers kill it — only **11 videos**
overlap between the two users (~3% of each ~350–400-video library), **0** cached-and-
unsummarized, and a new user's newest-20 unsummarized are **20/20 NOT cached**. Newest-first
and cached-first are structurally incompatible: fresh uploads are exactly what nobody has
fetched. An empty lever at pilot scale.
**Decision.**
1. **Persist `duration_s` at discovery.** `filterLowValue` (ADR-023) already has each candidate's
duration in hand; carry it onto the kept `domain.Video` and have `UpsertVideo` write it,
COALESCE-preserving a known value (the channel-title backfill stance, migration 014). No new
migration — the column exists. The connect-triggered discovery pass runs *before* the burst,
so a fresh user's candidates are enriched in time.
2. **Junk-avoiding selection.** A new `OnboardBurstVideoIDs(userID, limit, minSeconds, maxSeconds)`
keeps the newest-first order but drops a video when its duration is *known* and outside
`[minSeconds, maxSeconds]` — `minSeconds` = `TAPIR_MIN_VIDEO_SECONDS` (60, the Shorts floor),
`maxSeconds` = new `TAPIR_ONBOARD_MAX_VIDEO_SECONDS` (default 14400 = 4h, to drop multi-hour
livestream VODs that pass the live filter once ended). A NULL duration is **unknown** — kept
(degrade-open) but ranked after known-good rows. **has-captions stays un-gateable pre-fetch**
(only knowable after a gate fetch or a ~0-probability cache hit); selection only *avoids
known-junk*, it does not *promise* captions.
3. **Stronger model for the burst only.** `TAPIR_ONBOARD_SUMMARIZER_MODEL` (default
`iguana/gemma4-26b`) leads a burst-specific summarizer chain (onboard model first, then the
standard ADR-022 chain as resilience, deduped), wrapped in a burst-specific processor over the
*same* store/cache/sink — a pure wiring choice; the engine and ports are unchanged (ADR-003).
Empty or equal-to-primary collapses the burst back onto the shared processor.
**Not a throughput change.** The caption rate gate (ADR-014) and the foreground priority lane
(ADR-026) are untouched — same pacing, same cap. This changes *which* ≤3 videos the burst spends
its fetches on and *which model* summarizes them, never how fast or how many. The engine's
existing read-stored-first (ADR-021) is unchanged and still yields a free instant summary on the
rare cache hit — we simply do not *select* for cache hits.
**Consequences.** Better odds of a strong first session: the burst avoids the obvious junk and
runs the better model on the one impression that decides return. The selection improvement is
forward-looking — existing rows have NULL `duration_s` until their next discovery pass backfills
it (lazy, like channel_title); a brand-new user benefits immediately because connect-discovery
runs first. `duration_s` becoming live also unblocks future length-aware features (feed sorting,
"long read" badges) for free.
**Reversibility.** Pure config + wiring + one column write + one query, no migration.
`TAPIR_ONBOARD_MAX_VIDEO_SECONDS=0` (and `TAPIR_MIN_VIDEO_SECONDS=0`) restores pure newest-first;
`TAPIR_ONBOARD_SUMMARIZER_MODEL=""` collapses the burst back to the shared processor.
Spec: `docs/specs/onboarding-wow-burst.md`.
---
## ADR-029 — Stateless session cookie (survives restarts, browser-close, idle)
**Status:** Accepted (2026-06-11). Triggered by pilot feedback: "lots of clicking to log in again
on iPhone." Supersedes the in-memory session store in the ADR-011 login.
**Context.** Three compounding causes made users re-login constantly:
1. **In-memory session store** (`sessionStore` map) — wiped on every pod restart, so each deploy
logged everyone out. During the active build period that was ~15 logouts.
2. **1-hour session TTL** — for a "check back tomorrow" reader, idle > 1h forced a re-login on
nearly every visit.
3. **No cookie Max-Age** — a session cookie (deleted on browser/app close); iPhone Safari closing
the tab dropped it.
Each re-login is the full Dex/Authentik redirect dance — many taps on mobile.
**Decision.** Make the session **stateless**: the identity (subject + email) and an absolute
expiry live INSIDE the existing HMAC-signed (HS256) cookie — no server-side table. Plus:
- **30-day sliding TTL** (was 1h), re-signed on each request so an active user never lapses.
- **Persistent cookie** (`Max-Age` set) so it survives browser/app close.
The cookie is HttpOnly + Secure + SameSite=Lax; the HMAC (keyed by the stable ESO
`tapir-session-secret`, which does NOT rotate per deploy) makes it tamper-proof. The payload is
identity, not secrets — the OIDC access/ID tokens are still discarded after callback.
**Consequences.** A deploy/restart no longer logs anyone out (proven by a test: a cookie issued by
one instance is accepted by a fresh instance with the same secret); works across replicas for
free. **Trade:** no server-side revocation — `logout` clears the cookie client-side, but a copied
cookie stays valid until expiry. Accepted for the Stage-0 reader pilot; revisit (server-side
revocation list, or shorter TTL + refresh) if it ever holds sensitive actions. Rotating
`tapir-session-secret` invalidates all sessions — the global logout lever.
**Not addressed here:** the tap-count of the IdP login page itself is Authentik's UX; with
re-login now rare (30-day idle or explicit logout), it matters far less.
---
## ADR-030 — Observability: slog timing + Prometheus metrics (AI-focused)
**Status:** Proposed (2026-06-11). Issue #15. **Draft for review — no code yet.**
**Context / requirements.** Nothing measures the activities that drive Tapir's performance and
UX, and the Stage-0 eval gate (ADR-016) needs a *performance* dimension to sit beside the
return-usage one. We need timing for: caption fetches (the scarce op), summarization (which model
won, how long, fallbacks), Q&A latency, LLM token spend, and basic session/usage (request rate,
latency by route, logins). Requirements:
- R1: structured `slog` timing at each AI call site (human-readable, already the logging stack).
- R2: Prometheus metrics for the same, scrapeable by the cluster's prometheus-operator.
- R3: **AI metrics are the priority** — summarize latency by `model`/`outcome`/`fallback`,
caption-fetch latency by `outcome`, chat latency by `model`, and LLM `tokens` by model+kind.
- R4: HTTP/session metrics via middleware — request count + latency by route, logins.
- R5: bounded label cardinality (no per-user, no raw-path labels).
- R6: `/metrics` must NOT be publicly exposed.
**Decision / architecture.**
1. **New package `internal/metrics`** owns all Prometheus collectors + a typed API
(`ObserveSummarize`, `ObserveCaptionFetch`, `ObserveChat`, `RecordTokens`, `IncLogin`,
`HTTPMiddleware`, `Handler`). Adapters call this API; they never import prometheus types.
2. **New dependency `github.com/prometheus/client_golang`.** Justification: it is *the* standard
Go Prometheus client and the cluster already runs prometheus-operator; hand-rolling exposition
is not worth it. (Needs the dep-justification note in the commit per repo rules.)
3. **The copied `llm` package stays stdlib-only (ADR-004).** It must not import `internal/metrics`.
Token usage is surfaced via an **optional callback** `llm.WithUsageHook(func(model string, prompt, completion int))`
set at wiring time (`buildSummarizer`/`buildChat`) to `metrics.RecordTokens`; `llm.Client` only
gains parsing of the response `usage` block. Our own adapters (`summarizer`, `youtube`, `chat`)
may import `internal/metrics` directly.
4. **HTTP middleware** reads `r.Pattern` AFTER routing (Go 1.22 sets it during ServeMux match), so
the `route` label is the bounded registered pattern (`GET /v/{videoId}`), satisfying R5;
unmatched → `other`.
5. **Dedicated metrics port** (`TAPIR_METRICS_ADDR`, default `:9090`) served by a second
`http.Server` in `cmdServe`; `/metrics` is never on the public app mux (R6). A **PodMonitor**
in `mathias/infra` scrapes it; the deployment exposes the port.
6. **slog** elapsed fields are emitted alongside each metric at the call sites (R1).
**Hook points (where the instrumentation lands).**
- `summarizer.Summarize` — per-endpoint timing + outcome (`success`/`parse_error`/`error`) + fallback flag.
- `youtube.FetchTranscript` — fetch timing + outcome from `domain.Transcript.Source`.
- `chat.Service` answer — timing by model.
- `llm.Client.Complete` — parse `usage`, fire the usage hook.
- `oidc.handleCallback` — `IncLogin`.
- `cmdServe` — wrap `Router()` in `metrics.HTTPMiddleware`; start the metrics server.
**Out of scope / later.** Persisting per-summary latency into Postgres for `tapir report`
(derive UX latency — publish/discovery → summary — from existing timestamps first; only persist
op-latency if the scrape proves insufficient). SPA view (#16) and visual refresh (#17).
**Reversibility.** Additive: a new package + middleware + a metrics port. Removing the PodMonitor
stops scraping; the app is unaffected. No schema change.
**Next steps (gated):** on approval of this ADR → BDD scenarios (`docs/use-cases/observability.feature`
+ scenario-coverage map) → TDD → implement → SemVer + docs + PodMonitor.
---
## ADR-031 — SPA-like reader: inline-expand summary + Q&A in the list (HTMX, no framework)
**Status:** Proposed (2026-06-12). Issue #16. **Draft for review — no code yet.**
**Context / requirements.** The reader is multi-page: a list of compact cards (`/`), then a
navigation to a separate detail page (`/v/{id}`) for the full summary + the docked chat (ADR-027).
It feels less fluid than a single integrated view. We want the full summary AND the per-video
Q&A to open **in place in the list**, no page hop. Requirements:
- R1: clicking a summarized card expands it in place to the full summary (summary/highlights/
takeaways) + the chat dock; a collapse returns it to the compact card.
- R2: **no SPA framework** — stay HTMX + Templ (ADR-003); reuse existing fragments, not a rewrite.
- R3: **progressive enhancement** — with JS off, the card link still navigates to `/v/{id}`
(the detail page stays as the no-JS + deep-link surface). Nothing becomes JS-only.
- R4: only **summarized** cards expand; pending/rate-limited/no-caption cards keep their current
footer behaviour (Summarize button, waiting/none states).
- R5: chat inside an expanded card works exactly as on the detail page (reuse `chatReveal`/
`chatSection` + the existing `/v/{id}/chat` endpoints, unchanged).
**Decision / architecture.**
1. **Reuse the existing fragments.** `summaryBody(r)` and `chatReveal(videoID)` already exist and
render the detail page; a new `expandedCard(r, chatEnabled)` composes the compact header + a
collapse control + `summaryBody` + `chatReveal`. `DetailPage` is refactored to also compose
`summaryBody` so the two never drift (DRY).
2. **Two fragment endpoints** (mirroring the existing list/status HTMX fragment pattern):
`GET /v/{videoId}/expand` → `expandedCard`; collapse reuses the existing compact `VideoCard`
via `GET /v/{videoId}/card`. Both are list-card `<li>` fragments with the SAME `id`
(`video-{id}`), swapped `outerHTML` — same mechanism as `processingCard`/`VideoCard` today.
3. **The compact card's title/"Read" affordance** becomes `hx-get=/v/{id}/expand`,
`hx-target=#video-{id}`, `hx-swap=outerHTML`, with `href=/v/{id}` as the no-JS fallback (R3).
The expanded card's collapse control is the inverse (`hx-get=/v/{id}/card`).
4. **Only when `r.Summarized`** does the expand affordance render (R4); the other states are
unchanged.
5. **v1 does NOT push the URL** (`hx-push-url`) — expand/collapse is ephemeral list UI state; the
detail page remains the deep-link/shareable URL. Deep-linking the open state via `hx-push-url`
is noted as a later option (needs list-state restore on back).
**Out of scope / later.** URL push / deep-linkable open state; the visual refresh (#17) — though
the expanded-card markup is where #17's TUI/charm styling will land, so they pair.
**Reversibility.** Additive: two fragment endpoints + one templ + an affordance swap on the
compact card. Removing the affordance reverts to plain list→detail navigation; the detail page is
untouched. No schema change.
**Next steps (gated):** on approval → BDD (`docs/use-cases/inline_expand.feature` + coverage
map) → TDD → implement → SemVer + docs.
---
## ADR-032 — Visual refresh: one charm-reader layout, light + dark themes
**Status:** Proposed (2026-06-12). Issue #17. **Draft for review — no code yet.** Follows the
sketch-first explore step (3 throwaway mockups in `docs/sketches/`, screenshotted for review).
**Context / decision.** The UI is flat. From the mockups, directions **B (light reader + charm)**
and **C (dark cozy terminal)** are the SAME layout — readable sans body, monospace meta, charm
palette accents, lipgloss-style bordered cards — in two palettes. Direction A (full-monospace TUI)
is dropped as too heavy to read long summaries. Decision: ship that one layout with **both a light
theme (B) and a dark theme (C)**, user-toggleable, defaulting to the OS preference.
**Requirements.**
- R1: one set of markup/structure; the two themes are pure palette (CSS variables), no duplicate templates.
- R2: a **theme toggle** persisted across visits; default to `prefers-color-scheme` when no choice stored.
- R3: charm language in both — mint/purple/pink accents, mono meta + section labels, lipgloss
bordered/gradient cards, the (fixed) ASCII tapir; readable sans body.
- R4: style the existing pieces — list, compact card, **expanded card incl. the already-present
video embed (ADR-031/summaryBody)**, detail page, chat dock, the queue note, the charm spinner.
- R5: WCAG-AA contrast for body text in BOTH themes; keep `prefers-reduced-motion` (already honoured).
- R6: stay HTMX+Templ; no CSS framework.
**Architecture.**
1. **Palette as CSS variables.** `:root` holds the light (B) tokens; `:root[data-theme="dark"]`
holds the dark (C) tokens; a `prefers-color-scheme: dark` media block sets the dark tokens when
no explicit `data-theme` is set. All component CSS references variables only (R1). The existing
`CharmMint/Purple/Pink/Cream/Dim` Go consts remain the source for the spinner's inline colours.
2. **Theme toggle** = a small inline script (a dozen lines, no framework) in `Layout`: on load,
apply stored theme (localStorage) or fall through to the media query; a header toggle button
flips `data-theme` on `<html>` and stores it. This is the one new bit of JS; everything else
stays server-rendered + HTMX. (Considered: cookie + server-render — rejected, a full round-trip
per toggle is clunky for a pure presentation flip.)
3. **Scope = the `styleTag` CSS in `view.go`** (the single style source) plus tiny class hooks in
the templ where needed; content/structure are unchanged, so existing view tests keep passing.
**Verification.** The deployed UI is behind auth (web-shot can't log in), so visual review is via
the mockups now + a styled full-set mockup screenshot before merge, then a live eyeball on device.
Automated tests stay structural/behavioural (theme tokens present, toggle persists, expanded card
embeds the video, no-JS still renders a readable default) — colours are not unit-tested.
**Reversibility.** A CSS theme swap + one small script + a few class hooks; revert `styleTag` to
roll back. No schema, no structural change.
**Next steps (gated):** on approval → BDD (`docs/use-cases/visual_theme.feature` + coverage map)
→ TDD → implement → SemVer + docs.
---
## ADR-033 — Upload audio, get a private Swedish transcript, discuss it (user-supplied STT)
**Status:** Proposed (2026-10-01). **Decider: Mathias — pending.** Issue #33 (audio-only subset of
#28). **Supersedes ADR-007's speech-to-text deferral for user-supplied audio only.** The
YouTube audio-download rejection in ADR-007 **stands untouched**. Builds on ADR-012 (isolation),
ADR-018 (single replica), ADR-021 (**the constraint**), ADR-022 (endpoint chain), ADR-025
(honest status), ADR-027 (chat). ADR-014 item 4 ("Whisper reconsideration is a future ADR,
gated on caption data") is a different question — whether to transcribe YouTube videos with no
captions — and this ADR does not answer it. Measurements: `docs/research/audio-length-ceiling-2026-10-01.md`.
**Context.** The maintainer has a concrete need: a Swedish Teams voice-only export, 30–90
minutes, to upload and then *discuss* (what was decided, what is open, who owes what). Every
piece exists — a Whisper endpoint on the gateway (infra#274), chat-with-transcript (ADR-027) —
and nothing connects them.
ADR-007 deferred STT for three reasons: yt-dlp fragility, ToS-grey scraping, and GPU contention
with the JEPA PoC. Upload-STT shares **none** of them: the user owns the file, nothing is
scraped, there is no caption rate gate, and the contention moved from koala to iguana. What it
does share is landing Whisper in the codebase, which ADR-007 deferred — so a new ADR is required,
and ADR-017's process note records the one time a guardrail-reversing change shipped without one.
**The constraint that shapes everything.** ADR-021 made `transcripts` a shared, **non-RLS**
table on the explicit grounds that it holds only public caption content and nothing
user-identifying. A recorded internal meeting is the opposite: private third-party content owned
by one user. The existing isolation test asserts `transcripts` *is* non-RLS, so a naive reuse
would leak across tenants **and pass CI green**.
**What was measured, not argued** (research doc above):
- **No transcription ceiling up to 90 minutes.** One 174 MB request through the LiteLLM NodePort
took 440 s (~12× realtime). The first failure is the public ingress `llm-api.d-ma.be`
returning `413` above roughly 8–10 MB (~5 min of WAV) — not whisper, and not a Metal OOM.
The 8:20 failure recorded in #33 is consistent with that cap.
- **`response_format=srt` survives LiteLLM, wrapped** in a JSON envelope (`{"text": "<srt>"}`).
- **The two backends differ.** `iguana/kb-whisper` returns whole-second segments;
`berget/kb-whisper` returns sub-second times and a different `language` token.
- **The existing chat path truncates at 18,000 chars (~22 min of speech), keeping the head.**
#33 assumed chat "retrieves rather than generates" and so has no length ceiling. It does
not retrieve; it stuffs the transcript into the prompt and cuts the tail. Measured against
the offered models: `phi4-mini` 400s at ~13k tokens; `gemma4-26b` is right to 90,000 chars and
silently drops the head at 120,000.
**Unknowns, stated.** Real-speech accuracy; compressed-input behaviour; concurrent long jobs on
iguana; and the Metal OOM, deliberately not provoked. (Whether the offered chat models can hold
an hour of Swedish *was* an unknown here; it is now measured, see decision 8.)
**Decision.**
1. **Audio is what is used; the transcript is the only thing kept.** An authenticated user
uploads a recording; Tapir transcribes its audio and stores the transcript. The audio itself
is deleted after transcription (temp file, removed in a `defer`, swept at start-up) — no
custody problem to solve, and nothing to hand to #31. **A file that also carries a video
track is accepted: its audio is extracted, the video is discarded, and the user is told so**
(see 4). *Amended 2026-10-01, the same day this ADR was drafted: the first draft rejected
video and named #28. The maintainer's first real recording — a Teams meeting with the camera
off — is still an `.mp4` with a video track, so rejecting video rejected the primary use
case.* This keeps no video and analyses none: playback, keyframes and what-is-shown analysis
remain #28. YouTube audio-download stays rejected. No public/anonymous upload (ADR-008).
2. **Private store: a separate RLS-scoped table, `uploaded_transcripts`** (resolves #28's
decision 1). `user_id NOT NULL REFERENCES users ON DELETE CASCADE`, `ENABLE` + `FORCE ROW LEVEL
SECURITY`, the same `tapir.current_user_id` policy as every other user-owned table. It carries
the transcript text and its timestamped segments (re-transcribing 90 minutes costs ~7
minutes; discarding the timestamps is irreversible), `status`, the `endpoint` that served
it, `audio_seconds`, and a typed `error_kind`. **Not** a `user_id` column on `transcripts`
with a partial policy, even though that would reuse more code: ADR-021's exception is
precisely that *that table holds no private row*, the isolation test asserts it has no
`user_id`, and a partial policy would turn "single exception" into "single exception, with a
private lane inside it" — the shape under which a future query drops the policy clause and
leaks. Two tables keep both boundaries independently assertable.
3. **The isolation test asserts both boundaries and the set.** (a) User A cannot read, update or
delete user B's uploaded transcript; no GUC set → deny-all. (b) `transcripts` is still
non-RLS with no `user_id`. (c) **The set of public-schema tables with row security off is
asserted exactly** — so a new table added without RLS fails CI instead of passing it. (c) is
new; the existing test only checks the named tables.
4. **Validate by probing, reject clearly, and say what was ignored.** Format, duration and
what a file contains cannot be read from a filename or MIME type — a Teams recording is an
`.mp4` whether or not the camera was on. The upload is probed (`ffprobe`): **accept any
container with an audio stream, whether or not it also has a video stream** (an mp3's cover
art counts as neither). Whether a real video track was present is recorded on the row
(`had_video`) and the transcript page says so plainly — *this file contained a video track;
only its audio was used and the video was discarded* — so the user is never left to wonder
what happened to the picture. Reject a file with **no** audio stream (including a screen
recording with no microphone) and an unreadable one. Limits, all configurable, all rejected
synchronously with a specific message rather than by timeout: **200 MB** upload and
**90 minutes** duration. 90 minutes is *the longest length measured*, not a ceiling that was
found; raising it is a measurement, not an edit. The size cap now matters more: a video file
is much larger than its audio and is spooled whole before the audio is extracted, so a long
recording with screen sharing can exceed 200 MB while its audio is a few tens of MB.
5. **Transcode server-side, always, to 16 kHz mono PCM WAV** — even when the input already is,
rather than assuming. whisper-server's `--convert` is rejected (its `--tmp-dir` is created at
process start and rots under macOS periodic cleanup). The transcoder sits behind a port; the
tests use a fake. **This puts `ffmpeg` and `ffprobe` in the runtime image**, which is
distroless-static today. **Provisional: copy static ffmpeg/ffprobe binaries into the existing
distroless image** (no shell, no package manager retained). This is a heavy new dependency
and is **the maintainer's call** — see Open items.
6. **Transcription adapter via the gateway, never the public hostname.** A `Transcriber` port;
the implementation calls `POST {TAPIR_GATEWAY_URL}/audio/transcriptions` where the gateway
URL is the in-cluster NodePort (`koala:30401/v1`). Primary `iguana/kb-whisper`, fallback
`berget/kb-whisper` (immune to iguana contention, which can wedge permanently — infra#273).
- **Fallback triggers on any primary failure** (transport error, timeout, non-2xx). It never
branches on the body: `{"error":"failed to process audio"}` has had at least five distinct
causes.
- **`response_format=verbose_json`**, normalised to one segment shape: times clamped to the
audio duration (the final segment can end past it), one language token. Not `srt`: the
envelope adds a parse step and iguana's segment timing is the same either way.
- **No chunking in v1.** Nothing failed to 90 minutes (measured). Chunk-and-stitch is the
answer *if* the cap is raised or smaller requests become necessary — see triggers.
- **The serving endpoint is recorded per row**, and surfaced, because the two backends
differ in timestamp granularity: which one served a transcript is a quality fact.
- **Berget is a cloud endpoint.** On fallback the audio of a private meeting leaves the
homelab. #33 asks for this fallback; it is configurable (`TAPIR_TRANSCRIBE_FALLBACK_MODEL`,
empty disables), recorded per row, and called out in Open items rather than defaulted
silently.
- Swedish only (`language=sv`). Language detection and routing (#28 decision 2) are not
touched.
- `-H 'Expect:'`, multipart field `file` — encoded in the adapter and its tests, not rediscovered.
7. **Jobs run in-process, one at a time (ADR-018 stands).** Transcription is minutes (~7.5 min
for 90) but a single goroutine behind a one-slot semaphore: two simultaneous 90-minute jobs on
iguana were not measured, so concurrency is 1 until they are. No worker, no new deployable,
no second replica. Honest status per ADR-025: `queued → transcribing → done | failed`, polled
by the same `/status` pattern, every terminal state a rendered card, never a stuck spinner.
Audio is not retained, so a restart mid-job **cannot resume**: at start-up any
`queued`/`transcribing` row is marked `failed` with `error_kind = interrupted`, and the user
re-uploads. The single-replica assumption gains a second dependent (the semaphore is
process-local) and is recorded here.
8. **Chat reuses `chat.Service`; it is not forked.** The uploaded-transcript chat handlers
look the row up through the RLS-scoped store, so a non-owner gets 404 by the same mechanism
as ADR-027's summary gate, **before any model is called**. ADR-027's decisions (ephemeral,
switchable models, trust-the-model) carry over unchanged. **Two things must differ, and both
were measured** (`docs/research/chat-context-over-long-transcripts-2026-10-01.md`):
- **The transcript budget.** The 18,000-char caption default cuts a one-hour meeting to its
first third. Uploaded transcripts take their own `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS`,
default **90,000** — the measured ceiling of the best offered model (`iguana/gemma4-26b`),
which answers correctly there but **silently loses the head of a 120,000-char transcript**
with no error and no truncation flag. (An earlier draft of this ADR proposed 120,000 as a
guess; the measurement refuted it.) The mechanism is a per-request `chat.Request.MaxChars`.
- **The opening model.** The offered list leads with the summarizer default `koala/phi4-mini`,
whose 8,192-token context returns HTTP 400 on an hour of speech (~13,000 tokens). Chat over
an upload opens on `TAPIR_UPLOAD_CHAT_MODEL` (default `iguana/gemma4-26b`); the switcher
still lists every model, and picking `phi4-mini` on a long recording fails loudly and
inline. A per-model budget is not built.
Past ~90,000 chars (≈100 minutes) no offered model can answer by stuffing; see triggers.
9. **Not decided here.** Rich `AVSNITT`/`ALTERNATIV` analysis (gated on #30 — `analyze_srt.py`
hardcodes `max_tokens: 6000`); video, keyframes, synced playback (#28); language detection;
per-transcript deletion UX (account deletion cascades; see Open items); retention of audio.
**On the sequencing question #28 asks of itself** — *is this the thing that best serves the
Stage-0 gate, or the most interesting thing to build?* Recorded answer: it proceeds **ahead of
#32** (the demand test) as the maintainer's explicit, knowing call (#28, comment of 2026-10-01),
on a concrete need rather than speculation. It is not a return-nudge (ADR-020/027): it adds
nothing that prompts a user to come back, so it does not contaminate the unprompted-return signal
ADR-016 measures. The honest risk is the one ADR-016 names: it is larger than anything ADR-020
deferred. The scope cut — chat first, no rich analysis, no video — is what keeps it bounded.
**Consequences.**
- **Positive.** The user's request is met end to end with existing parts. ADR-021's exception stays
singular and now provably so. The measured ceiling removes chunking from v1. No audio is stored.
- **Negative / watch.**
- Tapir's runtime image gains `ffmpeg`: a larger image and a new CVE surface to patch.
- iguana becomes load-bearing for a user-facing feature (#28 decision 3; infra#273 is still the
dependency that matters) — a Metal OOM now means *every upload fails over to berget*, i.e.
private audio leaving the homelab, silently unless the per-row endpoint is surfaced.
- A 90-minute job holds the single slot for ~7.5 minutes; a second user's upload queues behind it.
- Chat over a long transcript is bounded by model context, not by anything in Tapir, and the
failure at the edge is *silent* for gemma (the head is dropped, no error). The 90,000 default
sits just inside the measured limit and is only as good as that single measurement.
- **The default chat model for uploads is a large ollama model on iguana — the same host as
`iguana/kb-whisper`.** infra#273 is exactly that a large resident ollama model can Metal-OOM
whisper-server permanently. Chatting with a transcript can therefore load the model that
wedges the transcriber. Not solved here; the berget fallback limits the blast radius.
- Tapir's own ingress body limit for a 200 MB upload is **unverified** (it lives in `infra`).
- Accepting video means large files are spooled before their audio is extracted, and ffmpeg
demuxes (but, with `-vn`, does not decode) video it then throws away. A client-side audio
extraction would avoid the upload cost; it is not built.
**Validation / resolution triggers** (each observable by someone other than the author):
- *Chunking*: build it when the duration cap is raised above 90 min, or when a measured
request at the current cap fails with whisper-side (not ingress) errors.
- *Berget-first*: if more than a quarter of uploads in any week are served by the fallback,
swap the primary/fallback order and revisit infra#273.
- *Retrieval chat*: build chunked retrieval when a recording's transcript exceeds the measured
90,000-char ceiling (≈100 minutes) — i.e. the moment the duration cap is raised above that, or a
real recording turns out denser than the fixture — or when the offered model list changes and
the ceiling is re-measured lower.
- *Worker*: if a transcription job ever degrades the reading UI, or Tapir runs more than one
replica, ADR-018 is revisited.
- *The decision itself*: if the maintainer has not used the feature unprompted within the
ADR-016 window, it was the interesting thing, not the useful one — remove it.
**Open items this ADR creates.**
- **DECISION NEEDED — ffmpeg in the runtime image.** *Options:* (A) static ffmpeg/ffprobe copied
into distroless **(recommended — the only option that satisfies "accept m4a/mp3/opus/mp4,
validate duration, reject video"; pure-Go decoders cover wav/mp3 only)**; (B) accept WAV
only, convert client-side (fails "I just want to upload it"); (C) a transcode sidecar
(second deployable, ADR-018's reasoning against). *Blocks:* the Dockerfile slice, not the
tests, which use the port.
- **DECISION NEEDED — berget fallback for private audio.** *Options:* (A) enabled by default as
#33 specifies, with the serving endpoint shown **(recommended, matches the issue)**; (B)
disabled by default, a failed iguana job fails honestly and the user retries. *Blocks:*
the production default only; the adapter ships either way.
- Verify Tapir's ingress `proxy-body-size` against the 200 MB cap (`infra`).
- Per-transcript deletion is not built; account deletion cascades. A private-content store
without a delete button is a gap worth a ticket, not a silent omission.
**Reversibility.** Additive: one table (down-migration drops it), one adapter, one upload
surface, one image change. Removing the upload route removes the feature; `transcripts`, `videos`
and ADR-021 are untouched.
---
## ADR-034 — Transcribe with two engines and let a local model reconcile their disagreements
**Status:** Proposed (2026-10-01). **Decider: Mathias — pending.** Issue #40 (decision B) and #33.
Builds on ADR-033 (private upload transcripts), ADR-022 (endpoint chain, a pattern reused), ADR-018
(single replica), ADR-025 (honest status). Evidence:
`docs/research/asr-kb-whisper-vs-pianissimo-2026-10-01.md` and
`docs/research/local-model-adjudicator-sweep-2026-10-01.md`.
**Context.** On the first real one-hour recording `iguana/kb-whisper` and `iguana/pianissimo` were
a near tie on accuracy (the speaker's blind verdicts on the 30 meaning-changing disagreements:
pianissimo 17, kb-whisper 13) and they **err in different places**: pianissimo is better on terms and
numbers, kb-whisper on ordinary words; kb-whisper's errors garble, pianissimo's read as plausible.
Pianissimo is ~6x faster (40 s vs 4m08s for 63.7 min) and runs through the same adapter. Two engines
that disagree in ~380 places per hour, with a ~92% word agreement, is information a single-engine
pipeline throws away. Separately, eleven local models were scored on adjudicating those disagreements
against the speaker's answers: **`gemma4-31b` 97% and `gemma4-26b` 92%** passed the pre-set bar
(>= 80% accuracy, >= 85% order-swap consistency, <= 10% unparseable), nothing else did.
**Unknowns, stated.** The adjudicator evidence is 30 items from one speaker and one domain, chosen by
Opus as decidable; the other ~350 disagreements (fillers, compounds, small words) have no ground
truth. Whether a *merged* transcript is better than the better single engine has not been measured.
Reconcile time per hour is estimated, not measured. `gemma4-31b`'s slowness is unexplained (Low Power
Mode on iguana is the unproven suspect).
**Decision.**
1. **Two engines, one WAV, both local, one after the other.** Each upload is transcribed by
`iguana/pianissimo` and `iguana/kb-whisper` from the same converted WAV, sequentially (concurrency
stays 1, ADR-033). Each engine fails independently; the berget fallback (ADR-033) applies only when
**both** fail. Off by default: with no second engine configured nothing changes.
2. **Align by word, with pianissimo as the base text.** Both texts are tokenised (lower case, letters
and digits only) and diffed word by word (Myers); neighbouring differences separated by a single
equal word are merged into one **dispute**. The base is pianissimo: sentence-level prose with
punctuation and millisecond timestamps, where kb-whisper's output is wrapped fragments. **This
choice is about readability and timing, not accuracy — accuracy was a tie — so it is the cheaper
mistake, not a measured winner.** A replaced span takes the alternative's own words.
3. **Resolve the trivial ones without a model.** A dispute that is identical once spaces are removed
(a compound split or joined) keeps the base; so does a dispute that is only a short filler word
on one side. Everything else is adjudicated. (Opus's classification of 380 disputes found ~46
compound and ~26 punctuation-only residue, ~65 filler.)
4. **A local model adjudicates the rest, bounded.** Each dispute goes to a local chat model (default
`iguana/gemma4-26b`; `gemma4-31b` is configurable and scored higher but is ~5x slower) with ~100
words of context either side and the two readings, in the wording that was measured. The reading
order alternates by dispute so position bias averages out. The verdict is *base*, *alternative* or
*unsure*; **unsure keeps the base and is flagged**. An optional order-check mode asks twice with the
order swapped and treats disagreement as unsure; it is **off by default** because it doubles the time
and the measured consistency of the passing models is 93%.
- **The model must be local.** Its alias must begin `koala/` or `iguana/`; a cloud alias is a
start-up configuration error. Private meeting content never goes to a cloud model for this step.
- **Bounded impact.** The reconciler can only choose between two readings the engines already
produced; it cannot write new text. Speech that says "answer 2" can at worst flip one dispute.
5. **Reconciliation never fails a job.** One engine fails → its text is used, recorded as
single-engine. The adjudicator is down (three consecutive errors) → the base text is used
unreconciled, recorded as skipped with the reason, and the remaining disputes are not retried one
timeout at a time. Both engines fail → ADR-033's fallback path.
6. **What is stored and shown.** `content` is the merged text. `segments` stay the base engine's timed
segments, untouched: `content` is authoritative for chat, `segments` only for time. `endpoint`
lists every contributor and the reconciler. A new `reconciliation` column (migration 019, same
RLS-scoped table) records the status, counts, and the **uncertain and changed spots with a minute
mark and both readings**, so the user can check them against the audio. The transcript page says in
plain words what was done — "reconciled from two transcripts: N differences, K uncertain" — and
lists the uncertain spots. A reconciled transcript is never presented as simply "the" transcript.
7. **Configuration.** `TAPIR_RECONCILE_BASE_MODEL` (empty = off; the engine whose text is the base,
e.g. `iguana/pianissimo` — `TAPIR_TRANSCRIBE_MODEL` stays the other engine, e.g. kb-whisper),
`TAPIR_RECONCILE_MODEL` (the adjudicator; required when a base engine is set),
`TAPIR_RECONCILE_CHECK_ORDER` (off). All three models must be local aliases (`koala/` or `iguana/`):
a cloud alias is a start-up error, and the base engine must differ from the other one. The default
job timeout rises from 30 to 60 minutes: ~4 min of kb-whisper and ~40 s of pianissimo per hour of
audio, plus the adjudication.
8. **Not decided here.** A cloud adjudicator (rejected: private content); tuning the prompt; making
`gemma4-31b` the default; a "reconciling" progress state (the page says "transcribing" for the whole
job); feeding the uncertain spots to chat; whether pianissimo should also be a middle tier in the
fallback chain (#40, option A).
**Consequences.**
- **Positive.** Uses the finding that the engines err differently; stays local end to end; degrades
to today's behaviour at every failure; and is honest on the page about what it did.
- **Negative / watch.**
- Job time grows: roughly 300 adjudicated disputes x ~2 s on `gemma4-26b` ~ 10 minutes per hour of
audio — an estimate, not a measurement; double with order-check on, ~5x with `gemma4-31b`.
- Two gemma-class calls sit on iguana beside whisper. During the sweep a 17-20 GB model was resident
beside whisper-server without wedging it, far below the 52 GB that did (infra#273); the hazard is
reduced, not removed.
- A merged transcript can be wrong where both engines are wrong, and where the adjudicator picks
the wrong one (it was right 92% on a hand-picked set, ~97% for 31b).
- Reading the page takes more words: the reconciliation summary is new surface to keep honest.
**First live smoke (2026-10-01, synthetic audio, 3 disputes):** all 3 were changed to the
alternative's reading; against the known script 2 were right and **1 was wrong** (a name replaced by a
pronoun, a correct base reading lost). That is one run of three, not a measurement, but it is the failure
this design must expect: the reconciler can overwrite a correct reading, which is why changed spots, not
only uncertain ones, are listed on the page, and why the decision below is a blind check on a real
recording.
**Validation / resolution triggers.**
- **The test that decides it:** re-upload the same real recording after merge; the speaker judges, blind,
~30 disputes where the reconciled text differs from pianissimo alone. Keep the feature only if the
reconciled reading is right at least 85% of the time **and** no worse than the better single engine.
If not, switch it off (config) and keep the single-engine path.
- Reconcile time above 25 minutes per hour of audio: drop to a smaller adjudicator or skip more
disputes automatically.
- A different recording or speaker in which the engines agree far less (or far more): re-measure before
trusting the 92%/97% figures.
**Reversibility.** Off by default and additive: the second engine, the reconciler and the column are
unused when the settings are empty; the down-migration drops the column; the single-engine path is
unchanged.
---
## Rejected alternatives ## Rejected alternatives
Approaches considered during the 2026-06-02 planning + grill session and **deliberately not Approaches considered during the 2026-06-02 planning + grill session and **deliberately not
@@ -757,7 +1664,7 @@ maps to the ADR that settles it.
| Lifting shared packages into a `brain-common` module | Couples Tapir's release cycle to the monolith for negligible code savings | ADR-004 | | Lifting shared packages into a `brain-common` module | Couples Tapir's release cycle to the monolith for negligible code savings | ADR-004 |
| Importing/replicating the filesystem `brain` package | Assumes co-location with the brain git checkout; wrong for a standalone networked service | ADR-005 | | Importing/replicating the filesystem `brain` package | Assumes co-location with the brain git checkout; wrong for a standalone networked service | ADR-005 |
| Reusing `ingestion`'s `oauth` package for YouTube/Vimeo | Same name, opposite direction — it's inbound MCP-server auth, not outbound provider OAuth | ADR-006 | | Reusing `ingestion`'s `oauth` package for YouTube/Vimeo | Same name, opposite direction — it's inbound MCP-server auth, not outbound provider OAuth | ADR-006 |
| Global cross-tenant `videos`/`transcripts` table (dedup) | Reintroduces the cross-domain DB coupling the homelab review is removing; at 1–5 users, re-summarizing is cheaper than the coupling | data-model.md | | Global cross-tenant `videos`/`transcripts` table (dedup) | Reintroduces the cross-domain DB coupling the homelab review is removing; at 1–5 users, re-summarizing is cheaper than the coupling. **Transcripts half reopened by ADR-021** — the avoided cost there is a rate-gated, reputation-risky *caption fetch*, not LLM re-summarization, so it outweighs the coupling; **videos stay per-user.** | data-model.md, **ADR-021** (transcripts only) |
| Audio-download + Whisper STT in the core path | ToS-grey, breakage-prone (yt-dlp), contends for koala GPU with the JEPA PoC; captions alone test the core hypothesis | ADR-007 | | Audio-download + Whisper STT in the core path | ToS-grey, breakage-prone (yt-dlp), contends for koala GPU with the JEPA PoC; captions alone test the core hypothesis | ADR-007 |
| Building multi-tenant SaaS / Google OAuth verification now | "Real users soon" was lowered to Future B; SaaS machinery before the Stage 0 self-use gate is the primary documented anti-goal | ADR-008, VISION | | Building multi-tenant SaaS / Google OAuth verification now | "Real users soon" was lowered to Future B; SaaS machinery before the Stage 0 self-use gate is the primary documented anti-goal | ADR-008, VISION |
| Delegating the S5 reuse spike to an agent swarm | A 1-hour sequential read-and-judge with a single coupled conclusion; orchestration overhead exceeds the work, and it's Diamond-1 judgment the maintainer wanted to own | (process note) | | Delegating the S5 reuse spike to an agent swarm | A 1-hour sequential read-and-judge with a single coupled conclusion; orchestration overhead exceeds the work, and it's Diamond-1 judgment the maintainer wanted to own | (process note) |
@@ -766,6 +1673,12 @@ maps to the ADR that settles it.
| Feedback-based Stage 0 gate (friends saying it's useful) | Politeness bias makes asked-for feedback the least reliable signal; return-usage is the real test | ADR-016 | | Feedback-based Stage 0 gate (friends saying it's useful) | Politeness bias makes asked-for feedback the least reliable signal; return-usage is the real test | ADR-016 |
| Reverse the Dex-write invite flow (Google OIDC only) | Some intended Future-B users won't use Google; OIDC-only leaves them with no onboarding path — invite flow is load-bearing | ADR-017 | | Reverse the Dex-write invite flow (Google OIDC only) | Some intended Future-B users won't use Google; OIDC-only leaves them with no onboarding path — invite flow is load-bearing | ADR-017 |
| k8s CronJob for scheduled discovery (vs in-process) | At Future-B scale the in-process scheduler is simpler to deploy; CronJob's failure-isolation benefit was weighed and traded away knowingly (revisit if >1 replica or load grows) | ADR-018 | | k8s CronJob for scheduled discovery (vs in-process) | At Future-B scale the in-process scheduler is simpler to deploy; CronJob's failure-isolation benefit was weighed and traded away knowingly (revisit if >1 replica or load grows) | ADR-018 |
| Cached-transcript-first onboarding burst (instant, zero-fetch picks) | Live pilot DB: ~3% cross-user video overlap, 0 cached-and-unsummarized, a new user's newest-20 are 20/20 uncached — newest-first and cached-first are structurally incompatible. Empty lever at pilot scale | ADR-028 |
| Uploaded transcripts as a `user_id` column + partial RLS policy on `transcripts` | Turns ADR-021's single exception into an exception with a private lane inside it; the isolation test asserts `transcripts` has no `user_id`. A separate RLS table keeps both boundaries independently assertable | ADR-033 |
| whisper-server `--convert` for server-side transcode | Its `--tmp-dir` is created at process start and rots under macOS periodic cleanup | ADR-033 |
| Calling the public `llm-api.d-ma.be` from in-cluster Tapir | Its ingress returns 413 above ~8–10 MB (~5 min of WAV); the NodePort has no such cap | ADR-033 |
| A cloud model (e.g. Opus) as the adjudicator | It scored 30/30 but would send private meeting content to a cloud API; local gemma4-26b/31b passed the same bar | ADR-034 |
| Pick one engine globally (pianissimo only, or kb-whisper only) | A near tie on accuracy with different error profiles; choosing one discards the information in their disagreement | ADR-034 |
If a future case genuinely reopens one of these, that's a new ADR superseding the relevant one — If a future case genuinely reopens one of these, that's a new ADR superseding the relevant one —
not a silent reversal. not a silent reversal.
+26
View File
@@ -0,0 +1,26 @@
# Vocabulary (hand-written pilot 2026-06 — will be generated by langgen in Phase 2)
| Term | Means | Never say |
|---|---|---|
| transcript | Captions-first text of a video; the source for summarizing. Shared store keyed by `(provider, video_id)`. | "subtitles file", "the audio" |
| video | Per-user record of a seen video; the unit of work. Not globally deduped. | "the shared video" |
| subscription | A watched channel on a user's connection that Tapir polls for new videos. | "feed", "follow" |
| summary | A video's produced output: summary text + highlights + takeaways + AI provenance. | "transcript" |
| highlight | A notable point pulled from a video (`Summary.Highlights`). | "takeaway" |
| takeaway | An actionable conclusion from a video (`Summary.Takeaways`). | "highlight" |
| sink | A delivery destination for a summary (`store`, `brain`). New one = new adapter. | "the database" |
| AI router | Local-first chain: local Primary → local fallback → external worst-case. | "the API" |
| BYO-AI fallback | User's own external AI key, used only when local fails; opt-in. Without it, content is never sent externally. | "the default AI" |
| brain | Persistent homelab knowledge store; in Tapir, one optional HTTP sink (`brain_ingest`), not the filesystem package. | "the database", "the store" |
| LiteLLM gateway | Local AI gateway (`koala:30401/v1`); the Primary in the AI router. | "piguard:4000", "koala:4000", "the cloud" |
| connection | A connected video account a user authorizes via OAuth; subscriptions hang off it. | "login", "session" |
## Caveman rubric
Before any HIGH/CRITICAL operation, output one line:
`caveman: me <verb> <object>, not <excluded thing>`
Valid iff: (1) only vocabulary terms + plain verbs, (2) a stranger could identify
the exact operation, (3) names one thing explicitly NOT being done.
Examples:
- `caveman: me delete user connection, not the shared transcript`
- `caveman: me send transcript to BYO-AI fallback, not the LiteLLM gateway`
+64
View File
@@ -0,0 +1,64 @@
package main
import (
"strings"
"testing"
"git.d-ma.be/mathias/tapir/internal/config"
)
// TestChatModelsReuseTheChainLocalFirst: the switcher offers the ADR-022 chain in
// order, deduped — primary, local fallback, cloud.
func TestChatModelsReuseTheChainLocalFirst(t *testing.T) {
got := chatModels(config.Config{
SummarizerModel: "koala/phi4-mini",
FallbackModel: "iguana/gemma4-26b",
CloudFallbackModel: "berget/mistral-small",
})
want := []string{"koala/phi4-mini", "iguana/gemma4-26b", "berget/mistral-small"}
if strings.Join(got, ",") != strings.Join(want, ",") {
t.Fatalf("chatModels = %v, want %v", got, want)
}
}
// TestChatCloudModelAbsentWhenDisabled: the local-first / NDA lever — with the
// cloud fallback empty (TAPIR_CLOUD_FALLBACK_MODEL=""), no external model is
// offered in the switcher, so chat content never leaves the local stack (ADR-027,
// honouring ADR-022's "content stays local" guarantee).
func TestChatCloudModelAbsentWhenDisabled(t *testing.T) {
got := chatModels(config.Config{
SummarizerModel: "koala/phi4-mini",
FallbackModel: "iguana/gemma4-26b",
CloudFallbackModel: "",
})
for _, m := range got {
if strings.HasPrefix(m, "berget/") || strings.Contains(m, "mistral") {
t.Fatalf("cloud model %q offered though the cloud fallback is disabled", m)
}
}
want := []string{"koala/phi4-mini", "iguana/gemma4-26b"}
if strings.Join(got, ",") != strings.Join(want, ",") {
t.Fatalf("chatModels = %v, want %v", got, want)
}
}
// TestChatModelsDedup: a config that reuses one alias across slots collapses to a
// single switcher entry (no duplicate options).
func TestChatModelsDedup(t *testing.T) {
got := chatModels(config.Config{
SummarizerModel: "koala/phi4-mini",
FallbackModel: "koala/phi4-mini",
CloudFallbackModel: "",
})
if len(got) != 1 || got[0] != "koala/phi4-mini" {
t.Fatalf("chatModels = %v, want a single deduped entry", got)
}
}
// TestBuildChatNilWithoutGateway: no gateway → no chat backend (routes unmounted,
// read path unaffected).
func TestBuildChatNilWithoutGateway(t *testing.T) {
if c := buildChat(config.Config{GatewayURL: ""}); c != nil {
t.Fatal("buildChat must return nil without a gateway URL")
}
}
+1 -1
View File
@@ -5,7 +5,7 @@ import (
"log/slog" "log/slog"
"sync" "sync"
"gitea.d-ma.be/mathias/tapir/internal/runner" "git.d-ma.be/mathias/tapir/internal/runner"
) )
// discoveryRunner runs one user's discovery pass. // discoveryRunner runs one user's discovery pass.
+1 -1
View File
@@ -10,7 +10,7 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/runner" "git.d-ma.be/mathias/tapir/internal/runner"
) )
// serialize must guarantee at most one discovery pass runs at a time, so a // serialize must guarantee at most one discovery pass runs at a time, so a
+190
View File
@@ -0,0 +1,190 @@
package main
import (
"context"
"fmt"
"io"
"mime/multipart"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"strings"
"testing"
"time"
embeddedpostgres "github.com/fergusstrange/embedded-postgres"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/store"
"git.d-ma.be/mathias/tapir/internal/config"
"git.d-ma.be/mathias/tapir/internal/web"
)
// TestE2E_UploadTranscribeAndChat is the end-to-end check for tapir#33: upload a
// real audio file, watch it become a transcript through the REAL ffprobe, ffmpeg
// and gateway, then ask a question and get a grounded answer from a REAL model.
// It is opt-in — it needs real binaries and the live gateway — and skipped unless
// TAPIR_E2E_AUDIO is set:
//
// TAPIR_E2E_AUDIO=/path/meeting.m4a TAPIR_E2E_FFMPEG=/path/ffmpeg \
// TAPIR_E2E_FFPROBE=/path/ffprobe TAPIR_E2E_QUESTION='Vem ansvarar för virket?' \
// TAPIR_E2E_EXPECT=anders op run --env-file ~/.op-env -- \
// go test ./cmd/tapir -run E2E_Upload -v -count=1
//
// Use SYNTHETIC audio only: this repo is public, and a fallback to berget sends
// the audio to a cloud endpoint (#33).
func TestE2E_UploadTranscribeAndChat(t *testing.T) {
audioPath := os.Getenv("TAPIR_E2E_AUDIO")
if audioPath == "" {
t.Skip("set TAPIR_E2E_AUDIO (and friends) to run the live end-to-end check")
}
key := os.Getenv("LITELLM_MASTER_KEY")
require.NotEmpty(t, key, "LITELLM_MASTER_KEY must be set (via op run)")
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
// A private embedded Postgres: nothing here may touch a real database.
port := uint32(55000 + os.Getpid()%1000)
rt := filepath.Join(os.TempDir(), fmt.Sprintf("tapir-epg-e2e-%d", os.Getpid()))
pg := embeddedpostgres.NewDatabase(embeddedpostgres.DefaultConfig().Port(port).
RuntimePath(rt).DataPath(filepath.Join(rt, "data")).BinariesPath(filepath.Join(rt, "bin")).
CachePath(filepath.Join(os.TempDir(), "tapir-epg-cache")))
require.NoError(t, pg.Start())
t.Cleanup(func() { _ = pg.Stop(); _ = os.RemoveAll(rt) })
dsn := fmt.Sprintf("postgres://postgres:postgres@localhost:%d/postgres?sslmode=disable", port)
st, err := store.New(ctx, dsn)
require.NoError(t, err)
t.Cleanup(st.Close)
raw, err := pgxpool.New(ctx, dsn)
require.NoError(t, err)
t.Cleanup(raw.Close)
const uid = "11111111-1111-1111-1111-111111111111"
_, err = raw.Exec(ctx, `INSERT INTO users (id) VALUES ($1)`, uid)
require.NoError(t, err)
_, err = raw.Exec(ctx, `INSERT INTO user_identities (dex_subject, user_id) VALUES ('e2e', $1)`, uid)
require.NoError(t, err)
cfg, err := config.Load()
require.NoError(t, err)
cfg.GatewayKey = key
cfg.UploadsEnabled = true
cfg.UploadTmpDir = filepath.Join(t.TempDir(), "spool")
if os.Getenv("TAPIR_E2E_RECONCILE") == "1" {
// Two engines and a local model, as ADR-034 describes.
cfg.ReconcileBaseModel = envOrDefault("TAPIR_E2E_BASE_MODEL", "iguana/pianissimo")
cfg.ReconcileModel = envOrDefault("TAPIR_E2E_RECONCILE_MODEL", "iguana/gemma4-26b")
}
cfg.FFmpegBin = envOrDefault("TAPIR_E2E_FFMPEG", "ffmpeg")
cfg.FFprobeBin = envOrDefault("TAPIR_E2E_FFPROBE", "ffprobe")
intake, worker, err := buildUploads(ctx, cfg, st, quietLog())
require.NoError(t, err)
go worker.Run(ctx)
app := &web.App{
Store: st, Identity: st, Auth: web.StubAuth{U: web.User{Subject: "e2e"}},
Chat: buildChat(cfg),
}
wireUploads(app, cfg, intake) // the same wiring cmdServe uses
srv := httptest.NewServer(app.Router())
t.Cleanup(srv.Close)
noRedirect := &http.Client{CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
// 1. Upload.
f, err := os.Open(audioPath)
require.NoError(t, err)
defer func() { _ = f.Close() }()
pr, pw := io.Pipe()
mw := multipart.NewWriter(pw)
go func() {
_ = mw.WriteField("title", "E2E recording")
fw, _ := mw.CreateFormFile("file", filepath.Base(audioPath))
_, _ = io.Copy(fw, f)
_ = mw.Close()
_ = pw.Close()
}()
req, err := http.NewRequest(http.MethodPost, srv.URL+"/uploads", pr)
require.NoError(t, err)
req.Header.Set("Content-Type", mw.FormDataContentType())
resp, err := noRedirect.Do(req)
require.NoError(t, err)
_ = resp.Body.Close()
require.Equal(t, http.StatusSeeOther, resp.StatusCode, "an accepted upload redirects to its page")
loc := resp.Header.Get("Location")
require.True(t, strings.HasPrefix(loc, "/uploads/"), loc)
t.Logf("accepted -> %s", loc)
// 2. Wait for the transcript: unfinished states poll, terminal states do not.
var page string
require.Eventually(t, func() bool {
r, err := http.Get(srv.URL + loc + "/status")
if err != nil {
return false
}
b, _ := io.ReadAll(r.Body)
_ = r.Body.Close()
page = string(b)
return !strings.Contains(page, "hx-trigger")
}, 12*time.Minute, 3*time.Second, "the upload never reached a terminal state")
require.Contains(t, page, "Transcribed", "terminal state was not 'done': %.300s", page)
var status, endpoint, content string
var secs float64
var hadVideo bool
require.NoError(t, raw.QueryRow(ctx,
`SELECT status, endpoint, content, audio_seconds, had_video FROM uploaded_transcripts WHERE user_id = $1`, uid).
Scan(&status, &endpoint, &content, &secs, &hadVideo))
t.Logf("done: endpoint=%s audio_s=%.1f transcript_chars=%d had_video=%v", endpoint, secs, len(content), hadVideo)
if os.Getenv("TAPIR_E2E_RECONCILE") == "1" {
var recJSON string
require.NoError(t, raw.QueryRow(ctx,
`SELECT COALESCE(reconciliation::text, 'NULL') FROM uploaded_transcripts WHERE user_id = $1`, uid).Scan(&recJSON))
require.NotEqual(t, "NULL", recJSON, "a two-engine run must record its reconciliation")
t.Logf("reconciliation: %.600s", recJSON)
assert.Contains(t, endpoint, "reconciled by", "every contributor is named in the endpoint")
assert.Contains(t, page, "Reconciled from two transcripts")
}
if os.Getenv("TAPIR_E2E_HAD_VIDEO") == "1" {
// A Teams meeting with the camera off: a video file whose audio is what matters.
assert.True(t, hadVideo, "the file had a video track and the row must say so")
assert.Contains(t, page, "Only its audio was used", "and the user must be told")
}
assert.Equal(t, "done", status)
assert.NotEmpty(t, endpoint)
assert.NotEmpty(t, content)
entries, err := os.ReadDir(cfg.UploadTmpDir)
require.NoError(t, err)
assert.Empty(t, entries, "the audio must be deleted once transcribed")
// 3. Ask a question; the answer must come from the transcript.
q := envOrDefault("TAPIR_E2E_QUESTION", "Who is responsible for the timber order?")
chatResp, err := http.PostForm(srv.URL+loc+"/chat", url.Values{"question": {q}})
require.NoError(t, err)
answerPage, _ := io.ReadAll(chatResp.Body)
_ = chatResp.Body.Close()
require.Equal(t, http.StatusOK, chatResp.StatusCode)
t.Logf("question: %s", q)
if want := os.Getenv("TAPIR_E2E_EXPECT"); want != "" {
assert.Contains(t, strings.ToLower(string(answerPage)), strings.ToLower(want),
"the answer should be grounded in the transcript")
}
t.Logf("answer page excerpt: %.600s", afterMarker(string(answerPage), `class="chat-turn chat-a"`))
}
func envOrDefault(k, d string) string {
if v := os.Getenv(k); v != "" {
return v
}
return d
}
func afterMarker(s, m string) string {
if i := strings.Index(s, m); i >= 0 {
return s[i:]
}
return "(no answer turn found)"
}
+1 -1
View File
@@ -5,7 +5,7 @@ import (
"fmt" "fmt"
"os" "os"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
// Env var names for the read-only CLI. DSN and user id are never hardcoded — the // Env var names for the read-only CLI. DSN and user id are never hardcoded — the
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
func TestFormatListColumnsAndOrdering(t *testing.T) { func TestFormatListColumnsAndOrdering(t *testing.T) {
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"strings" "strings"
"text/tabwriter" "text/tabwriter"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
// runList prints the user's stored summaries as a table, most recent first. // runList prints the user's stored summaries as a table, most recent first.
+82 -20
View File
@@ -23,14 +23,15 @@ import (
"sync" "sync"
"time" "time"
"gitea.d-ma.be/mathias/tapir/internal/adapters/secrets" "git.d-ma.be/mathias/tapir/internal/adapters/secrets"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
"gitea.d-ma.be/mathias/tapir/internal/adapters/youtube" "git.d-ma.be/mathias/tapir/internal/adapters/youtube"
"gitea.d-ma.be/mathias/tapir/internal/auth" "git.d-ma.be/mathias/tapir/internal/auth"
"gitea.d-ma.be/mathias/tapir/internal/config" "git.d-ma.be/mathias/tapir/internal/config"
"gitea.d-ma.be/mathias/tapir/internal/runner" "git.d-ma.be/mathias/tapir/internal/metrics"
"gitea.d-ma.be/mathias/tapir/internal/web" "git.d-ma.be/mathias/tapir/internal/runner"
"gitea.d-ma.be/mathias/tapir/internal/web/oidc" "git.d-ma.be/mathias/tapir/internal/web"
"git.d-ma.be/mathias/tapir/internal/web/oidc"
) )
func main() { func main() {
@@ -136,7 +137,8 @@ func cmdRun(ctx context.Context, log *slog.Logger) error {
r := runner.New(engine.Source, st, engine, cfg.UserID, log, r := runner.New(engine.Source, st, engine, cfg.UserID, log,
runner.WithBackoff(cfg.FetchBackoff), runner.WithBackoff(cfg.FetchBackoff),
runner.WithAutoWindow(cfg.AutoSummarizeWindow)) runner.WithAutoWindow(cfg.AutoSummarizeWindow),
runner.WithCaptionMemory(cfg.ChannelCaptionlessThreshold, cfg.ChannelCaptionlessWindow))
log.Info("starting run", "user", cfg.UserID, "model", cfg.SummarizerModel, log.Info("starting run", "user", cfg.UserID, "model", cfg.SummarizerModel,
"gateway", cfg.GatewayURL, "poll_interval", cfg.PollInterval, "fetch_backoff", cfg.FetchBackoff, "gateway", cfg.GatewayURL, "poll_interval", cfg.PollInterval, "fetch_backoff", cfg.FetchBackoff,
@@ -196,6 +198,35 @@ func cmdServe(ctx context.Context, log *slog.Logger) error {
secretStore := secrets.NewFileStore(cfg.SecretsFile) secretStore := secrets.NewFileStore(cfg.SecretsFile)
app := &web.App{Store: st, Identity: st, Auth: authn, Secrets: secretStore, Log: log, RecencyWindow: cfg.AutoSummarizeWindow} app := &web.App{Store: st, Identity: st, Auth: authn, Secrets: secretStore, Log: log, RecencyWindow: cfg.AutoSummarizeWindow}
// Per-video deeper-dive chat over the STORED transcript (ADR-027). Enabled
// whenever a gateway is configured — it needs no YouTube credentials because it
// never fetches. Guarded so a typed-nil never lands in the interface field
// (which would mount the routes over a nil backend).
if c := buildChat(cfg); c != nil {
app.Chat = c
log.Info("web chat enabled (stored-transcript only)", "models", chatModels(cfg))
}
// Audio upload -> private transcript (ADR-033). Off unless TAPIR_UPLOADS_ENABLED:
// the image must carry ffmpeg first. The worker is IN-PROCESS and one job at a
// time, so it inherits the single-replica assumption below (its queue is
// process-local); it stops with the same ctx as the server.
intake, worker, err := buildUploads(ctx, cfg, st, log)
if err != nil {
return err
}
wireUploads(app, cfg, intake)
if intake != nil {
go worker.Run(ctx)
log.Info("audio upload enabled", "model", cfg.TranscribeModel, "fallback", cfg.TranscribeFallbackModel,
"max_mb", cfg.UploadMaxBytes>>20, "max_minutes", int(cfg.UploadMaxDuration.Minutes()))
log.Warn("audio upload assumes a SINGLE replica: the transcription queue is process-local (ADR-033, ADR-018)")
if cfg.TranscribeFallbackModel != "" {
log.Warn("audio upload fallback is a CLOUD endpoint: if the primary fails, private audio leaves the homelab",
"fallback", cfg.TranscribeFallbackModel)
}
}
// User onboarding is handled by the IdP (Authentik invite flow), not Tapir — // User onboarding is handled by the IdP (Authentik invite flow), not Tapir —
// the Dex local-password provisioning path was removed (ADR-019). An // the Dex local-password provisioning path was removed (ADR-019). An
// authenticated subject with no Tapir user is routed to /register. // authenticated subject with no Tapir user is routed to /register.
@@ -252,23 +283,36 @@ func cmdServe(ctx context.Context, log *slog.Logger) error {
// One lock shared by the scheduler and connect-triggered passes (#6) so // One lock shared by the scheduler and connect-triggered passes (#6) so
// they never fetch concurrently — the single-fetcher invariant (ADR-018). // they never fetch concurrently — the single-fetcher invariant (ADR-018).
runUser := serialize(&sync.Mutex{}, rawRunUser) runUser := serialize(&sync.Mutex{}, rawRunUser)
// Onboarding burst (Feature 1): after the connect-triggered discovery pass, // Onboarding burst (Feature 1, refined by ADR-028): after the connect-triggered
// summarize up to OnboardSummarizeCount of the user's NEWEST unsummarized // discovery pass, summarize up to OnboardSummarizeCount of the user's newest
// videos so a fresh account gets real summaries in its first session. Hard // LIKELY-GOOD unsummarized videos so a fresh account gets a strong first
// cap; explicit, so it bypasses the recency window — but every fetch still // session. Selection avoids known-junk (Shorts/over-long/livestream VODs via
// goes through globalFetchGate via the Processor. No-op when disabled // the persisted duration); the burst leads its chain with the stronger onboard
// (count 0) or queue-only (no Processor). // model. Hard cap; explicit, so it bypasses the recency window — but every
// fetch still goes through globalFetchGate. No-op when disabled (count 0) or
// queue-only (no processor).
//
// burstProcessor leads with the stronger model (ADR-028); it collapses onto the
// shared Processor when the onboard model is empty/equal-to-primary or the
// engine config is incomplete.
burstProcessor := app.Processor
if burstEngine, berr := buildBurstProcessor(cfg, st); berr != nil {
return berr
} else if burstEngine != nil {
burstProcessor = &engineProcessor{engine: burstEngine, store: st}
log.Info("onboarding burst uses a stronger model", "onboard_model", cfg.OnboardSummarizerModel)
}
onboard := func(ctx context.Context, userID string) { onboard := func(ctx context.Context, userID string) {
if cfg.OnboardSummarizeCount <= 0 || app.Processor == nil { if cfg.OnboardSummarizeCount <= 0 || burstProcessor == nil {
return return
} }
ids, err := st.NewestUnsummarizedVideoIDs(ctx, userID, cfg.OnboardSummarizeCount) ids, err := st.OnboardBurstVideoIDs(ctx, userID, cfg.OnboardSummarizeCount, cfg.MinVideoSeconds, cfg.OnboardMaxVideoSeconds)
if err != nil { if err != nil {
log.Warn("onboarding: list newest unsummarized", "user", userID, "err", err) log.Warn("onboarding: list burst candidates", "user", userID, "err", err)
return return
} }
for _, id := range ids { for _, id := range ids {
if err := app.Processor.ProcessVideo(ctx, userID, id); err != nil { if err := burstProcessor.ProcessVideo(ctx, userID, id); err != nil {
log.Warn("onboarding: summarize", "user", userID, "video", id, "err", err) log.Warn("onboarding: summarize", "user", userID, "video", id, "err", err)
} }
} }
@@ -287,16 +331,34 @@ func cmdServe(ctx context.Context, log *slog.Logger) error {
srv := &http.Server{ srv := &http.Server{
Addr: cfg.HTTPAddr, Addr: cfg.HTTPAddr,
Handler: app.Router(), Handler: metrics.HTTPMiddleware(app.Router()),
ReadHeaderTimeout: 10 * time.Second, ReadHeaderTimeout: 10 * time.Second,
} }
// Prometheus /metrics on a SEPARATE port (ADR-030) — never on the public app
// mux, so a scrape is in-cluster only. Empty TAPIR_METRICS_ADDR disables it.
var metricsSrv *http.Server
if cfg.MetricsAddr != "" {
mmux := http.NewServeMux()
mmux.Handle("GET /metrics", metrics.Handler())
metricsSrv = &http.Server{Addr: cfg.MetricsAddr, Handler: mmux, ReadHeaderTimeout: 10 * time.Second}
go func() {
log.Info("serving metrics", "addr", cfg.MetricsAddr)
if err := metricsSrv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Error("metrics server", "err", err)
}
}()
}
// Graceful shutdown on signal: stop accepting, drain in-flight requests. // Graceful shutdown on signal: stop accepting, drain in-flight requests.
go func() { go func() {
<-ctx.Done() <-ctx.Done()
shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel() defer cancel()
_ = srv.Shutdown(shutdownCtx) _ = srv.Shutdown(shutdownCtx)
if metricsSrv != nil {
_ = metricsSrv.Shutdown(shutdownCtx)
}
}() }()
log.Info("serving web ui", "addr", cfg.HTTPAddr, "user", cfg.UserID) log.Info("serving web ui", "addr", cfg.HTTPAddr, "user", cfg.UserID)
+194 -21
View File
@@ -3,17 +3,20 @@ package main
import ( import (
"context" "context"
"fmt" "fmt"
"strings"
"gitea.d-ma.be/mathias/tapir/internal/adapters/llm" "git.d-ma.be/mathias/tapir/internal/adapters/chat"
"gitea.d-ma.be/mathias/tapir/internal/adapters/secrets" "git.d-ma.be/mathias/tapir/internal/adapters/llm"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/secrets"
"gitea.d-ma.be/mathias/tapir/internal/adapters/summarizer" "git.d-ma.be/mathias/tapir/internal/adapters/store"
"gitea.d-ma.be/mathias/tapir/internal/adapters/youtube" "git.d-ma.be/mathias/tapir/internal/adapters/summarizer"
"gitea.d-ma.be/mathias/tapir/internal/config" "git.d-ma.be/mathias/tapir/internal/adapters/youtube"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/config"
"gitea.d-ma.be/mathias/tapir/internal/ports" "git.d-ma.be/mathias/tapir/internal/domain"
"gitea.d-ma.be/mathias/tapir/internal/usecase" "git.d-ma.be/mathias/tapir/internal/metrics"
"gitea.d-ma.be/mathias/tapir/internal/web" "git.d-ma.be/mathias/tapir/internal/ports"
"git.d-ma.be/mathias/tapir/internal/usecase"
"git.d-ma.be/mathias/tapir/internal/web"
) )
// videoFetcher adapts the YouTube adapter to web.VideoFetcher for the paste flow // videoFetcher adapts the YouTube adapter to web.VideoFetcher for the paste flow
@@ -42,28 +45,172 @@ func (f videoFetcher) FetchVideo(ctx context.Context, userID, videoID string) (d
// queue-only fallback: the web UI keeps working (the button just queues) and // queue-only fallback: the web UI keeps working (the button just queues) and
// `tapir run` reports the gap via its own ValidateForRun. Missing engine config // `tapir run` reports the gap via its own ValidateForRun. Missing engine config
// is never an error here. // is never an error here.
// buildSummarizer wires the summarization endpoint chain (ADR-022) shared by the
// web "Summarize now" path and the scheduler's per-user runners. The chain is:
// primary (local, fast) → local fallback → cloud fallback (worst case). Each
// endpoint reaches the same LiteLLM gateway with a different model alias — the
// gateway fronts both llama-swap and berget — so a fallback is just a different
// alias, not a second client config. Empty model entries are skipped, so a
// client deployment can set the cloud fallback empty to keep content local.
func buildSummarizer(cfg config.Config) *summarizer.Summarizer {
mk := summarizerEndpoint(cfg)
eps := []summarizer.Endpoint{mk(cfg.SummarizerModel)}
if cfg.FallbackModel != "" && cfg.FallbackModel != cfg.SummarizerModel {
eps = append(eps, mk(cfg.FallbackModel))
}
if cfg.CloudFallbackModel != "" && cfg.CloudFallbackModel != cfg.SummarizerModel {
eps = append(eps, mk(cfg.CloudFallbackModel))
}
return summarizer.NewChain(eps, cfg.MaxTranscriptChars)
}
// summarizerEndpoint returns a constructor for a chain endpoint over the one
// LiteLLM gateway, varying only the model alias (the gateway fronts both
// llama-swap and berget). Shared by the standard and burst chains.
func summarizerEndpoint(cfg config.Config) func(model string) summarizer.Endpoint {
return func(model string) summarizer.Endpoint {
return summarizer.Endpoint{
Client: llm.New(cfg.GatewayURL, cfg.GatewayKey, model, cfg.SummarizerTimeout, llm.WithMaxTokens(cfg.SummaryMaxTokens), llm.WithUsageHook(metrics.RecordTokens)),
Provider: providerOf(model),
Model: model,
}
}
}
// burstChainModels is the ordered, deduped model list for the onboarding burst
// (ADR-028): the stronger onboard model leads, then the standard ADR-022 chain
// (primary → local fallback → cloud) follows as resilience. Empty entries are
// dropped and duplicates collapsed, so the NDA lever (empty cloud fallback) keeps
// the burst chain fully local exactly as the standard chain does.
func burstChainModels(cfg config.Config) []string {
var models []string
add := func(m string) {
if m == "" {
return
}
for _, e := range models {
if e == m {
return
}
}
models = append(models, m)
}
add(cfg.OnboardSummarizerModel)
add(cfg.SummarizerModel)
add(cfg.FallbackModel)
add(cfg.CloudFallbackModel)
return models
}
// buildBurstSummarizer builds the onboarding-burst summarizer chain (ADR-028):
// the onboard model first, then the standard chain as fallback, deduped.
func buildBurstSummarizer(cfg config.Config) *summarizer.Summarizer {
mk := summarizerEndpoint(cfg)
var eps []summarizer.Endpoint
for _, m := range burstChainModels(cfg) {
eps = append(eps, mk(m))
}
return summarizer.NewChain(eps, cfg.MaxTranscriptChars)
}
// chatModels is the ordered, local-first set of models offered in the chat
// switcher (ADR-027), reusing the ADR-022 chain: primary → local fallback →
// cloud. Empty entries are dropped and duplicates collapsed, so a client/NDA
// deployment that sets the cloud fallback empty simply has no external model in
// the switcher — the same local-first lever the summarizer honours.
func chatModels(cfg config.Config) []string {
var models []string
add := func(m string) {
if m == "" {
return
}
for _, e := range models {
if e == m {
return
}
}
models = append(models, m)
}
add(cfg.SummarizerModel)
add(cfg.FallbackModel)
add(cfg.CloudFallbackModel)
return models
}
// buildChat wires the per-video chat service (ADR-027): a Completer factory over
// the SAME LiteLLM gateway the summarizer uses (a different alias per model, not a
// second client config) and the same transcript-truncation budget. It returns nil
// when no gateway is configured — chat is simply not mounted, the read path is
// unaffected. It deliberately takes NO YouTube source: chat is stored-only.
func buildChat(cfg config.Config) *chat.Service {
if cfg.GatewayURL == "" {
return nil
}
models := chatModels(cfg)
if len(models) == 0 {
return nil
}
newClient := func(model string) chat.Completer {
return llm.New(cfg.GatewayURL, cfg.GatewayKey, model, cfg.SummarizerTimeout, llm.WithMaxTokens(cfg.SummaryMaxTokens), llm.WithUsageHook(metrics.RecordTokens))
}
return chat.New(newClient, models, cfg.MaxTranscriptChars)
}
// providerOf maps a model alias to the domain AIProvider recorded on summaries.
// A "berget/" alias is an external provider; everything else is the local stack.
func providerOf(model string) string {
if strings.HasPrefix(model, "berget/") {
return "berget"
}
return "local"
}
func buildProcessor(cfg config.Config, st *store.Store) (*usecase.Engine, error) { func buildProcessor(cfg config.Config, st *store.Store) (*usecase.Engine, error) {
if cfg.GatewayURL == "" || cfg.YTClientID == "" || cfg.YTClientSecret == "" || cfg.SecretsFile == "" { if cfg.GatewayURL == "" || cfg.YTClientID == "" || cfg.YTClientSecret == "" || cfg.SecretsFile == "" {
return nil, nil return nil, nil
} }
secretStore := secrets.NewFileStore(cfg.SecretsFile) src := newYouTubeSource(cfg, secrets.NewFileStore(cfg.SecretsFile))
src := youtube.New(youtube.Config{ sum := buildSummarizer(cfg)
// The store is both the summary sink and the shared transcript cache (ADR-021):
// the engine reads stored transcripts before any caption fetch and writes
// resolved ones back, so re-analysis never re-touches YouTube.
eng := usecase.NewEngine(src, sum, st)
eng.Transcripts = st
return eng, nil
}
// newYouTubeSource builds the captions-first VideoSource shared by the standard
// and burst processors — same per-process YouTube credentials and ADR-023 Shorts
// filter; only the summarizer chain differs between them.
func newYouTubeSource(cfg config.Config, secretStore ports.SecretStore) ports.VideoSource {
return youtube.New(youtube.Config{
ClientID: cfg.YTClientID, ClientID: cfg.YTClientID,
ClientSecret: cfg.YTClientSecret, ClientSecret: cfg.YTClientSecret,
TokenSecretRef: cfg.YTTokenRef, TokenSecretRef: cfg.YTTokenRef,
PreferredLanguages: []string{"en"}, PreferredLanguages: []string{"en"},
MinVideoSeconds: cfg.MinVideoSeconds,
}, secretStore) }, secretStore)
}
// Local Primary only; no BYO fallback for the demo (fallback nil). // buildBurstProcessor wires a processor whose summarizer leads with the stronger
primary := summarizer.Endpoint{ // onboard model (ADR-028), used only by the connect-time burst over the SAME
Client: llm.New(cfg.GatewayURL, cfg.GatewayKey, cfg.SummarizerModel, cfg.SummarizerTimeout), // store / transcript cache / sink — a wiring choice; the engine and ports are
Provider: "local", // unchanged. Returns (nil, nil) — the collapse lever — when the onboard model is
Model: cfg.SummarizerModel, // empty or equal to the primary (the burst then reuses the shared processor), or
// when the engine config is incomplete (queue-only, same as buildProcessor).
func buildBurstProcessor(cfg config.Config, st *store.Store) (*usecase.Engine, error) {
if cfg.OnboardSummarizerModel == "" || cfg.OnboardSummarizerModel == cfg.SummarizerModel {
return nil, nil
} }
sum := summarizer.New(primary, nil) if cfg.GatewayURL == "" || cfg.YTClientID == "" || cfg.YTClientSecret == "" || cfg.SecretsFile == "" {
return nil, nil
return usecase.NewEngine(src, sum, st), nil }
src := newYouTubeSource(cfg, secrets.NewFileStore(cfg.SecretsFile))
eng := usecase.NewEngine(src, buildBurstSummarizer(cfg), st)
eng.Transcripts = st
return eng, nil
} }
// engineProcessor adapts the engine (which works in terms of a domain.Video) to // engineProcessor adapts the engine (which works in terms of a domain.Video) to
@@ -78,6 +225,11 @@ type engineProcessor struct {
} }
func (p *engineProcessor) ProcessVideo(ctx context.Context, userID, videoID string) error { func (p *engineProcessor) ProcessVideo(ctx context.Context, userID, videoID string) error {
// This is the user-initiated (foreground) path — a click on "Summarize",
// "Try now", or a pasted URL. Mark the context so the caption gate gives it
// priority over the background sweep (ADR-026, Pillar A).
ctx = youtube.ForegroundContext(ctx)
row, err := p.store.GetVideoRow(ctx, userID, videoID) row, err := p.store.GetVideoRow(ctx, userID, videoID)
if err != nil { if err != nil {
return fmt.Errorf("load video %q: %w", videoID, err) return fmt.Errorf("load video %q: %w", videoID, err)
@@ -97,7 +249,28 @@ func (p *engineProcessor) ProcessVideo(ctx context.Context, userID, videoID stri
if err != nil { if err != nil {
return fmt.Errorf("process video %q: %w", videoID, err) return fmt.Errorf("process video %q: %w", videoID, err)
} }
if res.Summary != nil {
// Record the outcome so the status endpoint can show honest state (ADR-025):
// a 429'd or caption-less click used to leave transcript_status unset, so the
// poll silently reverted to the "Summarize" button. Mirror the runner: stamp
// rate_limited / none / fetched. A rate-limited video keeps its requested flag
// so the background sweep retries it; none and fetched are terminal here.
switch {
case res.Skipped && res.TranscriptSource == string(domain.SourceRateLimited):
if err := p.store.SetTranscriptStatus(ctx, userID, videoID, "rate_limited"); err != nil {
return fmt.Errorf("set rate_limited status %q: %w", videoID, err)
}
case res.Skipped:
if err := p.store.SetTranscriptStatus(ctx, userID, videoID, "none"); err != nil {
return fmt.Errorf("set none status %q: %w", videoID, err)
}
if err := p.store.ClearSummarizeRequested(ctx, userID, videoID); err != nil {
return fmt.Errorf("clear summarize flag %q: %w", videoID, err)
}
case res.Summary != nil:
if err := p.store.SetTranscriptStatus(ctx, userID, videoID, "fetched"); err != nil {
return fmt.Errorf("set fetched status %q: %w", videoID, err)
}
if err := p.store.ClearSummarizeRequested(ctx, userID, videoID); err != nil { if err := p.store.ClearSummarizeRequested(ctx, userID, videoID); err != nil {
return fmt.Errorf("clear summarize flag %q: %w", videoID, err) return fmt.Errorf("clear summarize flag %q: %w", videoID, err)
} }
+63 -1
View File
@@ -3,7 +3,7 @@ package main
import ( import (
"testing" "testing"
"gitea.d-ma.be/mathias/tapir/internal/config" "git.d-ma.be/mathias/tapir/internal/config"
) )
// TestBuildProcessorNilOnIncompleteConfig asserts the queue-only fallback: when a // TestBuildProcessorNilOnIncompleteConfig asserts the queue-only fallback: when a
@@ -45,3 +45,65 @@ func TestBuildProcessorNilOnIncompleteConfig(t *testing.T) {
}) })
} }
} }
// TestBurstChainModelsLeadsWithOnboardModel: the onboarding burst chain (ADR-028)
// leads with the stronger onboard model, then falls back through the standard
// ADR-022 chain (primary -> local fallback -> cloud), deduped.
func TestBurstChainModelsLeadsWithOnboardModel(t *testing.T) {
got := burstChainModels(config.Config{
OnboardSummarizerModel: "iguana/gemma4-26b",
SummarizerModel: "koala/phi4-mini",
FallbackModel: "iguana/gemma4-26b", // also the onboard model -> dedup
CloudFallbackModel: "berget/mistral-small",
})
want := []string{"iguana/gemma4-26b", "koala/phi4-mini", "berget/mistral-small"}
if len(got) != len(want) {
t.Fatalf("burstChainModels = %v, want %v", got, want)
}
for i := range want {
if got[i] != want[i] {
t.Fatalf("burstChainModels = %v, want %v", got, want)
}
}
}
// TestBurstChainModelsCloudAbsentWhenDisabled: the NDA lever holds for the burst
// too — empty cloud fallback keeps the burst chain fully local.
func TestBurstChainModelsCloudAbsentWhenDisabled(t *testing.T) {
got := burstChainModels(config.Config{
OnboardSummarizerModel: "iguana/gemma4-26b",
SummarizerModel: "koala/phi4-mini",
CloudFallbackModel: "",
})
for _, m := range got {
if m == "" || m == "berget/mistral-small" {
t.Fatalf("cloud model leaked into burst chain: %v", got)
}
}
}
// TestBuildBurstProcessorNilWhenCollapsed: an empty or primary-equal onboard model
// collapses the burst onto the shared processor (buildBurstProcessor returns nil).
func TestBuildBurstProcessorNilWhenCollapsed(t *testing.T) {
base := config.Config{
GatewayURL: "http://gw/v1",
YTClientID: "id",
YTClientSecret: "secret",
SecretsFile: "/tmp/secrets.json",
SummarizerModel: "koala/phi4-mini",
}
t.Run("empty onboard model", func(t *testing.T) {
base.OnboardSummarizerModel = ""
eng, err := buildBurstProcessor(base, nil)
if err != nil || eng != nil {
t.Fatalf("buildBurstProcessor = (%v, %v), want (nil, nil)", eng, err)
}
})
t.Run("onboard model equals primary", func(t *testing.T) {
base.OnboardSummarizerModel = "koala/phi4-mini"
eng, err := buildBurstProcessor(base, nil)
if err != nil || eng != nil {
t.Fatalf("buildBurstProcessor = (%v, %v), want (nil, nil)", eng, err)
}
})
}
+34 -4
View File
@@ -6,14 +6,36 @@ import (
"io" "io"
"os" "os"
"text/tabwriter" "text/tabwriter"
"time"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
// gateThreshold is the Stage-0 gate (VISION/ADR-016): usage in >= 2 distinct // gateThreshold is the Stage-0 gate (VISION/ADR-016): usage in >= 2 distinct
// weeks. The gate passes when any user reaches it. // weeks. The gate passes when any user reaches it.
const gateThreshold = 2 const gateThreshold = 2
// defaultGateStart is the date Stage-0 return-usage tracking begins: the morning
// the pilot was actually unblocked and summaries started flowing (2026-06-11).
// Activity before this — testing, the period the pilot was stuck on zero — is
// noise and must not count toward the gate. Override with TAPIR_USAGE_GATE_START
// (YYYY-MM-DD). The gate measures whether users RETURN once it genuinely works.
const defaultGateStart = "2026-06-11"
// gateStart resolves the baseline date from TAPIR_USAGE_GATE_START or the default,
// parsed as a UTC calendar day.
func gateStart() (time.Time, error) {
v := os.Getenv("TAPIR_USAGE_GATE_START")
if v == "" {
v = defaultGateStart
}
t, err := time.Parse("2006-01-02", v)
if err != nil {
return time.Time{}, fmt.Errorf("TAPIR_USAGE_GATE_START=%q: want YYYY-MM-DD: %w", v, err)
}
return t, nil
}
// runReport prints the Stage-0 usage gate: per-user distinct active weeks (reads // runReport prints the Stage-0 usage gate: per-user distinct active weeks (reads
// UNION acts) and the pass/fail verdict. Read-only, cross-user — needs only // UNION acts) and the pass/fail verdict. Read-only, cross-user — needs only
// TAPIR_DB_DSN (not TAPIR_USER_ID; the report enumerates all users itself). // TAPIR_DB_DSN (not TAPIR_USER_ID; the report enumerates all users itself).
@@ -28,16 +50,24 @@ func runReport(ctx context.Context, _ []string) error {
} }
defer s.Close() defer s.Close()
rows, err := s.ActiveWeeks(ctx) since, err := gateStart()
if err != nil { if err != nil {
return err return err
} }
return formatReport(os.Stdout, rows)
rows, err := s.ActiveWeeks(ctx, since)
if err != nil {
return err
}
return formatReport(os.Stdout, rows, since)
} }
// formatReport renders the per-user week counts and the gate verdict. Pure: no DB, // formatReport renders the per-user week counts and the gate verdict. Pure: no DB,
// no env — so the layout and verdict logic are unit-testable without Postgres. // no env — so the layout and verdict logic are unit-testable without Postgres.
func formatReport(w io.Writer, rows []store.UserActiveWeeks) error { func formatReport(w io.Writer, rows []store.UserActiveWeeks, since time.Time) error {
if _, err := fmt.Fprintf(w, "Counting usage since %s (Stage-0 gate baseline)\n\n", since.Format("2006-01-02")); err != nil {
return err
}
if len(rows) == 0 { if len(rows) == 0 {
_, err := fmt.Fprintln(w, "no users yet") _, err := fmt.Fprintln(w, "no users yet")
return err return err
+8 -4
View File
@@ -3,12 +3,15 @@ package main
import ( import (
"strings" "strings"
"testing" "testing"
"time"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
var testSince = time.Date(2026, 6, 11, 0, 0, 0, 0, time.UTC)
func TestFormatReportColumnsAndGatePass(t *testing.T) { func TestFormatReportColumnsAndGatePass(t *testing.T) {
rows := []store.UserActiveWeeks{ rows := []store.UserActiveWeeks{
{UserID: "user-a", DisplayName: "Ada", ActiveWeeks: 3}, {UserID: "user-a", DisplayName: "Ada", ActiveWeeks: 3},
@@ -16,9 +19,10 @@ func TestFormatReportColumnsAndGatePass(t *testing.T) {
} }
var b strings.Builder var b strings.Builder
require.NoError(t, formatReport(&b, rows)) require.NoError(t, formatReport(&b, rows, testSince))
out := b.String() out := b.String()
require.Contains(t, out, "since 2026-06-11", "report states the gate baseline date")
require.Contains(t, out, "USER") require.Contains(t, out, "USER")
require.Contains(t, out, "ACTIVE_WEEKS") require.Contains(t, out, "ACTIVE_WEEKS")
require.Contains(t, out, "Ada") require.Contains(t, out, "Ada")
@@ -34,12 +38,12 @@ func TestFormatReportGateNotMet(t *testing.T) {
rows := []store.UserActiveWeeks{{UserID: "user-a", ActiveWeeks: 1}} rows := []store.UserActiveWeeks{{UserID: "user-a", ActiveWeeks: 1}}
var b strings.Builder var b strings.Builder
require.NoError(t, formatReport(&b, rows)) require.NoError(t, formatReport(&b, rows, testSince))
require.Contains(t, b.String(), "NOT YET MET", "no user at >= 2 weeks fails the gate") require.Contains(t, b.String(), "NOT YET MET", "no user at >= 2 weeks fails the gate")
} }
func TestFormatReportEmpty(t *testing.T) { func TestFormatReportEmpty(t *testing.T) {
var b strings.Builder var b strings.Builder
require.NoError(t, formatReport(&b, nil)) require.NoError(t, formatReport(&b, nil, testSince))
require.Contains(t, b.String(), "no users yet") require.Contains(t, b.String(), "no users yet")
} }
+76 -24
View File
@@ -6,15 +6,13 @@ import (
"log/slog" "log/slog"
"time" "time"
"gitea.d-ma.be/mathias/tapir/internal/adapters/llm" "git.d-ma.be/mathias/tapir/internal/adapters/store"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/youtube"
"gitea.d-ma.be/mathias/tapir/internal/adapters/summarizer" "git.d-ma.be/mathias/tapir/internal/config"
"gitea.d-ma.be/mathias/tapir/internal/adapters/youtube" "git.d-ma.be/mathias/tapir/internal/ports"
"gitea.d-ma.be/mathias/tapir/internal/config" "git.d-ma.be/mathias/tapir/internal/runner"
"gitea.d-ma.be/mathias/tapir/internal/ports" "git.d-ma.be/mathias/tapir/internal/usecase"
"gitea.d-ma.be/mathias/tapir/internal/runner" "git.d-ma.be/mathias/tapir/internal/web"
"gitea.d-ma.be/mathias/tapir/internal/usecase"
"gitea.d-ma.be/mathias/tapir/internal/web"
) )
// buildUserRunner constructs a runner.Runner for one user, reusing the same // buildUserRunner constructs a runner.Runner for one user, reusing the same
@@ -35,18 +33,20 @@ func buildUserRunner(cfg config.Config, st *store.Store, secretStore ports.Secre
ClientSecret: cfg.YTClientSecret, ClientSecret: cfg.YTClientSecret,
TokenSecretRef: web.YouTubeTokenRef(userID), TokenSecretRef: web.YouTubeTokenRef(userID),
PreferredLanguages: []string{"en"}, PreferredLanguages: []string{"en"},
MinVideoSeconds: cfg.MinVideoSeconds,
}, secretStore) }, secretStore)
primary := summarizer.Endpoint{ engine := usecase.NewEngine(src, buildSummarizer(cfg), st)
Client: llm.New(cfg.GatewayURL, cfg.GatewayKey, cfg.SummarizerModel, cfg.SummarizerTimeout), // Share the transcript cache (ADR-021) on the scheduler path too — without
Provider: "local", // this every scheduled pass re-fetches transcripts it already had, burning the
Model: cfg.SummarizerModel, // scarce per-IP caption budget (ADR-014) and starving other users. The web
} // "Summarize now" path already sets this; the scheduler omitting it was a bug.
engine := usecase.NewEngine(src, summarizer.New(primary, nil), st) engine.Transcripts = st
return runner.New(src, st, engine, userID, log, return runner.New(src, st, engine, userID, log,
runner.WithBackoff(cfg.FetchBackoff), runner.WithBackoff(cfg.FetchBackoff),
runner.WithAutoWindow(cfg.AutoSummarizeWindow)), nil runner.WithAutoWindow(cfg.AutoSummarizeWindow),
runner.WithCaptionMemory(cfg.ChannelCaptionlessThreshold, cfg.ChannelCaptionlessWindow)), nil
} }
// userLister enumerates every registered user and reports a user's video // userLister enumerates every registered user and reports a user's video
@@ -64,6 +64,7 @@ type userLister interface {
// isolation). Returns the stats summed across users. // isolation). Returns the stats summed across users.
func runDiscoveryPass( func runDiscoveryPass(
ctx context.Context, ctx context.Context,
pass int,
lister userLister, lister userLister,
runUser func(context.Context, string) (runner.Stats, error), runUser func(context.Context, string) (runner.Stats, error),
log *slog.Logger, log *slog.Logger,
@@ -74,15 +75,17 @@ func runDiscoveryPass(
return runner.Stats{} return runner.Stats{}
} }
log.Info("scheduler: starting discovery pass", "users", len(users)) // Keep only users with a video connection. A pass for a connectionless user
var total runner.Stats // (e.g. a stale Dex-era orphan identity) only tries to resolve a token that
// was never minted, logging a spurious "ref not found" every tick. Filtering
// here — BEFORE rotation — also keeps fairness honest: rotation is over the
// users that actually consume the caption budget, so a dead identity can't eat
// a rotation slot and skew the lead share.
var connected []store.UserIdentity
for _, u := range users { for _, u := range users {
if ctx.Err() != nil { if ctx.Err() != nil {
break // shutting down: stop enumerating return runner.Stats{} // shutting down
} }
// Skip users with no video connection. A discovery pass for them only
// attempts to resolve a token that was never minted, logging a spurious
// "ref not found" every tick (e.g. stale Dex-era orphan identities).
conns, err := lister.ConnectionsForUser(ctx, u.UserID) conns, err := lister.ConnectionsForUser(ctx, u.UserID)
if err != nil { if err != nil {
log.Warn("scheduler: list connections failed", "user", u.UserID, "err", err) log.Warn("scheduler: list connections failed", "user", u.UserID, "err", err)
@@ -92,6 +95,22 @@ func runDiscoveryPass(
log.Debug("scheduler: skipping user with no video connections", "user", u.UserID) log.Debug("scheduler: skipping user with no video connections", "user", u.UserID)
continue continue
} }
connected = append(connected, u)
}
// Rotate who goes first each pass. Caption fetches share one per-egress-IP
// rate budget (ADR-014); whoever runs first each pass spends the pre-throttle
// window, so a FIXED order permanently starves whoever is last (a new pilot
// user got 0 fetches for 12h while the first-listed user got all of them).
// Rotation over the connected set gives each real user the lead in turn.
connected = rotateUsers(connected, pass)
log.Info("scheduler: starting discovery pass", "users", len(connected))
var total runner.Stats
for _, u := range connected {
if ctx.Err() != nil {
break // shutting down: stop enumerating
}
stats, err := runUser(ctx, u.UserID) stats, err := runUser(ctx, u.UserID)
total = sumStats(total, stats) total = sumStats(total, stats)
if err != nil { if err != nil {
@@ -103,6 +122,7 @@ func runDiscoveryPass(
"skipped_seen", total.SkippedSeen, "skipped_no_text", total.SkippedNoText, "skipped_seen", total.SkippedSeen, "skipped_no_text", total.SkippedNoText,
"skipped_manual", total.SkippedManual, "skipped_too_old", total.SkippedTooOld, "skipped_manual", total.SkippedManual, "skipped_too_old", total.SkippedTooOld,
"skipped_rate_limited", total.SkippedRateLimited, "skipped_rate_limited", total.SkippedRateLimited,
"skipped_no_caption_channel", total.SkippedNoCaptionChannel,
"channel_unavailable", total.ChannelUnavailable, "errors", total.Errors) "channel_unavailable", total.ChannelUnavailable, "errors", total.Errors)
return total return total
} }
@@ -123,7 +143,18 @@ func runScheduler(
return // disabled return // disabled
} }
runDiscoveryPass(ctx, lister, runUser, log) // Derive the rotation offset from wall-clock, NOT an in-memory counter. A
// counter reset to 0 on every pod restart always hands the lead to the
// first-listed user — so frequent deploys re-starve whoever is last (exactly
// what happened to the first pilot user during a deploy-heavy session). A
// time-based offset advances with real time and is identical across restarts,
// so the lead rotates fairly regardless of how often the pod bounces.
runPass := func() {
pass := int(time.Now().Unix() / int64(interval/time.Second))
runDiscoveryPass(ctx, pass, lister, runUser, log)
}
runPass()
ticker := time.NewTicker(interval) ticker := time.NewTicker(interval)
defer ticker.Stop() defer ticker.Stop()
@@ -132,11 +163,31 @@ func runScheduler(
case <-ctx.Done(): case <-ctx.Done():
return return
case <-ticker.C: case <-ticker.C:
runDiscoveryPass(ctx, lister, runUser, log) runPass()
} }
} }
} }
// rotateUsers left-rotates users by pass positions so a different user leads each
// pass. With n users, user i leads on every pass where pass ≡ i (mod n). A pass
// offset that is negative or exceeds n is normalised. Order within the rotation
// is otherwise preserved, so the set of users run is unchanged — only who is
// first (and thus wins the scarce caption-fetch budget) rotates.
func rotateUsers(users []store.UserIdentity, pass int) []store.UserIdentity {
n := len(users)
if n <= 1 {
return users
}
off := ((pass % n) + n) % n
if off == 0 {
return users
}
out := make([]store.UserIdentity, 0, n)
out = append(out, users[off:]...)
out = append(out, users[:off]...)
return out
}
// sumStats adds two passes' stats field-wise, so runDiscoveryPass can report a // sumStats adds two passes' stats field-wise, so runDiscoveryPass can report a
// per-tick aggregate across all users. // per-tick aggregate across all users.
func sumStats(a, b runner.Stats) runner.Stats { func sumStats(a, b runner.Stats) runner.Stats {
@@ -148,6 +199,7 @@ func sumStats(a, b runner.Stats) runner.Stats {
SkippedManual: a.SkippedManual + b.SkippedManual, SkippedManual: a.SkippedManual + b.SkippedManual,
SkippedTooOld: a.SkippedTooOld + b.SkippedTooOld, SkippedTooOld: a.SkippedTooOld + b.SkippedTooOld,
SkippedRateLimited: a.SkippedRateLimited + b.SkippedRateLimited, SkippedRateLimited: a.SkippedRateLimited + b.SkippedRateLimited,
SkippedNoCaptionChannel: a.SkippedNoCaptionChannel + b.SkippedNoCaptionChannel,
ChannelUnavailable: a.ChannelUnavailable + b.ChannelUnavailable, ChannelUnavailable: a.ChannelUnavailable + b.ChannelUnavailable,
Errors: a.Errors + b.Errors, Errors: a.Errors + b.Errors,
} }
+47 -6
View File
@@ -11,8 +11,8 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
"gitea.d-ma.be/mathias/tapir/internal/runner" "git.d-ma.be/mathias/tapir/internal/runner"
) )
func quietLog() *slog.Logger { func quietLog() *slog.Logger {
@@ -45,6 +45,7 @@ func (f fakeLister) ConnectionsForUser(_ context.Context, userID string) ([]stor
type countingRunUser struct { type countingRunUser struct {
mu sync.Mutex mu sync.Mutex
calls map[string]int calls map[string]int
order []string // userIDs in the order they were run, across all passes
failFor map[string]bool failFor map[string]bool
} }
@@ -60,12 +61,19 @@ func (c *countingRunUser) run(_ context.Context, userID string) (runner.Stats, e
c.mu.Lock() c.mu.Lock()
defer c.mu.Unlock() defer c.mu.Unlock()
c.calls[userID]++ c.calls[userID]++
c.order = append(c.order, userID)
if c.failFor[userID] { if c.failFor[userID] {
return runner.Stats{Errors: 1}, errors.New("boom") return runner.Stats{Errors: 1}, errors.New("boom")
} }
return runner.Stats{Summarized: 1}, nil return runner.Stats{Summarized: 1}, nil
} }
func (c *countingRunUser) runOrder() []string {
c.mu.Lock()
defer c.mu.Unlock()
return append([]string(nil), c.order...)
}
func (c *countingRunUser) count(userID string) int { func (c *countingRunUser) count(userID string) int {
c.mu.Lock() c.mu.Lock()
defer c.mu.Unlock() defer c.mu.Unlock()
@@ -94,7 +102,7 @@ func TestDiscoveryPassRunsEveryUserOnce(t *testing.T) {
lister := fakeLister{users: usersN("a", "b", "c")} lister := fakeLister{users: usersN("a", "b", "c")}
rc := newCountingRunUser() rc := newCountingRunUser()
stats := runDiscoveryPass(context.Background(), lister, rc.run, quietLog()) stats := runDiscoveryPass(context.Background(), 0, lister, rc.run, quietLog())
require.Equal(t, 1, rc.count("a")) require.Equal(t, 1, rc.count("a"))
require.Equal(t, 1, rc.count("b")) require.Equal(t, 1, rc.count("b"))
@@ -102,13 +110,46 @@ func TestDiscoveryPassRunsEveryUserOnce(t *testing.T) {
require.Equal(t, 3, stats.Summarized, "stats are summed across users") require.Equal(t, 3, stats.Summarized, "stats are summed across users")
} }
// Caption fetches share one per-IP budget; a fixed user order starves whoever is
// last. Each pass must rotate which user leads so the lead slot is shared.
func TestDiscoveryPassRotatesLeadUser(t *testing.T) {
lister := fakeLister{users: usersN("a", "b", "c")}
rc := newCountingRunUser()
runDiscoveryPass(context.Background(), 0, lister, rc.run, quietLog())
runDiscoveryPass(context.Background(), 1, lister, rc.run, quietLog())
runDiscoveryPass(context.Background(), 2, lister, rc.run, quietLog())
require.Equal(t, []string{"a", "b", "c", "b", "c", "a", "c", "a", "b"}, rc.runOrder(),
"each pass left-rotates the user order so every user leads in turn")
// Fairness: over a full rotation cycle every user ran the same number of times.
require.Equal(t, 3, rc.count("a"))
require.Equal(t, 3, rc.count("b"))
require.Equal(t, 3, rc.count("c"))
}
// A connectionless orphan must not consume a rotation slot: rotation is over the
// connected users only, so two real users alternate the lead 50/50 even with a
// dead identity listed between them.
func TestDiscoveryPassRotationIgnoresConnectionlessUsers(t *testing.T) {
lister := fakeLister{users: usersN("a", "orphan", "c"), noConn: map[string]bool{"orphan": true}}
rc := newCountingRunUser()
runDiscoveryPass(context.Background(), 0, lister, rc.run, quietLog())
runDiscoveryPass(context.Background(), 1, lister, rc.run, quietLog())
require.Equal(t, []string{"a", "c", "c", "a"}, rc.runOrder(),
"only connected users rotate; the orphan never runs and never holds a slot")
require.Equal(t, 0, rc.count("orphan"))
}
func TestDiscoveryPassSkipsUsersWithoutConnections(t *testing.T) { func TestDiscoveryPassSkipsUsersWithoutConnections(t *testing.T) {
// b never connected a video source (e.g. a stale Dex-era orphan identity). // b never connected a video source (e.g. a stale Dex-era orphan identity).
// It must be skipped silently — not run and logged as a token error every pass. // It must be skipped silently — not run and logged as a token error every pass.
lister := fakeLister{users: usersN("a", "b", "c"), noConn: map[string]bool{"b": true}} lister := fakeLister{users: usersN("a", "b", "c"), noConn: map[string]bool{"b": true}}
rc := newCountingRunUser() rc := newCountingRunUser()
stats := runDiscoveryPass(context.Background(), lister, rc.run, quietLog()) stats := runDiscoveryPass(context.Background(), 0, lister, rc.run, quietLog())
require.Equal(t, 1, rc.count("a")) require.Equal(t, 1, rc.count("a"))
require.Equal(t, 0, rc.count("b"), "a user with no connection must be skipped, not run") require.Equal(t, 0, rc.count("b"), "a user with no connection must be skipped, not run")
@@ -121,7 +162,7 @@ func TestDiscoveryPassOneUserFailureDoesNotStopOthers(t *testing.T) {
lister := fakeLister{users: usersN("a", "b", "c")} lister := fakeLister{users: usersN("a", "b", "c")}
rc := newCountingRunUser("b") // user b's pass errors rc := newCountingRunUser("b") // user b's pass errors
stats := runDiscoveryPass(context.Background(), lister, rc.run, quietLog()) stats := runDiscoveryPass(context.Background(), 0, lister, rc.run, quietLog())
require.Equal(t, 1, rc.count("a")) require.Equal(t, 1, rc.count("a"))
require.Equal(t, 1, rc.count("b")) require.Equal(t, 1, rc.count("b"))
@@ -134,7 +175,7 @@ func TestDiscoveryPassListerErrorIsContained(t *testing.T) {
lister := fakeLister{err: errors.New("db down")} lister := fakeLister{err: errors.New("db down")}
rc := newCountingRunUser() rc := newCountingRunUser()
stats := runDiscoveryPass(context.Background(), lister, rc.run, quietLog()) stats := runDiscoveryPass(context.Background(), 0, lister, rc.run, quietLog())
require.Equal(t, 0, rc.total(), "no users enumerated → no passes") require.Equal(t, 0, rc.total(), "no users enumerated → no passes")
require.Equal(t, runner.Stats{}, stats) require.Equal(t, runner.Stats{}, stats)
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"os" "os"
"strings" "strings"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
// runShow prints the full summary for one video: text, highlights, takeaways, // runShow prints the full summary for one video: text, highlights, takeaways,
+130
View File
@@ -0,0 +1,130 @@
package main
import (
"context"
"fmt"
"log/slog"
"net/http"
"os"
"path/filepath"
"time"
"git.d-ma.be/mathias/tapir/internal/adapters/audio"
"git.d-ma.be/mathias/tapir/internal/adapters/llm"
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
"git.d-ma.be/mathias/tapir/internal/config"
"git.d-ma.be/mathias/tapir/internal/reconcile"
"git.d-ma.be/mathias/tapir/internal/uploads"
"git.d-ma.be/mathias/tapir/internal/web"
)
// uploadsStore is the store surface the upload wiring needs. *store.Store
// satisfies it.
type uploadsStore interface {
uploads.Store
uploads.WorkerStore
FailInterruptedUploads(ctx context.Context) (int, error)
}
// buildUploads wires audio upload and private transcription (ADR-033). It returns
// nils when uploads are disabled — never a typed-nil Intake, which would mount the
// /uploads routes over nothing. When enabled it, in order: prepares a private spool
// directory, deletes any audio a crashed run left in it, fails the rows such a run
// left queued or transcribing (the audio is not retained, so they can never
// resume), and only then builds the intake and the worker. The caller runs the
// worker. The fallback endpoint is a cloud service; an empty
// TAPIR_TRANSCRIBE_FALLBACK_MODEL keeps private audio on the homelab.
func buildUploads(ctx context.Context, cfg config.Config, st uploadsStore, log *slog.Logger) (*uploads.Intake, *uploads.Worker, error) {
if !cfg.UploadsEnabled {
return nil, nil, nil
}
if cfg.GatewayURL == "" {
return nil, nil, fmt.Errorf("uploads: TAPIR_UPLOADS_ENABLED is set but there is no TAPIR_GATEWAY_URL to transcribe through")
}
if err := os.MkdirAll(cfg.UploadTmpDir, 0o700); err != nil {
return nil, nil, fmt.Errorf("uploads: spool dir: %w", err)
}
removeSpooledAudio(cfg.UploadTmpDir, log)
n, err := st.FailInterruptedUploads(ctx)
if err != nil {
return nil, nil, fmt.Errorf("uploads: fail interrupted rows: %w", err)
}
if n > 0 {
log.Warn("uploads: marked interrupted by a restart (audio is not retained; they must be re-uploaded)", "count", n)
}
worker := uploads.NewWorker(st, &audio.FFmpeg{Bin: cfg.FFmpegBin}, buildTranscriber(cfg),
cfg.UploadTmpDir, cfg.TranscribeTimeout)
intake := uploads.NewIntake(st, &audio.FFprobe{Bin: cfg.FFprobeBin}, worker,
uploadLimits(cfg), cfg.UploadTmpDir)
return intake, worker, nil
}
func uploadLimits(cfg config.Config) uploads.Limits {
return uploads.Limits{MaxBytes: cfg.UploadMaxBytes, MaxDuration: cfg.UploadMaxDuration}
}
// removeSpooledAudio deletes the spool files this process family creates
// (upload-*, wav-*) and nothing else.
func removeSpooledAudio(dir string, log *slog.Logger) {
for _, pat := range []string{"upload-*", "wav-*"} {
matches, _ := filepath.Glob(filepath.Join(dir, pat))
for _, m := range matches {
if err := os.Remove(m); err != nil {
log.Warn("uploads: remove leftover audio", "path", m, "err", err)
}
}
}
}
// wireUploads connects a built Intake to the web App: the routes, the limits the
// form states, and the chat settings for an uploaded recording. A nil intake
// leaves the App untouched — assigning a nil *Intake to the Uploader interface
// would produce a non-nil interface and mount the routes over nothing.
func wireUploads(app *web.App, cfg config.Config, in *uploads.Intake) {
if in == nil {
return
}
app.Uploads = in
app.UploadLimits = uploadLimits(cfg)
app.UploadChatMaxChars = cfg.MaxUploadedTranscriptChars
app.UploadChatModel = cfg.UploadChatModel
}
// adjudicatorTimeout bounds one adjudication call. A call takes ~2 s on gemma4-26b and ~10 s
// on 31b; this is the wait before an unresponsive model is counted as an error.
const adjudicatorTimeout = 2 * time.Minute
// buildTranscriber builds what the worker transcribes with. By default that is the
// single-engine chain (the primary, then the optional cloud fallback). When
// TAPIR_RECONCILE_BASE_MODEL is set it is the two-engine ensemble (ADR-034): the base
// engine and the primary run on the same WAV and a LOCAL model reconciles them. Config
// has already refused any non-local model, so nothing here can send private audio or
// text to a cloud model for reconciliation; the cloud fallback is unchanged.
func buildTranscriber(cfg config.Config) uploads.Transcriber {
// No overall client timeout: a 90 minute recording takes minutes. The deadline is
// the worker's per-job context.
client := &http.Client{}
primary := transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, cfg.TranscribeModel, client)
var fallback transcribe.Backend
if cfg.TranscribeFallbackModel != "" && cfg.TranscribeFallbackModel != cfg.TranscribeModel {
fallback = transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, cfg.TranscribeFallbackModel, client)
}
if cfg.ReconcileBaseModel == "" {
return transcribe.NewChain(primary, fallback)
}
base := transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, cfg.ReconcileBaseModel, client)
// Temperature 0 and a short reply: the adjudicator's accuracy was measured that way.
chat := llm.New(cfg.GatewayURL, cfg.GatewayKey, cfg.ReconcileModel, adjudicatorTimeout,
llm.WithTemperature(0), llm.WithMaxTokens(200))
var adj reconcile.Adjudicator = reconcile.NewLLMAdjudicator(chat)
if cfg.ReconcileCheckOrder {
adj = reconcile.NewOrderChecked(adj)
}
return transcribe.NewEnsemble(base, primary, fallback, adj, reconcile.Options{
BaseEngine: cfg.ReconcileBaseModel, AltEngine: cfg.TranscribeModel, Reconciler: cfg.ReconcileModel,
})
}
+187
View File
@@ -0,0 +1,187 @@
package main
import (
"context"
"errors"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
"git.d-ma.be/mathias/tapir/internal/config"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/web"
)
// fakeUploadsStore satisfies the store surface buildUploads needs, and counts the
// start-up sweep.
type fakeUploadsStore struct {
sweeps int
sweepErr error
}
func (f *fakeUploadsStore) CreateUpload(context.Context, string, string, bool) (string, error) {
return "id", nil
}
func (f *fakeUploadsStore) FailUpload(context.Context, string, string, domain.UploadErrorKind) error {
return nil
}
func (f *fakeUploadsStore) StartTranscribing(context.Context, string, string) error { return nil }
func (f *fakeUploadsStore) CompleteUpload(context.Context, string, string, domain.Transcription) error {
return nil
}
func (f *fakeUploadsStore) FailInterruptedUploads(context.Context) (int, error) {
f.sweeps++
return 2, f.sweepErr
}
func uploadCfg(t *testing.T, enabled bool) config.Config {
t.Helper()
var cfg config.Config
cfg.GatewayURL = "http://gateway.invalid/v1"
cfg.UploadsEnabled = enabled
cfg.UploadMaxBytes = 200 << 20
cfg.UploadTmpDir = filepath.Join(t.TempDir(), "spool")
cfg.TranscribeModel = "iguana/kb-whisper"
cfg.TranscribeFallbackModel = "berget/kb-whisper"
return cfg
}
func TestBuildUploads_DisabledMountsNothingAndSweepsNothing(t *testing.T) {
st := &fakeUploadsStore{}
in, w, err := buildUploads(context.Background(), uploadCfg(t, false), st, quietLog())
require.NoError(t, err)
assert.Nil(t, in, "a typed-nil Intake must never reach web.App.Uploads: it would mount the routes over nothing")
assert.Nil(t, w)
assert.Zero(t, st.sweeps)
}
func TestBuildUploads_EnabledFailsInterruptedRowsBeforeAnythingElseStarts(t *testing.T) {
st := &fakeUploadsStore{}
in, w, err := buildUploads(context.Background(), uploadCfg(t, true), st, quietLog())
require.NoError(t, err)
require.NotNil(t, in)
require.NotNil(t, w)
assert.Equal(t, 1, st.sweeps,
"audio is not retained, so rows left queued/transcribing by a restart can never resume: fail them at start-up, not never")
}
func TestBuildUploads_SweepFailureStopsStartup(t *testing.T) {
st := &fakeUploadsStore{sweepErr: errors.New("db down")}
_, _, err := buildUploads(context.Background(), uploadCfg(t, true), st, quietLog())
require.Error(t, err)
assert.Contains(t, err.Error(), "db down")
}
func TestBuildUploads_CreatesTheSpoolDirectoryPrivately(t *testing.T) {
cfg := uploadCfg(t, true)
_, _, err := buildUploads(context.Background(), cfg, &fakeUploadsStore{}, quietLog())
require.NoError(t, err)
info, err := os.Stat(cfg.UploadTmpDir)
require.NoError(t, err)
assert.True(t, info.IsDir())
assert.Equal(t, os.FileMode(0o700), info.Mode().Perm(), "spooled audio is private third-party content")
}
func TestBuildUploads_EnabledWithoutAGatewayIsAnError(t *testing.T) {
cfg := uploadCfg(t, true)
cfg.GatewayURL = ""
_, _, err := buildUploads(context.Background(), cfg, &fakeUploadsStore{}, quietLog())
assert.Error(t, err, "uploads on with nowhere to transcribe must fail loudly at start-up, not at the first upload")
}
func TestBuildUploads_RemovesAudioLeftBehindByACrash(t *testing.T) {
cfg := uploadCfg(t, true)
require.NoError(t, os.MkdirAll(cfg.UploadTmpDir, 0o700))
leftovers := []string{"upload-123", "wav-456"}
for _, n := range leftovers {
require.NoError(t, os.WriteFile(filepath.Join(cfg.UploadTmpDir, n), []byte("audio"), 0o600))
}
unrelated := filepath.Join(cfg.UploadTmpDir, "someone-elses-file")
require.NoError(t, os.WriteFile(unrelated, []byte("keep"), 0o600))
_, _, err := buildUploads(context.Background(), cfg, &fakeUploadsStore{}, quietLog())
require.NoError(t, err)
for _, n := range leftovers {
_, err := os.Stat(filepath.Join(cfg.UploadTmpDir, n))
assert.True(t, os.IsNotExist(err), "%s: audio is never retained, so a spooled file from a crashed run must not survive start-up", n)
}
_, err = os.Stat(unrelated)
assert.NoError(t, err, "only our own spool files are removed")
}
func TestWireUploads_SetsEverythingTheWebAppNeedsForUploadsAndChat(t *testing.T) {
cfg := uploadCfg(t, true)
cfg.UploadMaxDuration = 90 * 60 * 1e9
cfg.MaxUploadedTranscriptChars = 90000
cfg.UploadChatModel = "iguana/gemma4-26b"
in, _, err := buildUploads(context.Background(), cfg, &fakeUploadsStore{}, quietLog())
require.NoError(t, err)
app := &web.App{}
wireUploads(app, cfg, in)
assert.NotNil(t, app.Uploads)
assert.Equal(t, cfg.UploadMaxBytes, app.UploadLimits.MaxBytes, "the form states the limits Intake enforces")
assert.Equal(t, cfg.UploadMaxDuration, app.UploadLimits.MaxDuration)
assert.Equal(t, 90000, app.UploadChatMaxChars,
"without this the chat would use the 18,000-char caption default and cut an hour to its first third")
assert.Equal(t, "iguana/gemma4-26b", app.UploadChatModel)
}
func TestWireUploads_NilIntakeLeavesTheAppUntouched(t *testing.T) {
app := &web.App{}
wireUploads(app, uploadCfg(t, false), nil)
assert.Nil(t, app.Uploads, "a nil *Intake must not become a non-nil interface that mounts the routes")
assert.Zero(t, app.UploadChatMaxChars)
}
func TestBuildTranscriber_DefaultIsTheSingleEngineChainAsBefore(t *testing.T) {
cfg := uploadCfg(t, true)
tr := buildTranscriber(cfg)
assert.IsType(t, &transcribe.Chain{}, tr, "with no base engine configured nothing changes")
}
func TestBuildTranscriber_ABaseEngineTurnsOnTheEnsemble(t *testing.T) {
cfg := uploadCfg(t, true)
cfg.ReconcileBaseModel = "iguana/pianissimo"
cfg.ReconcileModel = "iguana/gemma4-26b"
tr := buildTranscriber(cfg)
assert.IsType(t, &transcribe.Ensemble{}, tr)
}
func TestBuildUploads_WiresTheEnsembleWhenConfigured(t *testing.T) {
cfg := uploadCfg(t, true)
cfg.ReconcileBaseModel = "iguana/pianissimo"
cfg.ReconcileModel = "iguana/gemma4-26b"
in, w, err := buildUploads(context.Background(), cfg, &fakeUploadsStore{}, quietLog())
require.NoError(t, err)
assert.NotNil(t, in)
assert.NotNil(t, w)
}
+26
View File
@@ -250,6 +250,32 @@ After (newest-first): `[chanB-new, chanA-mid, chanA-old, chanB-null]`
The set of *processed* videos now also excludes auto-mode back-catalogue beyond the recency The set of *processed* videos now also excludes auto-mode back-catalogue beyond the recency
window (those stay listed, summarised on demand); within the processed set, only order changes. window (those stay listed, summarised on demand); within the processed set, only order changes.
### Connect-time onboarding burst (ADR-018 → ADR-028)
On a successful YouTube connect, `ConnectHandler` enqueues a connect-triggered discovery pass;
the `discoveryTrigger` runs that pass and then fires the **onboarding burst** — a third entry path
that summarises up to `TAPIR_ONBOARD_SUMMARIZE_COUNT` (default 3, hard-capped) of the new user's
videos so the first session is not empty. The burst still flows through `globalFetchGate` (it is
not a throughput change); ADR-028 sharpened *which* videos and *which model*:
- **Selection** is `OnboardBurstVideoIDs`, not pure newest-first. It keeps newest-first order but
excludes a video whose **known** duration is outside `[TAPIR_MIN_VIDEO_SECONDS,
TAPIR_ONBOARD_MAX_VIDEO_SECONDS]` (drops Shorts and multi-hour livestream VODs). An unknown
(NULL) duration is degrade-open — kept, but ranked after known-good rows. The connect-triggered
discovery pass runs *before* the burst, and ADR-023's `videos.list` enrichment now **persists**
`duration_s` (instead of discarding it after the Shorts filter), so a fresh user's candidates
carry a duration in time for selection.
- **Model**: the burst runs through a dedicated summarizer chain led by
`TAPIR_ONBOARD_SUMMARIZER_MODEL` (default `iguana/gemma4-26b`, the stronger local model), with
the standard ADR-022 chain following as fallback. This is a wiring choice — a second
`engineProcessor` over the same store / transcript cache / sink; the engine and ports are
unchanged. Empty / equal-to-primary collapses it back onto the shared processor.
`has-captions` is deliberately **not** a selection signal — it is only knowable after a gate fetch
(or a ~0-probability cache hit at pilot scale), so the burst can avoid known-junk but cannot
promise captions. Cached-transcript-first selection was investigated and rejected (ADR-028:
~3% cross-user overlap).
--- ---
## Sequence — core use case: new video summarized ## Sequence — core use case: new video summarized
+57 -15
View File
@@ -10,12 +10,20 @@ only opaque references to them; the secret material lives in ESO/1Password (ADR-
## Design decisions baked into this model ## Design decisions baked into this model
- **Per-user isolation, not a shared global video table.** The earlier draft proposed a - **Per-user isolation for everything except transcripts.** The earlier draft proposed a
global `videos`/`transcripts` table deduped across tenants. Rejected for Future B: it global `videos`/`transcripts` table deduped across tenants. **Videos** stay per-user and
reintroduces exactly the cross-domain coupling the homelab architecture review is RLS-scoped — a shared video table reintroduces exactly the cross-domain coupling the homelab
removing, and at 1–5 users the cost of occasionally re-summarizing the same video is architecture review is removing. **Transcripts**, however, are now shared (ADR-021): keyed by
trivial compared to the isolation it would cost. Each user's data is self-contained. `(provider, provider_video_id)`, no `user_id`, **not** RLS-scoped. The cost avoided there is
(Revisit only if Future C makes GPU/transcription cost dominate — a new ADR, not a default.) not LLM re-summarization but a rate-gated, reputation-risky caption fetch (ADR-010/014), which
is paid per re-fetch regardless of user count — so persisting public caption content once and
sharing it strictly beats the coupling it removes. Everything else each user owns is
self-contained; `rls_test.go` proves transcripts is the single exception for *content*.
**Uploaded-audio transcripts are the opposite of public caption content** (private, owned by
one user) and therefore live in their own RLS-scoped `uploaded_transcripts` table, never in
`transcripts` (ADR-033). The isolation test now also pins the **exact set** of tables with row
security off (`transcripts`, `user_identities`, `invitations`, `schema_migrations`), so a new
table added without RLS fails CI.
- **Secrets by reference only.** Tables hold a `secret_ref` (opaque string/UUID resolved via - **Secrets by reference only.** Tables hold a `secret_ref` (opaque string/UUID resolved via
the `SecretStore` port), never tokens or keys. the `SecretStore` port), never tokens or keys.
- **The brain sink is just a delivery target.** No brain-specific tables. Whether a summary - **The brain sink is just a delivery target.** No brain-specific tables. Whether a summary
@@ -34,10 +42,11 @@ erDiagram
USER ||--o{ AI_CREDENTIAL : "has (planned)" USER ||--o{ AI_CREDENTIAL : "has (planned)"
VIDEO_CONNECTION ||--o{ SUBSCRIPTION : "exposes (planned)" VIDEO_CONNECTION ||--o{ SUBSCRIPTION : "exposes (planned)"
SUBSCRIPTION ||--o{ VIDEO : "produces (per user)" SUBSCRIPTION ||--o{ VIDEO : "produces (per user)"
VIDEO ||--o| TRANSCRIPT : "has at most one" VIDEO }o--o| TRANSCRIPT : "shares one by (provider, provider_video_id) — not FK (ADR-021)"
VIDEO ||--o| SUMMARY : "has at most one" VIDEO ||--o| SUMMARY : "has at most one"
SUMMARY ||--o{ SINK_DELIVERY : "delivered via" SUMMARY ||--o{ SINK_DELIVERY : "delivered via"
USER ||--o{ CHANNEL_ERROR : "reports unavailable channels" USER ||--o{ CHANNEL_ERROR : "reports unavailable channels"
USER ||--o{ UPLOADED_TRANSCRIPT : "uploads (private, ADR-033)"
USER { USER {
uuid id PK uuid id PK
@@ -92,12 +101,12 @@ erDiagram
timestamptz rate_limited_at "backoff clock for 429 retries (migration 007)" timestamptz rate_limited_at "backoff clock for 429 retries (migration 007)"
} }
TRANSCRIPT { TRANSCRIPT {
uuid video_id PK_FK text provider PK "part of shared key (ADR-021)"
uuid user_id FK text provider_video_id PK "part of shared key — the cross-user dedup key"
text source "captions | none" text source "captions | none"
text language text language
text content "null when source = none" text content "null when source = none"
timestamptz resolved_at timestamptz fetched_at
} }
SUMMARY { SUMMARY {
uuid id PK uuid id PK
@@ -133,12 +142,29 @@ erDiagram
timestamptz first_seen timestamptz first_seen
timestamptz last_seen timestamptz last_seen
} }
UPLOADED_TRANSCRIPT {
uuid id PK
uuid user_id FK "-> USER, ON DELETE CASCADE, NOT NULL; RLS ENABLE+FORCE (migration 017)"
text title
text status "queued | transcribing | done | failed"
text content "nullable until done; the transcript text"
jsonb segments "nullable until done; [{start,end,text}] in seconds"
text endpoint "which backend served it: iguana/... | berget/... (berget = audio left the homelab)"
float audio_seconds
text error_kind "typed: interrupted | transcription_failed — never a provider string"
bool had_video "the file carried a video track; only its audio was used (migration 018)"
jsonb reconciliation "NULL for a single engine; else status, exact counts and a bounded list of contested spots (migration 019, ADR-034)"
timestamptz created_at
timestamptz finished_at
}
``` ```
`SUMMARY_ACTION` has `UNIQUE (user_id, video_id, action)`; `VIDEO_CONNECTION` has `SUMMARY_ACTION` has `UNIQUE (user_id, video_id, action)`; `VIDEO_CONNECTION` has
`UNIQUE (user_id, provider)` (one connection per provider — reconnect upserts in place). `UNIQUE (user_id, provider)` (one connection per provider — reconnect upserts in place).
RLS (`ENABLE` + `FORCE`) is on **every solid user-owned table above** — `users`, `videos`, RLS (`ENABLE` + `FORCE`) is on **every solid user-owned table above** — `users`, `videos`,
`transcripts`, `summaries`, `summary_actions`, `video_connections`, `channel_errors`. `summaries`, `summary_actions`, `video_connections`, `channel_errors`, `uploaded_transcripts`.
(`transcripts` is deliberately **not** in this list: since ADR-021 it is shared public caption
content with no `user_id`.)
`sink_deliveries` is RLS'd via an `EXISTS` on its parent summary; `user_identities` is `sink_deliveries` is RLS'd via an `EXISTS` on its parent summary; `user_identities` is
intentionally **not** RLS'd (auth plumbing). See the *Isolation invariant* section for the intentionally **not** RLS'd (auth plumbing). See the *Isolation invariant* section for the
mechanism. mechanism.
@@ -173,8 +199,12 @@ mechanism.
`transcript_status` and `rate_limited_at` (migration 007) track caption-fetch outcomes for `transcript_status` and `rate_limited_at` (migration 007) track caption-fetch outcomes for
rate-limit backoff: `NULL` = not attempted; `rate_limited` = 429 seen, skip until rate-limit backoff: `NULL` = not attempted; `rate_limited` = 429 seen, skip until
`NOW() - rate_limited_at > TAPIR_FETCH_BACKOFF`; `fetched` = resolved; `none` = no transcript. `NOW() - rate_limited_at > TAPIR_FETCH_BACKOFF`; `fetched` = resolved; `none` = no transcript.
- **TRANSCRIPT** — at most one per video. `source = none` records "checked, no usable - **TRANSCRIPT** — shared public caption content, one row per `(provider, provider_video_id)`,
transcript" so the watcher doesn't reprocess (ADR-007). `content` null in that case. **not** RLS-scoped and carrying no `user_id` (ADR-021). Two users who watch the same video
share the one row; the summarize path reads it before any caption fetch, so re-analysis never
re-touches YouTube (ADR-010/014). `source = none` records "checked, no usable transcript" so
no one reprocesses (ADR-007); `content` null in that case. A transient 429 is never stored
here — it stays a per-user retry via `VIDEO.transcript_status`.
- **SUMMARY** — at most one per video. `fallback_used` + `ai_provider`/`ai_model` make the - **SUMMARY** — at most one per video. `fallback_used` + `ai_provider`/`ai_model` make the
"is local good enough?" question queryable (the Stage 0 quality signal). `highlights`/ "is local good enough?" question queryable (the Stage 0 quality signal). `highlights`/
`takeaways` as jsonb to stay schema-flexible while the output format settles. `takeaways` as jsonb to stay schema-flexible while the output format settles.
@@ -190,6 +220,14 @@ mechanism.
the most recent discovery pass. Upserted per scheduler pass (`last_seen` refreshed each run); the most recent discovery pass. Upserted per scheduler pass (`last_seen` refreshed each run);
surfaced on the account page as a warning. Cascades on user deletion. Primary key is surfaced on the account page as a warning. Cascades on user deletion. Primary key is
`(user_id, channel_id)`. FORCE RLS'd. `(user_id, channel_id)`. FORCE RLS'd.
- **UPLOADED_TRANSCRIPTS** (migration 017, ADR-033) — the **private** transcript of audio a user
uploaded. Deliberately a separate table from the shared `transcripts`: that one is non-RLS on the
grounds that it holds only public caption content, and a recorded meeting is the opposite.
`user_id NOT NULL ... ON DELETE CASCADE`, FORCE RLS'd. `status` runs `queued → transcribing →
done | failed`; `done` requires `content`, `failed` requires a typed `error_kind` (CHECK
constraints). `reconciliation` (migration 019, ADR-034) is NULL when one engine produced the text, so the page can never claim a second opinion that was not there; otherwise it holds the status (`reconciled` / `single_engine` / `skipped`), exact counts (differences, settled automatically, decided by the model, changed, uncertain, model errors) and up to 200 contested spots (each a position in seconds, both readings and which was used). The spots hold fragments of the private transcript, which is why they live in this RLS-scoped row. `had_video` (migration 018) records that the file carried a video track — a Teams meeting with the camera off is still a video file — so the page can say only its audio was used and the video was discarded. `endpoint` records which backend served it — `berget/...` means the audio left the
homelab. The audio itself is never stored. `segments` keeps the timestamps (re-transcribing 90
minutes costs ~7). Unlike `summary_actions`/`login_events` it **does** cascade on user deletion.
- **LOGIN_EVENTS** (migration 010) — throttled one-row-per-(user, date) login stamp. Used by the - **LOGIN_EVENTS** (migration 010) — throttled one-row-per-(user, date) login stamp. Used by the
Stage-0 gate query (VISION §Stage 0: "returned and used in ≥2 distinct weeks"). Stage-0 gate query (VISION §Stage 0: "returned and used in ≥2 distinct weeks").
@@ -202,7 +240,10 @@ enforcement dormant); **ADR-012 opened Stage 1 and turned enforcement on in the
Enforcement is **Postgres Row-Level Security** (migration `003_rls.up.sql`): Enforcement is **Postgres Row-Level Security** (migration `003_rls.up.sql`):
- RLS is `ENABLE`d **and** `FORCE`d on every user-owned table — `users`, `videos`, - RLS is `ENABLE`d **and** `FORCE`d on every user-owned table — `users`, `videos`,
`transcripts`, `summaries`, `summary_actions`, `video_connections`, `channel_errors`. `summaries`, `summary_actions`, `video_connections`, `channel_errors`, and (migration 017)
`uploaded_transcripts`. `transcripts` is **not** one of them: ADR-021 made it shared public
content, and migration 003's original per-user `transcripts` policy was dropped with that
table in migration 015.
`FORCE` is load-bearing: the app connects as the table **owner** (`tapir` role), and owners `FORCE` is load-bearing: the app connects as the table **owner** (`tapir` role), and owners
bypass RLS unless forced. bypass RLS unless forced.
- Each policy keys off the per-request GUC `tapir.current_user_id`, set transaction-locally by - Each policy keys off the per-request GUC `tapir.current_user_id`, set transaction-locally by
@@ -231,7 +272,8 @@ queue, doesn't replace it). Deferred until there's a reason.
## Explicitly out of scope (Future C) ## Explicitly out of scope (Future C)
- Global cross-tenant video/transcript dedup (rejected above). - Global cross-tenant *video* dedup (rejected above). Note: cross-tenant *transcript* sharing
is now in scope and shipped (ADR-021); only the videos half stays per-user.
- Sharding / per-tenant physical databases. - Sharding / per-tenant physical databases.
- Soft-delete + full audit trail on connections/credentials (a Stage 2 hardening item; add - Soft-delete + full audit trail on connections/credentials (a Stage 2 hardening item; add
via ADR when Stage 2 work starts). via ADR when Stage 2 work starts).
+57
View File
@@ -12,6 +12,30 @@ it** — endpoints and aliases drift, and this file is a snapshot (2026-06-06),
- **Off-cluster (host / NodePort):** `http://koala:30401/v1/` — service `litellm-nodeport` (`4000:30401`). - **Off-cluster (host / NodePort):** `http://koala:30401/v1/` — service `litellm-nodeport` (`4000:30401`).
- NOTE: the earlier `31234` was **wrong** — `31234` is the `llama-swap` NodePort (`8080:31234`), - NOTE: the earlier `31234` was **wrong** — `31234` is the `llama-swap` NodePort (`8080:31234`),
not LiteLLM. Use `30401` for off-cluster LiteLLM. `confirm` resolved. not LiteLLM. Use `30401` for off-cluster LiteLLM. `confirm` resolved.
- **Speech-to-text (ADR-033):** the same gateway serves `POST /v1/audio/transcriptions` for
`iguana/kb-whisper` (primary, local) and `berget/kb-whisper` (fallback, EU cloud — private audio
leaves the homelab if it is used). **Call it via the NodePort / in-cluster address above, NOT
`https://llm-api.d-ma.be`:** that public ingress (openresty) returns `413` for bodies over
roughly 8–10 MB (~5 minutes of 16 kHz WAV), while the NodePort took 174 MB (90 minutes) in one
request (measured 2026-10-01, `docs/research/audio-length-ceiling-2026-10-01.md`). Send multipart
field `file`, `response_format=verbose_json`, no `Expect: 100-continue`. iguana's whisper-server
can wedge permanently after a Metal OOM (infra#273); never branch on its
`{"error":"failed to process audio"}` string — it has had at least five causes. Tapir's knobs:
`TAPIR_UPLOADS_ENABLED` (off by default), `TAPIR_UPLOAD_MAX_MB` (200), `TAPIR_UPLOAD_MAX_MINUTES`
(90), `TAPIR_TRANSCRIBE_MODEL`, `TAPIR_TRANSCRIBE_FALLBACK_MODEL` (empty disables the cloud
fallback), `TAPIR_TRANSCRIBE_TIMEOUT` (30m), `TAPIR_UPLOAD_TMP_DIR`, `TAPIR_FFMPEG_BIN`,
`TAPIR_FFPROBE_BIN`, `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS` (90000 — the measured ceiling of
`iguana/gemma4-26b`, which silently drops the head above it), `TAPIR_UPLOAD_CHAT_MODEL`
(`iguana/gemma4-26b`; `koala/phi4-mini`'s 8,192-token context 400s on an hour of speech). Needs `ffmpeg` + `ffprobe`
in the runtime image; Tapir's own ingress body limit for a 200 MB upload is unverified (infra).
**Two-engine mode (ADR-034), off unless `TAPIR_RECONCILE_BASE_MODEL` is set:** it runs
`iguana/pianissimo` (the base text) and `TAPIR_TRANSCRIBE_MODEL` (kb-whisper) on the same audio, then a
LOCAL chat model (`TAPIR_RECONCILE_MODEL`, e.g. `iguana/gemma4-26b`: 92% against a speaker's blind answers
at ~2 s a call; `iguana/gemma4-31b` 97% but ~5x slower) chooses between their disagreements.
`TAPIR_RECONCILE_CHECK_ORDER=true` asks twice per dispute with the order swapped (doubles the time).
Every model must start `koala/` or `iguana/`: a cloud alias is a start-up error. The default job timeout is
60 minutes. `iguana/pianissimo` is KlangAI's Swedish ASR (a Parakeet/FastConformer on MLX, served by a small
FastAPI shell on iguana:8102) and speaks the same `/v1/audio/transcriptions` API.
- **Auth key:** treat as a secret via the SecretStore port; do not hardcode in committed code. - **Auth key:** treat as a secret via the SecretStore port; do not hardcode in committed code.
NOTE: the documented `sk-local-123` returned `401` on `30401/v1/models` (2026-06-02) — it is NOTE: the documented `sk-local-123` returned `401` on `30401/v1/models` (2026-06-02) — it is
stale or the gateway now enforces the `LITELLM_MASTER_KEY` (1Password `op://HomeLab/LITELLM_MASTER_KEY`). stale or the gateway now enforces the `LITELLM_MASTER_KEY` (1Password `op://HomeLab/LITELLM_MASTER_KEY`).
@@ -27,6 +51,30 @@ it** — endpoints and aliases drift, and this file is a snapshot (2026-06-06),
`iguana/deepseek-r1-14b`) is preferred for summary quality if its latency/output is acceptable. `iguana/deepseek-r1-14b`) is preferred for summary quality if its latency/output is acceptable.
The `max_tokens` fix below means thinking models no longer return empty content, so they are now The `max_tokens` fix below means thinking models no longer return empty content, so they are now
viable choices, not blocked ones. Do not assume a coder alias is right for prose. viable choices, not blocked ones. Do not assume a coder alias is right for prose.
- **Summarizer fallback chain (ADR-022).** The primary alias is the *first* of an ordered chain;
on failure or unparseable output the summarizer advances to the next model. All reached through
the same gateway by alias.
- `TAPIR_FALLBACK_MODEL` — local fallback. **Default `iguana/gemma4-26b`** — on iguana, NOT
koala, so the fallback does not compete with koala's other GPU loads (and runs from a different
egress IP). Empty disables it.
- `TAPIR_CLOUD_FALLBACK_MODEL` — worst-case EXTERNAL fallback. **Default `berget/mistral-small`.**
**Set this empty (`""`) for any client/NDA deployment** so content never leaves the local
stack — the chain then contains only local endpoints.
- `TAPIR_SUMMARY_MAX_TOKENS` — per-summary completion budget. **Default `1500`.** Small on
purpose: with the old 8192 budget, prompt + completion overflowed `phi4-mini`'s 8k window.
- `TAPIR_MAX_TRANSCRIPT_CHARS` — transcript truncation budget sent to the model. **Default
`18000`** (~fits an 8k-context model). `0` disables truncation. Prevents the context-overflow
HTTP 400 a long transcript caused on `phi4-mini`.
- **Discovery low-value filter (ADR-023).** `TAPIR_MIN_VIDEO_SECONDS` — **default `60`**. At
discovery, `NewVideos` enriches candidates with one cheap `videos.list` call (quota API, NOT
the timedtext 429 path) and drops videos shorter than this plus any live/upcoming broadcast,
so the scarce caption-fetch budget isn't spent on Shorts. `0` disables the filter. The
paste-a-URL path is never filtered.
- **Per-channel caption memory (ADR-024).** `TAPIR_CHANNEL_CAPTIONLESS_THRESHOLD` — **default
`5`** consecutive no-caption results before a channel is suppressed (its videos listed but not
caption-fetched). `TAPIR_CHANNEL_CAPTIONLESS_WINDOW` — **default `336h`** (14d) suppression
before one video is re-probed. `THRESHOLD=0` disables. A successful fetch resets the channel;
a 429 does not count; an explicit manual request bypasses suppression.
- **Thinking models need an explicit `max_tokens`.** qwen3 / deepseek-r1 spend the budget on - **Thinking models need an explicit `max_tokens`.** qwen3 / deepseek-r1 spend the budget on
reasoning and return **empty content** if `max_tokens` is too low (or unset). The summarizer's reasoning and return **empty content** if `max_tokens` is too low (or unset). The summarizer's
parser treats an empty summary as an error for exactly this reason. **Done (2026-06-02, Worker F):** parser treats an empty summary as an error for exactly this reason. **Done (2026-06-02, Worker F):**
@@ -200,6 +248,15 @@ knobs plus one load-bearing deployment constraint:
- `TAPIR_DISCOVERY_INTERVAL` — Go duration, e.g. `2h`. The cadence the serve process runs a - `TAPIR_DISCOVERY_INTERVAL` — Go duration, e.g. `2h`. The cadence the serve process runs a
discovery pass for every registered user (run-once-on-startup, then every interval). discovery pass for every registered user (run-once-on-startup, then every interval).
**Unset or `0` = disabled** (dev/tests never auto-fetch). **Unset or `0` = disabled** (dev/tests never auto-fetch).
- `TAPIR_USAGE_GATE_START` — `YYYY-MM-DD`, default **`2026-06-11`** (the morning the pilot was
unblocked and summaries started flowing). `tapir report` counts return-usage (distinct active
weeks, ADR-016) only from this date, so pre-launch testing and the blocked period are excluded.
- `TAPIR_METRICS_ADDR` — listen address for the Prometheus `/metrics` endpoint (ADR-030).
**Default `:9090`** — a SEPARATE port from `TAPIR_HTTP_ADDR` so metrics are never on the public
app; scraped in-cluster only (PodMonitor). Empty disables the metrics server. Key series:
`tapir_summarize_duration_seconds{model,outcome,fallback}`, `tapir_caption_fetch_duration_seconds{outcome}`,
`tapir_chat_duration_seconds{model}`, `tapir_llm_tokens_total{model,kind}`,
`tapir_http_request_duration_seconds{method,route}`, `tapir_logins_total`.
- `TAPIR_FETCH_RATE` — Go duration, default `2s`. The **process-wide per-egress-IP caption-fetch - `TAPIR_FETCH_RATE` — Go duration, default `2s`. The **process-wide per-egress-IP caption-fetch
rate gate** (ADR-014 item 2). Every caption fetch — scheduler runners *and* the web "Summarize" rate gate** (ADR-014 item 2). Every caption fetch — scheduler runners *and* the web "Summarize"
click-path — serialises through this one limiter so the pod cannot collectively trip 429s. `0` click-path — serialises through this one limiter so the pod cannot collectively trip 429s. `0`
@@ -0,0 +1,81 @@
# kb-whisper vs pianissimo on one real hour of Swedish speech (tapir#33, 2026-10-01)
One recording, one speaker: a 63.7-minute Teams session of the maintainer dictating project
notes (domain-heavy, English loanwords and product names mixed into Swedish). **No transcript
text and no domain terms are recorded here**: this repo is public and the content is private.
Only aggregate numbers.
Backends, both through the LiteLLM gateway and Tapir's own adapter, unchanged:
`iguana/kb-whisper` (whisper.cpp, KB's Swedish fine-tune) and `iguana/pianissimo` (KlangAI
Swedish Parakeet/FastConformer-TDT, MLX 8-bit, served by a small FastAPI shell on iguana:8102).
## Result: a near tie on accuracy, a large difference in speed
| | kb-whisper | pianissimo |
|---|---|---|
| Wall time for 63.7 min | 4m08s (~15x realtime) | **40 s (~95x realtime)** |
| Segments | 1,302 (timed pieces, wrapped at ~60 chars) | 434 (sentences) |
| Words (normalised) | 6,606 | 6,510 |
| Punctuation | light, every segment ends in a full stop | heavier (138 vs 82 commas) |
| Repetition loops | none | none |
Word-level agreement between them: **92.4%** as first computed, **93.1%** after the segment
rejoin fix below. 380 aligned disagreement spans, median one word each.
**Blind A/B by the speaker** (the only ground truth available), on the 30 spans a model
(Claude Opus 5.5) had picked as changing meaning, shuffled and unlabelled: **pianissimo 17,
kb-whisper 13**, no "neither", no "can't tell".
| Category of the 30 | pianissimo | kb-whisper |
|---|---|---|
| Technical term / English loanword | 8 | 4 |
| Number / date / unit | 4 | 1 |
| Name / proper noun | 3 | 3 |
| Function word / small word | 1 | 2 |
| Other (everyday words) | 1 | 3 |
The reading: pianissimo is better on terminology and numbers, kb-whisper on ordinary Swedish
words; names are even. Their errors differ in kind: kb-whisper's tend to be garbled words, a
couple were wrong numbers; pianissimo's tend to be fluent substitutions that read as plausible
(a wrong institution, an invented person, a dropped sign), which a reader or an LLM is less
likely to catch.
Opus's own lean agreed with the speaker's blind answer on **30 of 30**. That is not a measure
of Opus over all 380 spans: it chose spans it could decide, and every one of the 30 carried a
lean. Its full classification of the 380 (counts only): function/small word 81, filler or
false start 65, technical term 58, compound split/join 46, other 39, name 28, casing or
punctuation residue 26, inflection 23, number/date/unit 10, content omission or insertion
(>= 3 words) 4.
## A bug this comparison found in Tapir (fixed)
kb-whisper's output is wrapped at ~60 characters and **a wrap can fall inside a word**; the
continuation segment has no leading space, the response's `text` field joins segments with
newlines, and a segment can be a lone `.`. Tapir stored that joined text, so chat received
broken words. Measured on real gateway output: 7% of segments (96/1,332) in a synthetic hour,
42/1,302 lines in the real one. berget's segments are whole sentences with no leading space, so
the two need different joins. `transcribe.joinSegments` now rebuilds the text from the segments.
Checked against the real transcript with pianissimo's vocabulary as an independent yardstick:
36 spans changed, 31 now entirely known words, words unknown to pianissimo 224 -> 177.
Five rejoined spans still contain an unknown word (a name or an unlucky glue): small residual
risk. Earlier tests used short sentences and could not see this.
## What this does not show
- **One recording, one speaker, one domain.** No claim about other voices, accents, crosstalk,
or room noise. 17 vs 13 is not a significant difference.
- **The other ~350 spans** were not judged by the speaker. They are mostly spelling, compounds,
fillers and small words; Opus's counts there are a model's judgement, not ground truth.
- Errors **both** models share in text they agree on are invisible to this method.
- Speed was measured once each, on an otherwise idle iguana; pianissimo's 40 s includes the
gateway hop and a 122 MB WAV upload.
- Whether pianissimo survives the infra#273 Metal-OOM contention the way whisper does not was
not tested; both run on iguana.
## What it suggests for Tapir (not decided, filed as an issue)
Pianissimo is a **second local backend on the same host, through the same adapter**. That
makes it a natural middle tier ahead of the cloud fallback (kb-whisper -> pianissimo ->
berget), so a wedged whisper no longer sends private audio off the homelab. At 40 s an hour it
is also cheap enough to run **both** on every upload and give chat both texts, or flag where
they disagree; the maintainer's blind answers show they err in different places.
@@ -0,0 +1,110 @@
# Audio length ceiling — measured before designing chunking (tapir#33, 2026-10-01)
Question (#33, "the main technical risk"): at what recording length does Swedish
transcription through the LiteLLM gateway first fail, and with what failure mode?
Measured **before** any chunking design, with iguana not holding a large ollama model,
so the result is the ceiling and not the Metal OOM of infra#273.
## Answer
**No transcription ceiling was found at or below 90 minutes.** A single request carrying
90 minutes of 16 kHz mono PCM (174 MB) transcribed in one pass in 440 s. The first thing
that actually fails is **not whisper**: it is the public gateway ingress, which rejects the
upload with `413` at roughly 5.5 minutes of WAV.
| Route | First failure | Failure mode |
|---|---|---|
| `https://llm-api.d-ma.be` (openresty ingress) | between 8.0 MB and 10.5 MB body (~4–5.5 min of WAV) | `413 Request Entity Too Large`, HTML body from `openresty`, returned instantly |
| `http://koala:30401` (LiteLLM NodePort) → `iguana/kb-whisper` | **none up to 90 min / 174 MB** | — |
| `http://koala:30401` → `berget/kb-whisper` | **none up to 22 min / 42 MB** (largest tried) | — |
The ingress limit is the reason the 8:20 / 15 MB recording in #33 "failed repeatedly" and a
30-second slice succeeded. That failure was attributed to a Metal OOM; at least in the
size-dependent form described, a body-size cap explains it without one. (The OOM
mechanism in infra#273 is separately real; this run did not provoke it.)
The exact ingress cap was bracketed (8.0 MB passed, 10.5 MB failed), not read from the
ingress configuration, which lives in `infra`. "About 10 MB" is an inference.
## Method
- **Audio.** No known-good Swedish recording is available that is safe to use: the only real
one (`IMG_1233.wav`) is the private `/bygge` recording (#31 custody still open) and was
deliberately not touched. Instead: macOS `say -v Alva` (Swedish) on iguana, a 1,171-char
five-paragraph meeting script → 82.5 s of 16 kHz mono PCM WAV (`ffmpeg -ar 16000 -ac 1`).
Concatenated ×4, ×8, ×16, ×32, ×44, ×66 with Python `wave` (the issue's own recipe).
Nothing here is a real transcript; none of it is committed.
- **Request.** `POST /v1/audio/transcriptions`, multipart `file=`, `model=iguana/kb-whisper`,
`response_format=verbose_json`, `-H 'Expect:'` (per #33's traps), master key supplied only
through `op run` and never on argv.
- **Confound control.** `ollama /api/ps` on iguana was read before every step. Resident:
nothing, or `nomic-embed-text` (0.37 GB). Never a large model.
## Results (`iguana/kb-whisper`, NodePort)
| Audio | WAV size | HTTP | Wall | Realtime factor | Segments | Chars |
|---|---|---|---|---|---|---|
| 0:30 | 0.96 MB | 200 | 2.8 s | 11× | — | — |
| 2:30 / 3:20 / 4:10 | 4.8 / 6.4 / 8.0 MB | 200 (ingress) | 12.9 / 16.0 / 19.9 s | 12× | — | — |
| 5:30 (×4) | 10.5 MB | **413 ingress**, 200 NodePort in 27 s | 27 s | 12× | 121 | 4,557 |
| 11:00 (×8) | 21.1 MB | 200 | 53 s | 12× | 241 | 9,094 |
| 22:00 (×16) | 42.3 MB | 200 | 105 s | 13× | 482 | 18,201 |
| 44:01 (×32) | 84.5 MB | 200 | 209 s | 13× | 966 | 36,438 |
| 60:32 (×44) | 116.2 MB | 200 | 291 s | 12× | 1,332 | 50,145 |
| 90:48 (×66) | 174.3 MB | 200 | 440 s | 12× | 2,005 | 75,165 |
- Throughput is flat at ~12× realtime. 90 minutes costs ~7.5 minutes of wall time. This is
a minutes-long job, not a request/response one (see ADR-033 on ADR-018).
- Segments per source repetition stay at 30.1–30.4 from ×4 to ×66, and segment times are
monotonic with `last_end` within 0.2 s of `duration`. No drift and no repetition-loop
hallucination appeared across 90 minutes. This is a weak check — the input is a repeated
loop and synthetic voice — not evidence of accuracy.
- **Only `iguana/kb-whisper`'s `x4` run reported `last_end` 360.0 s against `duration`
330.2 s.** Whisper's final segment can end past the audio; clamp to `duration`.
## Findings that change the design
1. **Use the NodePort route, not the public hostname.** Tapir runs in-cluster beside the
gateway; `llm-api.d-ma.be` is the external door and has a body cap. #33 said "use the
gateway, never the host" and named the public URL; the gateway is right, that URL is not.
`TAPIR_GATEWAY_URL` already points at `koala:30401/v1`.
2. **`response_format=srt` survives the LiteLLM hop, but wrapped.** The response is JSON
`{"text": "<the SRT>", "usage": …}`, not a bare SRT body — a client that parses the body
as SRT gets a parse error on `{`. #33 question 2 is answered: it passes through, inside
an envelope.
3. **The two backends return different timestamp granularity.** `iguana/kb-whisper`
segments are **whole seconds** (`0→1`, `1→4`, `4→7`); `berget/kb-whisper` returns real
sub-second times (`0.051→1.192`) and a different `language` token (`swedish` vs `sv`).
`verbose_json` and the default JSON were byte-identical on iguana. An adapter that
falls back between the two must normalise both, and "which endpoint served this" is
a quality fact, not just a log line.
4. **The existing chat path cannot answer about most of a long recording.** At
828 chars/min (this fixture; real speech is plausibly a little denser),
`TAPIR_MAX_TRANSCRIPT_CHARS`'s default of **18,000** is ~21.7 minutes. A 60-minute
transcript is 50 k chars, a 90-minute one 75 k. `chat.Service` truncates by keeping
the head (`internal/adapters/chat/chat.go`), so questions about the last two-thirds of
a one-hour meeting get an answer from a transcript that does not contain it. #33 says
chat "retrieves rather than generates" and therefore has no ceiling; it **stuffs and
truncates**, and its ceiling is the model context. See ADR-033.
5. **Chunking is not needed for v1 on length.** Nothing failed up to 90 minutes. The
chunking-and-stitching recipe in #33 (offset timestamps, renumber cues, strip `\r`) is
not built. It stays the answer *if* a longer cap is wanted or the OOM mechanism forces
smaller requests — see ADR-033's resolution triggers.
## What this did not measure
- **Metal OOM under contention.** Deliberately avoided. A big resident ollama model
during a long transcription is still the known way to wedge iguana permanently.
- **Real speech.** TTS is clean, single-voice, no crosstalk or room noise. WER and
speaker overlap in a Teams export are untested. Speed is probably representative;
accuracy is not claimed.
- **Compressed input.** Only WAV was sent. Whether `m4a`/`mp3`/`opus` can reach whisper
without a server-side transcode is not established here (ADR-033 treats it as needing one).
- **Concurrency.** One request at a time. Two simultaneous 90-minute jobs on iguana were
not tried.
- **`berget/kb-whisper` beyond 22 min / 42 MB.** Enough to know the fallback is not
size-capped at meeting length for 22 minutes; not proven at 90.
- **Time-of-day / other tenants.** One afternoon, one run per length.
Scripts were throwaway shell/Python in a session scratchpad and are not committed; the
method above reproduces them.
@@ -0,0 +1,66 @@
# Chat over a long transcript — what the offered models really hold (tapir#33, 2026-10-01)
Question: ADR-033 reuses `chat.Service`, which stuffs the transcript into the prompt and
cuts the tail. #33 assumed chat "retrieves rather than generates" and so has no length
ceiling. It does not retrieve, so its ceiling is the model's context. What is it, for the
models chat actually offers?
## Answer
| Model (via the NodePort gateway) | Result | Limit found |
|---|---|---|
| `koala/phi4-mini` | correct to 30,000 chars; **HTTP 400 at 50,000** (13,332 tokens vs an 8,192-token context) | ~22,000 chars in production, where the client adds `max_tokens: 1500` |
| `iguana/gemma4-26b` | correct at 18k, 50k and **90,000 chars** (26.5 k tokens); **at 120,000 chars it silently loses the head** | **90,000 chars** |
| `berget/mistral-small` | answered "the transcript does not contain it" for a fact **at the end** of an 18,000-char and a 50,000-char transcript | unreliable here, one run each — see caveats |
The dangerous one is gemma at 120,000 chars: **no error, no truncation flag** — the reported
prompt tokens fell from 26,548 (90 k chars) to 16,387 (120 k chars), and the model answered
"it does not say who the chair is" about a sentence at the very start of the transcript. The
serving layer truncated the prompt from the front. `chat.Service` reports `Truncated: false`
because *it* did not cut anything.
## What this changes
1. **The budget for an uploaded transcript is 90,000 chars, not a round 120,000.** That is the
best offered model's measured limit, ~100 minutes of speech at the fixture's density.
`TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS` defaults to it.
2. **Chat over an upload must not open on the summarizer default.** The offered list is
`[SummarizerModel, FallbackModel, CloudFallbackModel]`, so `koala/phi4-mini` is first — and
400s on an hour of speech. `TAPIR_UPLOAD_CHAT_MODEL` (default `iguana/gemma4-26b`) is what
the upload chat opens with; the switcher still lists every model.
3. **A per-model budget does not exist yet.** Someone who switches the dropdown to
`phi4-mini` on a long recording gets a loud HTTP 400, shown inline as "couldn't answer".
That fails honestly but unhelpfully. Filed as a follow-up.
4. **The "retrieval chat" trigger in ADR-033 is now concrete:** any recording past 90,000 chars
(≈100 min) cannot be answered by stuffing, for any offered model.
## Method
`POST /v1/chat/completions` through `http://koala:30401/v1`, `temperature: 0`,
`max_tokens: 200` (production sends 1500 for chat via the summary token cap), the same system
framing as `chat.Service`. The transcript is the 60-minute synthetic fixture from
`audio-length-ceiling-2026-10-01.md` truncated/repeated to N chars, with one unique sentence
planted either at the **end** ("lagret på Kvarngatan ska tömmas … av Pelle", question: who/when)
or at the **head** ("kvällens ordförande är Ingrid", question: who chairs). Needle at the
end alone would have missed the gemma failure — a serving layer that truncates from the front
keeps the tail, so the end-needle passed at 120,000 chars; only the head-needle exposed it.
Grounded = the answer contains the planted fact.
## What this did not measure
- **Real speech**: the fixture is a repeated 82-second script, so retrieval "difficulty" is
artificially easy except for the planted needle. One question per run, one run per cell.
- **`berget/mistral-small`'s failure**: it missed an end-needle at 18,000 chars where the
others succeeded. One run; could be the model, the prompt framing, or position bias. Not
investigated; do not read it as a ranking.
- **Latency under load**, **concurrent** chat while whisper is transcribing, and in particular
the hazard below.
- **`koala/qwen36-35b-a3b` or any frontier model** — not in the offered list.
## A hazard this surfaces and does not solve
`iguana/gemma4-26b` is a large ollama model on the same host as `iguana/kb-whisper`. infra#273
is precisely that a large resident ollama model can Metal-OOM whisper-server permanently. Making
gemma the default *chat* model for uploads means chatting with a transcript can load the model
that wedges the transcriber. The primary-fails-over-to-berget design limits the blast radius but
does not remove it. Recorded in ADR-033 and filed.
@@ -0,0 +1,75 @@
# Can a local model adjudicate two ASR transcripts? (tapir#33, 2026-10-01)
Question: when `iguana/kb-whisper` and `iguana/pianissimo` disagree about a word or phrase,
can a **local** model say which reading is right, well enough to merge the two transcripts?
Ground truth: the speaker's own blind answers on 30 disagreements of one real recording
(see `asr-kb-whisper-vs-pianissimo-2026-10-01.md`; no transcript text is recorded here,
this repo is public). Reference: Claude Opus 5.5 agreed with the speaker on all 30.
## Method
- **Items:** 30 disagreement spans that Opus had picked as changing meaning, the speaker's
verdict as truth (pianissimo right 17, kb-whisper right 13).
- **Prompt:** one fixed English instruction (two transcriptions of Swedish speech differ at
ONE marked spot; say which reading the speaker most likely said; reply `ANSWER: 1|2|UNSURE`
plus one sentence). Each item was asked in **both reading orders** and in **two contexts**:
narrow (+-8 words) and wide (+-100 words). 120 calls per model, temperature 0.
- **Models:** 11, all local (`koala/*`, `iguana/*`): nothing left the homelab. Run one iguana
model at a time, each unloaded afterwards, to avoid the memory pressure behind infra#273;
whisper was confirmed healthy afterwards.
- **Pass criteria, fixed before the run:** wide context, strict accuracy >= 80% (an
unparseable or "unsure" answer counts as wrong), order-swap consistency >= 85%,
unparseable <= 10%.
## Results (wide context, 60 calls per model)
| Model | Correct | 95% CI | Same pick when order swapped | Unparseable | s/call | Pass |
|---|---|---|---|---|---|---|
| `iguana/gemma4-31b` | 58 = 97% | 89-99% | 93% | 0 | 10.4 | **yes** |
| `iguana/gemma4-26b` | 55 = 92% | 82-96% | 93% | 0 | 2.0 | **yes** |
| `iguana/ornith-35b` | 51 = 85% | 74-92% | 77% | 0 | 2.5 | no (consistency) |
| `koala/phi4-14b` | 50 = 83% | 72-91% | 80% | 0 | 1.0 | no (consistency) |
| `koala/qwen36-35b-a3b` | 49 = 82% | 70-89% | 83% | 0 | 3.3 | no (consistency) |
| `iguana/granite4` | 47 = 78% | 66-87% | 83% | 0 | 4.6 | no |
| `iguana/qwen3-14b` | 44 = 73% | 61-83% | 87% | 0 | 4.4 | no (accuracy) |
| `iguana/gpt-oss-20b` | 40 = 67% | 54-77% | 100% | 17 | 11.2 | no (unparseable) |
| `koala/phi4-mini` | 37 = 62% | 49-73% | 24% | 1 | 0.5 | no |
| `iguana/gemma4-26b-think` | 23 = 38% | 27-51% | 100% | 37 | 15.3 | no (unparseable) |
| `koala/qwen35-9b-fast` | 0 | n/a | n/a | 60 | 7.2 | no (no output) |
Narrow context only (+-8 words) costs 6-12 points for the models that pass: gemma4-31b 92%,
gemma4-26b 83%, ornith-35b 80%. Context is not decoration: Opus also used evidence from
elsewhere in the recording (an amount stated earlier, a term that recurs).
Position bias: the passing models pick "Reading 1" 47-51% of the time. `phi4-mini` picks it 15%
and flips its answer when the order is swapped 76% of the time: it carries almost no signal.
## Reading it
- **Two models pass:** the gemma4 pair. Both clear the bar and neither shows position bias.
26b is ~5x faster; their intervals overlap, so 30 items cannot separate 92% from 97%.
- **The thinking models were not tested fairly.** `gpt-oss-20b` and `gemma4-26b-think` spend
the 600-token budget reasoning and often never reach the answer line. When gemma4-26b-think
did answer, it was right 23 of 23; gpt-oss-20b 93% of the answered. A larger budget would
make them eligible; at 15 s a call it would not make them practical.
## What this does not show
- **Thirty items, one recording, one speaker, one domain.** The 30 were chosen by Opus as
decidable, so they overstate how easy the typical disagreement is; ~350 others are mostly
spelling, compounds and fillers, which no ground truth covers.
- Opus agreeing 30/30 is not a measure of Opus over all 380 disagreements.
- Accuracy was measured at temperature 0 on one prompt; no prompt tuning was tried.
- The speaker's answers are the only truth; they are one person's recollection of what they said.
## Why gemma4-31b is slow, and what was ruled out
On iguana (M2 Ultra, 64 GB) the dense 31B runs at **103 tok/s prefill, 7.9 tok/s decode**; the
26B MoE at 641 and 39.6. Ruled out as the cause: memory pressure (no swap, 45% free, model 100%
on GPU), context length (32k vs 4k made no difference), and the **inference engine** — MLX 4-bit
of the same model measured 76 tok/s prefill and 8.6 decode, the same within noise. (Terse
output saves only ~3.6 s of ~9.3 s because prefill dominates.) Two unrelated engines at the same
speed points at the machine: ~140 GB/s effective memory bandwidth on a part rated for 800, and
**iguana has macOS Low Power Mode on** (`pmset` `lowpowermode 1`). That is the leading suspect
and is **not proven**: turning it off needs root and was not done. A first MLX benchmark was
wrong because `mlx_lm.server` caches prompts; the figures above are from the corrected run.
+63
View File
@@ -0,0 +1,63 @@
<!DOCTYPE html><html lang="en"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>tapir — A · TUI panels</title>
<style>
:root{--bg:#0d0d12;--panel:#14141c;--line:#2a2a3a;--mint:#0EF9B6;--pink:#F740A0;--purple:#7653fc;--cream:#F5E9D6;--dim:#6b6b86;--text:#d9d9e6}
*{box-sizing:border-box}
body{margin:0;background:var(--bg);color:var(--text);font:14px/1.5 ui-monospace,SFMono-Regular,Menlo,"Cascadia Code",monospace}
.wrap{max-width:760px;margin:0 auto;padding:18px 14px 60px}
header{display:flex;align-items:center;gap:10px;border-bottom:1px solid var(--line);padding-bottom:12px;margin-bottom:18px}
.brand{color:var(--mint);font-weight:700;letter-spacing:.5px}
.brand b{color:var(--cream)}
.tip{margin-left:auto;color:var(--dim);font-size:12px}
.note{color:var(--dim);font-size:12.5px;border-left:2px solid var(--purple);padding:6px 10px;margin:0 0 16px;background:#11111a}
/* lipgloss-style bordered panel */
.card{border:1px solid var(--line);border-radius:8px;background:var(--panel);padding:12px 14px;margin:0 0 12px;position:relative}
.card::before{content:"";position:absolute;left:0;top:10px;bottom:10px;width:3px;border-radius:3px;background:var(--purple)}
.card.ready::before{background:var(--mint)}
.title{color:var(--cream);font-size:15px;font-weight:600;margin:0 0 4px}
.meta{color:var(--dim);font-size:12px}
.chip{display:inline-block;border:1px solid var(--mint);color:var(--mint);border-radius:4px;padding:0 6px;font-size:11px;margin-left:6px}
.chip.q{border-color:var(--pink);color:var(--pink)}
.preview{color:var(--dim);margin:6px 0 0;font-size:13px}
/* expanded */
.exp{border-color:var(--purple)}
.exp .head{display:flex;justify-content:space-between;align-items:baseline}
.collapse{color:var(--mint);font-size:12px;text-decoration:none;border:1px solid var(--line);border-radius:4px;padding:1px 7px}
.sec h2{color:var(--mint);font-size:12px;text-transform:uppercase;letter-spacing:1px;margin:16px 0 6px;border-bottom:1px dashed var(--line);padding-bottom:3px}
.sec ul{margin:0;padding-left:18px}.sec li{margin:3px 0}
.body{color:var(--text)}
.dock{margin-top:16px;border-top:1px solid var(--line);padding-top:12px}
.ask{display:inline-block;background:linear-gradient(90deg,var(--purple),var(--pink));color:#fff;border:0;border-radius:6px;padding:7px 12px;font:inherit;font-size:13px;cursor:pointer}
pre.tapir{margin:0;color:var(--mint);font-size:10px;line-height:1.05}
</style></head><body><div class="wrap">
<header>
<pre class="tapir"> ▄█▓▓█▄ ∩
█▓( ◕ ◕ )▓█──┘</pre>
<span class="brand">◆ <b>tapir</b> · watch less, know more</span>
<span class="tip">261 in queue</span>
</header>
<p class="note">Tapir fetches captions slowly on purpose, to respect YouTube's limits — new summaries land gradually.</p>
<div class="card ready"><div class="title">How the Attention Economy Rewires Your Brain</div>
<div class="meta">youtube · 2026-06-11 · 18 min <span class="chip">ready</span></div>
<div class="preview">A tour of the incentive loops behind infinite feeds and three concrete ways to claw back focus…</div></div>
<article class="card exp ready">
<div class="head"><div class="title">Postgres 18 — What's Actually New</div><a class="collapse" href="#">collapse ↑</a></div>
<div class="meta">youtube · 2026-06-10 · 42 min · phi4-mini</div>
<div class="sec"><h2>Takeaways</h2><ul>
<li>Async I/O cuts cold-cache read latency materially on NVMe.</li>
<li>Skip-scan makes more multicolumn indexes usable without rewrites.</li>
<li>Upgrade path is smooth; test the new planner stats first.</li></ul></div>
<div class="sec"><h2>Highlights</h2><ul>
<li>Async I/O subsystem (effective_io_concurrency now matters more).</li>
<li>B-tree skip scan for leading-column gaps.</li>
<li>Better partition-wise joins.</li></ul></div>
<div class="sec"><h2>Summary</h2><p class="body">Postgres 18 is an incremental but meaningful release: the headline is the new asynchronous I/O path, with skip-scan and planner improvements close behind. For most homelabs the upgrade is low-risk and worth it for the read-latency wins.</p></div>
<div class="dock"><button class="ask">Dig deeper — ask about this video →</button></div>
</article>
<div class="card"><div class="title">RAG is dead, right?? — Conference Talk</div>
<div class="meta">youtube · 2026-06-09 · 31 min <span class="chip q">queued</span></div></div>
</div></body></html>
+62
View File
@@ -0,0 +1,62 @@
<!DOCTYPE html><html lang="en"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>tapir — B · reader + charm accents</title>
<style>
:root{--bg:#faf7f2;--card:#fff;--ink:#1c1b22;--soft:#6a6878;--line:#e7e2d8;--mint:#0bbf8c;--purple:#6a4cf0;--pink:#e0379a}
*{box-sizing:border-box}
body{margin:0;background:var(--bg);color:var(--ink);font:16px/1.6 -apple-system,BlinkMacSystemFont,"Segoe UI",Inter,sans-serif}
.mono{font-family:ui-monospace,SFMono-Regular,Menlo,monospace}
.wrap{max-width:680px;margin:0 auto;padding:20px 16px 60px}
header{display:flex;align-items:center;gap:10px;margin-bottom:18px}
.brand{font-weight:800;font-size:18px;letter-spacing:-.3px}
.brand .dot{color:var(--mint)}
.tag{color:var(--soft);font-size:13px}
.tip{margin-left:auto;color:var(--soft);font-size:12px}
.note{color:var(--soft);font-size:13px;background:#fff;border:1px solid var(--line);border-left:3px solid var(--mint);border-radius:8px;padding:8px 12px;margin:0 0 18px}
.card{background:var(--card);border:1px solid var(--line);border-radius:12px;padding:16px 18px;margin:0 0 14px;box-shadow:0 1px 2px rgba(20,18,40,.04)}
.title{font-size:18px;font-weight:700;letter-spacing:-.2px;margin:0 0 4px;line-height:1.3}
.meta{color:var(--soft);font-size:13px}
.meta .mono{font-size:12.5px}
.chip{display:inline-block;background:rgba(11,191,140,.12);color:var(--mint);border-radius:999px;padding:1px 9px;font-size:12px;font-weight:600;margin-left:6px}
.chip.q{background:rgba(224,55,154,.12);color:var(--pink)}
.preview{color:var(--soft);margin:8px 0 0}
.exp{border-color:#d9d0ee;box-shadow:0 6px 24px rgba(106,76,240,.10)}
.head{display:flex;justify-content:space-between;align-items:baseline;gap:10px}
.collapse{color:var(--purple);font-size:13px;font-weight:600;text-decoration:none;white-space:nowrap}
.sec h2{font-size:12px;text-transform:uppercase;letter-spacing:1.2px;color:var(--purple);margin:18px 0 6px}
.sec ul{margin:0;padding-left:20px}.sec li{margin:5px 0}
.body{line-height:1.7}
.dock{margin-top:18px;border-top:1px solid var(--line);padding-top:14px}
.ask{display:inline-flex;align-items:center;gap:6px;background:var(--ink);color:#fff;border:0;border-radius:999px;padding:9px 16px;font:inherit;font-size:14px;font-weight:600;cursor:pointer}
pre.tapir{margin:0;color:var(--mint);font-size:10px;line-height:1.05}
</style></head><body><div class="wrap">
<header>
<pre class="tapir"> ▄█▓▓█▄ ∩
█▓( ◕ ◕ )▓█──┘</pre>
<span class="brand">tapir<span class="dot">.</span></span><span class="tag">watch less, know more</span>
<span class="tip mono">261 in queue</span>
</header>
<p class="note">Tapir fetches captions slowly on purpose, to respect YouTube's limits — new summaries land gradually.</p>
<div class="card"><div class="title">How the Attention Economy Rewires Your Brain</div>
<div class="meta mono">youtube · 2026-06-11 · 18 min <span class="chip">ready</span></div>
<div class="preview">A tour of the incentive loops behind infinite feeds and three concrete ways to claw back focus…</div></div>
<article class="card exp">
<div class="head"><div class="title">Postgres 18 — What's Actually New</div><a class="collapse" href="#">collapse ↑</a></div>
<div class="meta mono">youtube · 2026-06-10 · 42 min · phi4-mini</div>
<div class="sec"><h2>Takeaways</h2><ul>
<li>Async I/O cuts cold-cache read latency materially on NVMe.</li>
<li>Skip-scan makes more multicolumn indexes usable without rewrites.</li>
<li>Upgrade path is smooth; test the new planner stats first.</li></ul></div>
<div class="sec"><h2>Highlights</h2><ul>
<li>Async I/O subsystem (effective_io_concurrency now matters more).</li>
<li>B-tree skip scan for leading-column gaps.</li>
<li>Better partition-wise joins.</li></ul></div>
<div class="sec"><h2>Summary</h2><p class="body">Postgres 18 is an incremental but meaningful release: the headline is the new asynchronous I/O path, with skip-scan and planner improvements close behind. For most homelabs the upgrade is low-risk and worth it for the read-latency wins.</p></div>
<div class="dock"><button class="ask">✦ Ask about this video</button></div>
</article>
<div class="card"><div class="title">RAG is dead, right?? — Conference Talk</div>
<div class="meta mono">youtube · 2026-06-09 · 31 min <span class="chip q">queued</span></div></div>
</div></body></html>
+65
View File
@@ -0,0 +1,65 @@
<!DOCTYPE html><html lang="en"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>tapir — C · cozy terminal</title>
<style>
:root{--bg:#16131d;--card:#1e1a28;--ink:#ece7f5;--soft:#9a92b4;--line:#322b44;--mint:#2ee6b6;--purple:#9d7bff;--pink:#ff6bdb;--cream:#f3ead8}
*{box-sizing:border-box}
body{margin:0;background:radial-gradient(1200px 600px at 70% -10%,#231b33,transparent),var(--bg);color:var(--ink);font:15px/1.6 -apple-system,BlinkMacSystemFont,"Segoe UI",Inter,sans-serif}
.mono{font-family:ui-monospace,SFMono-Regular,Menlo,monospace}
.wrap{max-width:700px;margin:0 auto;padding:20px 16px 60px}
header{display:flex;align-items:center;gap:12px;margin-bottom:18px}
.brand{font-weight:800;font-size:18px}.brand .c{color:var(--mint)}
.tag{color:var(--soft);font-size:13px}
.tip{margin-left:auto;color:var(--soft);font-size:12px}
.note{color:var(--soft);font-size:13px;background:#1b1726;border:1px solid var(--line);border-radius:10px;padding:9px 12px;margin:0 0 18px}
.note b{color:var(--mint);font-weight:600}
.card{position:relative;background:var(--card);border:1px solid var(--line);border-radius:12px;padding:14px 16px 14px 18px;margin:0 0 13px;overflow:hidden}
.card::before{content:"";position:absolute;left:0;top:0;bottom:0;width:4px;background:var(--purple)}
.card.ready::before{background:linear-gradient(var(--mint),var(--purple))}
.title{font-size:16px;font-weight:700;margin:0 0 4px;color:var(--cream)}
.meta{color:var(--soft);font-size:12.5px}
.chip{display:inline-block;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:600;margin-left:6px;background:rgba(46,230,182,.14);color:var(--mint)}
.chip.q{background:rgba(157,123,255,.18);color:var(--purple)}
.preview{color:var(--soft);margin:7px 0 0;font-size:14px}
.exp{border-color:#473a63;box-shadow:0 10px 40px rgba(0,0,0,.35)}
.head{display:flex;justify-content:space-between;align-items:baseline;gap:10px}
.collapse{color:var(--mint);font-size:12px;text-decoration:none;border:1px solid var(--line);border-radius:6px;padding:2px 8px;white-space:nowrap}
.sec h2{font-size:11px;text-transform:uppercase;letter-spacing:1.3px;color:var(--mint);margin:18px 0 7px;display:flex;align-items:center;gap:8px}
.sec h2::after{content:"";flex:1;height:1px;background:var(--line)}
.sec ul{margin:0;padding-left:18px}.sec li{margin:5px 0}
.body{line-height:1.7;color:var(--ink)}
.dock{margin-top:18px;border-top:1px dashed var(--line);padding-top:14px;display:flex;align-items:center;gap:10px}
.ask{display:inline-flex;align-items:center;gap:7px;background:linear-gradient(90deg,var(--purple),var(--mint));color:#10101a;border:0;border-radius:999px;padding:9px 16px;font:inherit;font-weight:700;font-size:14px;cursor:pointer}
.dockhint{color:var(--soft);font-size:12px}
pre.tapir{margin:0;color:var(--mint);font-size:11px;line-height:1.05}
</style></head><body><div class="wrap">
<header>
<pre class="tapir"> ▄█▓▓█▄ ∩
█▓( ◕ ◕ )▓█──┘</pre>
<span class="brand">tapir<span class="c">_</span></span><span class="tag">watch less, know more</span>
<span class="tip mono">261 in queue</span>
</header>
<p class="note">Tapir fetches captions slowly on purpose, to respect YouTube's limits — <b>new summaries land gradually</b>.</p>
<div class="card ready"><div class="title">How the Attention Economy Rewires Your Brain</div>
<div class="meta mono">youtube · 2026-06-11 · 18 min <span class="chip">ready</span></div>
<div class="preview">A tour of the incentive loops behind infinite feeds and three concrete ways to claw back focus…</div></div>
<article class="card exp ready">
<div class="head"><div class="title">Postgres 18 — What's Actually New</div><a class="collapse" href="#">collapse ↑</a></div>
<div class="meta mono">youtube · 2026-06-10 · 42 min · phi4-mini</div>
<div class="sec"><h2>Takeaways</h2><ul>
<li>Async I/O cuts cold-cache read latency materially on NVMe.</li>
<li>Skip-scan makes more multicolumn indexes usable without rewrites.</li>
<li>Upgrade path is smooth; test the new planner stats first.</li></ul></div>
<div class="sec"><h2>Highlights</h2><ul>
<li>Async I/O subsystem (effective_io_concurrency now matters more).</li>
<li>B-tree skip scan for leading-column gaps.</li>
<li>Better partition-wise joins.</li></ul></div>
<div class="sec"><h2>Summary</h2><p class="body">Postgres 18 is an incremental but meaningful release: the headline is the new asynchronous I/O path, with skip-scan and planner improvements close behind. For most homelabs the upgrade is low-risk and worth it for the read-latency wins.</p></div>
<div class="dock"><button class="ask">◆ Ask about this video</button><span class="dockhint mono">answers only from this video's transcript</span></div>
</article>
<div class="card"><div class="title">RAG is dead, right?? — Conference Talk</div>
<div class="meta mono">youtube · 2026-06-09 · 31 min <span class="chip q">queued</span></div></div>
</div></body></html>
+89
View File
@@ -0,0 +1,89 @@
# Spec — Chat with a video's stored transcript (ADR-027)
**Repo:** tapir · **Size:** medium · **Solo session.** Implements ADR-027. Read CLAUDE.md,
DECISIONS.md (ADR-021 transcript store, ADR-022 model chain, ADR-012 isolation, ADR-027), and
`docs/ui-spec.md` first. TBD, conventional commits, `task check` green per commit, `templ
generate` after view changes.
## FIRST: append ADR-027 to DECISIONS.md
ADR-027 text is provided separately (planning thread). Insert immediately before the
`## Rejected alternatives` heading, as the first commit, so the decision precedes the build.
## What this is
A per-video chat letting the user ask questions against a video's **already-stored** transcript,
entered from the summary view. Born from observed demand: the maintainer read real summaries and
some made him want to dig deeper — this gives that "I want more" reaction somewhere to go, without
watching the video.
## HARD CONSTRAINT — stored-transcript-only (the safety property)
Chat is available **ONLY** for videos that already have a stored transcript (ADR-021). It must
**never** trigger a caption fetch, never touch the rate gate, never reach YouTube. Entry being
"from a summarized video" guarantees the transcript exists. If somehow invoked on a video with no
stored transcript → show "transcript not available for chat", NO fetch. This is what makes the
feature safe by construction; do not add an on-demand-fetch path (explicitly deferred).
## 1. Entry point
- A "Dig deeper" / "Ask about this" affordance on the **summary view** of a summarized video
(not the list cards — the detail/summary page). Quiet, consistent with the existing card-state
styling.
- Opens a chat panel/view scoped to that one video, with its stored transcript as context.
## 2. The chat
- Read the stored transcript for the video (via the ADR-021 `TranscriptStore`, keyed by
`(provider, provider_video_id)`). No fetch.
- Send transcript + the user's question + minimal system framing to the chosen model via the
**existing LiteLLM gateway** (the same client the summarizer uses — a chat is a different
call, not a new integration).
- Stream or return the answer; render in the chat panel. HTMX/no-JS ethos — match the existing
app (the summarize status uses HTMX polling; chat can use a simple POST-and-render or HTMX
streaming if clean).
- **Transcript truncation:** reuse/respect `TAPIR_MAX_TRANSCRIPT_CHARS` (ADR-022) so a long
transcript fits the model context. If truncated, the chat should be honest that it's working
from a bounded portion (a quiet note), since answers about the tail of a long video may be
incomplete.
## 3. Model selection (the instrumentation win)
- **Default model = the model that produced this video's summary.** (Store/lookup which chain
model summarized it — if not already recorded, this is a small addition; if recording it is
non-trivial, default to the chain primary and note the gap.)
- **User can switch** among the ADR-022 chain models (`phi4-mini`, `gemma4-26b`,
`mistral-small` to start) via a simple selector in the chat panel. Switching re-runs against
the same transcript — this is deliberate model-comparison instrumentation.
- Respect the local-first / NDA posture: if `TAPIR_CLOUD_FALLBACK_MODEL=""` (cloud disabled),
the external model is NOT offered in the switcher — only local models. Chat must honor the same
"content stays local" guarantee as ADR-022.
## 4. Ephemeral (v1)
- No persisted chat history. Conversation lives for the session/page. No new table, no migration.
- (Multi-turn within a session is fine — keep the running messages in the request/page state —
but nothing is written to the DB.)
## 5. Isolation
- The transcript is shared/non-RLS (ADR-021) — fine, it's public content. But the chat is invoked
by a user about a video **in their feed**; confirm the entry path is reachable only for the
requesting user's own videos (the summary view is already RLS-scoped). Chat adds no new
user-data surface (ephemeral), so there's nothing new to RLS — but the test should confirm a
user can only open chat from their own summary view, not arbitrary video ids.
## Tests
- Chat on a video with a stored transcript → answer returned; assert NO caption-fetch / no
YouTube call occurs (the safety property — this is the key assertion).
- Chat invoked on a video with no stored transcript → honest "not available", NO fetch.
- Model switch → re-runs against the same transcript with the selected model; cloud model absent
from the switcher when `TAPIR_CLOUD_FALLBACK_MODEL=""`.
- Truncation honored for a long transcript; the bounded-context note shows.
- Entry is reachable only from the user's own summary view (isolation).
## Out of scope / deferred (record, don't build)
- **Persisted chat history** (per-user, RLS-scoped) — deferred until evidence anyone revisits a
conversation.
- **Show-source / transcript-verification UI** — the natural v2 (ADR-027 records it); v1 is
chat-only/trust-the-model. ADR-021's stored transcript makes v2 cheap when wanted.
- **On-demand fetch** for un-stored videos — would reintroduce the caption-fetch surface the
stored-only constraint removes. Not now.
- Anything that nudges the user to return (ADR-020 — gate contamination).
## Boundaries
Stored-transcript-only (HARD). No rate-gate/fetch surface. No auth changes. No new persisted
state in v1. Reuse the existing gateway client + truncation config; don't build a new model
integration.
+128
View File
@@ -0,0 +1,128 @@
# Spec — Onboarding "wow" burst: better picks, stronger model
**Repo:** tapir · **Size:** medium · **Solo session** (not a swarm).
> **Status: built (v0.25.0, ADR-028).** This supersedes the original investigate-first brief
> (committed as the prior version of this file): Phase 1 was run against the live pilot DB and its
> findings are folded into "Why this exists" below; Phase 2 was built as described here. The one
> brief lever NOT built — the honest "the rest fill in over the coming days" framing copy — is
> listed under *Explicitly NOT in this slice*.
**Why this exists.** A new user's first session decides whether they return (the Stage-0 gate,
VISION.md). On connect, Tapir fires a capped burst (≤`TAPIR_ONBOARD_SUMMARIZE_COUNT`, default 3)
that summarizes the user's newest unsummarized videos so the feed isn't empty (the burst itself
works — wired in `cmd/tapir/discovery.go` → `cmd/tapir/main.go` `onboard`). A Phase-1
investigation of the live pilot DB found the burst *fires* but delivers a **weak first
impression** for two concrete reasons, and ruled out a third idea:
1. **Picks are junk.** Selection is pure newest-first (`videos.NewestUnsummarizedVideoIDs`,
`ORDER BY published_at DESC`) with **zero quality signal**. Pilot user "Jonte"'s live burst-3
were a stock-ticker **livestream** + two regional news clips — the newest, not the best.
2. **Weakest model on the first impression.** All of Jonte's summaries ran on
`koala/phi4-mini` (the documented weak link — ADR-022 was born from its failures). The
stronger, brain-validated `iguana/gemma4-26b` was never used for the burst.
3. **Cached-first is empty at pilot scale — REJECTED.** The idea (summarize already-cached
transcripts instantly, zero fetch) dies on the numbers: only **11 videos** overlap between the
two pilot users (~3% of each library), **0** cached-and-unsummarized, and a new user's
newest-20 unsummarized are **20/20 NOT cached** — newest-first and cached-first are
structurally incompatible (fresh uploads are exactly what nobody has fetched yet). Not built.
This is a **curation/latency problem for ~3 videos, NOT a throughput/429 problem** — fetching 3
captions is nowhere near the rate limit. Nothing here fetches harder or pressures the rate gate;
it picks the right few videos and runs a better model on them.
Read `CLAUDE.md`, `DECISIONS.md` (esp. ADR-014, ADR-018, ADR-020, ADR-021, ADR-022, ADR-023,
and the new **ADR-028**), and `VISION.md` (the Stage-0 gate) first. TBD — commit directly to
`main`, one logical change per commit, conventional commits, `task check` green before each
commit, `templ generate` if any view changes (none expected).
## Decisions already made (do not reopen)
- **Not a throughput change.** The caption rate gate (ADR-014) is untouched — same pacing, same
priority lane (ADR-026). This slice changes *which* ≤3 videos the burst spends its fetches on
and *which model* summarizes them, never how fast or how many.
- **Cached-first is dropped** (ADR-028, the 3% overlap). The engine's existing read-stored-first
(ADR-021, `resolveTranscript`) stays — it already gives a free instant summary on the rare
cache hit, transparently. We do not *select* for cache hits.
- **has-captions is not a pre-fetch signal.** It is only knowable after a gate fetch (or a cache
hit, ~0 for new videos). Selection can only *avoid known-junk* (Shorts/live/over-long) — it
cannot *guarantee* captions. The spec is honest about this: better odds, not a promise.
- **No credentialed caption fetch** (ADR-010/ADR-026 dead end). **No client extension.**
## 1. Persist `duration_s` at discovery (the enabling change)
The `videos.duration_s` column exists (migration 001) but is **never written** — ADR-023's
`filterLowValue` (`internal/adapters/youtube/youtube.go`) already fetches each candidate's
duration via the cheap quota `videos.list` call, uses it to drop Shorts/live, then **discards
it**. Stop discarding:
- Add `DurationSeconds int` to `domain.Video`.
- In `filterLowValue`, set `DurationSeconds` on each kept video from the `videos.list` `meta`.
- `UpsertVideo` writes `duration_s`, **COALESCE-preserving** a known value (never overwrite a
real duration with 0/unknown), mirroring the `channel_title` backfill stance (migration 014).
- No new migration — the column is already there.
Consequence: a fresh user's connect-triggered discovery pass runs **before** the onboard burst
(`Enqueue`: `run()` then `onboard()`), so duration is populated for the burst's candidates at
connect. Existing rows backfill on their next discovery pass; until then their `duration_s` is
NULL and treated as "unknown" (§2).
## 2. Junk-avoiding burst selection
New store method, RLS-scoped via `withUser`:
```
OnboardBurstVideoIDs(ctx, userID string, limit, minSeconds, maxSeconds int) ([]string, error)
```
- Same base as the old `NewestUnsummarizedVideoIDs`: the user's videos with no summary yet,
`ORDER BY published_at DESC NULLS LAST, seen_at DESC`, `LIMIT limit`.
- **Exclude known-junk**: a row is dropped only when `duration_s IS NOT NULL` **and**
(`duration_s < minSeconds` OR `duration_s > maxSeconds`). A NULL duration is **unknown** — kept
(degrade-open: never starve the burst because metadata is missing), but ordered *after* rows
with a known-good duration so a freshly-enriched good pick wins when both exist.
- `minSeconds` reuses `TAPIR_MIN_VIDEO_SECONDS` (default 60 — the Shorts floor, ADR-023).
`maxSeconds` is new: `TAPIR_ONBOARD_MAX_VIDEO_SECONDS` (default 14400 = 4h) — drops the
multi-hour livestream VODs that pass the live filter once ended.
- `minSeconds<=0` and `maxSeconds<=0` each disable that bound (so `0/0` == the old
newest-first behaviour, the reversibility lever).
- The burst switches to this method; `NewestUnsummarizedVideoIDs` is removed (fully superseded —
`OnboardBurstVideoIDs(., 0, 0)` is identical pure-newest behaviour).
## 3. Stronger model for the burst
The burst summarizes only ≤3 videos, so a slower, stronger model is affordable exactly here.
- New config `TAPIR_ONBOARD_SUMMARIZER_MODEL` (default `iguana/gemma4-26b` — the brain-validated
homelab general-purpose model, already the ADR-022 fallback).
- Build a **burst-specific summarizer chain** that puts the onboard model **first**, then the
standard chain (primary → local fallback → cloud) as resilience, deduped. Wrap it in a
burst-specific `engineProcessor` reusing the same store/transcript-cache/sink — a pure wiring
choice, engine and ports unchanged (Clean Architecture, ADR-003).
- The `onboard` closure uses the burst processor instead of `app.Processor`.
- **Collapse cleanly**: when `OnboardSummarizerModel` is empty or equals `SummarizerModel`, the
onboard path reuses `app.Processor` (no separate chain) — the reversibility lever.
- Local-first preserved: the onboard model is a local alias; the cloud endpoint stays last in the
chain, so a client/NDA deployment with `TAPIR_CLOUD_FALLBACK_MODEL=""` keeps burst content
local too.
## 4. Behaviour spec + docs
- Add scenarios to `docs/use-cases/connect_account.feature` (the connect → burst flow): burst
skips a too-long/live video in favour of a reasonable-length one; burst summarizes with the
stronger model first. Map them in `scenarioCoverage` so `TestScenarioCoverage` stays green.
- Update `docs/architecture/architecture.md` (the onboarding-burst section) to describe the
junk-avoiding selection + the burst model override.
- ADR-028 in `DECISIONS.md` records the rationale (incl. the rejected cached-first lever).
## Success criteria
- `task check` green (fmt, vet, lint, `go test -p 1 ./...`).
- A unit test proves `OnboardBurstVideoIDs` drops a known too-long / sub-min video and keeps a
good one, newest-first, RLS-scoped, unsummarized-only.
- A test proves discovery persists `duration_s` and does not clobber it on re-upsert.
- A test proves the burst chain leads with the onboard model (then the standard chain).
- Config defaults + bounds tested (`OnboardMaxVideoSeconds`, `OnboardSummarizerModel`).
- No change to the rate gate, fetch pacing, or burst cap. `0/0` + empty model == prior behaviour.
## Explicitly NOT in this slice
- Cached-first selection (rejected, ADR-028).
- Any caption-availability *guarantee* (impossible pre-fetch).
- **Honest "taster" framing copy** ("summaries of a few of your videos to get you started — the
rest fill in over the coming days"). A good lever from the original brief, but it's a UI/copy
change with no backend dependency; deferred to a UI pass, tracked as an issue.
- Backfilling `duration_s` for existing rows via a migration (it backfills lazily on discovery).
- Return-nudges / digests (ADR-020: poisons the unprompted-return signal).
- Raising fetch throughput, multi-IP, or Whisper (out of scope; the gate is deliberate).
+1
View File
@@ -179,4 +179,5 @@ distinguishable.
| **"Summarize now" foreground path** | Unified quiet nudge button on actionable non-summarized cards. Five explicit card states — (1) summarized: chip + no button; (2) no captions (`transcript_status = 'none'`): "No transcript available", no button; (3) queued: "Queued" chip, no button; (4) rate-limited: "Fetching soon…" + "Summarize now" → `POST /v/{id}/retry-now` (clears `rate_limited_at`, triggers engine); (5) pending: "Not summarized" + "Summarize now" → `POST /v/{id}/summarize` (queues + triggers engine). One verb, one style (`.btn-quiet`); backend difference invisible to user. Both handlers call `ProcessVideo` through `globalFetchGate`. Rate gate respected, not bypassed — this is onboarding prioritisation. | Fast onboarding value; honest dead-end for no-captions videos (no button that fails). | `internal/web/handlers.go` (`handleRetryNow`, `handleRequestSummarize`); `internal/web/views.templ` (`VideoCard`) | | **"Summarize now" foreground path** | Unified quiet nudge button on actionable non-summarized cards. Five explicit card states — (1) summarized: chip + no button; (2) no captions (`transcript_status = 'none'`): "No transcript available", no button; (3) queued: "Queued" chip, no button; (4) rate-limited: "Fetching soon…" + "Summarize now" → `POST /v/{id}/retry-now` (clears `rate_limited_at`, triggers engine); (5) pending: "Not summarized" + "Summarize now" → `POST /v/{id}/summarize` (queues + triggers engine). One verb, one style (`.btn-quiet`); backend difference invisible to user. Both handlers call `ProcessVideo` through `globalFetchGate`. Rate gate respected, not bypassed — this is onboarding prioritisation. | Fast onboarding value; honest dead-end for no-captions videos (no button that fails). | `internal/web/handlers.go` (`handleRetryNow`, `handleRequestSummarize`); `internal/web/views.templ` (`VideoCard`) |
| **Pipeline stats bar** | A one-line status bar above the video list: `N summarized · M fetching soon · K no captions`. Computed from the unfiltered row set; hidden when all videos are summarized. Gives the user a clear read on pipeline state without any interaction. | Replaces the "why is nothing happening?" confusion when most videos are pending or rate-limited. | `internal/web/view.go` (`PipelineStats`, `pipelineStats`) | | **Pipeline stats bar** | A one-line status bar above the video list: `N summarized · M fetching soon · K no captions`. Computed from the unfiltered row set; hidden when all videos are summarized. Gives the user a clear read on pipeline state without any interaction. | Replaces the "why is nothing happening?" confusion when most videos are pending or rate-limited. | `internal/web/view.go` (`PipelineStats`, `pipelineStats`) |
| **Unavailable channels (account page)** | The `/account` page shows a "Unavailable channels" section when any channels returned HTTP 404 on the last discovery pass. Lists channel name, an "unavailable" badge, and the first-seen date. Data sourced from the `channel_errors` table (migration 013). | Surfaces silent failures so users know why some subscribed channels produce no new videos. | migration 013; `internal/web/account.go`; `internal/adapters/youtube/youtube.go` (`domain.ErrChannelUnavailable`) | | **Unavailable channels (account page)** | The `/account` page shows a "Unavailable channels" section when any channels returned HTTP 404 on the last discovery pass. Lists channel name, an "unavailable" badge, and the first-seen date. Data sourced from the `channel_errors` table (migration 013). | Surfaces silent failures so users know why some subscribed channels produce no new videos. | migration 013; `internal/web/account.go`; `internal/adapters/youtube/youtube.go` (`domain.ErrChannelUnavailable`) |
| **Visual refresh — charm-reader theme + light/dark toggle (ADR-032)** | One layout in two palettes expressed as CSS custom properties: a warm "reader" light theme (sketch B) and a "cozy terminal" dark theme (sketch C). Palette is chosen in cascade order — `:root` light default, an OS-preference dark block scoped to `:root:not([data-theme])` so it only applies absent an explicit choice, and `:root[data-theme="dark"\|"light"]` set by a header toggle that outranks the media query and persists in `localStorage` (guarded; degrades to OS default). An init script in `<head>` applies the stored choice before paint (no flash). Charm touches: monospace meta lines, accent uppercase section dividers with a trailing rule, pill buttons, a lifted/accent-edged expanded card. Error/danger shades became `--err-*` tokens so they follow the theme without per-block dark overrides. Sketches kept as the design record under `docs/sketches/`. | The UI read "flat and boring"; the charm/TUI aesthetic makes it distinctive and gives a real light/dark choice rather than OS-only. | ADR-032; `docs/use-cases/visual_theme.feature`; `internal/web/visual_theme_test.go`; `internal/web/view.go` (`stylesheet`, `themeScript`), `internal/web/views.templ` (Layout/PublicLayout) |
| **Recency window + sparse-state honesty (ADR-020)** | Supersedes the copy/sort in the rows above. Auto-summarize is bounded to videos published within `TAPIR_AUTO_SUMMARIZE_WINDOW` (~7d); older un-summarized videos collapse behind a single "Show N older videos — summarize on demand" disclosure, and caption-less videos collapse to a one-line count (not N cards). List order is now `summarized-first, published_at DESC NULLS LAST`. Copy reframed for honest scarcity: pipeline bar reads "N ready · M in queue · K no captions" (no "fetching soon"); a gradual-fill note explains the rate limit; the nudge verb is "Summarize" (not "Summarize now"); the queued card says "summarizing shortly"; the empty-connected state drops the impossible `tapir run` instruction. Detail leads with Takeaways. Filters slimmed (no date pickers; hidden when empty); watched/skipped segmented; back link on detail; empty terms checkbox removed. | Make the sparse reality legible and honest instead of implying abundance/imminence; bound auto load so the back-catalogue doesn't re-drive the caption gate. Never fetch harder — scarcity is surfaced, not engineered around. | ADR-020; `2384c47`, `3df0459`, `40b703e`, `a1a5217`, `4a0a56e`, `9bf1c31`, `980638d`, `12fb031`, `f775441`, `51aa5d9` | | **Recency window + sparse-state honesty (ADR-020)** | Supersedes the copy/sort in the rows above. Auto-summarize is bounded to videos published within `TAPIR_AUTO_SUMMARIZE_WINDOW` (~7d); older un-summarized videos collapse behind a single "Show N older videos — summarize on demand" disclosure, and caption-less videos collapse to a one-line count (not N cards). List order is now `summarized-first, published_at DESC NULLS LAST`. Copy reframed for honest scarcity: pipeline bar reads "N ready · M in queue · K no captions" (no "fetching soon"); a gradual-fill note explains the rate limit; the nudge verb is "Summarize" (not "Summarize now"); the queued card says "summarizing shortly"; the empty-connected state drops the impossible `tapir run` instruction. Detail leads with Takeaways. Filters slimmed (no date pickers; hidden when empty); watched/skipped segmented; back link on detail; empty terms checkbox removed. | Make the sparse reality legible and honest instead of implying abundance/imminence; bound auto load so the back-catalogue doesn't re-drive the caption gate. Never fetch harder — scarcity is surfaced, not engineered around. | ADR-020; `2384c47`, `3df0459`, `40b703e`, `a1a5217`, `4a0a56e`, `9bf1c31`, `980638d`, `12fb031`, `f775441`, `51aa5d9` |
+13 -1
View File
@@ -31,5 +31,17 @@ Feature: Local-first AI with optional BYO fallback
When any transcript is summarized When any transcript is summarized
Then my content is only ever sent to the local AI stack Then my content is only ever sent to the local AI stack
# "Reliably" is operationalized as: Primary returned without error within timeout. Scenario: A model returns unparseable output and the next endpoint succeeds
Given the local AI stack is available
But the primary model returns output that cannot be parsed into a summary
And a fallback model is configured
When a transcript is summarized
Then Tapir falls back to the next model in the chain
And the summary records fallback_used as true
# "Reliably" is operationalized as: an endpoint returned a PARSEABLE summary
# within timeout. A 200 with malformed JSON (or highlights emitted as a bare
# string) counts as a failure and advances the chain (ADR-022). Endpoints are
# tried in order, locals first, so the external worst-case model only ever sees
# content after every local endpoint has failed.
# Quality scoring may be added later without changing these scenarios. # Quality scoring may be added later without changing these scenarios.
+158
View File
@@ -0,0 +1,158 @@
Feature: Upload an audio recording, get a private Swedish transcript, discuss it
As someone with a recorded meeting on my computer
I want to upload it and then ask questions about what was said
So that I can find out what was decided, what is open and who owes what
# ADR-033, tapir#33. The load-bearing constraint: an uploaded transcript is
# PRIVATE content, so it lives in its own RLS-scoped store and never in the
# shared public-caption `transcripts` table (ADR-021). Rejection is synchronous,
# the audio is never retained, and a failed job ends in a terminal state with a
# message — never a spinner (ADR-025). A file with a video track (a Teams meeting
# with the camera off is one) is accepted: only its audio is used and the user is
# told. Video playback, keyframes and analysis of what is shown remain #28.
Background:
Given I am a registered user with audio upload enabled
Scenario: The upload form states the limits up front
When I open the upload page
Then I see the size limit and the length limit before I choose a file
Scenario: An accepted recording is queued and I am taken to its page
When I upload an audio file within the limits
Then it is recorded as queued for me
And I am redirected to its page
Scenario: A recording with no title is named after its file
When I upload an audio file and give no title
Then its title is the filename without its extension
Scenario: A file with a video track is accepted and I am told the video was ignored
When I upload a recording that also has a video track
Then it is accepted and queued like any other
And its page says only its audio was used and the video was discarded
And it says so while queued, while transcribing, when done and when failed
Scenario: A video file with no audio is rejected for having no audio
When I upload a video that has no audio track
Then it is rejected with a message that it has no audio
And nothing is recorded
Scenario: A recording over the length limit is rejected with both lengths
When I upload a recording longer than the limit
Then it is rejected stating the recording's length and the limit
And nothing is recorded
Scenario: A file over the size limit is rejected as too large
When I upload a file larger than the size limit
Then it is rejected with 413 and a message about the limit
And nothing is recorded
Scenario: Cover art on an audio file is not a video track
When I upload an mp3 whose only video stream is attached cover art
Then it is treated as audio and no video notice is shown
Scenario: A file that is not media is rejected as unreadable
When I upload a file ffprobe cannot read
Then it is rejected as the file's fault
Scenario: A broken ffprobe is reported as our fault, not as a bad file
When the prober itself fails
Then the upload is an error rather than a rejection of my file
Scenario: A recording becomes a transcript and the audio is deleted
Given I have uploaded a recording
When it has been transcribed
Then the transcript is stored privately for me
And the audio is deleted
Scenario: The endpoint that served a transcript is recorded and shown
Given a recording was transcribed by the fallback endpoint
When I view its page
Then it names the fallback endpoint, because that means the audio left the homelab
Scenario: The local endpoint is tried first and the fallback only on failure
When the primary transcription endpoint fails for any reason
Then the fallback endpoint transcribes it
But when the primary succeeds the fallback is never contacted
Scenario: A failed transcription ends in a message, never a spinner
When every transcription endpoint fails
Then the upload is failed with a typed reason
And its page shows that reason and stops polling
Scenario: A restart fails the jobs it interrupted
Given the service restarted while a recording was queued or transcribing
When it starts again
Then those uploads are failed as interrupted
And any audio left in the spool is deleted
Scenario: Only one recording is transcribed at a time
When several recordings are queued
Then they are transcribed one after another
Scenario: My uploads are invisible to other users
Given another user has uploaded a recording
When I ask for it by its id
Then I get 404 and no part of its title or text
And the chat for it calls no model
Scenario: I can ask about my recording and get a grounded answer
Given my recording has been transcribed
When I ask a question about it
Then the answer comes from my transcript through the same chat as a video's
Scenario: A long recording is not cut to its first third
Given my recording's transcript is longer than the caption default
When I ask a question about it
Then the transcript is sent with the upload budget, not the 18,000-character caption default
# ADR-034, two engines and a local reconciler (off unless TAPIR_RECONCILE_BASE_MODEL is set)
Scenario: Two engines transcribe the audio and their disagreements are reconciled
Given two-engine transcription is enabled
When a recording is transcribed
Then both engines run on the same audio, the fast one first
And a local model chooses between their disagreements
And the endpoint names every engine and the model that took part
Scenario: Trivial disagreements never reach the model
When the engines differ only by a compound written as one word or two, or by a filler word
Then the base reading is kept without a model call
Scenario: Differences close together are one decision
When two differences are separated by a single shared word
Then they are asked about as one dispute
Scenario: An unsure or changed reading is flagged for me
When the model changes a reading or cannot decide
Then the page lists the spot with a minute mark, both readings and which was used
And an unsure spot keeps the base reading and is marked uncertain
Scenario: The model cannot be a cloud model
When the reconciliation model or base engine is not a koala/ or iguana/ alias
Then the service refuses to start
And a cloud alias that merely contains a local host name is refused too
Scenario: One engine failing does not fail the upload
When one of the two engines fails
Then the other engine's transcript is used
And the page says only one engine produced it and why
And the cloud fallback is not used
Scenario: Only when both local engines fail does the cloud fallback run
When both local engines fail and a fallback is configured
Then the fallback transcribes it and the row records that it did
Scenario: A model that is down gives an unreconciled transcript, not a failed job
When the reconciliation model fails three times in a row
Then the base transcript is kept unreconciled with the reason
And the page says it could not be reconciled
Scenario: A cancelled job never reaches the cloud
When the job is cancelled during either engine
Then neither the second engine nor the cloud fallback is called
Scenario: A single-engine transcript claims no second opinion
When only one engine was configured
Then the page does not mention reconciliation
+69
View File
@@ -0,0 +1,69 @@
Feature: Chat with a video's stored transcript
As a reader whose summary made me want to dig deeper
I want to ask questions about the video without watching it
So that I can go further on the ones worth it, without leaving the reader
# ADR-027. The load-bearing constraint is safety-by-construction: chat runs
# ONLY against an already-stored transcript (ADR-021) and never fetches captions,
# never touches the rate gate, never reaches YouTube. Entry is from the summary
# view of one's OWN video; the conversation is ephemeral (no persisted history).
Background:
Given I have a summarized video with a stored transcript
Scenario: A summary view offers a deeper-dive into the video
When I view the summary
Then I see a "dig deeper" affordance that opens a chat about this video
And it opens the chat in place, below the summary, without leaving the page
Scenario: The summary and the chat are on one page
When I open the chat
Then the summary stays visible alongside the chat
And I can read the summary while I ask questions
Scenario: Ask a question answered from the stored transcript
When I ask a question in the chat
Then the answer is produced from the stored transcript
And no caption fetch and no YouTube call occurs
Scenario: Chat never fetches captions or reaches YouTube
When I ask a question in the chat
Then Tapir reads only the stored transcript
And the caption-fetch and video-fetch paths are never invoked
Scenario: A video with no stored transcript offers no chat
Given a video that has no stored transcript
When I open the chat for it
Then I am told chat is not available
And no fetch is attempted and no model is called
Scenario: The default model is the summary's model and is switchable
When I open the chat
Then the model defaults to the model that produced the summary
And I can switch among the offered chain models
Scenario: Switching models re-runs against the same transcript
When I ask a question with a different chain model selected
Then the chosen model answers
And it answers against the same stored transcript
Scenario: The cloud model is hidden when cloud is disabled
Given the cloud fallback model is disabled
Then the chat switcher offers only local models
Scenario: A long transcript is bounded and the chat says so
Given the stored transcript is longer than the model budget
When I ask a question
Then the answer is produced from a bounded portion
And the chat notes that it worked from a bounded portion
Scenario: A multi-turn conversation is ephemeral
When I ask a follow-up question
Then the prior turn is carried into the answer
And nothing about the conversation is written to the database
Scenario: Chat is reachable only from my own summary view
Given another user has a summarized video with a stored transcript
When I try to open the chat for their video
Then I get a not-found response
And no model is called
+9 -2
View File
@@ -24,12 +24,19 @@ Feature: Connect and manage video accounts
Then a discovery pass for my account is triggered right away Then a discovery pass for my account is triggered right away
And I do not have to wait for the next scheduled pass to see my videos And I do not have to wait for the next scheduled pass to see my videos
Scenario: Connecting summarizes my newest videos right away Scenario: Connecting summarizes my best recent videos right away
Given I have no connected video accounts Given I have no connected video accounts
When I connect my YouTube account When I connect my YouTube account
Then up to the onboarding cap of my newest videos are summarized through the rate gate Then up to the onboarding cap of my newest likely-good videos are summarized through the rate gate
And videos whose known duration is too short or too long are skipped
And the rest are left to the scheduled recency-bounded pass And the rest are left to the scheduled recency-bounded pass
Scenario: The onboarding burst summarizes with a stronger model
Given I have no connected video accounts
When I connect my YouTube account
Then the burst summarizes with the stronger onboarding model first
And the standard summarizer chain still follows as a fallback
Scenario: Tokens are never stored in the clear Scenario: Tokens are never stored in the clear
When I connect any video account When I connect any video account
Then no OAuth token value is stored in the database Then no OAuth token value is stored in the database
+37
View File
@@ -0,0 +1,37 @@
Feature: Inline-expand summary + Q&A in the list (ADR-031, #16)
As a reader skimming my summaries
I want to open a summary and its Q&A in place in the list
So that I get the full read and follow-up without leaving the list (SPA-like, no page hop)
# HTMX inline-expand, no SPA framework (ADR-031). Each scenario maps to a Go test
# in scenario_coverage_test.go (the BDD name-coverage gate).
Scenario: A summarized card expands to the full summary in place
Given a summarized video in my list
When I expand its card
Then the full summary, highlights, and takeaways are returned as an in-place card fragment, not a full page
Scenario: An expanded card collapses back to the compact card
Given an expanded card
When I collapse it
Then the compact card fragment is returned in its place
Scenario: The expanded card offers the Q&A dock
Given chat is enabled
When a summarized card is expanded
Then the expanded card includes the deeper-dive chat affordance for that video
Scenario: Only a summarized card offers expand
Given a discovered but not-yet-summarized card
When the card is rendered
Then it shows its summarize/queue footer and no expand affordance
Scenario: With JS off the card still reaches the full summary
Given a summarized card
When it is rendered
Then its expand affordance carries an href to the detail page as a no-JS fallback
Scenario: The detail page and the expanded card show the same summary
Given a summarized video
When I view it on the detail page and as an expanded card
Then both render the same summary body (one shared fragment, no drift)
+46
View File
@@ -0,0 +1,46 @@
Feature: Observability — timing and metrics for performance and UX (ADR-030, #15)
As the maintainer running Tapir for pilot users
I want timing and Prometheus metrics for the activities that drive performance and UX
So that I can see latency, model behaviour, and usage — and feed the Stage-0 eval gate
# AI metrics are the priority (ADR-030 R3). Each scenario maps to a Go test in
# test/acceptance/scenario_coverage_test.go (the BDD name-coverage gate).
Scenario: Summarization latency is recorded per endpoint
Given the summarizer runs a transcript through its endpoint chain
When an endpoint returns a parseable summary
Then the summarize latency is recorded with the model, outcome "success", and whether it was a fallback
Scenario: A failing summarizer endpoint records its failure outcome
Given the summarizer runs a transcript through its endpoint chain
When an endpoint errors or returns unparseable output
Then the summarize latency is recorded with outcome "error" or "parse_error" before the chain advances
Scenario: Caption fetch latency is recorded by outcome
Given a caption fetch is attempted for a video
When it resolves to captions, no captions, or a rate limit
Then the caption-fetch latency is recorded labelled by that outcome
Scenario: LLM token usage is recorded from the completion
Given an LLM completion returns a usage block with prompt and completion tokens
When the client finishes the call
Then the prompt and completion tokens are recorded for that model
Scenario: Q&A answer latency is recorded
Given a user asks a question about a video
When the answer is produced from the stored transcript
Then the chat answer latency is recorded for the answering model
Scenario: HTTP requests are counted by route, method, and status
Given the metrics HTTP middleware wraps the app
When a request is served against a registered route
Then it is counted and timed under the bounded route pattern, not the raw path
Scenario: A successful login is counted
Given a user completes the OIDC callback and a session is established
Then the login counter is incremented
Scenario: The metrics endpoint is not on the public app port
Given the service is running
When the public app mux is inspected
Then it exposes no /metrics route — metrics are served on the dedicated metrics port only
@@ -31,5 +31,16 @@ Feature: Summarize new videos from subscribed channels
When the watcher sees "Designing for Attention" again When the watcher sees "Designing for Attention" again
Then Tapir does not produce a second summary for it Then Tapir does not produce a second summary for it
Scenario: Re-analyzing a stored video does not re-fetch its transcript
Given a transcript for "Designing for Attention" is already stored
When the video is summarized again
Then Tapir reads the stored transcript
And Tapir does not fetch captions from YouTube
# Captions-first is the core path (ADR-007). Audio-download + speech-to-text is # Captions-first is the core path (ADR-007). Audio-download + speech-to-text is
# deferred and intentionally has no scenario here yet. # deferred and intentionally has no scenario here yet.
#
# Transcript persistence (ADR-021): the stored transcript is shared, keyed by
# (provider, provider_video_id) and read before any caption fetch, so the
# re-analysis scenario above also covers paste-a-URL and the onboarding burst —
# both summarize through the same engine chokepoint.
+29
View File
@@ -0,0 +1,29 @@
Feature: Visual refresh — one charm-reader layout, light + dark themes (ADR-032, #17)
As a reader who likes a TUI/charm aesthetic
I want a fresh look with a light and a dark theme
So that the app feels distinctive and reads well in either mode
# Direction B (light) + C (dark) are one layout in two palettes (CSS variables),
# plus a persisted toggle and an OS-preference default. Colours are reviewed via
# the mockups in docs/sketches/, not unit-tested; these scenarios cover the
# testable structure. Each maps to a Go test in scenario_coverage_test.go.
Scenario: Light and dark themes share one layout via CSS variables
Given the app stylesheet
When it is rendered
Then it defines a light palette on the root and a dark palette under data-theme="dark", with no duplicate markup
Scenario: Without a stored choice the theme follows the OS preference
Given a visitor with no saved theme
When the page loads
Then a prefers-color-scheme dark block applies the dark palette automatically
Scenario: A persisted toggle switches light and dark
Given any page
When it is rendered
Then it includes a theme-toggle control and a small script that flips data-theme and persists the choice
Scenario: The expanded card embeds the video player
Given a summarized video with a valid provider id
When its card is expanded
Then the expanded card includes the embedded video player
+12 -2
View File
@@ -1,4 +1,4 @@
module gitea.d-ma.be/mathias/tapir module git.d-ma.be/mathias/tapir
go 1.26.1 go 1.26.1
@@ -9,23 +9,33 @@ require (
github.com/go-jose/go-jose/v4 v4.1.4 github.com/go-jose/go-jose/v4 v4.1.4
github.com/golang-migrate/migrate/v4 v4.19.1 github.com/golang-migrate/migrate/v4 v4.19.1
github.com/jackc/pgx/v5 v5.9.2 github.com/jackc/pgx/v5 v5.9.2
github.com/prometheus/client_golang v1.23.2
github.com/prometheus/client_model v0.6.2
github.com/stretchr/testify v1.11.1 github.com/stretchr/testify v1.11.1
golang.org/x/crypto v0.45.0
golang.org/x/oauth2 v0.36.0 golang.org/x/oauth2 v0.36.0
golang.org/x/time v0.15.0 golang.org/x/time v0.15.0
) )
require ( require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa // indirect github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/lib/pq v1.10.9 // indirect github.com/lib/pq v1.10.9 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/common v0.66.1 // indirect
github.com/prometheus/procfs v0.16.1 // indirect
github.com/rogpeppe/go-internal v1.15.0 // indirect github.com/rogpeppe/go-internal v1.15.0 // indirect
github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 // indirect github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect
golang.org/x/sync v0.18.0 // indirect golang.org/x/sync v0.18.0 // indirect
golang.org/x/sys v0.41.0 // indirect
golang.org/x/text v0.31.0 // indirect golang.org/x/text v0.31.0 // indirect
google.golang.org/protobuf v1.36.8 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect
) )
+26 -6
View File
@@ -4,6 +4,10 @@ github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERo
github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU= github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
github.com/a-h/templ v0.3.1020 h1:ypAT/L5ySWEnZ6Zft/5yfoWXYYkhFNvEFOeeqecg4tw= github.com/a-h/templ v0.3.1020 h1:ypAT/L5ySWEnZ6Zft/5yfoWXYYkhFNvEFOeeqecg4tw=
github.com/a-h/templ v0.3.1020/go.mod h1:A2DlK61v+K+NRoGnhmYbNYVmtYHcFO5/AisMvBdDxTM= github.com/a-h/templ v0.3.1020/go.mod h1:A2DlK61v+K+NRoGnhmYbNYVmtYHcFO5/AisMvBdDxTM=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI= github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI=
github.com/containerd/errdefs v1.0.0/go.mod h1:+YBYIdtsnF4Iw6nWZhJcqGSg/dwvV7tyJ/kCkyJ2k+M= github.com/containerd/errdefs v1.0.0/go.mod h1:+YBYIdtsnF4Iw6nWZhJcqGSg/dwvV7tyJ/kCkyJ2k+M=
github.com/containerd/errdefs/pkg v0.3.0 h1:9IKJ06FvyNlexW690DXuQNx2KA2cUJXx151Xdx3ZPPE= github.com/containerd/errdefs/pkg v0.3.0 h1:9IKJ06FvyNlexW690DXuQNx2KA2cUJXx151Xdx3ZPPE=
@@ -37,8 +41,8 @@ github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang-migrate/migrate/v4 v4.19.1 h1:OCyb44lFuQfYXYLx1SCxPZQGU7mcaZ7gH9yH4jSFbBA= github.com/golang-migrate/migrate/v4 v4.19.1 h1:OCyb44lFuQfYXYLx1SCxPZQGU7mcaZ7gH9yH4jSFbBA=
github.com/golang-migrate/migrate/v4 v4.19.1/go.mod h1:CTcgfjxhaUtsLipnLoQRWCrjYXycRz/g5+RWDuYgPrE= github.com/golang-migrate/migrate/v4 v4.19.1/go.mod h1:CTcgfjxhaUtsLipnLoQRWCrjYXycRz/g5+RWDuYgPrE=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa h1:s+4MhCQ6YrzisK6hFJUX53drDT4UsSW3DEhKn0ifuHw= github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa h1:s+4MhCQ6YrzisK6hFJUX53drDT4UsSW3DEhKn0ifuHw=
github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa/go.mod h1:a/s9Lp5W7n/DD0VrVoyJ00FbP2ytTPDVOivvn2bMlds= github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa/go.mod h1:a/s9Lp5W7n/DD0VrVoyJ00FbP2ytTPDVOivvn2bMlds=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM= github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
@@ -49,10 +53,14 @@ github.com/jackc/pgx/v5 v5.9.2 h1:3ZhOzMWnR4yJ+RW1XImIPsD1aNSz4T4fyP7zlQb56hw=
github.com/jackc/pgx/v5 v5.9.2/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4= github.com/jackc/pgx/v5 v5.9.2/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo= github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0= github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk= github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw= github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw=
github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o= github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0= github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0=
@@ -61,6 +69,8 @@ github.com/moby/term v0.5.0 h1:xt8Q1nalod/v7BqbG21f8mQPqH+xAaC9C3N3wfWbVP0=
github.com/moby/term v0.5.0/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y= github.com/moby/term v0.5.0/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y=
github.com/morikuni/aec v1.0.0 h1:nP9CBfwrvYnBRgY6qfDQkygYDmYwOilePFkwzv4dU8A= github.com/morikuni/aec v1.0.0 h1:nP9CBfwrvYnBRgY6qfDQkygYDmYwOilePFkwzv4dU8A=
github.com/morikuni/aec v1.0.0/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc= github.com/morikuni/aec v1.0.0/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM= github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
github.com/opencontainers/image-spec v1.1.0 h1:8SG7/vwALn54lVB/0yZ/MMwhFrPYtpEHQb2IpWsCzug= github.com/opencontainers/image-spec v1.1.0 h1:8SG7/vwALn54lVB/0yZ/MMwhFrPYtpEHQb2IpWsCzug=
@@ -70,6 +80,14 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o=
github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9ZoGs=
github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA=
github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg=
github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is=
github.com/rogpeppe/go-internal v1.15.0 h1:D0RCU5rMAp+SpgkiNdrjfJ+LX4J1M32V2NeCY7EJ6hc= github.com/rogpeppe/go-internal v1.15.0 h1:D0RCU5rMAp+SpgkiNdrjfJ+LX4J1M32V2NeCY7EJ6hc=
github.com/rogpeppe/go-internal v1.15.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs= github.com/rogpeppe/go-internal v1.15.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
@@ -91,8 +109,8 @@ go.opentelemetry.io/otel/trace v1.37.0 h1:HLdcFNbRQBE2imdSEgm/kwqmQj1Or1l/7bW6mx
go.opentelemetry.io/otel/trace v1.37.0/go.mod h1:TlgrlQ+PtQO5XFerSPUYG0JSgGyryXewPGyayAWSBS0= go.opentelemetry.io/otel/trace v1.37.0/go.mod h1:TlgrlQ+PtQO5XFerSPUYG0JSgGyryXewPGyayAWSBS0=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
golang.org/x/crypto v0.45.0 h1:jMBrvKuj23MTlT0bQEOBcAE0mjg8mK9RXFhRH6nyF3Q= go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=
golang.org/x/crypto v0.45.0/go.mod h1:XTGrrkGJve7CYK7J8PEww4aY7gM3qMCElcJQ8n8JdX4= go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs= golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q= golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
golang.org/x/sync v0.18.0 h1:kr88TuHDroi+UVf+0hZnirlk8o8T+4MrK6mr60WkH/I= golang.org/x/sync v0.18.0 h1:kr88TuHDroi+UVf+0hZnirlk8o8T+4MrK6mr60WkH/I=
@@ -103,6 +121,8 @@ golang.org/x/text v0.31.0 h1:aC8ghyu4JhP8VojJ2lEHBnochRno1sgL6nEi9WGFGMM=
golang.org/x/text v0.31.0/go.mod h1:tKRAlv61yKIjGGHX/4tP1LTbc13YSec1pxVEWXzfoeM= golang.org/x/text v0.31.0/go.mod h1:tKRAlv61yKIjGGHX/4tP1LTbc13YSec1pxVEWXzfoeM=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U= golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno= golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc=
google.golang.org/protobuf v1.36.8/go.mod h1:fuxRtAxBytpl4zzqUh6/eyUujkJdNiuEkXntxiD/uRU=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
+47
View File
@@ -0,0 +1,47 @@
package audio
import (
"context"
"fmt"
"os"
"os/exec"
"strings"
)
// FFmpeg converts any audio container to the strict format whisper wants: 16 kHz
// mono signed 16-bit PCM WAV (ADR-033). It ALWAYS transcodes, even when the input
// already is that format — a Teams export often is, and the transcode is then
// nearly free; assuming it would be the bug. This replaces whisper-server's
// --convert, whose --tmp-dir is created at process start and rots under macOS
// periodic cleanup.
type FFmpeg struct {
// Bin is the ffmpeg executable ("ffmpeg" resolves via PATH).
Bin string
}
// ToWAV writes src, converted, to dst. Video streams are dropped (-vn): by now the
// file has already been probed as audio-only, but a cover-art stream must not end
// up in the WAV.
func (f *FFmpeg) ToWAV(ctx context.Context, src, dst string) error {
bin := f.Bin
if bin == "" {
bin = "ffmpeg"
}
// Input restrictions go BEFORE -i: placed after it they would apply to the output.
args := append([]string{"-y", "-v", "error"}, inputRestrictions...)
args = append(args, "-i", src, "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le", "-f", "wav", dst)
cmd := exec.CommandContext(ctx, bin, args...)
cmd.WaitDelay = killGrace
if out, err := cmd.CombinedOutput(); err != nil {
if ctxErr := ctx.Err(); ctxErr != nil {
return fmt.Errorf("audio: ffmpeg: %w", ctxErr)
}
return fmt.Errorf("audio: ffmpeg: %w: %s", err, strings.TrimSpace(string(out)))
}
// A zero exit that wrote nothing is still a failure: never hand whisper a
// missing file and let the error surface three layers later.
if info, err := os.Stat(dst); err != nil || info.Size() == 0 {
return fmt.Errorf("audio: ffmpeg wrote no output at %s", dst)
}
return nil
}
+79
View File
@@ -0,0 +1,79 @@
package audio_test
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/audio"
)
func TestFFmpegToWAV_AsksForSixteenKilohertzMonoPCMAlways(t *testing.T) {
// Always transcode, even when the input already is 16 kHz mono: a Teams export
// often is, and the point is to not ASSUME (ADR-033). Records the argv.
rec := filepath.Join(t.TempDir(), "argv")
bin := stub(t, `printf '%s\n' "$@" > `+rec+`
for a; do last="$a"; done
printf data > "$last"`)
err := (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/spool/in", filepath.Join(t.TempDir(), "out.wav"))
require.NoError(t, err)
raw, err := os.ReadFile(rec)
require.NoError(t, err)
argv := strings.Split(strings.TrimSpace(string(raw)), "\n")
joined := " " + strings.Join(argv, " ") + " "
for _, want := range []string{" -ac 1 ", " -ar 16000 ", " -c:a pcm_s16le ", " -vn ", " -i /spool/in "} {
assert.Contains(t, joined, want)
}
assert.True(t, strings.HasSuffix(strings.TrimSpace(joined), "out.wav"), "the output path is the last argument")
}
func TestFFmpegToWAV_NonzeroExitIsAnErrorCarryingStderr(t *testing.T) {
bin := stub(t, `echo "moov atom not found" >&2; exit 1`)
err := (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/in", filepath.Join(t.TempDir(), "o.wav"))
require.Error(t, err)
assert.Contains(t, err.Error(), "moov atom not found")
}
func TestFFmpegToWAV_ZeroExitWithNoOutputIsAnError(t *testing.T) {
bin := stub(t, `exit 0`)
err := (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/in", filepath.Join(t.TempDir(), "never-written.wav"))
require.Error(t, err)
}
func TestFFmpegToWAV_MissingBinaryIsAnError(t *testing.T) {
err := (&audio.FFmpeg{Bin: filepath.Join(t.TempDir(), "no-ffmpeg")}).ToWAV(context.Background(), "/in", "/out.wav")
require.Error(t, err)
assert.False(t, errors.Is(err, context.Canceled))
}
func TestFFmpegToWAV_RestrictsWhatTheFileMayMakeFFmpegOpen(t *testing.T) {
rec := filepath.Join(t.TempDir(), "argv")
bin := stub(t, `printf '%s\n' "$@" > `+rec+`
for a; do last="$a"; done
printf data > "$last"`)
err := (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/spool/in", filepath.Join(t.TempDir(), "o.wav"))
require.NoError(t, err)
args := argvOf(t, rec)
pw, fw, in := indexOf(args, "-protocol_whitelist"), indexOf(args, "-format_whitelist"), indexOf(args, "-i")
require.GreaterOrEqual(t, pw, 0, "argv: %v", args)
require.GreaterOrEqual(t, fw, 0, "argv: %v", args)
assert.Equal(t, "file,pipe", args[pw+1])
assert.Equal(t, audio.AllowedDemuxers, args[fw+1])
assert.Less(t, pw, in, "input options must come BEFORE -i or ffmpeg applies them to the output")
assert.Less(t, fw, in)
}
+109
View File
@@ -0,0 +1,109 @@
// Package audio holds the adapters that inspect and convert user-supplied audio
// with ffmpeg's tools (ADR-033). They sit behind small ports so the use case is
// tested with fakes and only this package knows a subprocess is involved.
package audio
import (
"context"
"encoding/json"
"errors"
"fmt"
"os/exec"
"strconv"
"time"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
// FFprobe implements uploads.Prober by running ffprobe on the spooled file.
type FFprobe struct {
// Bin is the ffprobe executable ("ffprobe" resolves via PATH).
Bin string
}
// AllowedDemuxers are the input formats an upload may be. The input is
// attacker-supplied, and a file's CONTENT can tell ffmpeg to open other files or
// URLs (HLS and concat playlists, image2 sequences, network demuxers). Recent
// ffmpeg refuses the ones tried (7.0.2: HLS without a .m3u8 name is not sniffed,
// concat rejects unsafe paths), but that is version luck; this list is not. It
// holds the containers real recordings use and none of the playlist or network
// formats. "mov" covers .mp4/.m4a/.3gp; "matroska" covers .webm.
const AllowedDemuxers = "wav,mov,mp4,m4a,3gp,mp3,ogg,matroska,webm,flac,aac,opus,aiff,amr,asf"
// inputRestrictions are ffmpeg/ffprobe INPUT options and must precede the input.
// Local files and pipes only: no network, no other protocols.
var inputRestrictions = []string{"-protocol_whitelist", "file,pipe", "-format_whitelist", AllowedDemuxers}
// killGrace bounds how long Probe waits for a cancelled ffprobe's pipes to close.
const killGrace = 2 * time.Second
// Probe runs ffprobe and classifies the outcome. A non-zero exit means ffprobe
// looked at the file and could not read it: that is the FILE's fault and returns
// uploads.ErrUnreadable. A missing binary, a cancelled context or unparsable
// output is OURS and is returned as an ordinary error, so it is never reported to
// the user as their file being bad.
func (f *FFprobe) Probe(ctx context.Context, path string) (uploads.Probe, error) {
bin := f.Bin
if bin == "" {
bin = "ffprobe"
}
args := append([]string{"-v", "error"}, inputRestrictions...)
args = append(args, "-print_format", "json", "-show_format", "-show_streams", path)
cmd := exec.CommandContext(ctx, bin, args...)
cmd.WaitDelay = killGrace
out, err := cmd.Output()
if err != nil {
if ctxErr := ctx.Err(); ctxErr != nil {
return uploads.Probe{}, fmt.Errorf("audio: ffprobe: %w", ctxErr)
}
var exit *exec.ExitError
if errors.As(err, &exit) {
return uploads.Probe{}, fmt.Errorf("%w: ffprobe: %s", uploads.ErrUnreadable, exit.Stderr)
}
return uploads.Probe{}, fmt.Errorf("audio: run ffprobe: %w", err)
}
p, err := ParseProbe(out)
if err != nil {
return uploads.Probe{}, fmt.Errorf("audio: ffprobe output: %w", err)
}
return p, nil
}
// probeJSON is the subset of `ffprobe -print_format json` output that matters.
type probeJSON struct {
Streams []struct {
CodecType string `json:"codec_type"`
Disposition struct {
AttachedPic int `json:"attached_pic"`
} `json:"disposition"`
} `json:"streams"`
Format struct {
Duration string `json:"duration"`
} `json:"format"`
}
// ParseProbe turns ffprobe's JSON into a uploads.Probe. A video stream that is
// only attached cover art (an mp3/m4a with an album picture) is not video. A
// missing or "N/A" duration yields zero, which uploads.Validate rejects as
// unreadable rather than accepting an empty recording.
func ParseProbe(data []byte) (uploads.Probe, error) {
var j probeJSON
if err := json.Unmarshal(data, &j); err != nil {
return uploads.Probe{}, fmt.Errorf("decode: %w", err)
}
var p uploads.Probe
for _, s := range j.Streams {
switch s.CodecType {
case "audio":
p.HasAudio = true
case "video":
if s.Disposition.AttachedPic == 0 {
p.HasVideo = true
}
}
}
if secs, err := strconv.ParseFloat(j.Format.Duration, 64); err == nil && secs > 0 {
p.Duration = time.Duration(secs * float64(time.Second))
}
return p, nil
}
+186
View File
@@ -0,0 +1,186 @@
package audio_test
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/audio"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
func TestParseProbe(t *testing.T) {
tests := []struct {
name string
json string
want uploads.Probe
}{
{
name: "audio-only m4a",
json: `{"streams":[{"codec_type":"audio","disposition":{"attached_pic":0}}],"format":{"duration":"2701.5"}}`,
want: uploads.Probe{Duration: 2701*time.Second + 500*time.Millisecond, HasAudio: true},
},
{
name: "mp4 with a real video stream",
json: `{"streams":[{"codec_type":"video","disposition":{"attached_pic":0}},{"codec_type":"audio"}],"format":{"duration":"240.0"}}`,
want: uploads.Probe{Duration: 240 * time.Second, HasAudio: true, HasVideo: true},
},
{
name: "mp3 whose only video stream is attached cover art is NOT video",
json: `{"streams":[{"codec_type":"audio"},{"codec_type":"video","disposition":{"attached_pic":1}}],"format":{"duration":"60"}}`,
want: uploads.Probe{Duration: 60 * time.Second, HasAudio: true},
},
{
name: "no streams at all",
json: `{"streams":[],"format":{"duration":"10"}}`,
want: uploads.Probe{Duration: 10 * time.Second},
},
{
name: "duration absent means zero, which Validate rejects as unreadable",
json: `{"streams":[{"codec_type":"audio"}],"format":{}}`,
want: uploads.Probe{HasAudio: true},
},
{
name: "duration reported as N/A means zero",
json: `{"streams":[{"codec_type":"audio"}],"format":{"duration":"N/A"}}`,
want: uploads.Probe{HasAudio: true},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := audio.ParseProbe([]byte(tt.json))
require.NoError(t, err)
assert.Equal(t, tt.want, got)
})
}
}
func TestParseProbe_GarbageIsAnError(t *testing.T) {
_, err := audio.ParseProbe([]byte("not json"))
assert.Error(t, err)
}
// stub writes an executable shell script standing in for ffprobe. This is a fake
// at the process boundary: the real binary is not on every host (it is not on
// koala), and what is under test is how Probe drives and interprets a process.
func stub(t *testing.T, body string) string {
t.Helper()
p := filepath.Join(t.TempDir(), "ffprobe")
require.NoError(t, os.WriteFile(p, []byte("#!/bin/sh\n"+body+"\n"), 0o755))
return p
}
func TestFFprobeProbe_ParsesWhatTheProcessPrints(t *testing.T) {
bin := stub(t, `echo '{"streams":[{"codec_type":"audio"}],"format":{"duration":"90"}}'`)
got, err := (&audio.FFprobe{Bin: bin}).Probe(context.Background(), "/ignored")
require.NoError(t, err)
assert.Equal(t, uploads.Probe{Duration: 90 * time.Second, HasAudio: true}, got)
}
func TestFFprobeProbe_PassesTheFilePathAsTheLastArgument(t *testing.T) {
out := filepath.Join(t.TempDir(), "args")
bin := stub(t, `for a; do last="$a"; done; printf '%s' "$last" > `+out+`
echo '{"streams":[{"codec_type":"audio"}],"format":{"duration":"1"}}'`)
_, err := (&audio.FFprobe{Bin: bin}).Probe(context.Background(), "/spool/upload-123")
require.NoError(t, err)
got, err := os.ReadFile(out)
require.NoError(t, err)
assert.Equal(t, "/spool/upload-123", string(got))
}
func TestFFprobeProbe_NonzeroExitMeansTheFileIsUnreadable(t *testing.T) {
bin := stub(t, `echo "Invalid data found when processing input" >&2; exit 1`)
_, err := (&audio.FFprobe{Bin: bin}).Probe(context.Background(), "/x")
assert.ErrorIs(t, err, uploads.ErrUnreadable)
}
func TestFFprobeProbe_MissingBinaryIsOurFaultNotAnUnreadableFile(t *testing.T) {
_, err := (&audio.FFprobe{Bin: filepath.Join(t.TempDir(), "no-such-ffprobe")}).Probe(context.Background(), "/x")
require.Error(t, err)
assert.False(t, errors.Is(err, uploads.ErrUnreadable),
"a missing ffprobe must not be reported to the user as their file being bad")
}
func TestFFprobeProbe_StopsWhenTheContextIsCancelled(t *testing.T) {
bin := stub(t, `sleep 30`)
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
start := time.Now()
_, err := (&audio.FFprobe{Bin: bin}).Probe(ctx, "/x")
require.Error(t, err)
assert.Less(t, time.Since(start), 5*time.Second)
assert.False(t, errors.Is(err, uploads.ErrUnreadable), "a timeout is not a verdict on the file")
}
// argvStub records the arguments it was called with into a file and then behaves as
// a successful ffprobe, so a test can assert exactly what the process was asked.
func argvStub(t *testing.T) (bin, argvFile string) {
t.Helper()
argvFile = filepath.Join(t.TempDir(), "argv")
bin = stub(t, `printf '%s\n' "$@" > `+argvFile+`
echo '{"streams":[{"codec_type":"audio"}],"format":{"duration":"1"}}'`)
return bin, argvFile
}
func argvOf(t *testing.T, file string) []string {
t.Helper()
raw, err := os.ReadFile(file)
require.NoError(t, err)
return strings.Split(strings.TrimSpace(string(raw)), "\n")
}
func indexOf(args []string, want string) int {
for i, a := range args {
if a == want {
return i
}
}
return -1
}
func TestFFprobeProbe_RestrictsWhatTheFileMayMakeFFmpegOpen(t *testing.T) {
// The input is attacker-supplied. ffmpeg can be told by a file's CONTENT to open
// other files or URLs (HLS and concat playlists); recent versions refuse by
// default, but that is version luck. Pin it: local files and pipes only, and
// only the demuxers real audio recordings use.
bin, argvFile := argvStub(t)
_, err := (&audio.FFprobe{Bin: bin}).Probe(context.Background(), "/spool/upload-1")
require.NoError(t, err)
args := argvOf(t, argvFile)
pw := indexOf(args, "-protocol_whitelist")
require.GreaterOrEqual(t, pw, 0, "argv: %v", args)
assert.Equal(t, "file,pipe", args[pw+1])
fw := indexOf(args, "-format_whitelist")
require.GreaterOrEqual(t, fw, 0, "argv: %v", args)
assert.Equal(t, audio.AllowedDemuxers, args[fw+1])
assert.Equal(t, "/spool/upload-1", args[len(args)-1], "the path stays the last argument")
}
func TestAllowedDemuxers_ExcludeThePlaylistFormats(t *testing.T) {
for _, bad := range []string{"concat", "hls", "applehttp", "dash", "rtsp", "sdp", "image2"} {
assert.NotContains(t, strings.Split(audio.AllowedDemuxers, ","), bad,
"%s can make ffmpeg open other files or URLs named inside the upload", bad)
}
for _, good := range []string{"wav", "mov", "mp4", "m4a", "mp3", "ogg", "matroska", "flac"} {
assert.Contains(t, strings.Split(audio.AllowedDemuxers, ","), good)
}
}
@@ -0,0 +1,32 @@
package audio_test
import (
"context"
"os"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/audio"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
// TestFFprobeLive_HostilePlaylistsAreRefusedAndRealAudioIsNot runs the REAL ffprobe
// (opt-in: TAPIR_LIVE_FFPROBE) on crafted concat/HLS files and on a real m4a.
func TestFFprobeLive_HostilePlaylistsAreRefusedAndRealAudioIsNot(t *testing.T) {
bin := os.Getenv("TAPIR_LIVE_FFPROBE")
if bin == "" {
t.Skip("set TAPIR_LIVE_FFPROBE (and TAPIR_LIVE_HOSTILE_DIR, TAPIR_LIVE_M4A) to run against the real ffprobe")
}
p := &audio.FFprobe{Bin: bin}
for _, name := range []string{"up-concat", "up-hls"} {
_, err := p.Probe(context.Background(), os.Getenv("TAPIR_LIVE_HOSTILE_DIR")+"/"+name)
assert.ErrorIs(t, err, uploads.ErrUnreadable, name)
}
got, err := p.Probe(context.Background(), os.Getenv("TAPIR_LIVE_M4A"))
require.NoError(t, err)
assert.True(t, got.HasAudio)
assert.False(t, got.HasVideo)
assert.InDelta(t, 82.5, got.Duration.Seconds(), 1)
}
+188
View File
@@ -0,0 +1,188 @@
// Package chat implements the per-video deeper-dive chat (ADR-027): a read-only
// QA over a video's ALREADY-STORED transcript (ADR-021). It is the enforcement
// point for the feature's load-bearing safety property — stored-transcript-only:
// the Service has NO VideoSource and NO caption-fetch dependency, only a
// Completer factory, so it CANNOT reach YouTube or the rate gate by construction.
// The caller supplies the stored transcript text; chat never fetches.
//
// It reuses the same LiteLLM gateway as the summarizer (a chat is a different
// call, not a new integration) and the same transcript-truncation discipline
// (TAPIR_MAX_TRANSCRIPT_CHARS) so a long transcript fits a small-context model.
package chat
import (
"context"
"fmt"
"log/slog"
"strings"
"time"
"unicode/utf8"
"git.d-ma.be/mathias/tapir/internal/metrics"
)
// Completer is the minimal LLM chat surface the Service needs. *llm.Client
// satisfies it; tests use a fake. It is the SAME surface the summarizer uses.
type Completer interface {
Complete(ctx context.Context, system, user string) (string, error)
}
// Turn is one completed exchange in an ephemeral, session-only conversation
// (ADR-027 v1: nothing is persisted).
type Turn struct {
Question string
Answer string
}
// Request is one chat turn: the chosen model, the stored transcript text, the
// prior turns (for multi-turn context within the session), and the new question.
type Request struct {
Model string
Transcript string
History []Turn
Question string
// MaxChars, when positive, overrides the Service's transcript budget for this
// request. The Service default is sized for YouTube captions (~22 minutes of
// speech); an uploaded recording is much longer, and truncation keeps only the
// head (ADR-033). Zero or negative means "use the Service default".
MaxChars int
}
// Reply is the model's answer plus whether the transcript was bounded to fit the
// model context (so the UI can be honest that an answer about the tail of a long
// video may be incomplete).
type Reply struct {
Answer string
Truncated bool
}
// Service answers questions against a stored transcript via a switchable set of
// models. models is the ordered, local-first list offered to the user (the cloud
// model is simply absent when disabled — see cmd wiring); maxChars bounds the
// transcript sent to any model (0 = unbounded). newClient builds a Completer for
// a chosen model alias (the same gateway, a different alias).
type Service struct {
newClient func(model string) Completer
models []string
maxChars int
}
// New constructs a Service. models must be non-empty and already filtered to the
// offerable set (cloud excluded when disabled) and de-duplicated by the caller.
func New(newClient func(model string) Completer, models []string, maxChars int) *Service {
return &Service{newClient: newClient, models: models, maxChars: maxChars}
}
// Models returns a copy of the offerable model list (local-first order).
func (s *Service) Models() []string {
out := make([]string, len(s.models))
copy(out, s.models)
return out
}
// offers reports whether model is in the offerable set — the guard that keeps an
// arbitrary, un-offered alias (e.g. a forged form value) from reaching the gateway.
func (s *Service) offers(model string) bool {
for _, m := range s.models {
if m == model {
return true
}
}
return false
}
// DefaultModel resolves the model a fresh chat opens with: the summary's own
// model when it is still an offered option (the ADR-027 default — chat continues
// in the model that produced the summary), otherwise the first offered model.
// Returns "" only when no models are configured.
func (s *Service) DefaultModel(summaryModel string) string {
if summaryModel != "" && s.offers(summaryModel) {
return summaryModel
}
if len(s.models) > 0 {
return s.models[0]
}
return ""
}
// Answer runs one chat turn. The model is forced back to a default if the request
// names an un-offered alias, so chat can never call the gateway with an arbitrary
// model. The transcript is truncated up front (reporting whether it was cut) and
// passed as system context; the running conversation is the user message.
func (s *Service) Answer(ctx context.Context, req Request) (Reply, error) {
if len(s.models) == 0 {
return Reply{}, fmt.Errorf("chat: no models configured")
}
model := req.Model
if !s.offers(model) {
model = s.DefaultModel("")
}
budget := s.maxChars
if req.MaxChars > 0 {
budget = req.MaxChars
}
transcript, truncated := truncate(req.Transcript, budget)
system := buildSystem(transcript, truncated)
user := buildUser(req.History, req.Question)
start := time.Now()
out, err := s.newClient(model).Complete(ctx, system, user)
if err != nil {
return Reply{}, fmt.Errorf("chat: %s: %w", model, err)
}
dur := time.Since(start)
metrics.ObserveChat(model, dur)
slog.Default().Info("chat answer", "model", model, "elapsed_ms", dur.Milliseconds())
answer := strings.TrimSpace(out)
if answer == "" {
return Reply{}, fmt.Errorf("chat: %s returned an empty answer", model)
}
return Reply{Answer: answer, Truncated: truncated}, nil
}
const systemPreamble = `You are Tapir, answering questions about ONE video using ONLY the transcript below.
Ground every answer in the transcript. If the transcript does not contain the answer, say so plainly rather than guessing.`
const truncatedNote = `
The transcript below is truncated to fit the model — if a question seems to concern something missing, note it may be beyond the available portion.`
// buildSystem frames the model as a transcript-grounded QA assistant and embeds
// the (possibly truncated) transcript as context.
func buildSystem(transcript string, truncated bool) string {
var b strings.Builder
b.WriteString(systemPreamble)
if truncated {
b.WriteString(truncatedNote)
}
b.WriteString("\n\nTranscript:\n")
b.WriteString(transcript)
return b.String()
}
// buildUser renders the running conversation as the user message: prior turns as
// Q/A pairs followed by the new question. Folding history into one message keeps
// the Completer surface (a single system+user call) unchanged — no new llm method.
func buildUser(history []Turn, question string) string {
var b strings.Builder
for _, t := range history {
fmt.Fprintf(&b, "Q: %s\nA: %s\n\n", t.Question, t.Answer)
}
fmt.Fprintf(&b, "Q: %s", question)
return b.String()
}
// truncate caps content to max bytes on a UTF-8 rune boundary, reporting whether
// it cut. It mirrors the summarizer's truncation discipline (ADR-022) but returns
// the cut flag so the chat UI can be honest about a bounded transcript. A
// non-positive max (or content already within budget) returns content unchanged.
func truncate(content string, max int) (string, bool) {
if max <= 0 || len(content) <= max {
return content, false
}
cut := max
for cut > 0 && !utf8.RuneStart(content[cut]) {
cut--
}
return content[:cut], true
}
+164
View File
@@ -0,0 +1,164 @@
package chat
import (
"context"
"errors"
"strings"
"testing"
)
// recordingCompleter captures the system+user it was asked with and returns a
// canned answer (or error). It also records which model alias built it.
type recordingCompleter struct {
model string
lastSystem string
lastUser string
answer string
err error
calls *int
}
func (c *recordingCompleter) Complete(_ context.Context, system, user string) (string, error) {
*c.calls++
c.lastSystem = system
c.lastUser = user
if c.err != nil {
return "", c.err
}
return c.answer, nil
}
// factory builds a recordingCompleter per model and records the last one built so
// the test can assert which model alias was actually used for the gateway call.
type factory struct {
answer string
err error
calls int
used *recordingCompleter
}
func (f *factory) make(model string) Completer {
c := &recordingCompleter{model: model, answer: f.answer, err: f.err, calls: &f.calls}
f.used = c
return c
}
func TestModelsAreOfferedLocalFirstAndCopied(t *testing.T) {
f := &factory{answer: "ok"}
s := New(f.make, []string{"koala/phi4-mini", "iguana/gemma4-26b"}, 0)
got := s.Models()
want := []string{"koala/phi4-mini", "iguana/gemma4-26b"}
if len(got) != len(want) || got[0] != want[0] || got[1] != want[1] {
t.Fatalf("Models() = %v, want %v", got, want)
}
// Mutating the returned slice must not corrupt the Service's list.
got[0] = "tampered"
if s.Models()[0] != "koala/phi4-mini" {
t.Fatal("Models() leaked its backing slice")
}
}
func TestDefaultModelIsTheSummarysModelWhenOffered(t *testing.T) {
f := &factory{answer: "ok"}
s := New(f.make, []string{"koala/phi4-mini", "iguana/gemma4-26b", "berget/mistral-small"}, 0)
if got := s.DefaultModel("iguana/gemma4-26b"); got != "iguana/gemma4-26b" {
t.Fatalf("DefaultModel(summary) = %q, want the summary's model", got)
}
// A summary model no longer offered (e.g. cloud disabled) falls back to first.
if got := s.DefaultModel("berget/old-model"); got != "koala/phi4-mini" {
t.Fatalf("DefaultModel(un-offered) = %q, want the first offered model", got)
}
// No summary model recorded → first offered.
if got := s.DefaultModel(""); got != "koala/phi4-mini" {
t.Fatalf("DefaultModel(\"\") = %q, want the first offered model", got)
}
}
func TestAnswerGroundsOnTranscriptAndCarriesHistory(t *testing.T) {
f := &factory{answer: " The video is about attention. "}
s := New(f.make, []string{"koala/phi4-mini"}, 0)
reply, err := s.Answer(context.Background(), Request{
Model: "koala/phi4-mini",
Transcript: "ATTENTION-TRANSCRIPT-MARKER",
History: []Turn{{Question: "who", Answer: "the host"}},
Question: "what is it about",
})
if err != nil {
t.Fatalf("Answer: %v", err)
}
if reply.Answer != "The video is about attention." {
t.Fatalf("answer not trimmed: %q", reply.Answer)
}
if reply.Truncated {
t.Fatal("short transcript must not report truncated")
}
// The transcript rides in the system prompt; the conversation in the user msg.
if !strings.Contains(f.used.lastSystem, "ATTENTION-TRANSCRIPT-MARKER") {
t.Fatal("transcript not grounded into the system prompt")
}
if !strings.Contains(f.used.lastUser, "Q: who") || !strings.Contains(f.used.lastUser, "A: the host") {
t.Fatalf("history not carried into the user message: %q", f.used.lastUser)
}
if !strings.Contains(f.used.lastUser, "what is it about") {
t.Fatal("new question missing from the user message")
}
}
func TestAnswerTruncatesLongTranscriptAndReportsIt(t *testing.T) {
f := &factory{answer: "answer"}
s := New(f.make, []string{"koala/phi4-mini"}, 10)
reply, err := s.Answer(context.Background(), Request{
Model: "koala/phi4-mini",
Transcript: strings.Repeat("x", 500),
Question: "summarize",
})
if err != nil {
t.Fatalf("Answer: %v", err)
}
if !reply.Truncated {
t.Fatal("a transcript past maxChars must report Truncated")
}
if strings.Count(f.used.lastSystem, "x") != 10 {
t.Fatalf("transcript not bounded to maxChars: got %d x's", strings.Count(f.used.lastSystem, "x"))
}
}
func TestAnswerForcesAnUnofferedModelBackToDefault(t *testing.T) {
f := &factory{answer: "answer"}
s := New(f.make, []string{"koala/phi4-mini", "iguana/gemma4-26b"}, 0)
// A forged/un-offered model must never reach the gateway as-is — it is forced
// to the default offered model (the cloud-absent guarantee depends on this).
_, err := s.Answer(context.Background(), Request{
Model: "berget/secret-cloud-model",
Transcript: "t",
Question: "q",
})
if err != nil {
t.Fatalf("Answer: %v", err)
}
if f.used.model != "koala/phi4-mini" {
t.Fatalf("un-offered model reached the gateway as %q, want the default", f.used.model)
}
}
func TestAnswerPropagatesCompleterError(t *testing.T) {
f := &factory{err: errors.New("gateway down")}
s := New(f.make, []string{"koala/phi4-mini"}, 0)
_, err := s.Answer(context.Background(), Request{Model: "koala/phi4-mini", Transcript: "t", Question: "q"})
if err == nil {
t.Fatal("expected the gateway error to propagate")
}
}
func TestAnswerRejectsEmptyModelSet(t *testing.T) {
s := New(func(string) Completer { return nil }, nil, 0)
if _, err := s.Answer(context.Background(), Request{Question: "q"}); err == nil {
t.Fatal("expected an error when no models are configured")
}
}
+52
View File
@@ -0,0 +1,52 @@
package chat
import (
"context"
"strings"
"testing"
)
// A Request may carry its own transcript budget. The Service's cap (18,000 chars by
// default) is sized for YouTube captions; an uploaded recording is much longer
// (~50,000 chars an hour), and keeping only the head of it would answer questions
// about the last two-thirds of a meeting from text that does not contain them
// (ADR-033). The per-request value overrides; zero means "the service default".
func TestAnswerMaxCharsOverridesTheServiceCap(t *testing.T) {
tests := []struct {
name string
serviceCap int
requestCap int
transcriptLen int
wantKept int
wantTruncated bool
}{
{"request cap above the service cap keeps the whole transcript", 10, 100, 50, 50, false},
{"request cap below the service cap bounds tighter", 100, 10, 50, 10, true},
{"zero request cap falls back to the service cap", 20, 0, 50, 20, true},
{"negative request cap is treated as unset", 20, -1, 50, 20, true},
{"request cap exceeded is reported as truncated", 10, 30, 50, 30, true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
f := &factory{answer: "answer"}
s := New(f.make, []string{"koala/phi4-mini"}, tt.serviceCap)
reply, err := s.Answer(context.Background(), Request{
Model: "koala/phi4-mini",
Transcript: strings.Repeat("x", tt.transcriptLen),
Question: "what was decided?",
MaxChars: tt.requestCap,
})
if err != nil {
t.Fatalf("Answer: %v", err)
}
if got := strings.Count(f.used.lastSystem, "x"); got != tt.wantKept {
t.Errorf("transcript chars sent = %d, want %d", got, tt.wantKept)
}
if reply.Truncated != tt.wantTruncated {
t.Errorf("Truncated = %v, want %v", reply.Truncated, tt.wantTruncated)
}
})
}
}
+50 -3
View File
@@ -25,24 +25,64 @@ import (
// docs/homelab-integration.md. // docs/homelab-integration.md.
const defaultMaxTokens = 8192 const defaultMaxTokens = 8192
// defaultTemperature is what every caller got before WithTemperature existed.
const defaultTemperature = 0.2
// Client calls an OpenAI-compatible chat completions endpoint. // Client calls an OpenAI-compatible chat completions endpoint.
type Client struct { type Client struct {
baseURL string baseURL string
apiKey string apiKey string
model string model string
maxTokens int maxTokens int
temp float64
httpClient *http.Client httpClient *http.Client
usageHook func(model string, prompt, completion int)
}
// Option configures a Client at construction. Variadic so the existing 4-arg
// call sites stay valid as new knobs are added.
type Option func(*Client)
// WithMaxTokens overrides the per-request completion budget. The summarizer uses
// this to cap completion for small-context models (e.g. koala/phi4-mini, 8k):
// with the default 8192 budget, prompt + max_tokens overflows an 8k context and
// the gateway returns HTTP 400. A non-positive n is ignored (keeps the default).
func WithMaxTokens(n int) Option {
return func(c *Client) {
if n > 0 {
c.maxTokens = n
}
}
}
// WithTemperature overrides the sampling temperature. Zero IS a valid value and is sent
// as zero: the reconciler's accuracy was measured at temperature 0 and must run there.
func WithTemperature(t float64) Option {
return func(c *Client) { c.temp = t }
}
// WithUsageHook registers a callback fired after a successful completion with the
// model and the prompt/completion token counts from the response usage block. It
// keeps this copied, stdlib-only package (ADR-004) decoupled from metrics: the
// caller wires it to internal/metrics, the client imports nothing. nil is ignored.
func WithUsageHook(fn func(model string, prompt, completion int)) Option {
return func(c *Client) { c.usageHook = fn }
} }
// New constructs a Client. // New constructs a Client.
func New(baseURL, apiKey, model string, timeout time.Duration) *Client { func New(baseURL, apiKey, model string, timeout time.Duration, opts ...Option) *Client {
return &Client{ c := &Client{
baseURL: strings.TrimRight(baseURL, "/"), baseURL: strings.TrimRight(baseURL, "/"),
apiKey: apiKey, apiKey: apiKey,
model: model, model: model,
maxTokens: defaultMaxTokens, maxTokens: defaultMaxTokens,
temp: defaultTemperature,
httpClient: &http.Client{Timeout: timeout}, httpClient: &http.Client{Timeout: timeout},
} }
for _, opt := range opts {
opt(c)
}
return c
} }
type chatRequest struct { type chatRequest struct {
@@ -61,6 +101,10 @@ type chatResponse struct {
Choices []struct { Choices []struct {
Message message `json:"message"` Message message `json:"message"`
} `json:"choices"` } `json:"choices"`
Usage struct {
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
} `json:"usage"`
} }
// Complete sends a system + user message and returns the assistant's reply. // Complete sends a system + user message and returns the assistant's reply.
@@ -72,7 +116,7 @@ func (c *Client) Complete(ctx context.Context, system, user string) (string, err
{Role: "system", Content: system}, {Role: "system", Content: system},
{Role: "user", Content: user}, {Role: "user", Content: user},
}, },
Temperature: 0.2, Temperature: c.temp,
MaxTokens: c.maxTokens, MaxTokens: c.maxTokens,
} }
b, err := json.Marshal(body) b, err := json.Marshal(body)
@@ -132,5 +176,8 @@ func (c *Client) Complete(ctx context.Context, system, user string) (string, err
if len(cr.Choices) == 0 { if len(cr.Choices) == 0 {
return "", fmt.Errorf("LLM returned no choices") return "", fmt.Errorf("LLM returned no choices")
} }
if c.usageHook != nil {
c.usageHook(c.model, cr.Usage.PromptTokens, cr.Usage.CompletionTokens)
}
return cr.Choices[0].Message.Content, nil return cr.Choices[0].Message.Content, nil
} }
+45
View File
@@ -64,6 +64,51 @@ func TestClient_SendsMaxTokens(t *testing.T) {
} }
} }
// TestClient_WithMaxTokens overrides the completion budget — the summarizer caps
// it small so prompt + max_tokens fits a small-context model's window (8k).
func TestClient_WithMaxTokens(t *testing.T) {
var body chatRequest
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = json.NewDecoder(r.Body).Decode(&body)
_ = json.NewEncoder(w).Encode(map[string]any{
"choices": []map[string]any{{"message": map[string]any{"content": "ok"}}},
})
}))
defer srv.Close()
c := New(srv.URL, "", "test-model", 10*time.Second, WithMaxTokens(1500))
if _, err := c.Complete(context.Background(), "sys", "user"); err != nil {
t.Fatalf("Complete: %v", err)
}
if body.MaxTokens != 1500 {
t.Errorf("max_tokens = %d, want 1500", body.MaxTokens)
}
}
// TestClient_UsageHookRecordsTokens: the usage hook fires with the model and the
// prompt/completion token counts parsed from the response usage block.
func TestClient_UsageHookRecordsTokens(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{
"choices": []map[string]any{{"message": map[string]any{"content": "ok"}}},
"usage": map[string]any{"prompt_tokens": 123, "completion_tokens": 45},
})
}))
defer srv.Close()
var gotModel string
var gotPrompt, gotCompletion int
c := New(srv.URL, "", "test-model", 10*time.Second, WithUsageHook(func(model string, p, comp int) {
gotModel, gotPrompt, gotCompletion = model, p, comp
}))
if _, err := c.Complete(context.Background(), "sys", "user"); err != nil {
t.Fatalf("Complete: %v", err)
}
if gotModel != "test-model" || gotPrompt != 123 || gotCompletion != 45 {
t.Errorf("usage hook got (%q, %d, %d), want (test-model, 123, 45)", gotModel, gotPrompt, gotCompletion)
}
}
func TestClient_ReturnsErrorOnNon200(t *testing.T) { func TestClient_ReturnsErrorOnNon200(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Error(w, "overloaded", http.StatusServiceUnavailable) http.Error(w, "overloaded", http.StatusServiceUnavailable)
+57
View File
@@ -0,0 +1,57 @@
package llm
import (
"context"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"testing"
"time"
)
// capturedTemperature runs one Complete against a fake gateway and returns the
// "temperature" field the request carried, and whether it was present at all.
func capturedTemperature(t *testing.T, opts ...Option) (float64, bool) {
t.Helper()
var body map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
raw, _ := io.ReadAll(r.Body)
_ = json.Unmarshal(raw, &body)
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{"choices": []map[string]any{{"message": map[string]any{"role": "assistant", "content": "ok"}}}})
}))
defer srv.Close()
if _, err := New(srv.URL, "", "m", 5*time.Second, opts...).Complete(context.Background(), "s", "u"); err != nil {
t.Fatalf("Complete: %v", err)
}
v, ok := body["temperature"].(float64)
return v, ok
}
func TestClient_DefaultTemperatureIsUnchanged(t *testing.T) {
got, ok := capturedTemperature(t)
if !ok || got != 0.2 {
t.Fatalf("default temperature = %v (present %v), want 0.2: existing callers must not change behaviour", got, ok)
}
}
func TestClient_WithTemperatureZeroSendsExactlyZero(t *testing.T) {
// The reconciler's accuracy (92%/97%) was measured at temperature 0. Zero must be
// SENT, not treated as "unset" and replaced by the default.
got, ok := capturedTemperature(t, WithTemperature(0))
if !ok || got != 0 {
t.Fatalf("temperature = %v (present %v), want an explicit 0", got, ok)
}
}
func TestClient_WithTemperatureSendsTheGivenValue(t *testing.T) {
got, ok := capturedTemperature(t, WithTemperature(0.7))
if !ok || got != 0.7 {
t.Fatalf("temperature = %v (present %v), want 0.7", got, ok)
}
}
+1 -1
View File
@@ -18,7 +18,7 @@ import (
"path/filepath" "path/filepath"
"sync" "sync"
"gitea.d-ma.be/mathias/tapir/internal/ports" "git.d-ma.be/mathias/tapir/internal/ports"
) )
// FileStore is a SecretStore backed by a single 0600 JSON file mapping opaque // FileStore is a SecretStore backed by a single 0600 JSON file mapping opaque
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"path/filepath" "path/filepath"
"testing" "testing"
"gitea.d-ma.be/mathias/tapir/internal/adapters/secrets" "git.d-ma.be/mathias/tapir/internal/adapters/secrets"
) )
func TestPutThenGet(t *testing.T) { func TestPutThenGet(t *testing.T) {
+7 -4
View File
@@ -10,10 +10,13 @@ import (
// DeleteUser permanently removes a user and all of their data. It runs through // DeleteUser permanently removes a user and all of their data. It runs through
// withUser so RLS confines every statement to the calling user's own rows. // withUser so RLS confines every statement to the calling user's own rows.
// //
// Deleting the users row cascades (ON DELETE CASCADE) to videos, transcripts, // Deleting the users row cascades (ON DELETE CASCADE) to videos, summaries
// summaries (→ sink_deliveries), video_connections, and the user_identities map // (→ sink_deliveries), video_connections, and the user_identities map —
// — referential-integrity cascades bypass RLS, so a user's child rows are removed // referential-integrity cascades bypass RLS, so a user's child rows are removed
// even though the deleting connection is scoped. summary_actions and login_events // even though the deleting connection is scoped. Transcripts are NOT removed:
// since ADR-021 they are shared public content keyed by (provider,
// provider_video_id) with no user_id, so another user may still reference the
// same row — a user deletion must not strip shared caption content. summary_actions and login_events
// are the exceptions: each carries a user_id but has NO foreign key to users // are the exceptions: each carries a user_id but has NO foreign key to users
// (migrations 002 and 010), so the cascade does not reach them; they are deleted // (migrations 002 and 010), so the cascade does not reach them; they are deleted
// explicitly in the same scoped transaction. Deleting an absent user is a no-op // explicitly in the same scoped transaction. Deleting an absent user is a no-op
@@ -0,0 +1,83 @@
package store
import (
"context"
"fmt"
"time"
"github.com/jackc/pgx/v5"
)
// CaptionlessChannels returns the set of channel ids currently suppressed for the
// user — channels whose recent videos all yielded no captions, within their
// suppression window (ADR-024). The runner skips caption fetches for these
// channels' videos. A channel whose window has expired is not returned, so its
// next video is re-probed (auto-recovery).
func (s *Store) CaptionlessChannels(ctx context.Context, userID string) (map[string]bool, error) {
out := map[string]bool{}
err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
rows, err := tx.Query(ctx, `
SELECT channel_id FROM channel_caption_state
WHERE user_id = $1 AND captionless_until IS NOT NULL AND captionless_until > now()`,
userID)
if err != nil {
return fmt.Errorf("store: caption-less channels: %w", err)
}
defer rows.Close()
for rows.Next() {
var ch string
if err := rows.Scan(&ch); err != nil {
return fmt.Errorf("store: scan caption-less channel: %w", err)
}
out[ch] = true
}
return rows.Err()
})
return out, err
}
// RecordChannelCaptionOutcome updates a channel's caption-availability memory
// after a fetch attempt (ADR-024). hadCaptions resets the channel (consecutive
// count to 0, suppression cleared). Otherwise the consecutive no-caption count is
// incremented; once it reaches threshold the channel is suppressed for window.
// threshold <= 0 is a no-op (feature disabled). An empty channelID is ignored
// (some sources may not carry one).
func (s *Store) RecordChannelCaptionOutcome(ctx context.Context, userID, channelID string, hadCaptions bool, threshold int, window time.Duration) error {
if channelID == "" || threshold <= 0 {
return nil
}
return s.withUser(ctx, userID, func(tx pgx.Tx) error {
if hadCaptions {
_, err := tx.Exec(ctx, `
INSERT INTO channel_caption_state (user_id, channel_id, consecutive_none, captionless_until, updated_at)
VALUES ($1, $2, 0, NULL, now())
ON CONFLICT (user_id, channel_id)
DO UPDATE SET consecutive_none = 0, captionless_until = NULL, updated_at = now()`,
userID, channelID)
if err != nil {
return fmt.Errorf("store: reset channel caption state: %w", err)
}
return nil
}
// No captions: increment the streak; suppress once it reaches threshold.
// captionless_until is set from the NEW count inside the same statement so
// the decision is atomic with the increment.
until := time.Now().Add(window)
_, err := tx.Exec(ctx, `
INSERT INTO channel_caption_state (user_id, channel_id, consecutive_none, captionless_until, updated_at)
VALUES ($1, $2, 1, CASE WHEN 1 >= $3 THEN $4::timestamptz ELSE NULL END, now())
ON CONFLICT (user_id, channel_id)
DO UPDATE SET
consecutive_none = channel_caption_state.consecutive_none + 1,
captionless_until = CASE
WHEN channel_caption_state.consecutive_none + 1 >= $3 THEN $4::timestamptz
ELSE channel_caption_state.captionless_until
END,
updated_at = now()`,
userID, channelID, threshold, until)
if err != nil {
return fmt.Errorf("store: record channel no-caption: %w", err)
}
return nil
})
}
@@ -0,0 +1,67 @@
package store_test
import (
"context"
"testing"
"time"
"github.com/stretchr/testify/require"
)
func TestChannelCaptionMemory_SuppressesAfterThreshold(t *testing.T) {
ctx := context.Background()
s := newStore(t)
super := rawPool(t)
resetDB(t, super)
seedUser(t, super, userA)
const threshold = 3
window := time.Hour
// Below threshold: not yet suppressed.
require.NoError(t, s.RecordChannelCaptionOutcome(ctx, userA, "chanX", false, threshold, window))
require.NoError(t, s.RecordChannelCaptionOutcome(ctx, userA, "chanX", false, threshold, window))
got, err := s.CaptionlessChannels(ctx, userA)
require.NoError(t, err)
require.NotContains(t, got, "chanX", "2 < threshold 3: not suppressed yet")
// Crossing the threshold suppresses the channel.
require.NoError(t, s.RecordChannelCaptionOutcome(ctx, userA, "chanX", false, threshold, window))
got, err = s.CaptionlessChannels(ctx, userA)
require.NoError(t, err)
require.Contains(t, got, "chanX", "3 consecutive no-caption results suppress the channel")
// A successful caption fetch resets it.
require.NoError(t, s.RecordChannelCaptionOutcome(ctx, userA, "chanX", true, threshold, window))
got, err = s.CaptionlessChannels(ctx, userA)
require.NoError(t, err)
require.NotContains(t, got, "chanX", "a captioned video clears suppression")
}
func TestChannelCaptionMemory_WindowExpiryReProbes(t *testing.T) {
ctx := context.Background()
s := newStore(t)
super := rawPool(t)
resetDB(t, super)
seedUser(t, super, userA)
// A negative window means captionless_until lands in the past — modelling an
// elapsed suppression window, which must make the channel eligible again.
require.NoError(t, s.RecordChannelCaptionOutcome(ctx, userA, "chanY", false, 1, -time.Hour))
got, err := s.CaptionlessChannels(ctx, userA)
require.NoError(t, err)
require.NotContains(t, got, "chanY", "an expired window re-enables the channel for a re-probe")
}
func TestChannelCaptionMemory_DisabledThresholdIsNoOp(t *testing.T) {
ctx := context.Background()
s := newStore(t)
super := rawPool(t)
resetDB(t, super)
seedUser(t, super, userA)
require.NoError(t, s.RecordChannelCaptionOutcome(ctx, userA, "chanZ", false, 0, time.Hour))
got, err := s.CaptionlessChannels(ctx, userA)
require.NoError(t, err)
require.Empty(t, got, "threshold 0 disables the memory — nothing recorded")
}
+1 -1
View File
@@ -6,7 +6,7 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
// seedUserRow inserts a bare users row (FK target for a connection) as the // seedUserRow inserts a bare users row (FK target for a connection) as the
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
const ( const (
+70 -17
View File
@@ -3,6 +3,7 @@ package store_test
import ( import (
"context" "context"
"database/sql" "database/sql"
"errors"
"os" "os"
"testing" "testing"
@@ -34,6 +35,30 @@ func fileMigrator(t *testing.T) *migrate.Migrate {
return m return m
} }
// headVersion reports the current (HEAD) schema version so a test can restore
// to it after stepping down, without hard-coding what HEAD is. Adding a
// migration on top changes HEAD but no test that uses this needs editing.
func headVersion(t *testing.T, m *migrate.Migrate) uint {
t.Helper()
v, dirty, err := m.Version()
require.NoError(t, err)
require.False(t, dirty, "schema must not be dirty")
return v
}
// migrateTo drives the schema to an exact version *by version number*, not by
// step count. This is the whole point of the migrate-test design: a migration
// added above the target does not shift any count here, so unrelated tests stay
// green (see issue #8). ErrNoChange (already at that version) is not a failure.
func migrateTo(t *testing.T, m *migrate.Migrate, version uint) {
t.Helper()
err := m.Migrate(version)
if errors.Is(err, migrate.ErrNoChange) {
return
}
require.NoError(t, err)
}
// loginEventsExists reports whether the login_events relation is present. // loginEventsExists reports whether the login_events relation is present.
func loginEventsExists(t *testing.T) bool { func loginEventsExists(t *testing.T) bool {
t.Helper() t.Helper()
@@ -53,19 +78,15 @@ func TestMigration010LoginEventsUpDown(t *testing.T) {
require.True(t, loginEventsExists(t), "login_events must exist at latest migration") require.True(t, loginEventsExists(t), "login_events must exist at latest migration")
m := fileMigrator(t) m := fileMigrator(t)
// 011, 012, 013 sit above 010; step them down first so 010 is exercised in isolation. head := headVersion(t, m)
require.NoError(t, m.Steps(-1), "down 013 drops channel_errors, login_events intact")
require.True(t, loginEventsExists(t), "013 down leaves login_events intact")
require.NoError(t, m.Steps(-1), "down 012 is a no-op, login_events intact")
require.True(t, loginEventsExists(t), "012 down leaves login_events intact")
require.NoError(t, m.Steps(-1), "down 011 must not touch login_events")
require.True(t, loginEventsExists(t), "011 down leaves login_events intact")
require.NoError(t, m.Steps(-1), "down 010 must drop login_events") migrateTo(t, m, 9) // just below 010 — everything above steps down
require.False(t, loginEventsExists(t), "login_events must be gone after the down migration") require.False(t, loginEventsExists(t), "login_events must be gone after the down migration")
require.NoError(t, m.Steps(4), "up must recreate 010 then re-apply 011, 012, 013") migrateTo(t, m, 10) // up 010
require.True(t, loginEventsExists(t), "login_events must be restored after the up migration") require.True(t, loginEventsExists(t), "login_events must be restored after the up migration")
migrateTo(t, m, head) // restore to HEAD for sibling tests
} }
// autoSummarizeDefault reads the users.auto_summarize column default as text // autoSummarizeDefault reads the users.auto_summarize column default as text
@@ -87,15 +108,43 @@ func TestMigration011AutoSummarizeDefaultUpDown(t *testing.T) {
require.Equal(t, "true", autoSummarizeDefault(t), "011 sets the default to TRUE") require.Equal(t, "true", autoSummarizeDefault(t), "011 sets the default to TRUE")
m := fileMigrator(t) m := fileMigrator(t)
require.NoError(t, m.Steps(-1), "down 013 drops channel_errors") head := headVersion(t, m)
require.NoError(t, m.Steps(-1), "down 012 is a no-op")
require.NoError(t, m.Steps(-1), "down 011 reverts the column default") migrateTo(t, m, 10) // just below 011 — reverts the column default
require.Equal(t, "false", autoSummarizeDefault(t), "default is FALSE after the down migration") require.Equal(t, "false", autoSummarizeDefault(t), "default is FALSE after the down migration")
require.NoError(t, m.Steps(1), "up 011 re-applies the TRUE default") migrateTo(t, m, 11) // up 011 re-applies the TRUE default
require.Equal(t, "true", autoSummarizeDefault(t)) require.Equal(t, "true", autoSummarizeDefault(t))
require.NoError(t, m.Steps(1), "up 012 runs clean (no FORCE RLS on fresh schema)")
require.NoError(t, m.Steps(1), "up 013 creates channel_errors") migrateTo(t, m, head) // restore to HEAD for sibling tests
}
// channelTitleExists reports whether videos.channel_title is present.
func channelTitleExists(t *testing.T) bool {
t.Helper()
var exists bool
require.NoError(t, rawPool(t).QueryRow(context.Background(),
`SELECT EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_name = 'videos' AND column_name = 'channel_title')`).Scan(&exists))
return exists
}
// TestMigration014VideoChannelTitleUpDown proves 014 is reversible: down drops
// videos.channel_title, up recreates it.
func TestMigration014VideoChannelTitleUpDown(t *testing.T) {
newStore(t) // latest (014 applied)
require.True(t, channelTitleExists(t), "channel_title exists at latest migration")
m := fileMigrator(t)
head := headVersion(t, m)
migrateTo(t, m, 13) // just below 014 — drops channel_title
require.False(t, channelTitleExists(t), "channel_title must be gone after the down migration")
migrateTo(t, m, 14) // up 014 recreates channel_title
require.True(t, channelTitleExists(t), "channel_title must be restored after the up migration")
migrateTo(t, m, head) // restore to HEAD for sibling tests
} }
// TestMigration012FixAutoSummarizeRLS proves 012 runs cleanly and flips any // TestMigration012FixAutoSummarizeRLS proves 012 runs cleanly and flips any
@@ -106,7 +155,11 @@ func TestMigration012FixAutoSummarizeRLS(t *testing.T) {
// Round-trip: down 012, then up 012 — must be idempotent. // Round-trip: down 012, then up 012 — must be idempotent.
m := fileMigrator(t) m := fileMigrator(t)
require.NoError(t, m.Steps(-1), "down 012 must not error") head := headVersion(t, m)
require.NoError(t, m.Steps(1), "up 012 must re-apply cleanly")
migrateTo(t, m, 11) // down 012 must not error
migrateTo(t, m, 12) // up 012 must re-apply cleanly
require.Equal(t, "true", autoSummarizeDefault(t), "default still TRUE after 012 re-applied") require.Equal(t, "true", autoSummarizeDefault(t), "default still TRUE after 012 re-applied")
migrateTo(t, m, head) // restore to HEAD for sibling tests
} }
@@ -0,0 +1 @@
ALTER TABLE videos DROP COLUMN channel_title;
@@ -0,0 +1,8 @@
-- Store the source channel's title per video so the list can offer a real
-- channel filter (multi-select of the user's channels) instead of the dead
-- free-text field that only ever matched the provider string. Nullable: existing
-- rows backfill on the next discovery pass (UpsertVideo writes it); pasted videos
-- get it immediately from videos.list. No FK to a channels table at Stage 0 — the
-- title is a denormalised display/filter value, consistent with the existing
-- subscription_id-stays-NULL stance (data-model.md).
ALTER TABLE videos ADD COLUMN channel_title TEXT;
@@ -0,0 +1,19 @@
-- Down 015: restore the per-user RLS-scoped transcripts shape (001 + 003).
DROP TABLE transcripts;
CREATE TABLE transcripts (
video_id UUID PRIMARY KEY REFERENCES videos(id) ON DELETE CASCADE,
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
source TEXT NOT NULL,
language TEXT,
content TEXT,
resolved_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_transcripts_user_id ON transcripts(user_id);
ALTER TABLE transcripts ENABLE ROW LEVEL SECURITY;
ALTER TABLE transcripts FORCE ROW LEVEL SECURITY;
CREATE POLICY transcripts_isolation ON transcripts
FOR ALL
USING (user_id = current_setting('tapir.current_user_id', true)::uuid);
@@ -0,0 +1,30 @@
-- Migration 015: transcripts become SHARED public-content storage (ADR-021).
--
-- The per-user transcripts table from 001 (PK videos.id, user_id NOT NULL, RLS
-- FORCEd in 003) was dead: no application code ever read or wrote it — only the
-- transcript_status columns on `videos` (007) carried fetch outcomes. ADR-021
-- repurposes it as the single shared store of public caption content, keyed by
-- the cross-user dedup key (provider, provider_video_id) — the video's public
-- identity, not Tapir's per-user videos.id — so re-analysis never re-fetches
-- from YouTube (ADR-010/014).
--
-- It holds ONLY public caption content + the video's public id (nothing
-- user-identifying), so it is deliberately NOT RLS-scoped: no user_id, no
-- policy, no FORCE. This is the single, intentional exception to the ADR-012
-- isolation boundary; rls_test.go asserts the boundary is exactly here and
-- nowhere else. Dropping the old table drops its RLS policy with it; it held no
-- real data, so drop+recreate loses nothing.
DROP TABLE transcripts;
CREATE TABLE transcripts (
provider TEXT NOT NULL,
provider_video_id TEXT NOT NULL,
source TEXT NOT NULL, -- 'captions' (content set) | 'none' (no captions; content NULL)
language TEXT,
content TEXT,
fetched_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
PRIMARY KEY (provider, provider_video_id)
);
COMMENT ON TABLE transcripts IS
'Shared public caption content keyed by (provider, provider_video_id). NOT RLS-scoped — public content only, de-facto cross-user dedup (ADR-021).';
@@ -0,0 +1 @@
DROP TABLE channel_caption_state;
@@ -0,0 +1,30 @@
-- Migration 016: per-(user, channel) caption-availability memory (ADR-024).
--
-- Some channels never publish English captions (foreign-language news, music,
-- etc.). Each of their new videos still costs ONE rate-limited caption fetch
-- (ADR-014) before resolving to "none" — and on a throttled egress IP that fetch
-- may 429 and churn through the backoff machinery first. This table remembers
-- channels that repeatedly yield no captions so discovery can stop attempting
-- their videos, freeing the scarce fetch budget for channels that do have them.
--
-- consecutive_none counts no-caption outcomes in a row; a successful fetch resets
-- it to 0. Once it crosses the threshold the channel is suppressed until
-- captionless_until, after which one video is re-probed (auto-recovery for a
-- channel that starts adding captions). Per-user + RLS-scoped, consistent with
-- the rest of the user-owned schema (subscriptions are per-user; ADR-012).
CREATE TABLE channel_caption_state (
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
channel_id TEXT NOT NULL,
consecutive_none INT NOT NULL DEFAULT 0,
captionless_until TIMESTAMPTZ,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (user_id, channel_id)
);
CREATE INDEX idx_channel_caption_state_user_id ON channel_caption_state(user_id);
ALTER TABLE channel_caption_state ENABLE ROW LEVEL SECURITY;
ALTER TABLE channel_caption_state FORCE ROW LEVEL SECURITY;
CREATE POLICY channel_caption_state_isolation ON channel_caption_state
FOR ALL
USING (user_id = current_setting('tapir.current_user_id', true)::uuid);
@@ -0,0 +1,3 @@
-- Drops the private uploaded-transcript store and every transcript in it. There
-- is no revert of the data: the original audio was never kept (ADR-033).
DROP TABLE uploaded_transcripts;
@@ -0,0 +1,48 @@
-- Migration 017: uploaded_transcripts — PRIVATE transcripts of audio a user
-- uploaded themselves (ADR-033, tapir#33).
--
-- This is deliberately NOT the shared `transcripts` table (migration 015,
-- ADR-021). That table is non-RLS on the explicit grounds that it holds only
-- public caption content and nothing user-identifying. A recorded meeting is the
-- opposite: private third-party content owned by exactly one user. Reusing
-- `transcripts` — or adding a user_id to it with a partial policy — would put a
-- private row in the one table the isolation test asserts is shared, and would
-- pass CI green. A separate table keeps ADR-021's exception singular.
--
-- Row-level security follows migration 003 exactly: ENABLE + FORCE (the app
-- connects as the table owner, which bypasses RLS unless forced), and a policy
-- keyed off the per-request GUC tapir.current_user_id. With no GUC set the
-- predicate is NULL and the table is deny-all.
--
-- The original audio is never stored (ADR-033): this row is all that is kept.
-- segments holds the timestamped cues; re-transcribing 90 minutes costs ~7
-- minutes, so discarding the timestamps would be irreversible.
CREATE TABLE uploaded_transcripts (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
title TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'queued'
CHECK (status IN ('queued', 'transcribing', 'done', 'failed')),
content TEXT,
segments JSONB,
endpoint TEXT, -- which backend served it (iguana | berget); a quality fact
audio_seconds DOUBLE PRECISION,
error_kind TEXT, -- typed, never a provider's error string (ADR-033)
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
finished_at TIMESTAMPTZ,
CONSTRAINT uploaded_transcripts_done_has_content
CHECK (status <> 'done' OR content IS NOT NULL),
CONSTRAINT uploaded_transcripts_failed_has_error_kind
CHECK (status <> 'failed' OR error_kind IS NOT NULL)
);
CREATE INDEX idx_uploaded_transcripts_user ON uploaded_transcripts (user_id, created_at DESC);
ALTER TABLE uploaded_transcripts ENABLE ROW LEVEL SECURITY;
ALTER TABLE uploaded_transcripts FORCE ROW LEVEL SECURITY;
CREATE POLICY uploaded_transcripts_isolation ON uploaded_transcripts
FOR ALL
USING (user_id = current_setting('tapir.current_user_id', true)::uuid);
COMMENT ON TABLE uploaded_transcripts IS
'PRIVATE per-user transcripts of user-supplied audio. RLS-scoped (ADR-012). Never merge into the shared transcripts table (ADR-021, ADR-033).';
@@ -0,0 +1 @@
ALTER TABLE uploaded_transcripts DROP COLUMN had_video;
@@ -0,0 +1,9 @@
-- Migration 018: record whether an uploaded file carried a video track (ADR-033,
-- amended 2026-10-01).
--
-- A Teams meeting with the camera off is still an .mp4 with a video track, so video
-- files are ACCEPTED: only the audio is extracted and the video is discarded. This
-- flag lets the transcript page say so — "this file contained a video track; only its
-- audio was used" — so the user is never left to wonder what happened to the picture.
-- It outlives the job: completing or failing an upload does not clear it.
ALTER TABLE uploaded_transcripts ADD COLUMN had_video BOOLEAN NOT NULL DEFAULT FALSE;
@@ -0,0 +1 @@
ALTER TABLE uploaded_transcripts DROP COLUMN reconciliation;
@@ -0,0 +1,12 @@
-- Migration 019: how a merged transcript came about (ADR-034).
--
-- When two engines transcribed the same audio and a local model chose between their
-- disagreements, this records the status, the exact counts, and a bounded list of the
-- contested spots (changed or unsure, each with a position and both readings) so the
-- page can say what was done and the user can check the uncertain spots against the
-- audio. NULL means a single engine: no second opinion, and the page must not claim one.
--
-- It lives on the already-FORCE'd, RLS-scoped uploaded_transcripts, so the exact-set
-- isolation test is unaffected. The contested spots hold fragments of the private
-- transcript, which is why they are here and nowhere else.
ALTER TABLE uploaded_transcripts ADD COLUMN reconciliation JSONB;
+4 -1
View File
@@ -29,6 +29,7 @@ type SummaryRow struct {
ProviderVideoID string // videos.provider_video_id; empty when no videos row ProviderVideoID string // videos.provider_video_id; empty when no videos row
Title string // videos.title; empty when no videos row Title string // videos.title; empty when no videos row
Channel string // videos.provider for now; empty when no videos row Channel string // videos.provider for now; empty when no videos row
ChannelTitle string // videos.channel_title; the source channel, for display + filtering
URL string // videos.url; empty when no videos row URL string // videos.url; empty when no videos row
PublishedAt time.Time // videos.published_at; zero when absent PublishedAt time.Time // videos.published_at; zero when absent
Summary string Summary string
@@ -137,7 +138,8 @@ const selectVideo = `
COALESCE(s.created_at, v.seen_at), COALESCE(s.created_at, v.seen_at),
(s.id IS NOT NULL) AS summarized, (s.id IS NOT NULL) AS summarized,
v.summarize_requested, v.summarize_requested,
COALESCE(v.transcript_status, '') COALESCE(v.transcript_status, ''),
COALESCE(v.channel_title, '')
FROM videos v FROM videos v
LEFT JOIN summaries s ON s.video_id = v.id AND s.user_id = v.user_id` LEFT JOIN summaries s ON s.video_id = v.id AND s.user_id = v.user_id`
@@ -254,6 +256,7 @@ func scanVideoRow(rows pgx.Row) (SummaryRow, error) {
&row.Summarized, &row.Summarized,
&row.SummarizeRequested, &row.SummarizeRequested,
&row.TranscriptStatus, &row.TranscriptStatus,
&row.ChannelTitle,
); err != nil { ); err != nil {
return SummaryRow{}, fmt.Errorf("store: scan video: %w", err) return SummaryRow{}, fmt.Errorf("store: scan video: %w", err)
} }
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"github.com/jackc/pgx/v5/pgxpool" "github.com/jackc/pgx/v5/pgxpool"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
// seedVideo inserts a videos row whose id matches a summary's video_id, so the // seedVideo inserts a videos row whose id matches a summary's video_id, so the
+12 -6
View File
@@ -4,6 +4,7 @@ import (
"context" "context"
"fmt" "fmt"
"sort" "sort"
"time"
"github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5"
) )
@@ -32,7 +33,12 @@ type UserActiveWeeks struct {
// Scope note: the enumeration covers users with a Dex identity (the web users the // Scope note: the enumeration covers users with a Dex identity (the web users the
// gate is about). A CLI-only user created by the store sink without an identity // gate is about). A CLI-only user created by the store sink without an identity
// row would not appear — out of scope for this gate. // row would not appear — out of scope for this gate.
func (s *Store) ActiveWeeks(ctx context.Context) ([]UserActiveWeeks, error) { // ActiveWeeks counts each user's distinct active weeks from `since` onward. A zero
// `since` means no lower bound (count all history). The Stage-0 gate baseline is
// set by the caller (the report command) to the date real usage tracking began,
// so pre-launch noise — testing, the period the pilot was blocked — does not count
// toward the return-usage signal (ADR-016).
func (s *Store) ActiveWeeks(ctx context.Context, since time.Time) ([]UserActiveWeeks, error) {
userIDs, err := s.identityUserIDs(ctx) userIDs, err := s.identityUserIDs(ctx)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -40,7 +46,7 @@ func (s *Store) ActiveWeeks(ctx context.Context) ([]UserActiveWeeks, error) {
out := make([]UserActiveWeeks, 0, len(userIDs)) out := make([]UserActiveWeeks, 0, len(userIDs))
for _, uid := range userIDs { for _, uid := range userIDs {
row, err := s.activeWeeksFor(ctx, uid) row, err := s.activeWeeksFor(ctx, uid, since)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -85,18 +91,18 @@ func (s *Store) identityUserIDs(ctx context.Context) ([]string, error) {
// activeWeeksFor counts one user's distinct active weeks (reads UNION acts) and // activeWeeksFor counts one user's distinct active weeks (reads UNION acts) and
// reads their display name, RLS-scoped via withUser. The UNION dedups a week that // reads their display name, RLS-scoped via withUser. The UNION dedups a week that
// has both a login and an action so it counts once. // has both a login and an action so it counts once.
func (s *Store) activeWeeksFor(ctx context.Context, userID string) (UserActiveWeeks, error) { func (s *Store) activeWeeksFor(ctx context.Context, userID string, since time.Time) (UserActiveWeeks, error) {
res := UserActiveWeeks{UserID: userID} res := UserActiveWeeks{UserID: userID}
if err := s.withUser(ctx, userID, func(tx pgx.Tx) error { if err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
if err := tx.QueryRow(ctx, if err := tx.QueryRow(ctx,
`WITH weeks AS ( `WITH weeks AS (
SELECT date_trunc('week', seen_at) AS wk SELECT date_trunc('week', seen_at) AS wk
FROM login_events WHERE user_id = $1 FROM login_events WHERE user_id = $1 AND seen_at >= $2
UNION UNION
SELECT date_trunc('week', acted_at) SELECT date_trunc('week', acted_at)
FROM summary_actions WHERE user_id = $1 FROM summary_actions WHERE user_id = $1 AND acted_at >= $2
) )
SELECT count(DISTINCT wk) FROM weeks`, userID).Scan(&res.ActiveWeeks); err != nil { SELECT count(DISTINCT wk) FROM weeks`, userID, since).Scan(&res.ActiveWeeks); err != nil {
return fmt.Errorf("store: count active weeks: %w", err) return fmt.Errorf("store: count active weeks: %w", err)
} }
if err := tx.QueryRow(ctx, if err := tx.QueryRow(ctx,
+29 -2
View File
@@ -3,6 +3,7 @@ package store_test
import ( import (
"context" "context"
"testing" "testing"
"time"
"github.com/jackc/pgx/v5/pgxpool" "github.com/jackc/pgx/v5/pgxpool"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
@@ -54,7 +55,7 @@ func TestActiveWeeksCountsDistinctWeeksAcrossReadsAndActs(t *testing.T) {
($1, 'vid-2', 'saved', '2026-01-19T18:00:00Z')`, userA) ($1, 'vid-2', 'saved', '2026-01-19T18:00:00Z')`, userA)
require.NoError(t, err) require.NoError(t, err)
got, err := s.ActiveWeeks(ctx) got, err := s.ActiveWeeks(ctx, time.Time{}) // zero since = no lower bound
require.NoError(t, err) require.NoError(t, err)
require.Len(t, got, 2, "both identity users must appear") require.Len(t, got, 2, "both identity users must appear")
@@ -72,7 +73,33 @@ func TestActiveWeeksEmptyWhenNoUsers(t *testing.T) {
s := newStore(t) s := newStore(t)
resetDB(t, rawPool(t)) resetDB(t, rawPool(t))
got, err := s.ActiveWeeks(ctx) got, err := s.ActiveWeeks(ctx, time.Time{})
require.NoError(t, err) require.NoError(t, err)
require.Empty(t, got) require.Empty(t, got)
} }
// TestActiveWeeksExcludesBeforeGateStart proves the baseline cutoff: activity
// before `since` does not count, so pre-launch noise (testing, the pilot's blocked
// period) is excluded from the Stage-0 return-usage gate (ADR-016).
func TestActiveWeeksExcludesBeforeGateStart(t *testing.T) {
ctx := context.Background()
s := newStore(t)
p := rawPool(t)
resetDB(t, p)
seedReportUser(t, p, userA, "subject-a", "Ada")
// One read well before the baseline, two reads in distinct weeks after it.
_, err := p.Exec(ctx,
`INSERT INTO login_events (user_id, seen_at) VALUES
($1, '2026-05-01T09:00:00Z'),
($1, '2026-06-12T09:00:00Z'),
($1, '2026-06-19T09:00:00Z')`, userA)
require.NoError(t, err)
since := time.Date(2026, 6, 11, 0, 0, 0, 0, time.UTC)
got, err := s.ActiveWeeks(ctx, since)
require.NoError(t, err)
require.Len(t, got, 1)
require.Equal(t, 2, got[0].ActiveWeeks, "only the two post-baseline weeks count; the May read is excluded")
}
+70 -14
View File
@@ -22,9 +22,11 @@ import (
// (no GUC set → zero rows) proves the enforcement path is live, not bypassed. // (no GUC set → zero rows) proves the enforcement path is live, not bypassed.
// userIsolatedTables are the tables that carry a user_id and whose policy keys // userIsolatedTables are the tables that carry a user_id and whose policy keys
// directly off the tapir.current_user_id GUC. // directly off the tapir.current_user_id GUC. transcripts is deliberately ABSENT
// — ADR-021 made it shared public content (non-RLS); TestTranscriptsTableIsSharedNotRLS
// proves that is the only place the isolation boundary moved.
var userIsolatedTables = []string{ var userIsolatedTables = []string{
"users", "videos", "transcripts", "summaries", "summary_actions", "login_events", "video_connections", "users", "videos", "summaries", "summary_actions", "login_events", "video_connections",
} }
// allIsolatedTables adds sink_deliveries, whose ownership is derived from its // allIsolatedTables adds sink_deliveries, whose ownership is derived from its
@@ -38,9 +40,10 @@ type seeded struct {
summaryID string summaryID string
} }
// seedUser inserts one full chain (user → video → transcript → summary → // seedUser inserts one full chain (user → video → summary → action → delivery)
// action → delivery) as the superuser pool, which bypasses RLS so both users' // as the superuser pool, which bypasses RLS so both users' data lands regardless
// data lands regardless of the GUC. // of the GUC. Transcripts are NOT seeded here: they are shared, non-RLS public
// content (ADR-021), so they have no place in a per-user isolation chain.
func seedUser(t *testing.T, p *pgxpool.Pool, userID string) seeded { func seedUser(t *testing.T, p *pgxpool.Pool, userID string) seeded {
t.Helper() t.Helper()
ctx := context.Background() ctx := context.Background()
@@ -54,11 +57,6 @@ func seedUser(t *testing.T, p *pgxpool.Pool, userID string) seeded {
VALUES ($1, 'youtube', $2, 'title') RETURNING id`, VALUES ($1, 'youtube', $2, 'title') RETURNING id`,
userID, "vid-"+userID).Scan(&videoID)) userID, "vid-"+userID).Scan(&videoID))
_, err = p.Exec(ctx,
`INSERT INTO transcripts (video_id, user_id, source, content)
VALUES ($1, $2, 'captions', 'words')`, videoID, userID)
require.NoError(t, err)
var summaryID string var summaryID string
require.NoError(t, p.QueryRow(ctx, require.NoError(t, p.QueryRow(ctx,
`INSERT INTO summaries (user_id, video_id, summary) VALUES ($1, $2, 'sum') `INSERT INTO summaries (user_id, video_id, summary) VALUES ($1, $2, 'sum')
@@ -92,9 +90,16 @@ func appPool(t *testing.T, super *pgxpool.Pool) *pgxpool.Pool {
t.Helper() t.Helper()
ctx := context.Background() ctx := context.Background()
// Idempotent across test runs (schema/role persist for the TestMain PG). // Idempotent across tests AND runs: the role persists for the TestMain PG and
_, _ = super.Exec(ctx, `DROP ROLE IF EXISTS app`) // owns granted privileges, so a plain DROP ROLE fails once any GRANT exists
_, err := super.Exec(ctx, `CREATE ROLE app LOGIN PASSWORD 'app'`) // (and more than one test now builds an app pool). Create only if absent; the
// GRANTs below are themselves idempotent.
_, err := super.Exec(ctx,
`DO $$ BEGIN
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'app') THEN
CREATE ROLE app LOGIN PASSWORD 'app';
END IF;
END $$`)
require.NoError(t, err) require.NoError(t, err)
_, err = super.Exec(ctx, `GRANT USAGE ON SCHEMA public TO app`) _, err = super.Exec(ctx, `GRANT USAGE ON SCHEMA public TO app`)
require.NoError(t, err) require.NoError(t, err)
@@ -186,7 +191,6 @@ func TestRLSEnforcesPerUserIsolation(t *testing.T) {
{"update users", `UPDATE users SET display_name = 'hacked' WHERE id = $1`, b.userID}, {"update users", `UPDATE users SET display_name = 'hacked' WHERE id = $1`, b.userID},
{"update videos", `UPDATE videos SET title = 'hacked' WHERE user_id = $1`, b.userID}, {"update videos", `UPDATE videos SET title = 'hacked' WHERE user_id = $1`, b.userID},
{"queue videos summarize", `UPDATE videos SET summarize_requested = TRUE WHERE id = $1`, b.videoID}, {"queue videos summarize", `UPDATE videos SET summarize_requested = TRUE WHERE id = $1`, b.videoID},
{"update transcripts", `UPDATE transcripts SET content = 'hacked' WHERE user_id = $1`, b.userID},
{"update summaries", `UPDATE summaries SET summary = 'hacked' WHERE user_id = $1`, b.userID}, {"update summaries", `UPDATE summaries SET summary = 'hacked' WHERE user_id = $1`, b.userID},
{"update summary_actions", `UPDATE summary_actions SET action = 'skipped' WHERE user_id = $1`, b.userID}, {"update summary_actions", `UPDATE summary_actions SET action = 'skipped' WHERE user_id = $1`, b.userID},
{"update login_events", `UPDATE login_events SET seen_at = NOW() WHERE user_id = $1`, b.userID}, {"update login_events", `UPDATE login_events SET seen_at = NOW() WHERE user_id = $1`, b.userID},
@@ -235,3 +239,55 @@ func TestRLSEnforcesPerUserIsolation(t *testing.T) {
_ = a // a's ids are seeded for the symmetric read assertions above _ = a // a's ids are seeded for the symmetric read assertions above
} }
// TestTranscriptsTableIsSharedNotRLS is the ADR-021 isolation proof: transcripts
// is the ONE shared, non-RLS surface, and the public-content classification
// leaked to nothing else. It is the inverse of TestRLSEnforcesPerUserIsolation —
// where that asserts deny-all on every user-owned table, this asserts transcripts
// is readable and writable with no user scope at all, holds no user_id, and is
// the single table with row-level security switched off.
func TestTranscriptsTableIsSharedNotRLS(t *testing.T) {
newStore(t)
super := rawPool(t)
resetDB(t, super)
app := appPool(t, super)
ctx := context.Background()
// 1. Shared + non-RLS: with NO GUC set, the app role both writes and reads a
// transcript. On an RLS table this would be deny-all (zero rows), exactly as
// the main isolation test asserts for every user-owned table.
_, err := app.Exec(ctx,
`INSERT INTO transcripts (provider, provider_video_id, source, content)
VALUES ('youtube', 'shared-vid', 'captions', 'public words')`)
require.NoError(t, err, "app role must write shared transcript content with no user scope")
require.Equal(t, 1, scopedCount(t, app, "", "transcripts"),
"transcripts must be readable with NO user scope — it is shared, non-RLS (ADR-021)")
// 2. No user_id column: the table holds only public caption content + the
// video's public id, nothing user-identifying.
var hasUserID bool
require.NoError(t, super.QueryRow(ctx,
`SELECT EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_name = 'transcripts' AND column_name = 'user_id')`).Scan(&hasUserID))
require.False(t, hasUserID, "transcripts must carry no user_id (ADR-021 public content)")
// 3. The boundary is EXACTLY here: every user-owned table still has row-level
// security enabled; transcripts alone has it off. This is the proof the
// non-RLS classification was applied to transcripts and leaked nowhere else.
for _, table := range allIsolatedTables {
require.True(t, rlsEnabled(t, super, table),
"%s must still enforce row-level security — isolation must not have regressed", table)
}
require.False(t, rlsEnabled(t, super, "transcripts"),
"transcripts must be the single table with row-level security OFF (the one shared surface)")
}
// rlsEnabled reports whether a public table has ROW LEVEL SECURITY enabled.
func rlsEnabled(t *testing.T, p *pgxpool.Pool, table string) bool {
t.Helper()
var enabled bool
require.NoError(t, p.QueryRow(context.Background(),
`SELECT relrowsecurity FROM pg_class
WHERE relname = $1 AND relnamespace = 'public'::regnamespace`, table).Scan(&enabled))
return enabled
}
+1 -1
View File
@@ -24,7 +24,7 @@ import (
_ "github.com/jackc/pgx/v5/stdlib" // register the "pgx" database/sql driver for migrate _ "github.com/jackc/pgx/v5/stdlib" // register the "pgx" database/sql driver for migrate
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
) )
//go:embed migrations/*.sql //go:embed migrations/*.sql
+18 -5
View File
@@ -4,15 +4,16 @@ import (
"context" "context"
"fmt" "fmt"
"os" "os"
"path/filepath"
"testing" "testing"
embeddedpostgres "github.com/fergusstrange/embedded-postgres" embeddedpostgres "github.com/fergusstrange/embedded-postgres"
"github.com/jackc/pgx/v5/pgxpool" "github.com/jackc/pgx/v5/pgxpool"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
"gitea.d-ma.be/mathias/tapir/internal/ports" "git.d-ma.be/mathias/tapir/internal/ports"
) )
// Static check: Store satisfies the Sink port. // Static check: Store satisfies the Sink port.
@@ -24,11 +25,22 @@ var _ ports.Sink = (*store.Store)(nil)
var dsn string var dsn string
func TestMain(m *testing.M) { func TestMain(m *testing.M) {
const port = 54329 // Port + runtime/data dirs are per-process (PID-derived) so two concurrent
// `go test` invocations — e.g. a push-run and a tag-run firing together in CI —
// don't collide on a fixed port or a shared data dir (which silently failed
// both runs). CachePath is shared so the PG archive is downloaded once, not
// per process. Base 54000 keeps this package's range distinct from web's.
port := uint32(54000 + os.Getpid()%1000)
dsn = fmt.Sprintf("postgres://postgres:postgres@localhost:%d/postgres?sslmode=disable", port) dsn = fmt.Sprintf("postgres://postgres:postgres@localhost:%d/postgres?sslmode=disable", port)
rt := filepath.Join(os.TempDir(), fmt.Sprintf("tapir-epg-store-%d", os.Getpid()))
pg := embeddedpostgres.NewDatabase( pg := embeddedpostgres.NewDatabase(
embeddedpostgres.DefaultConfig().Port(port), embeddedpostgres.DefaultConfig().
Port(port).
RuntimePath(rt).
DataPath(filepath.Join(rt, "data")).
BinariesPath(filepath.Join(rt, "bin")).
CachePath(filepath.Join(os.TempDir(), "tapir-epg-cache")),
) )
if err := pg.Start(); err != nil { if err := pg.Start(); err != nil {
fmt.Fprintf(os.Stderr, "embedded-postgres start: %v\n", err) fmt.Fprintf(os.Stderr, "embedded-postgres start: %v\n", err)
@@ -40,6 +52,7 @@ func TestMain(m *testing.M) {
if err := pg.Stop(); err != nil { if err := pg.Stop(); err != nil {
fmt.Fprintf(os.Stderr, "embedded-postgres stop: %v\n", err) fmt.Fprintf(os.Stderr, "embedded-postgres stop: %v\n", err)
} }
_ = os.RemoveAll(rt)
os.Exit(code) os.Exit(code)
} }
@@ -8,7 +8,7 @@ import (
"github.com/jackc/pgx/v5/pgxpool" "github.com/jackc/pgx/v5/pgxpool"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
// seedBareVideo inserts a videos row with no summary, so the all-videos read and // seedBareVideo inserts a videos row with no summary, so the all-videos read and
+66
View File
@@ -0,0 +1,66 @@
package store
import (
"context"
"errors"
"fmt"
"github.com/jackc/pgx/v5"
"git.d-ma.be/mathias/tapir/internal/domain"
)
// GetTranscript returns the shared, stored transcript for a video keyed by the
// cross-user dedup key (provider, providerVideoID), and whether one exists
// (ADR-021). It reads via the raw pool, NOT withUser: the table holds public
// content with no user_id and no RLS policy, so it is shared across users by
// construction. A stored SourceNone is a real hit (ok == true, HasText() ==
// false) — a known caption-less video, so the caller skips without re-fetching.
func (s *Store) GetTranscript(ctx context.Context, provider, providerVideoID string) (domain.Transcript, bool, error) {
var source, lang, content string
err := s.pool.QueryRow(ctx,
`SELECT source, COALESCE(language, ''), COALESCE(content, '')
FROM transcripts WHERE provider = $1 AND provider_video_id = $2`,
provider, providerVideoID).Scan(&source, &lang, &content)
if errors.Is(err, pgx.ErrNoRows) {
return domain.Transcript{}, false, nil
}
if err != nil {
return domain.Transcript{}, false, fmt.Errorf("store: get transcript: %w", err)
}
return domain.Transcript{
Source: domain.TranscriptSource(source),
Language: lang,
Content: content,
}, true, nil
}
// SaveTranscript upserts the shared transcript for (provider, providerVideoID).
// Only terminal outcomes belong here: SourceCaptions (with text) or SourceNone
// (no captions). A transient SourceRateLimited is rejected so persistence never
// masks a 429 as a permanent absence — that stays a per-user retry (ADR-014).
// Last write wins on conflict (a later re-fetch may correct an entry). It writes
// via the raw pool, NOT withUser — public content, shared, non-RLS (ADR-021).
func (s *Store) SaveTranscript(ctx context.Context, provider, providerVideoID string, t domain.Transcript) error {
switch t.Source {
case domain.SourceCaptions, domain.SourceNone:
// terminal — persist
case domain.SourceRateLimited:
return fmt.Errorf("store: refusing to persist transient rate-limited transcript for %s/%s", provider, providerVideoID)
default:
return fmt.Errorf("store: invalid transcript source %q", t.Source)
}
_, err := s.pool.Exec(ctx,
`INSERT INTO transcripts (provider, provider_video_id, source, language, content)
VALUES ($1, $2, $3, NULLIF($4, ''), NULLIF($5, ''))
ON CONFLICT (provider, provider_video_id)
DO UPDATE SET source = EXCLUDED.source,
language = EXCLUDED.language,
content = EXCLUDED.content,
fetched_at = NOW()`,
provider, providerVideoID, string(t.Source), t.Language, t.Content)
if err != nil {
return fmt.Errorf("store: save transcript: %w", err)
}
return nil
}
@@ -6,7 +6,7 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
func TestSetTranscriptStatus_RoundTrip(t *testing.T) { func TestSetTranscriptStatus_RoundTrip(t *testing.T) {
@@ -0,0 +1,87 @@
package store_test
import (
"context"
"testing"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/store"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/ports"
)
// Static check: Store satisfies the shared TranscriptStore port (ADR-021).
var _ ports.TranscriptStore = (*store.Store)(nil)
func TestSaveAndGetTranscript_RoundTrip(t *testing.T) {
s := newStore(t)
resetDB(t, rawPool(t))
ctx := context.Background()
want := domain.Transcript{Source: domain.SourceCaptions, Language: "en", Content: "the words"}
require.NoError(t, s.SaveTranscript(ctx, "youtube", "vid-1", want))
got, ok, err := s.GetTranscript(ctx, "youtube", "vid-1")
require.NoError(t, err)
require.True(t, ok, "a saved transcript must be found")
require.Equal(t, domain.SourceCaptions, got.Source)
require.Equal(t, "en", got.Language)
require.Equal(t, "the words", got.Content)
require.True(t, got.HasText())
}
func TestGetTranscript_Miss(t *testing.T) {
s := newStore(t)
resetDB(t, rawPool(t))
_, ok, err := s.GetTranscript(context.Background(), "youtube", "absent")
require.NoError(t, err, "a miss is not an error")
require.False(t, ok)
}
// A stored "no captions" outcome is a real hit: callers must skip without
// re-fetching, so ok is true even though there is no text (ADR-021 / ADR-007).
func TestSaveAndGetTranscript_NoneIsAStoredHit(t *testing.T) {
s := newStore(t)
resetDB(t, rawPool(t))
ctx := context.Background()
require.NoError(t, s.SaveTranscript(ctx, "youtube", "vid-none", domain.Transcript{Source: domain.SourceNone}))
got, ok, err := s.GetTranscript(ctx, "youtube", "vid-none")
require.NoError(t, err)
require.True(t, ok, "a stored SourceNone is a hit, not a miss")
require.Equal(t, domain.SourceNone, got.Source)
require.False(t, got.HasText())
}
// A transient 429 must never be persisted as a terminal transcript, or a later
// read would mask the rate-limit as a permanent "no transcript" (ADR-014).
func TestSaveTranscript_RejectsRateLimited(t *testing.T) {
s := newStore(t)
resetDB(t, rawPool(t))
err := s.SaveTranscript(context.Background(), "youtube", "vid-429",
domain.Transcript{Source: domain.SourceRateLimited})
require.Error(t, err)
_, ok, _ := s.GetTranscript(context.Background(), "youtube", "vid-429")
require.False(t, ok, "a rejected rate-limited save must leave nothing stored")
}
func TestSaveTranscript_UpsertLastWriteWins(t *testing.T) {
s := newStore(t)
resetDB(t, rawPool(t))
ctx := context.Background()
require.NoError(t, s.SaveTranscript(ctx, "youtube", "vid-up", domain.Transcript{Source: domain.SourceNone}))
require.NoError(t, s.SaveTranscript(ctx, "youtube", "vid-up",
domain.Transcript{Source: domain.SourceCaptions, Language: "en", Content: "now resolved"}))
got, ok, err := s.GetTranscript(ctx, "youtube", "vid-up")
require.NoError(t, err)
require.True(t, ok)
require.Equal(t, domain.SourceCaptions, got.Source)
require.Equal(t, "now resolved", got.Content)
}
@@ -0,0 +1,216 @@
package store
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
"github.com/jackc/pgx/v5"
"git.d-ma.be/mathias/tapir/internal/domain"
)
// ErrUploadNotFound is returned when an uploaded transcript does not exist OR is
// not the caller's. The two are deliberately indistinguishable: a non-owner learns
// nothing about whether someone else's upload exists (ADR-033).
var ErrUploadNotFound = errors.New("store: uploaded transcript not found")
// Uploaded transcripts are PRIVATE per-user content, stored in their own
// RLS-scoped table (migration 017) — never in the shared `transcripts` (ADR-021).
// Every method goes through withUser, so row-level security confines it to the
// caller's rows; each statement ALSO names user_id in its predicate, so the
// isolation does not rest on a single mechanism and stays testable under the
// superuser connection the unit tests use.
// CreateUpload records a new upload in the queued state and returns its id.
// hadVideo records that the file carried a video track (only its audio is used);
// it is kept for the life of the row so the page can say so.
func (s *Store) CreateUpload(ctx context.Context, userID, title string, hadVideo bool) (string, error) {
var id string
if err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
return tx.QueryRow(ctx,
`INSERT INTO uploaded_transcripts (user_id, title, had_video) VALUES ($1, $2, $3) RETURNING id`,
userID, title, hadVideo).Scan(&id)
}); err != nil {
return "", fmt.Errorf("store: create upload: %w", err)
}
return id, nil
}
// StartTranscribing moves an upload to transcribing.
func (s *Store) StartTranscribing(ctx context.Context, userID, id string) error {
return s.updateUpload(ctx, userID, id, "start transcribing",
`UPDATE uploaded_transcripts SET status = 'transcribing'
WHERE id = $1 AND user_id = $2`)
}
// CompleteUpload stores the finished transcription and moves the upload to done.
func (s *Store) CompleteUpload(ctx context.Context, userID, id string, t domain.Transcription) error {
segs, err := json.Marshal(t.Segments)
if err != nil {
return fmt.Errorf("store: complete upload: marshal segments: %w", err)
}
// A nil Reconciliation must be SQL NULL, not the JSON text "null": the page decides
// whether to claim a second opinion by whether this column is NULL.
var rec []byte
if t.Reconciliation != nil {
if rec, err = json.Marshal(t.Reconciliation); err != nil {
return fmt.Errorf("store: complete upload: marshal reconciliation: %w", err)
}
}
return s.updateUpload(ctx, userID, id, "complete upload",
`UPDATE uploaded_transcripts
SET status = 'done', content = $3, segments = $4, endpoint = $5,
audio_seconds = $6, reconciliation = $7, error_kind = NULL, finished_at = NOW()
WHERE id = $1 AND user_id = $2`,
t.Text, segs, t.Endpoint, t.AudioSeconds, rec)
}
// FailUpload moves an upload to failed with a typed reason.
func (s *Store) FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error {
return s.updateUpload(ctx, userID, id, "fail upload",
`UPDATE uploaded_transcripts
SET status = 'failed', error_kind = $3, finished_at = NOW()
WHERE id = $1 AND user_id = $2`,
string(kind))
}
// updateUpload runs one UPDATE keyed on ($1 = id, $2 = userID) plus extra args,
// and maps "no row touched" to ErrUploadNotFound — which is what a write to another
// user's upload looks like under RLS (the row is invisible, not forbidden).
func (s *Store) updateUpload(ctx context.Context, userID, id, op, sql string, args ...any) error {
all := append([]any{id, userID}, args...)
return s.withUser(ctx, userID, func(tx pgx.Tx) error {
ct, err := tx.Exec(ctx, sql, all...)
if err != nil {
return fmt.Errorf("store: %s: %w", op, err)
}
if ct.RowsAffected() == 0 {
return ErrUploadNotFound
}
return nil
})
}
// GetUpload returns one of the caller's uploads, including the transcript text.
func (s *Store) GetUpload(ctx context.Context, userID, id string) (*domain.UploadedTranscript, error) {
var u domain.UploadedTranscript
err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
row := tx.QueryRow(ctx, uploadSelect+` WHERE id = $1 AND user_id = $2`, id, userID)
return scanUpload(row, &u, true)
})
if errors.Is(err, pgx.ErrNoRows) {
return nil, ErrUploadNotFound
}
if err != nil {
return nil, fmt.Errorf("store: get upload: %w", err)
}
return &u, nil
}
// ListUploads returns the caller's uploads, newest first, WITHOUT transcript text
// or segments: a list view must not drag every transcript along.
func (s *Store) ListUploads(ctx context.Context, userID string) ([]domain.UploadedTranscript, error) {
var out []domain.UploadedTranscript
err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
rows, err := tx.Query(ctx, uploadSelect+` WHERE user_id = $1 ORDER BY created_at DESC, id`, userID)
if err != nil {
return err
}
defer rows.Close()
for rows.Next() {
var u domain.UploadedTranscript
if err := scanUpload(rows, &u, false); err != nil {
return err
}
out = append(out, u)
}
return rows.Err()
})
if err != nil {
return nil, fmt.Errorf("store: list uploads: %w", err)
}
return out, nil
}
// FailInterruptedUploads is the start-up sweep. The audio of an upload is never
// retained (ADR-033), so a job that was queued or transcribing when the process
// stopped can never resume; leaving it would be a stuck spinner. Every such row,
// for every user, becomes failed/interrupted and the user re-uploads. It
// enumerates users from the un-RLS'd identity map and scopes each pass with
// withUser, exactly as scheduled discovery does (ADR-018). Returns rows failed.
func (s *Store) FailInterruptedUploads(ctx context.Context) (int, error) {
users, err := s.ListAllUsers(ctx)
if err != nil {
return 0, fmt.Errorf("store: fail interrupted uploads: %w", err)
}
total := 0
for _, u := range users {
err := s.withUser(ctx, u.UserID, func(tx pgx.Tx) error {
ct, err := tx.Exec(ctx,
`UPDATE uploaded_transcripts
SET status = 'failed', error_kind = $2, finished_at = NOW()
WHERE user_id = $1 AND status IN ('queued', 'transcribing')`,
u.UserID, string(domain.UploadErrInterrupted))
total += int(ct.RowsAffected())
return err
})
if err != nil {
return total, fmt.Errorf("store: fail interrupted uploads: user %s: %w", u.UserID, err)
}
}
return total, nil
}
const uploadSelect = `SELECT id, user_id, title, status, content, segments, endpoint,
audio_seconds, error_kind, had_video, reconciliation, created_at, finished_at FROM uploaded_transcripts`
// scanUpload scans one uploadSelect row. withText controls whether the transcript
// body and segments are kept (GetUpload) or dropped (ListUploads).
func scanUpload(row pgx.Row, u *domain.UploadedTranscript, withText bool) error {
var (
status, errKind, endpoint, content *string
segments, reconciliation []byte
seconds *float64
finished *time.Time
)
if err := row.Scan(&u.ID, &u.UserID, &u.Title, &status, &content, &segments, &endpoint,
&seconds, &errKind, &u.HadVideo, &reconciliation, &u.CreatedAt, &finished); err != nil {
return err
}
u.Status = domain.UploadStatus(deref(status))
u.Endpoint = deref(endpoint)
u.ErrorKind = domain.UploadErrorKind(deref(errKind))
if seconds != nil {
u.AudioSeconds = *seconds
}
if finished != nil {
u.FinishedAt = *finished
}
if !withText {
return nil
}
if len(reconciliation) > 0 {
var r domain.Reconciliation
if err := json.Unmarshal(reconciliation, &r); err != nil {
return fmt.Errorf("unmarshal reconciliation: %w", err)
}
u.Reconciliation = &r
}
u.Content = deref(content)
if len(segments) > 0 {
if err := json.Unmarshal(segments, &u.Segments); err != nil {
return fmt.Errorf("unmarshal segments: %w", err)
}
}
return nil
}
func deref(s *string) string {
if s == nil {
return ""
}
return *s
}
@@ -0,0 +1,196 @@
package store_test
import (
"context"
"sort"
"testing"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/stretchr/testify/require"
)
// Isolation proof for ADR-033: an uploaded transcript is PRIVATE content owned by
// one user, so it lives in its own RLS-scoped table — never in the shared,
// non-RLS `transcripts` (ADR-021). These tests drive the database directly (raw
// SQL as the non-superuser `app` role) so they observe row-level security itself,
// not any application-layer WHERE clause. The superuser pool is used only to seed
// and to read pg_catalog; it bypasses RLS and so must never be the pool whose
// answers the assertions rest on.
// seedUploadedTranscript inserts one finished upload for userID as the superuser
// (RLS bypassed so both users' rows land regardless of any GUC) and returns its id.
// The text is synthetic — this repo is public, and a real transcript must never
// appear in it (#33).
func seedUploadedTranscript(t *testing.T, super *pgxpool.Pool, userID, text string) string {
t.Helper()
ctx := context.Background()
_, err := super.Exec(ctx, `INSERT INTO users (id) VALUES ($1) ON CONFLICT DO NOTHING`, userID)
require.NoError(t, err)
var id string
require.NoError(t, super.QueryRow(ctx,
`INSERT INTO uploaded_transcripts (user_id, title, status, content)
VALUES ($1, 'synthetic fixture', 'done', $2) RETURNING id`,
userID, text).Scan(&id))
return id
}
func TestUploadedTranscriptsRLS_ScopedReadsSeeOnlyOwnRows(t *testing.T) {
newStore(t)
super := rawPool(t)
resetDB(t, super)
seedUploadedTranscript(t, super, userA, "text owned by A")
seedUploadedTranscript(t, super, userB, "text owned by B")
app := appPool(t, super)
require.Equal(t, 0, scopedCount(t, app, "", "uploaded_transcripts"),
"no tapir.current_user_id must be deny-all: this proves RLS is live, not bypassed")
require.Equal(t, 1, scopedCount(t, app, userA, "uploaded_transcripts"),
"user A must see exactly its own upload")
require.Equal(t, 1, scopedCount(t, app, userB, "uploaded_transcripts"),
"user B must see exactly its own upload")
// Count alone cannot distinguish "A's row" from "B's row" — read the content.
var got string
ctx := context.Background()
tx, err := app.Begin(ctx)
require.NoError(t, err)
defer tx.Rollback(ctx) //nolint:errcheck
_, err = tx.Exec(ctx, `SELECT set_config('tapir.current_user_id', $1, true)`, userA)
require.NoError(t, err)
require.NoError(t, tx.QueryRow(ctx, `SELECT content FROM uploaded_transcripts`).Scan(&got))
require.Equal(t, "text owned by A", got, "A's scoped read must return A's text, never B's")
}
func TestUploadedTranscriptsRLS_CrossUserWritesAffectZeroRows(t *testing.T) {
newStore(t)
super := rawPool(t)
resetDB(t, super)
seedUploadedTranscript(t, super, userA, "text owned by A")
bID := seedUploadedTranscript(t, super, userB, "text owned by B")
app := appPool(t, super)
require.Equal(t, int64(0), scopedRowsAffected(t, app, userA,
`UPDATE uploaded_transcripts SET content = 'hacked' WHERE id = $1`, bID),
"user A scoped UPDATE must touch zero of B's uploads")
require.Equal(t, int64(0), scopedRowsAffected(t, app, userA,
`DELETE FROM uploaded_transcripts WHERE id = $1`, bID),
"user A scoped DELETE must touch zero of B's uploads")
var content string
require.NoError(t, super.QueryRow(context.Background(),
`SELECT content FROM uploaded_transcripts WHERE id = $1`, bID).Scan(&content))
require.Equal(t, "text owned by B", content, "B's upload must survive A's writes unchanged")
}
func TestUploadedTranscriptsRLS_InsertForAnotherUserIsRejected(t *testing.T) {
newStore(t)
super := rawPool(t)
resetDB(t, super)
seedUploadedTranscript(t, super, userA, "text owned by A")
_, err := super.Exec(context.Background(), `INSERT INTO users (id) VALUES ($1)`, userB)
require.NoError(t, err)
app := appPool(t, super)
ctx := context.Background()
tx, err := app.Begin(ctx)
require.NoError(t, err)
defer tx.Rollback(ctx) //nolint:errcheck
_, err = tx.Exec(ctx, `SELECT set_config('tapir.current_user_id', $1, true)`, userA)
require.NoError(t, err)
// Scoped as A, plant a row that claims to belong to B. The policy's WITH CHECK
// (defaulting to USING under FOR ALL) must refuse it.
_, err = tx.Exec(ctx,
`INSERT INTO uploaded_transcripts (user_id, title, status, content)
VALUES ($1, 'planted', 'done', 'planted')`, userB)
require.Error(t, err, "a row scoped as A must not be insertable on behalf of B")
require.Contains(t, err.Error(), "row-level security")
}
func TestUploadedTranscripts_RowSecurityIsEnabledForcedAndKeyedByUser(t *testing.T) {
newStore(t)
super := rawPool(t)
ctx := context.Background()
var enabled, forced bool
require.NoError(t, super.QueryRow(ctx,
`SELECT relrowsecurity, relforcerowsecurity FROM pg_class
WHERE oid = 'public.uploaded_transcripts'::regclass`).Scan(&enabled, &forced))
require.True(t, enabled, "uploaded_transcripts must have row-level security enabled")
require.True(t, forced,
"FORCE is what binds the table-owner role tapir; without it the policy is dead in production")
var userIDNotNull bool
require.NoError(t, super.QueryRow(ctx,
`SELECT is_nullable = 'NO' FROM information_schema.columns
WHERE table_name = 'uploaded_transcripts' AND column_name = 'user_id'`).Scan(&userIDNotNull))
require.True(t, userIDNotNull, "an upload with no owner would match no policy and be unreachable")
}
func TestUploadedTranscripts_DeleteUserRemovesTheirUploads(t *testing.T) {
s := newStore(t)
super := rawPool(t)
resetDB(t, super)
seedUploadedTranscript(t, super, userA, "text owned by A")
bID := seedUploadedTranscript(t, super, userB, "text owned by B")
require.NoError(t, s.DeleteUser(context.Background(), userA))
var aRows, bRows int
ctx := context.Background()
require.NoError(t, super.QueryRow(ctx,
`SELECT count(*) FROM uploaded_transcripts WHERE user_id = $1`, userA).Scan(&aRows))
require.NoError(t, super.QueryRow(ctx,
`SELECT count(*) FROM uploaded_transcripts WHERE id = $1`, bID).Scan(&bRows))
require.Equal(t, 0, aRows, "deleting a user must remove their private transcripts (ON DELETE CASCADE)")
require.Equal(t, 1, bRows, "deleting A must not touch B's transcripts")
}
// rowSecurityOffTables are the public-schema tables that deliberately have row
// security OFF. The list is exact on purpose: a NEW table added without RLS must
// fail this test and force a decision, instead of passing CI by omission — the
// failure mode ADR-033 names, where a naive reuse "leaks private content across
// tenants and passes CI green".
//
// - transcripts: shared public caption content, the single exception for CONTENT
// (ADR-021). It is the one that must never gain a private row.
// - user_identities: the Dex-subject -> user_id map, read at login before any
// user is known (migration 004). Holds no user content.
// - invitations: exists before the user does, so there is no user to scope by;
// the single-use token is the capability (migration 009).
// - schema_migrations: golang-migrate bookkeeping.
//
// #33 describes `transcripts` as "the only non-RLS table". That is true of content
// and false of tables; this test asserts the real set.
var rowSecurityOffTables = []string{
"invitations",
"schema_migrations",
"transcripts",
"user_identities",
}
func TestRowSecurityOffTablesAreExactlyTheKnownSet(t *testing.T) {
newStore(t)
super := rawPool(t)
rows, err := super.Query(context.Background(),
`SELECT c.relname FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public' AND c.relkind IN ('r', 'p') AND NOT c.relrowsecurity
ORDER BY c.relname`)
require.NoError(t, err)
defer rows.Close()
var got []string
for rows.Next() {
var name string
require.NoError(t, rows.Scan(&name))
got = append(got, name)
}
require.NoError(t, rows.Err())
want := append([]string{}, rowSecurityOffTables...)
sort.Strings(want)
require.Equal(t, want, got,
"every public table must have row security ON unless it is in rowSecurityOffTables; "+
"adding a table here is a decision that needs an ADR")
}
@@ -0,0 +1,286 @@
package store_test
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/store"
"git.d-ma.be/mathias/tapir/internal/domain"
)
// These exercise the Store API over uploaded_transcripts. The test Store connects
// as the superuser, which BYPASSES row-level security — so the cross-user cases
// here prove the query's own user_id predicate (defence in depth), while the
// database-enforced guarantee is proven separately, as the non-superuser app
// role, in uploaded_transcripts_isolation_test.go. Text is synthetic; this repo is
// public and no real transcript may appear in it (#33).
func seedUsers(t *testing.T, ids ...string) {
t.Helper()
super := rawPool(t)
resetDB(t, super)
for _, id := range ids {
_, err := super.Exec(context.Background(), `INSERT INTO users (id) VALUES ($1)`, id)
require.NoError(t, err)
}
}
func TestCreateUpload_StartsQueuedAndIsOwnedByTheCreator(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "Teams export", false)
require.NoError(t, err)
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, id, got.ID)
assert.Equal(t, userA, got.UserID)
assert.Equal(t, "Teams export", got.Title)
assert.Equal(t, domain.UploadQueued, got.Status)
assert.Empty(t, got.Content)
assert.True(t, got.FinishedAt.IsZero(), "an unfinished upload has no finish time")
}
func TestUploadLifecycle_CompletedKeepsTextSegmentsEndpointAndDuration(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id))
mid, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, domain.UploadTranscribing, mid.Status)
want := domain.Transcription{
Text: "God morgon allihopa. Vi går igenom läget.",
Segments: []domain.Segment{
{Start: 0.0, End: 1.5, Text: "God morgon allihopa."},
{Start: 1.5, End: 4.25, Text: "Vi går igenom läget."},
},
Endpoint: "iguana/kb-whisper",
AudioSeconds: 4.25,
}
require.NoError(t, s.CompleteUpload(ctx, userA, id, want))
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, domain.UploadDone, got.Status)
assert.Equal(t, want.Text, got.Content)
assert.Equal(t, want.Segments, got.Segments, "segment times must survive the JSONB round trip")
assert.Equal(t, "iguana/kb-whisper", got.Endpoint, "the serving endpoint is a quality fact and is recorded per row")
assert.InDelta(t, 4.25, got.AudioSeconds, 1e-9)
assert.False(t, got.FinishedAt.IsZero())
assert.Empty(t, got.ErrorKind)
}
func TestUploadLifecycle_FailedRecordsTypedKindAndNoContent(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id))
require.NoError(t, s.FailUpload(ctx, userA, id, domain.UploadErrTranscriptionFailed))
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, domain.UploadFailed, got.Status)
assert.Equal(t, domain.UploadErrTranscriptionFailed, got.ErrorKind)
assert.Empty(t, got.Content)
assert.False(t, got.FinishedAt.IsZero())
}
func TestGetUpload_AnotherUsersIDIsNotFound(t *testing.T) {
s := newStore(t)
seedUsers(t, userA, userB)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "A's meeting", false)
require.NoError(t, err)
_, err = s.GetUpload(ctx, userB, id)
assert.ErrorIs(t, err, store.ErrUploadNotFound,
"a non-owner must get not-found, not forbidden: existence is not disclosed")
}
func TestUploadWrites_ByAnotherUserChangeNothing(t *testing.T) {
s := newStore(t)
seedUsers(t, userA, userB)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "A's meeting", false)
require.NoError(t, err)
assert.ErrorIs(t, s.StartTranscribing(ctx, userB, id), store.ErrUploadNotFound)
assert.ErrorIs(t, s.CompleteUpload(ctx, userB, id, domain.Transcription{Text: "planted"}), store.ErrUploadNotFound)
assert.ErrorIs(t, s.FailUpload(ctx, userB, id, domain.UploadErrTranscriptionFailed), store.ErrUploadNotFound)
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, domain.UploadQueued, got.Status, "B's writes must leave A's upload exactly as it was")
assert.Empty(t, got.Content)
}
func TestListUploads_ReturnsOnlyTheCallersNewestFirstWithoutTranscriptText(t *testing.T) {
s := newStore(t)
seedUsers(t, userA, userB)
ctx := context.Background()
first, err := s.CreateUpload(ctx, userA, "first", false)
require.NoError(t, err)
second, err := s.CreateUpload(ctx, userA, "second", false)
require.NoError(t, err)
_, err = s.CreateUpload(ctx, userB, "B's", false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, first))
require.NoError(t, s.CompleteUpload(ctx, userA, first, domain.Transcription{Text: "long text body", Endpoint: "e"}))
got, err := s.ListUploads(ctx, userA)
require.NoError(t, err)
require.Len(t, got, 2, "A must see only A's uploads")
assert.Equal(t, second, got[0].ID, "newest first")
assert.Equal(t, first, got[1].ID)
assert.Empty(t, got[1].Content, "a list row must not drag the whole transcript along")
assert.Equal(t, domain.UploadDone, got[1].Status)
}
func TestFailInterruptedUploads_FailsOnlyUnfinishedRowsForEveryUser(t *testing.T) {
s := newStore(t)
seedUsers(t, userA, userB)
super := rawPool(t)
ctx := context.Background()
for subject, uid := range map[string]string{"sub-a": userA, "sub-b": userB} {
_, err := super.Exec(ctx, `INSERT INTO user_identities (dex_subject, user_id) VALUES ($1, $2)`, subject, uid)
require.NoError(t, err)
}
queued, err := s.CreateUpload(ctx, userA, "queued", false)
require.NoError(t, err)
running, err := s.CreateUpload(ctx, userB, "running", false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userB, running))
done, err := s.CreateUpload(ctx, userA, "done", false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, done))
require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "kept", Endpoint: "e"}))
n, err := s.FailInterruptedUploads(ctx)
require.NoError(t, err)
assert.Equal(t, 2, n, "the queued and the transcribing row, across both users")
for _, c := range []struct{ user, id string }{{userA, queued}, {userB, running}} {
got, err := s.GetUpload(ctx, c.user, c.id)
require.NoError(t, err)
assert.Equal(t, domain.UploadFailed, got.Status)
assert.Equal(t, domain.UploadErrInterrupted, got.ErrorKind,
"audio is not retained, so a restart cannot resume: the honest state is failed/interrupted")
}
kept, err := s.GetUpload(ctx, userA, done)
require.NoError(t, err)
assert.Equal(t, domain.UploadDone, kept.Status, "a finished transcript must survive the sweep")
assert.Equal(t, "kept", kept.Content)
}
func TestCreateUpload_RecordsWhetherTheFileHadAVideoTrack(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
withVideo, err := s.CreateUpload(ctx, userA, "camera off", true)
require.NoError(t, err)
audioOnly, err := s.CreateUpload(ctx, userA, "voice only", false)
require.NoError(t, err)
gotVideo, err := s.GetUpload(ctx, userA, withVideo)
require.NoError(t, err)
gotAudio, err := s.GetUpload(ctx, userA, audioOnly)
require.NoError(t, err)
listed, err := s.ListUploads(ctx, userA)
require.NoError(t, err)
assert.True(t, gotVideo.HadVideo, "the page must be able to say the video was ignored")
assert.False(t, gotAudio.HadVideo)
byID := map[string]bool{}
for _, u := range listed {
byID[u.ID] = u.HadVideo
}
assert.Equal(t, map[string]bool{withVideo: true, audioOnly: false}, byID, "the list carries it too")
}
func TestUploadHadVideo_SurvivesCompletionAndFailure(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
done, err := s.CreateUpload(ctx, userA, "d", true)
require.NoError(t, err)
failed, err := s.CreateUpload(ctx, userA, "f", true)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, done))
require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "x", Endpoint: "e"}))
require.NoError(t, s.FailUpload(ctx, userA, failed, domain.UploadErrTranscriptionFailed))
for _, id := range []string{done, failed} {
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.True(t, got.HadVideo, "completing or failing must not clear it: the notice outlives the job")
}
}
func TestUploadLifecycle_ReconciliationRoundTripsWithItsContestedSpots(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id))
want := &domain.Reconciliation{
Status: domain.ReconcileDone, Base: "iguana/pianissimo", Alt: "iguana/kb-whisper", Reconciler: "iguana/gemma4-26b",
Disputes: 380, AutoResolved: 90, Adjudicated: 285, Changed: 120, Unsure: 5, Errors: 0,
Contested: []domain.ContestedSpot{
{Seconds: 2210.5, Base: "SEB", Alt: "SCB", Used: "alt"},
{Seconds: 3400, Base: "T+1", Alt: "en T", Used: "base", Unsure: true},
},
}
require.NoError(t, s.CompleteUpload(ctx, userA, id, domain.Transcription{Text: "x", Endpoint: "e", Reconciliation: want}))
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, want, got.Reconciliation)
}
func TestUploadLifecycle_ASingleEngineTranscriptHasNoReconciliation(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id))
require.NoError(t, s.CompleteUpload(ctx, userA, id, domain.Transcription{Text: "x", Endpoint: "e"}))
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Nil(t, got.Reconciliation, "no reconciliation is not an empty one: the page must not claim a second opinion")
}
func TestUploadLifecycle_ASkippedReconciliationKeepsItsReason(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id))
rec := &domain.Reconciliation{Status: domain.ReconcileSkipped, Reason: "the model failed 3 times in a row: gateway down", Errors: 3}
require.NoError(t, s.CompleteUpload(ctx, userA, id, domain.Transcription{Text: "x", Endpoint: "e", Reconciliation: rec}))
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, rec, got.Reconciliation)
}
+65 -15
View File
@@ -7,7 +7,7 @@ import (
"github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
) )
// UpsertVideo persists a video's metadata and returns its durable store id (the // UpsertVideo persists a video's metadata and returns its durable store id (the
@@ -46,14 +46,16 @@ func (s *Store) UpsertVideo(ctx context.Context, v domain.Video) (string, error)
} }
if err := tx.QueryRow(ctx, if err := tx.QueryRow(ctx,
`INSERT INTO videos (user_id, provider, provider_video_id, title, url, published_at) `INSERT INTO videos (user_id, provider, provider_video_id, title, url, published_at, channel_title, duration_s)
VALUES ($1, $2, $3, $4, $5, $6) VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
ON CONFLICT (user_id, provider, provider_video_id) DO UPDATE SET ON CONFLICT (user_id, provider, provider_video_id) DO UPDATE SET
title = EXCLUDED.title, title = EXCLUDED.title,
url = EXCLUDED.url, url = EXCLUDED.url,
published_at = EXCLUDED.published_at published_at = EXCLUDED.published_at,
channel_title = COALESCE(NULLIF(EXCLUDED.channel_title, ''), videos.channel_title),
duration_s = COALESCE(EXCLUDED.duration_s, videos.duration_s)
RETURNING id`, RETURNING id`,
v.UserID, provider, v.ProviderVideoID, v.Title, v.URL, nullTime(v.PublishedAt), v.UserID, provider, v.ProviderVideoID, v.Title, v.URL, nullTime(v.PublishedAt), v.ChannelTitle, nullDuration(v.DurationSeconds),
).Scan(&id); err != nil { ).Scan(&id); err != nil {
return fmt.Errorf("store: upsert video: %w", err) return fmt.Errorf("store: upsert video: %w", err)
} }
@@ -73,12 +75,27 @@ func nullTime(t time.Time) *time.Time {
return &t return &t
} }
// NewestUnsummarizedVideoIDs returns up to limit of the user's videos that have // nullDuration maps an unknown duration (0) to SQL NULL so the upsert's
// no summary yet, newest first (published_at DESC, NULLS LAST). It caps the // COALESCE(EXCLUDED.duration_s, videos.duration_s) preserves a previously-known
// connect-time onboarding burst (Feature 1) at a fixed count: the caller marks // value instead of clobbering it with 0 (ADR-028; the channel_title backfill
// these for summarization through the shared rate gate. RLS-scoped via withUser, // stance, migration 014).
// so it only ever sees the requesting user's rows. limit <= 0 returns nil. func nullDuration(seconds int) *int {
func (s *Store) NewestUnsummarizedVideoIDs(ctx context.Context, userID string, limit int) ([]string, error) { if seconds <= 0 {
return nil
}
return &seconds
}
// OnboardBurstVideoIDs returns up to limit of the user's unsummarized videos for
// the connect-time onboarding burst (ADR-028), newest-first but quality-aware: a
// video is excluded when its duration is KNOWN and outside [minSeconds, maxSeconds]
// — dropping Shorts (below min) and multi-hour livestream VODs (above max) that
// would waste a scarce caption fetch on a poor first impression. A NULL/unknown
// duration is kept (degrade-open) but ranked AFTER known-good rows, so a freshly
// enriched good pick wins when both exist. minSeconds<=0 / maxSeconds<=0 each
// disable that bound (0/0 == pure newest-first, the reversibility lever).
// RLS-scoped via withUser; limit <= 0 returns nil.
func (s *Store) OnboardBurstVideoIDs(ctx context.Context, userID string, limit, minSeconds, maxSeconds int) ([]string, error) {
if limit <= 0 { if limit <= 0 {
return nil, nil return nil, nil
} }
@@ -91,16 +108,21 @@ func (s *Store) NewestUnsummarizedVideoIDs(ctx context.Context, userID string, l
AND NOT EXISTS ( AND NOT EXISTS (
SELECT 1 FROM summaries su SELECT 1 FROM summaries su
WHERE su.user_id = v.user_id AND su.video_id = v.id) WHERE su.user_id = v.user_id AND su.video_id = v.id)
ORDER BY v.published_at DESC NULLS LAST, v.seen_at DESC AND NOT (
LIMIT $2`, userID, limit) v.duration_s IS NOT NULL
AND ( ($3 > 0 AND v.duration_s < $3)
OR ($4 > 0 AND v.duration_s > $4) ))
ORDER BY (v.duration_s IS NOT NULL) DESC,
v.published_at DESC NULLS LAST, v.seen_at DESC
LIMIT $2`, userID, limit, minSeconds, maxSeconds)
if err != nil { if err != nil {
return fmt.Errorf("store: newest unsummarized: %w", err) return fmt.Errorf("store: onboard burst videos: %w", err)
} }
defer rows.Close() defer rows.Close()
for rows.Next() { for rows.Next() {
var id string var id string
if err := rows.Scan(&id); err != nil { if err := rows.Scan(&id); err != nil {
return fmt.Errorf("store: scan newest unsummarized: %w", err) return fmt.Errorf("store: scan onboard burst video: %w", err)
} }
ids = append(ids, id) ids = append(ids, id)
} }
@@ -110,3 +132,31 @@ func (s *Store) NewestUnsummarizedVideoIDs(ctx context.Context, userID string, l
} }
return ids, nil return ids, nil
} }
// DistinctChannels returns the user's distinct, non-empty source channel titles
// (the channels they have videos from), alphabetically — the option list for the
// feed's channel filter. RLS-scoped via withUser.
func (s *Store) DistinctChannels(ctx context.Context, userID string) ([]string, error) {
var out []string
if err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
rows, err := tx.Query(ctx,
`SELECT DISTINCT channel_title FROM videos
WHERE user_id = $1 AND channel_title IS NOT NULL AND channel_title <> ''
ORDER BY channel_title`, userID)
if err != nil {
return fmt.Errorf("store: distinct channels: %w", err)
}
defer rows.Close()
for rows.Next() {
var c string
if err := rows.Scan(&c); err != nil {
return fmt.Errorf("store: scan channel: %w", err)
}
out = append(out, c)
}
return rows.Err()
}); err != nil {
return nil, err
}
return out, nil
}
+90 -15
View File
@@ -7,7 +7,7 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
) )
func ytVideo(userID, provVideoID, title string) domain.Video { func ytVideo(userID, provVideoID, title string) domain.Video {
@@ -52,6 +52,36 @@ func TestUpsertVideo_ReturnsStableID(t *testing.T) {
require.Equal(t, 1, count, "must not duplicate the row") require.Equal(t, 1, count, "must not duplicate the row")
} }
func TestUpsertVideo_PersistsAndPreservesDuration(t *testing.T) {
ctx := context.Background()
s := newStore(t)
resetDB(t, rawPool(t))
// First upsert carries a known duration (ADR-028: discovery enriches it).
v := ytVideo(userA, "dur0000001x", "with duration")
v.DurationSeconds = 750
id, err := s.UpsertVideo(ctx, v)
require.NoError(t, err)
p := rawPool(t)
readDuration := func() *int {
var d *int
require.NoError(t, p.QueryRow(ctx, `SELECT duration_s FROM videos WHERE id = $1`, id).Scan(&d))
return d
}
require.NotNil(t, readDuration())
require.Equal(t, 750, *readDuration(), "duration must persist")
// A later upsert that does NOT know the duration (0) must not clobber it —
// the channel_title backfill stance (migration 014): COALESCE-preserve.
v2 := ytVideo(userA, "dur0000001x", "title updated, duration unknown")
v2.DurationSeconds = 0
_, err = s.UpsertVideo(ctx, v2)
require.NoError(t, err)
require.NotNil(t, readDuration(), "a 0/unknown re-upsert must not erase a known duration")
require.Equal(t, 750, *readDuration())
}
func TestUpsertVideo_IDMatchesSummaryDedup(t *testing.T) { func TestUpsertVideo_IDMatchesSummaryDedup(t *testing.T) {
ctx := context.Background() ctx := context.Background()
s := newStore(t) s := newStore(t)
@@ -82,34 +112,79 @@ func TestUpsertVideo_PerUserIsolation(t *testing.T) {
require.NotEqual(t, idA, idB, "same provider video for two users must be two distinct rows") require.NotEqual(t, idA, idB, "same provider video for two users must be two distinct rows")
} }
func TestNewestUnsummarizedVideoIDs(t *testing.T) { func TestOnboardBurstVideoIDs(t *testing.T) {
ctx := context.Background() ctx := context.Background()
s := newStore(t) s := newStore(t)
resetDB(t, rawPool(t)) resetDB(t, rawPool(t))
mk := func(user, pid string, day int) string { mk := func(user, pid string, day, dur int) string {
v := ytVideo(user, pid, pid) v := ytVideo(user, pid, pid)
v.PublishedAt = time.Date(2026, 6, day, 12, 0, 0, 0, time.UTC) v.PublishedAt = time.Date(2026, 6, day, 12, 0, 0, 0, time.UTC)
v.DurationSeconds = dur // 0 == unknown (NULL)
id, err := s.UpsertVideo(ctx, v) id, err := s.UpsertVideo(ctx, v)
require.NoError(t, err) require.NoError(t, err)
return id return id
} }
_ = mk(userA, "a1vid000001", 1) summarized := mk(userA, "summ0000001", 6, 600) // newest known-good, but already summarized
id2 := mk(userA, "a2vid000002", 2) good1 := mk(userA, "good0000001", 5, 600) // 10m, newest UNsummarized known-good
id3 := mk(userA, "a3vid000003", 3) tooLong := mk(userA, "toolong0001", 4, 20000) // > maxSeconds -> dropped
id4 := mk(userA, "a4vid000004", 4) tooShort := mk(userA, "tooshort001", 3, 30) // < minSeconds -> dropped
mk(userB, "b1vid000009", 9) // userB's newest — must never leak via RLS unknown := mk(userA, "unknown0001", 2, 0) // NULL duration -> kept, ranked last
good2 := mk(userA, "good0000002", 1, 800) // known-good but oldest
mk(userB, "bvid0000009", 9, 600) // userB -> must not leak via RLS
// The newest (v4) is summarized, so it's excluded from "unsummarized". // The newest video is summarized, so it is excluded from the burst.
require.NoError(t, s.Deliver(ctx, summary(userA, id4, "done"))) require.NoError(t, s.Deliver(ctx, summary(userA, summarized, "done")))
// Cap 2, newest-first unsummarized: v3 then v2 (v4 excluded; userB excluded). const minSec, maxSec = 60, 14400
got, err := s.NewestUnsummarizedVideoIDs(ctx, userA, 2)
// Known-good ranked before unknown, each newest-first within its group; the
// too-long and too-short videos are excluded by their known duration.
got, err := s.OnboardBurstVideoIDs(ctx, userA, 5, minSec, maxSec)
require.NoError(t, err) require.NoError(t, err)
require.Equal(t, []string{id3, id2}, got) require.Equal(t, []string{good1, good2, unknown}, got,
"known-good first (newest-first), then unknown-duration; junk excluded")
none, err := s.NewestUnsummarizedVideoIDs(ctx, userA, 0) // Cap is honoured.
capped, err := s.OnboardBurstVideoIDs(ctx, userA, 2, minSec, maxSec)
require.NoError(t, err) require.NoError(t, err)
require.Empty(t, none, "limit 0 returns nothing") require.Equal(t, []string{good1, good2}, capped)
// Bounds disabled (0/0) == pure newest-first, nothing excluded.
all, err := s.OnboardBurstVideoIDs(ctx, userA, 10, 0, 0)
require.NoError(t, err)
require.ElementsMatch(t, []string{good1, tooLong, tooShort, unknown, good2}, all,
"0/0 bounds disable the duration filter (prior newest-first behaviour)")
// limit <= 0 returns nothing.
none, err := s.OnboardBurstVideoIDs(ctx, userA, 0, minSec, maxSec)
require.NoError(t, err)
require.Empty(t, none)
}
func TestUpsertVideoPersistsChannelAndDistinctChannels(t *testing.T) {
ctx := context.Background()
s := newStore(t)
resetDB(t, rawPool(t))
mk := func(pid, channel string) {
v := ytVideo(userA, pid, pid)
v.ChannelTitle = channel
_, err := s.UpsertVideo(ctx, v)
require.NoError(t, err)
}
mk("aa11111aaaa", "Acme Talks")
mk("bb22222bbbb", "Acme Talks") // same channel
mk("cc33333cccc", "Zeta Channel")
// userB's channel must not leak.
vb := ytVideo(userB, "dd44444dddd", "x")
vb.ChannelTitle = "Bravo Only"
_, err := s.UpsertVideo(ctx, vb)
require.NoError(t, err)
got, err := s.DistinctChannels(ctx, userA)
require.NoError(t, err)
require.Equal(t, []string{"Acme Talks", "Zeta Channel"}, got,
"distinct, alphabetical, user-scoped (no Bravo Only)")
} }
+150 -37
View File
@@ -1,19 +1,25 @@
// Package summarizer implements ports.Summarizer backed by the copied llm // Package summarizer implements ports.Summarizer backed by the copied llm
// package's local-Primary -> BYO-Fallback routing (ADR-004). It is the only // package's routing (ADR-004, extended by ADR-022). It is the only place content
// place content ever leaves the engine toward an AI model, so it is also the // ever leaves the engine toward an AI model, so it is also the enforcement point
// enforcement point for the local-first guarantee in // for the local-first guarantee in docs/use-cases/ai_routing.feature: endpoints
// docs/use-cases/ai_routing.feature: a user with no BYO provider configured has // are tried in order, locals first, so content only reaches an external model
// their content sent to the local stack and nowhere else. // after every local endpoint has failed — and never at all when no external
// endpoint is configured.
package summarizer package summarizer
import ( import (
"bytes"
"context" "context"
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"log/slog"
"strings" "strings"
"time" "time"
"unicode/utf8"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/metrics"
) )
// Completer is the minimal LLM chat surface the Summarizer needs. // Completer is the minimal LLM chat surface the Summarizer needs.
@@ -29,19 +35,41 @@ type Endpoint struct {
Model string // resolved alias, e.g. "iguana/deepseek-r1-14b" Model string // resolved alias, e.g. "iguana/deepseek-r1-14b"
} }
// Summarizer routes a transcript through the local endpoint first, then the // Summarizer routes a transcript through an ordered chain of endpoints, trying
// optional BYO endpoint. It owns its routing (rather than delegating to // each in turn until one returns a parseable summary. It owns its routing
// llm.Router) so it can record which provider answered and whether the fallback // (rather than delegating to llm.Router) so it can record which provider answered
// was used — information llm.Router collapses away. // and whether a fallback was used — information llm.Router collapses away. The
// chain ordering is the local-first guarantee: callers place local endpoints
// first and any external endpoint last, so content only reaches an external model
// after every local endpoint has failed.
type Summarizer struct { type Summarizer struct {
primary Endpoint endpoints []Endpoint
fallback *Endpoint // nil => no BYO; primary errors are returned, never sent externally maxInputChars int // transcript truncation budget; 0 = no limit
now func() time.Time now func() time.Time
} }
// New constructs a Summarizer. fallback may be nil (no BYO provider configured). // New constructs a Summarizer from a primary endpoint and an optional fallback
// (the historical local-Primary -> BYO-Fallback shape, ADR-004). A nil fallback
// means a single-endpoint chain: errors are returned, content never leaves it.
func New(primary Endpoint, fallback *Endpoint) *Summarizer { func New(primary Endpoint, fallback *Endpoint) *Summarizer {
return &Summarizer{primary: primary, fallback: fallback, now: time.Now} eps := []Endpoint{primary}
if fallback != nil {
eps = append(eps, *fallback)
}
return &Summarizer{endpoints: eps, now: time.Now}
}
// NewChain constructs a Summarizer over an ordered endpoint chain (ADR-022).
// endpoints are tried in order; the first to return a parseable summary wins, and
// FallbackUsed is recorded true for any endpoint past the first. maxInputChars
// bounds the transcript text sent to every endpoint (0 = unbounded), so a long
// transcript does not overflow a small-context primary model's window. It panics
// on an empty chain — a wiring bug, not a runtime condition.
func NewChain(endpoints []Endpoint, maxInputChars int) *Summarizer {
if len(endpoints) == 0 {
panic("summarizer: NewChain requires at least one endpoint")
}
return &Summarizer{endpoints: endpoints, maxInputChars: maxInputChars, now: time.Now}
} }
const systemPrompt = `You are Tapir, a video-summarization assistant. const systemPrompt = `You are Tapir, a video-summarization assistant.
@@ -53,31 +81,42 @@ Respond with ONLY a JSON object, no prose and no code fences:
- "takeaways": the actionable conclusions a viewer should leave with. - "takeaways": the actionable conclusions a viewer should leave with.
Output the JSON object and nothing else.` Output the JSON object and nothing else.`
// Summarize implements ports.Summarizer. // Summarize implements ports.Summarizer. It walks the endpoint chain in order:
// the first endpoint whose reply parses into a non-empty summary wins. An
// endpoint is considered failed — and the next one tried — when the model call
// errors OR when its reply cannot be parsed (a 200 with malformed JSON or a
// highlights field the model emitted as a bare string). Truncation is applied
// once, up front, so every endpoint sees the same bounded prompt. When the whole
// chain fails, the joined error is returned so the engine queues the work for
// retry and delivers no summary.
func (s *Summarizer) Summarize(ctx context.Context, v domain.Video, t domain.Transcript) (domain.Summary, error) { func (s *Summarizer) Summarize(ctx context.Context, v domain.Video, t domain.Transcript) (domain.Summary, error) {
if !t.HasText() { if !t.HasText() {
return domain.Summary{}, fmt.Errorf("summarize: transcript for video %s has no text", v.ID) return domain.Summary{}, fmt.Errorf("summarize: transcript for video %s has no text", v.ID)
} }
user := buildUserPrompt(v, t) user := buildUserPrompt(v, t, s.maxInputChars)
// Primary = local stack. Only on its failure is anything sent externally, var errs []error
// and only when a BYO fallback is configured. for i, ep := range s.endpoints {
out, err := s.primary.Client.Complete(ctx, systemPrompt, user) fallback := i > 0
if err == nil { start := time.Now()
return s.build(v, s.primary, false, out) out, err := ep.Client.Complete(ctx, systemPrompt, user)
dur := time.Since(start)
if err != nil {
metrics.ObserveSummarize(ep.Model, "error", fallback, dur)
errs = append(errs, fmt.Errorf("%s/%s call: %w", ep.Provider, ep.Model, err))
continue
} }
sum, perr := s.build(v, ep, fallback, out)
if s.fallback == nil { if perr != nil {
// No BYO: content was sent to the local stack only. Surface the error so metrics.ObserveSummarize(ep.Model, "parse_error", fallback, dur)
// the engine can queue the work for retry; deliver no summary. errs = append(errs, fmt.Errorf("%s/%s output: %w", ep.Provider, ep.Model, perr))
return domain.Summary{}, fmt.Errorf("summarize: local AI failed and no BYO provider configured: %w", err) continue
} }
metrics.ObserveSummarize(ep.Model, "success", fallback, dur)
out, ferr := s.fallback.Client.Complete(ctx, systemPrompt, user) slog.Default().Info("summarized", "model", ep.Model, "fallback", fallback, "elapsed_ms", dur.Milliseconds())
if ferr != nil { return sum, nil
return domain.Summary{}, fmt.Errorf("summarize: local AI failed: %w; BYO %s failed: %v", err, s.fallback.Provider, ferr)
} }
return s.build(v, *s.fallback, true, out) return domain.Summary{}, fmt.Errorf("summarize: all %d endpoint(s) failed: %w", len(s.endpoints), errors.Join(errs...))
} }
func (s *Summarizer) build(v domain.Video, ep Endpoint, fallbackUsed bool, raw string) (domain.Summary, error) { func (s *Summarizer) build(v domain.Video, ep Endpoint, fallbackUsed bool, raw string) (domain.Summary, error) {
@@ -89,8 +128,8 @@ func (s *Summarizer) build(v domain.Video, ep Endpoint, fallbackUsed bool, raw s
UserID: v.UserID, UserID: v.UserID,
VideoID: v.ID, VideoID: v.ID,
Summary: parsed.Summary, Summary: parsed.Summary,
Highlights: parsed.Highlights, Highlights: []string(parsed.Highlights),
Takeaways: parsed.Takeaways, Takeaways: []string(parsed.Takeaways),
AIProvider: ep.Provider, AIProvider: ep.Provider,
AIModel: ep.Model, AIModel: ep.Model,
FallbackUsed: fallbackUsed, FallbackUsed: fallbackUsed,
@@ -98,20 +137,94 @@ func (s *Summarizer) build(v domain.Video, ep Endpoint, fallbackUsed bool, raw s
}, nil }, nil
} }
func buildUserPrompt(v domain.Video, t domain.Transcript) string { func buildUserPrompt(v domain.Video, t domain.Transcript, maxInputChars int) string {
var b strings.Builder var b strings.Builder
fmt.Fprintf(&b, "Title: %s\n", v.Title) fmt.Fprintf(&b, "Title: %s\n", v.Title)
if v.URL != "" { if v.URL != "" {
fmt.Fprintf(&b, "URL: %s\n", v.URL) fmt.Fprintf(&b, "URL: %s\n", v.URL)
} }
fmt.Fprintf(&b, "\nTranscript:\n%s", t.Content) fmt.Fprintf(&b, "\nTranscript:\n%s", truncate(t.Content, maxInputChars))
return b.String() return b.String()
} }
// truncate caps content to max bytes on a UTF-8 rune boundary, appending a
// marker so the model knows the transcript was cut. A non-positive max (or a
// content already within budget) returns content unchanged. Bounding the input
// keeps a long transcript from overflowing a small-context model's window — the
// production failure mode where koala/phi4-mini's 8k context returned HTTP 400 on
// a 11.6k-token transcript.
func truncate(content string, max int) string {
if max <= 0 || len(content) <= max {
return content
}
cut := max
for cut > 0 && !utf8.RuneStart(content[cut]) {
cut--
}
return content[:cut] + "\n…[transcript truncated to fit the model context]"
}
// flexStrings is a []string that also unmarshals from a single JSON string or a
// JSON array of scalars. Small local models (koala/phi4-mini) sometimes emit
// "highlights": "one point" instead of an array, or mix in a number; rather than
// fail the whole summary on that quirk, coerce to []string. Empty/whitespace
// elements are dropped.
type flexStrings []string
func (f *flexStrings) UnmarshalJSON(b []byte) error {
b = bytes.TrimSpace(b)
if len(b) == 0 || string(b) == "null" {
*f = nil
return nil
}
if b[0] == '[' {
var raw []json.RawMessage
if err := json.Unmarshal(b, &raw); err != nil {
return err
}
out := make([]string, 0, len(raw))
for _, r := range raw {
s, err := rawToString(r)
if err != nil {
return err
}
if strings.TrimSpace(s) != "" {
out = append(out, s)
}
}
*f = out
return nil
}
s, err := rawToString(b)
if err != nil {
return err
}
if strings.TrimSpace(s) == "" {
*f = nil
} else {
*f = flexStrings{s}
}
return nil
}
// rawToString renders a JSON scalar as text: a quoted string is unquoted; any
// other scalar (number, bool) is kept as its literal source so no content is lost.
func rawToString(r json.RawMessage) (string, error) {
r = bytes.TrimSpace(r)
if len(r) > 0 && r[0] == '"' {
var s string
if err := json.Unmarshal(r, &s); err != nil {
return "", err
}
return s, nil
}
return string(r), nil
}
type parsedSummary struct { type parsedSummary struct {
Summary string `json:"summary"` Summary string `json:"summary"`
Highlights []string `json:"highlights"` Highlights flexStrings `json:"highlights"`
Takeaways []string `json:"takeaways"` Takeaways flexStrings `json:"takeaways"`
} }
// parse extracts the JSON object from a model reply. Thinking models (qwen3, // parse extracts the JSON object from a model reply. Thinking models (qwen3,
+121 -4
View File
@@ -7,13 +7,36 @@ package summarizer
import ( import (
"context" "context"
"errors" "errors"
"net/http"
"net/http/httptest"
"strings" "strings"
"testing" "testing"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
"gitea.d-ma.be/mathias/tapir/internal/ports" "git.d-ma.be/mathias/tapir/internal/metrics"
"git.d-ma.be/mathias/tapir/internal/ports"
) )
// TestSummarizerRecordsMetric verifies the summarizer→metrics wiring (ADR-030)
// black-box: after a successful summarize, the public /metrics scrape shows a
// success observation for that endpoint's model.
func TestSummarizerRecordsMetric(t *testing.T) {
const model = "metrics-test-model"
s := New(Endpoint{Client: &fakeClient{reply: goodReply}, Provider: "local", Model: model}, nil)
if _, err := s.Summarize(context.Background(), testVideo(), testTranscript()); err != nil {
t.Fatalf("Summarize: %v", err)
}
rec := httptest.NewRecorder()
metrics.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/metrics", nil))
body := rec.Body.String()
if !strings.Contains(body, `tapir_summarize_duration_seconds`) ||
!strings.Contains(body, `model="`+model+`"`) ||
!strings.Contains(body, `outcome="success"`) {
t.Errorf("metrics scrape missing summarize success for %s", model)
}
}
// compile-time check: Summarizer satisfies the port. // compile-time check: Summarizer satisfies the port.
var _ ports.Summarizer = (*Summarizer)(nil) var _ ports.Summarizer = (*Summarizer)(nil)
@@ -129,8 +152,8 @@ func TestSummarize_NoBYO_ContentOnlyLocal(t *testing.T) {
local := &fakeClient{reply: goodReply} local := &fakeClient{reply: goodReply}
s := New(Endpoint{Client: local, Provider: "local", Model: "iguana/deepseek-r1-14b"}, nil) s := New(Endpoint{Client: local, Provider: "local", Model: "iguana/deepseek-r1-14b"}, nil)
if s.fallback != nil { if len(s.endpoints) != 1 {
t.Fatal("no BYO configured but fallback endpoint is non-nil") t.Fatalf("no BYO configured but chain has %d endpoints, want 1", len(s.endpoints))
} }
for i := 0; i < 3; i++ { for i := 0; i < 3; i++ {
sum, err := s.Summarize(context.Background(), testVideo(), testTranscript()) sum, err := s.Summarize(context.Background(), testVideo(), testTranscript())
@@ -176,3 +199,97 @@ func TestParse_EmptySummaryRejected(t *testing.T) {
t.Fatal("want error for empty summary (thinking model returned no content)") t.Fatal("want error for empty summary (thinking model returned no content)")
} }
} }
// parse tolerates a small model emitting "highlights" as a bare string instead
// of an array — the production koala/phi4-mini quirk that errored with
// "cannot unmarshal string into Go struct field ... highlights of type []string".
func TestParse_ToleratesStringHighlights(t *testing.T) {
p, err := parse(`{"summary":"s","highlights":"one big point","takeaways":["a","b"]}`)
if err != nil {
t.Fatalf("parse: %v", err)
}
if len(p.Highlights) != 1 || p.Highlights[0] != "one big point" {
t.Errorf("highlights = %v, want [\"one big point\"]", p.Highlights)
}
if len(p.Takeaways) != 2 {
t.Errorf("takeaways = %v, want 2", p.Takeaways)
}
}
// Chain: an endpoint that returns a 200 with unparseable output is treated as a
// failure, and the next endpoint in the chain is tried. This is the case the old
// primary->fallback shape missed — a parse error short-circuited instead of
// falling back.
func TestSummarize_FallsBackOnMalformedOutput(t *testing.T) {
bad := &fakeClient{reply: `{"summary": not json`}
good := &fakeClient{reply: goodReply}
s := NewChain([]Endpoint{
{Client: bad, Provider: "local", Model: "koala/phi4-mini"},
{Client: good, Provider: "local", Model: "koala/phi4-14b"},
}, 0)
sum, err := s.Summarize(context.Background(), testVideo(), testTranscript())
if err != nil {
t.Fatalf("Summarize: %v", err)
}
if sum.AIModel != "koala/phi4-14b" {
t.Errorf("AIModel = %q, want koala/phi4-14b (fell back past malformed primary)", sum.AIModel)
}
if !sum.FallbackUsed {
t.Error("FallbackUsed = false, want true")
}
if bad.calls != 1 || good.calls != 1 {
t.Errorf("calls: bad=%d good=%d, want 1 and 1", bad.calls, good.calls)
}
}
// Chain: when every endpoint fails, no summary is produced and the joined error
// names each failure so the engine queues the work for retry.
func TestSummarize_ChainAllEndpointsFail(t *testing.T) {
a := &fakeClient{err: errors.New("context overflow")}
b := &fakeClient{reply: "not even json"}
s := NewChain([]Endpoint{
{Client: a, Provider: "local", Model: "m1"},
{Client: b, Provider: "berget", Model: "m2"},
}, 0)
if _, err := s.Summarize(context.Background(), testVideo(), testTranscript()); err == nil {
t.Fatal("want error when all endpoints fail")
}
if a.calls != 1 || b.calls != 1 {
t.Errorf("calls: a=%d b=%d, want 1 and 1", a.calls, b.calls)
}
}
// A transcript longer than the chain's input budget is truncated before it
// reaches any model, so a small-context primary does not overflow its window.
func TestSummarize_TruncatesLongTranscript(t *testing.T) {
local := &fakeClient{reply: goodReply}
const budget = 100
s := NewChain([]Endpoint{{Client: local, Provider: "local", Model: "m"}}, budget)
long := domain.Transcript{
VideoID: "vid-1", UserID: "user-1", Source: domain.SourceCaptions,
Content: strings.Repeat("word ", 1000), // 5000 bytes, well over budget
}
if _, err := s.Summarize(context.Background(), testVideo(), long); err != nil {
t.Fatalf("Summarize: %v", err)
}
// The prompt carries title/URL framing plus the truncation marker, so allow
// headroom over the raw transcript budget — but it must be far below 5000.
if len(local.lastUser) > budget+300 {
t.Errorf("prompt length = %d, want <= %d (transcript not truncated)", len(local.lastUser), budget+300)
}
if !strings.Contains(local.lastUser, "truncated") {
t.Error("truncation marker missing from prompt")
}
}
func TestNewChain_PanicsOnEmptyChain(t *testing.T) {
defer func() {
if recover() == nil {
t.Fatal("want panic on empty endpoint chain")
}
}()
NewChain(nil, 0)
}
+53
View File
@@ -0,0 +1,53 @@
package transcribe
import (
"context"
"errors"
"fmt"
"git.d-ma.be/mathias/tapir/internal/domain"
)
// Backend is one transcription endpoint. *Gateway satisfies it; Name is the value
// recorded as the serving endpoint.
type Backend interface {
Name() string
Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error)
}
// Chain tries a primary (local) backend and, if it fails for ANY reason, a
// fallback. The fallback exists because iguana can wedge permanently on a Metal
// OOM (infra#273) — which is also why a fallback is a data-egress decision: the
// berget endpoint is a cloud service, so the result records which one served it.
type Chain struct {
primary Backend
fallback Backend // nil disables the fallback
}
// NewChain builds a Chain. A nil fallback means a primary failure is the result.
func NewChain(primary, fallback Backend) *Chain {
return &Chain{primary: primary, fallback: fallback}
}
// Transcribe returns the first backend's success. It falls back on any primary
// error — transport, timeout, non-2xx, empty transcript — and never inspects a
// response body to decide. A cancelled context stops the chain: a job the user or
// shutdown cancelled must not ship its audio to a cloud endpoint as a last act.
func (c *Chain) Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error) {
t, err := c.primary.Transcribe(ctx, wavPath)
if err == nil {
t.Endpoint = c.primary.Name()
return t, nil
}
if c.fallback == nil || ctx.Err() != nil {
return domain.Transcription{}, fmt.Errorf("%s: %w", c.primary.Name(), err)
}
t, ferr := c.fallback.Transcribe(ctx, wavPath)
if ferr != nil {
return domain.Transcription{}, errors.Join(
fmt.Errorf("%s: %w", c.primary.Name(), err),
fmt.Errorf("%s: %w", c.fallback.Name(), ferr))
}
t.Endpoint = c.fallback.Name()
return t, nil
}
@@ -0,0 +1,99 @@
package transcribe_test
import (
"context"
"errors"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
"git.d-ma.be/mathias/tapir/internal/domain"
)
type fakeBackend struct {
name string
text string
err error
calls int
}
func (f *fakeBackend) Name() string { return f.name }
func (f *fakeBackend) Transcribe(_ context.Context, _ string) (domain.Transcription, error) {
f.calls++
if f.err != nil {
return domain.Transcription{}, f.err
}
return domain.Transcription{Text: f.text, Endpoint: f.name}, nil
}
func TestChain_PrimarySuccessNeverTouchesTheFallback(t *testing.T) {
primary := &fakeBackend{name: "iguana/kb-whisper", text: "primary text"}
fallback := &fakeBackend{name: "berget/kb-whisper", text: "fallback text"}
got, err := transcribe.NewChain(primary, fallback).Transcribe(context.Background(), "/a.wav")
require.NoError(t, err)
assert.Equal(t, "primary text", got.Text)
assert.Equal(t, "iguana/kb-whisper", got.Endpoint)
assert.Zero(t, fallback.calls, "audio must not leave the homelab when the local endpoint worked")
}
func TestChain_AnyPrimaryFailureFallsBackAndRecordsTheFallbackEndpoint(t *testing.T) {
tests := []struct {
name string
err error
}{
{"server error", &transcribe.StatusError{Code: 500}},
{"gateway timeout", &transcribe.StatusError{Code: 504}},
{"client error", &transcribe.StatusError{Code: 400}},
{"transport failure", errors.New("dial tcp: connection refused")},
{"empty transcript", transcribe.ErrEmptyTranscript},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
primary := &fakeBackend{name: "iguana/kb-whisper", err: tt.err}
fallback := &fakeBackend{name: "berget/kb-whisper", text: "fallback text"}
got, err := transcribe.NewChain(primary, fallback).Transcribe(context.Background(), "/a.wav")
require.NoError(t, err)
assert.Equal(t, "fallback text", got.Text)
assert.Equal(t, "berget/kb-whisper", got.Endpoint,
"the row must say berget served it: that means the audio left the homelab")
})
}
}
func TestChain_BothFailingReportsBothErrors(t *testing.T) {
primary := &fakeBackend{name: "iguana/kb-whisper", err: errors.New("iguana down")}
fallback := &fakeBackend{name: "berget/kb-whisper", err: errors.New("berget down")}
_, err := transcribe.NewChain(primary, fallback).Transcribe(context.Background(), "/a.wav")
require.Error(t, err)
assert.Contains(t, err.Error(), "iguana down")
assert.Contains(t, err.Error(), "berget down")
}
func TestChain_WithoutAFallbackAPrimaryFailureIsTheResult(t *testing.T) {
primary := &fakeBackend{name: "iguana/kb-whisper", err: errors.New("iguana down")}
_, err := transcribe.NewChain(primary, nil).Transcribe(context.Background(), "/a.wav")
require.Error(t, err)
assert.Contains(t, err.Error(), "iguana down")
}
func TestChain_ACancelledJobDoesNotFallBack(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
primary := &fakeBackend{name: "iguana/kb-whisper", err: context.Canceled}
fallback := &fakeBackend{name: "berget/kb-whisper", text: "x"}
_, err := transcribe.NewChain(primary, fallback).Transcribe(ctx, "/a.wav")
assert.ErrorIs(t, err, context.Canceled)
assert.Zero(t, fallback.calls, "a cancelled job must not ship the audio to a cloud endpoint")
}
+77
View File
@@ -0,0 +1,77 @@
package transcribe
import (
"context"
"errors"
"fmt"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/reconcile"
)
// Ensemble transcribes the same WAV with two engines, one after the other, and has a
// local model reconcile their disagreements (ADR-034). It satisfies uploads.Transcriber,
// so the worker neither knows nor cares that two engines were involved.
//
// It degrades in steps and never turns a recoverable problem into a failed job:
// one engine down → the other's text, said so on the result; the model down → the
// base text unreconciled; both engines down → the cloud fallback, if configured, and
// recorded as such. A cancelled job stops everything and never reaches the cloud.
type Ensemble struct {
base, alt Backend
fallback Backend // nil = no cloud fallback
adj reconcile.Adjudicator
opts reconcile.Options
}
// NewEnsemble builds an Ensemble. base is the engine whose text stands where the two
// disagree and is the faster one, so it runs first; alt supplies the other reading.
func NewEnsemble(base, alt, fallback Backend, adj reconcile.Adjudicator, opts reconcile.Options) *Ensemble {
return &Ensemble{base: base, alt: alt, fallback: fallback, adj: adj, opts: opts}
}
// Transcribe implements uploads.Transcriber.
func (e *Ensemble) Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error) {
b, berr := e.base.Transcribe(ctx, wavPath)
if err := ctx.Err(); err != nil {
return domain.Transcription{}, err
}
a, aerr := e.alt.Transcribe(ctx, wavPath)
if err := ctx.Err(); err != nil {
return domain.Transcription{}, err
}
switch {
case berr == nil && aerr == nil:
out, err := reconcile.Reconcile(ctx, b, a, e.adj, e.opts)
if err != nil {
return domain.Transcription{}, err
}
out.Endpoint = fmt.Sprintf("%s + %s, reconciled by %s", e.base.Name(), e.alt.Name(), e.opts.Reconciler)
return out, nil
case berr == nil:
return e.single(b, e.base.Name(), fmt.Sprintf("%s failed: %v", e.alt.Name(), aerr)), nil
case aerr == nil:
return e.single(a, e.alt.Name(), fmt.Sprintf("%s failed: %v", e.base.Name(), berr)), nil
}
both := errors.Join(fmt.Errorf("%s: %w", e.base.Name(), berr), fmt.Errorf("%s: %w", e.alt.Name(), aerr))
if e.fallback == nil {
return domain.Transcription{}, both
}
f, ferr := e.fallback.Transcribe(ctx, wavPath)
if ferr != nil {
return domain.Transcription{}, errors.Join(both, fmt.Errorf("%s: %w", e.fallback.Name(), ferr))
}
return e.single(f, e.fallback.Name(), "both local engines failed"), nil
}
// single is a transcript from one engine only: nothing to reconcile, and the result
// says why, because a reader should know the second opinion was missing.
func (e *Ensemble) single(t domain.Transcription, engine, reason string) domain.Transcription {
t.Endpoint = engine
t.Reconciliation = &domain.Reconciliation{
Status: domain.ReconcileSingle, Reason: reason, Base: e.opts.BaseEngine, Alt: e.opts.AltEngine, Reconciler: e.opts.Reconciler,
}
return t
}
@@ -0,0 +1,206 @@
package transcribe_test
import (
"context"
"errors"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/reconcile"
)
// orderedBackend records, in a shared slice, when it was called.
type orderedBackend struct {
name string
text string
err error
calls *[]string
}
func (o *orderedBackend) Name() string { return o.name }
func (o *orderedBackend) Transcribe(_ context.Context, _ string) (domain.Transcription, error) {
*o.calls = append(*o.calls, o.name)
if o.err != nil {
return domain.Transcription{}, o.err
}
return domain.Transcription{Text: o.text, Endpoint: o.name, AudioSeconds: 10}, nil
}
type verdictAdj struct {
v reconcile.Verdict
err error
calls int
}
func (a *verdictAdj) Adjudicate(context.Context, reconcile.Request) (reconcile.Verdict, error) {
a.calls++
return a.v, a.err
}
var ensembleOpts = reconcile.Options{Reconciler: "iguana/gemma4-26b", BaseEngine: "iguana/pianissimo", AltEngine: "iguana/kb-whisper"}
type rig struct {
e *transcribe.Ensemble
calls *[]string
adj *verdictAdj
}
func newEnsemble(baseText, altText string, baseErr, altErr error, withFallback bool, adj *verdictAdj) *rig {
calls := &[]string{}
base := &orderedBackend{name: "iguana/pianissimo", text: baseText, err: baseErr, calls: calls}
alt := &orderedBackend{name: "iguana/kb-whisper", text: altText, err: altErr, calls: calls}
var fb transcribe.Backend
if withFallback {
fb = &orderedBackend{name: "berget/kb-whisper", text: "fallback text", calls: calls}
}
return &rig{e: transcribe.NewEnsemble(base, alt, fb, adj, ensembleOpts), calls: calls, adj: adj}
}
func TestEnsemble_BothEnginesRunInOrderAndTheirDisagreementIsReconciled(t *testing.T) {
adj := &verdictAdj{v: reconcile.VerdictAlt}
r := newEnsemble("Totalt 1015 kronor idag.", "Totalt 1050 kronor idag.", nil, nil, true, adj)
got, err := r.e.Transcribe(context.Background(), "/a.wav")
require.NoError(t, err)
assert.Equal(t, []string{"iguana/pianissimo", "iguana/kb-whisper"}, *r.calls,
"the 40-second engine runs first, so a failure there is known early; the fallback is not touched")
assert.Equal(t, "Totalt 1050 kronor idag.", got.Text)
assert.Equal(t, "iguana/pianissimo + iguana/kb-whisper, reconciled by iguana/gemma4-26b", got.Endpoint,
"every contributor is recorded: which engines touched the audio is a quality and a privacy fact")
require.NotNil(t, got.Reconciliation)
assert.Equal(t, domain.ReconcileDone, got.Reconciliation.Status)
}
func TestEnsemble_IdenticalTextsNeverCallTheModel(t *testing.T) {
adj := &verdictAdj{v: reconcile.VerdictAlt}
r := newEnsemble("Vi går igenom läget.", "Vi går igenom läget.", nil, nil, false, adj)
got, err := r.e.Transcribe(context.Background(), "/a.wav")
require.NoError(t, err)
assert.Zero(t, adj.calls)
assert.Equal(t, "Vi går igenom läget.", got.Text)
}
func TestEnsemble_WhenOneEngineFailsTheOtherIsUsedAndItIsSaidSo(t *testing.T) {
tests := []struct {
name string
baseErr error
altErr error
wantText string
wantEndpoint string
wantReasonHas string
}{
{"pianissimo down", errors.New("pianissimo wedged"), nil, "alt text", "iguana/kb-whisper", "iguana/pianissimo failed"},
{"kb-whisper down", nil, errors.New("metal oom"), "base text", "iguana/pianissimo", "iguana/kb-whisper failed"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
adj := &verdictAdj{v: reconcile.VerdictAlt}
r := newEnsemble("base text", "alt text", tt.baseErr, tt.altErr, true, adj)
got, err := r.e.Transcribe(context.Background(), "/a.wav")
require.NoError(t, err)
assert.Equal(t, tt.wantText, got.Text)
assert.Equal(t, tt.wantEndpoint, got.Endpoint)
assert.Zero(t, adj.calls, "there is nothing to reconcile")
require.NotNil(t, got.Reconciliation)
assert.Equal(t, domain.ReconcileSingle, got.Reconciliation.Status)
assert.Contains(t, got.Reconciliation.Reason, tt.wantReasonHas)
assert.NotContains(t, *r.calls, "berget/kb-whisper", "one local engine working means audio does not leave the homelab")
})
}
}
func TestEnsemble_OnlyWhenBothLocalEnginesFailDoesTheCloudFallbackRun(t *testing.T) {
r := newEnsemble("", "", errors.New("a"), errors.New("b"), true, &verdictAdj{})
got, err := r.e.Transcribe(context.Background(), "/a.wav")
require.NoError(t, err)
assert.Equal(t, "fallback text", got.Text)
assert.Equal(t, "berget/kb-whisper", got.Endpoint, "the row must say berget served it: the audio left the homelab")
assert.Equal(t, domain.ReconcileSingle, got.Reconciliation.Status)
assert.Contains(t, got.Reconciliation.Reason, "both local engines failed")
}
func TestEnsemble_BothFailingWithNoFallbackReportsBothErrors(t *testing.T) {
r := newEnsemble("", "", errors.New("pianissimo down"), errors.New("whisper down"), false, &verdictAdj{})
_, err := r.e.Transcribe(context.Background(), "/a.wav")
require.Error(t, err)
assert.Contains(t, err.Error(), "pianissimo down")
assert.Contains(t, err.Error(), "whisper down")
}
func TestEnsemble_ACancelledJobNeverReachesTheCloud(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
r := newEnsemble("", "", context.Canceled, context.Canceled, true, &verdictAdj{})
_, err := r.e.Transcribe(ctx, "/a.wav")
assert.ErrorIs(t, err, context.Canceled)
assert.NotContains(t, *r.calls, "berget/kb-whisper", "a cancelled job must not ship its audio to a cloud endpoint as a last act")
}
func TestEnsemble_AModelThatIsDownGivesTheBaseTextUnreconciledNotAFailedJob(t *testing.T) {
adj := &verdictAdj{err: errors.New("gateway down")}
// Four separate disagreements, each separated by two shared words, so the
// breaker trips after three consecutive failures.
r := newEnsemble("a1 s t b1 s t c1 s t d1", "a2 s t b2 s t c2 s t d2", nil, nil, false, adj)
got, err := r.e.Transcribe(context.Background(), "/a.wav")
require.NoError(t, err)
assert.Equal(t, "a1 s t b1 s t c1 s t d1", got.Text)
assert.Equal(t, domain.ReconcileSkipped, got.Reconciliation.Status)
assert.Equal(t, 3, adj.calls)
}
// cancelling is a backend that cancels the job's context while it runs, the way a
// shutdown arrives in the middle of a long transcription.
type cancelling struct {
name string
cancel context.CancelFunc
calls *[]string
}
func (c *cancelling) Name() string { return c.name }
func (c *cancelling) Transcribe(_ context.Context, _ string) (domain.Transcription, error) {
*c.calls = append(*c.calls, c.name)
c.cancel()
return domain.Transcription{}, context.Canceled
}
func TestEnsemble_AShutdownDuringTheFirstEngineStopsBeforeTheSecondOrTheCloud(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
calls := &[]string{}
base := &cancelling{name: "iguana/pianissimo", cancel: cancel, calls: calls}
alt := &orderedBackend{name: "iguana/kb-whisper", text: "alt", calls: calls}
fb := &orderedBackend{name: "berget/kb-whisper", text: "fb", calls: calls}
_, err := transcribe.NewEnsemble(base, alt, fb, &verdictAdj{}, ensembleOpts).Transcribe(ctx, "/a.wav")
assert.ErrorIs(t, err, context.Canceled)
assert.Equal(t, []string{"iguana/pianissimo"}, *calls, "no second engine and no cloud once the job is cancelled")
}
func TestEnsemble_AShutdownDuringTheSecondEngineIsAnErrorNotASingleEngineSuccess(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
calls := &[]string{}
base := &orderedBackend{name: "iguana/pianissimo", text: "base", calls: calls}
alt := &cancelling{name: "iguana/kb-whisper", cancel: cancel, calls: calls}
got, err := transcribe.NewEnsemble(base, alt, nil, &verdictAdj{}, ensembleOpts).Transcribe(ctx, "/a.wav")
assert.ErrorIs(t, err, context.Canceled, "a cancelled job must not be recorded as 'kb-whisper failed'")
assert.Empty(t, got.Text)
}

Some files were not shown because too many files have changed in this diff Show More