Compare commits

..
167 Commits
Author SHA1 Message Date
mathiasandClaude Sonnet 5.5 7029dd1d7f test(e2e): a finished job's phases are all closed and it was reporting while it ran (ADR-035)
CI / Lint / Test / Vet (pull_request) Successful in 23s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:05:36 +02:00
mathiasandClaude Sonnet 5.5 c977618979 docs(claude): the progress tracker, the two timestamps, and the log's privacy rule (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:04:32 +02:00
mathiasandClaude Sonnet 5.5 0763e6238b docs(data-model): progress, progress_at, alive_at, and every judged spot stored (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:04:32 +02:00
mathiasandClaude Sonnet 5.5 6722433eb0 test(acceptance): map the progress, log and judged-spot scenarios to their covering tests
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:04:23 +02:00
mathiasandClaude Sonnet 5.5 ac75971030 docs(use-cases): scenarios for progress, the log and the judged spots (ADR-035) -- the coverage gate is red until they are mapped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:04:19 +02:00
mathiasandClaude Sonnet 5.5 02a065b826 test(web): opt-in renderer of every progress state for a screenshot review (TAPIR_SHOT_DIR)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:03:55 +02:00
mathiasandClaude Sonnet 5.5 105777f1aa fix(web): the spinner sits in the same mark column as the other glyphs; a lost job's running phase gets a warning mark (screenshot review)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:03:55 +02:00
mathiasandClaude Sonnet 5.5 18d8050a2b fix(web): the progress bar ends at the time column and the lost mark is error-coloured (screenshot review)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:03:55 +02:00
mathiasandClaude Sonnet 5.5 c06d285a3a test(web): the lost job's running phase carries a warning mark (red; found in the screenshot review)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:03:32 +02:00
mathiasandClaude Sonnet 5.5 f74fbbc40e test(cmd): the uploads store double has the progress methods the worker now needs
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:01:37 +02:00
mathiasandClaude Sonnet 5.5 cd7595a048 feat(web): a spot shows the words before it, and an empty list shows no section (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:01:21 +02:00
mathiasandClaude Sonnet 5.5 24d1a99ed0 feat(web): list only the spots to check, in order and capped (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:01:21 +02:00
mathiasandClaude Sonnet 5.5 8cb0b0a12d test(web): the kept-spot assertion looks for what the page actually renders (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:00:56 +02:00
mathiasandClaude Sonnet 5.5 6dfd53ac71 test(web): the page lists only changed and uncertain spots, with the words before each, though every judged spot is stored (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:00:49 +02:00
mathiasandClaude Sonnet 5.5 d62331b5f9 test(web): the quiet note is capitalised as the page writes it
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:00:23 +02:00
mathiasandClaude Sonnet 5.5 a7668b670e feat(web): the upload page shows the job's phases, a bar only where the count is real, and says when the service is lost (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:00:23 +02:00
mathiasandClaude Sonnet 5.5 89c9dd6705 feat(web): tell running from quiet from lost; poll every two seconds (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:00:23 +02:00
mathiasandClaude Sonnet 5.5 d1dcb131f2 feat(web): progress styles in tapir's own tokens -- a mono step log, a CSS spinner that respects reduced motion (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:00:23 +02:00
mathiasandClaude Sonnet 5.5 d0f2dcf838 test(web): the page shows real phases, tells running from quiet from lost, and never invents a percentage (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:57:58 +02:00
mathiasandClaude Sonnet 5.5 a97598586c test(reconcile): a span across a segment edge leaves neither segment holding a replaced word -- a mutation that skipped this survived
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:56:53 +02:00
mathiasandClaude Sonnet 5.5 976aeb71c6 feat(reconcile): report the comparison and deciding phases, store every judged spot, merge the words into the segments (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:56:53 +02:00
mathiasandClaude Sonnet 5.5 f74ad726e2 feat(domain): a judged spot carries the few words before it (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:56:53 +02:00
mathiasandClaude Sonnet 5.5 9297f728c2 test(reconcile): every judged spot with its left context; segments rewritten per segment, or left alone when they do not line up (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:55:43 +02:00
mathiasandClaude Sonnet 5.5 520ab666f8 test(reconcile): the comparison and the deciding phase report real counts, and a shutdown is neither done nor failed (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:55:43 +02:00
mathiasandClaude Sonnet 5.5 0b7bf08c95 test(reconcile): kept spots are stored, segments carry the merged words, the cap is an option -- three old assertions changed on purpose (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:55:43 +02:00
mathiasandClaude Sonnet 5.5 aefc8103f7 feat(transcribe): the ensemble declares and reports its phases; a failed engine shows failed, a shutdown does not (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:54:31 +02:00
mathiasandClaude Sonnet 5.5 f064dec185 test(transcribe): the ensemble declares and reports its phases; a failed engine is shown failed, a shutdown is not (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:54:07 +02:00
mathiasandClaude Sonnet 5.5 c6319b9bc6 refactor(uploads): the worker's own failure is FailRunning
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:53:43 +02:00
mathiasandClaude Sonnet 5.5 5f279d7e71 feat(progress): keyed Fail for a soft failure, Note for the operator, FailRunning for the job's own (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:53:43 +02:00
mathiasandClaude Sonnet 5.5 a6b8a2b744 test(progress): a keyed soft failure that logs why, a note that costs no write, and FailRunning for the job's own failure (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:53:33 +02:00
mathiasandClaude Sonnet 5.5 9393c27cdb test(uploads): the worker waits for its heartbeat to finish -- a mutation that skipped the wait survived until now
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:53:10 +02:00
mathiasandClaude Sonnet 5.5 538ed93ccd test(uploads): a slow heartbeat write, so one is in flight when the job ends
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:53:10 +02:00
mathiasandClaude Sonnet 5.5 27a88a0a1f feat(uploads): the worker reports real phases, beats on its own, and logs each one (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:53:10 +02:00
mathiasandClaude Sonnet 5.5 eb4d4b67e6 test(uploads): the worker reports real phases, beats on its own, and logs them in order without the transcript (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:52:20 +02:00
mathiasandClaude Sonnet 5.5 3f66c35497 test(uploads): the worker's store double records progress saves and heartbeats
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:52:20 +02:00
mathiasandClaude Sonnet 5.5 f8526d4549 feat(store): SaveProgress and Heartbeat; Get returns the phases and both timestamps (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:51:39 +02:00
mathiasandClaude Sonnet 5.5 41ebba3963 feat(store): migration 020 -- uploaded_transcripts.progress, progress_at, alive_at (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:51:11 +02:00
mathiasandClaude Sonnet 5.5 256e5b1a66 test(store): progress and heartbeat on an upload row -- round trip, alive is not progress, another user's write changes nothing (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:51:04 +02:00
mathiasandClaude Sonnet 5.5 8ec43ad393 test(progress): read the log buffer safely while the heartbeat writes to it
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:50:39 +02:00
mathiasandClaude Sonnet 5.5 4bfe578a05 feat(progress): Tracker -- one writer for the page and the log, throttled counts, a heartbeat of its own (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:50:39 +02:00
mathiasandClaude Sonnet 5.5 c9ebe526cc feat(domain): progress steps and the two timestamps that tell alive from progressing (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:50:39 +02:00
mathiasandClaude Sonnet 5.5 7bf5f6b8b7 test(progress): a tracker that shows the plan, times each step, throttles counts and heartbeats on its own (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:49:44 +02:00
mathiasandClaude Sonnet 5.5 7e9f5d7ca5 docs(adr): ADR-035 -- progress as real phases, the same transitions in the log, every judged spot kept (#42 #43 #44)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:49:06 +02:00
mathiasandClaude Sonnet 5.5 1a89eb3868 docs(adr-034): record the first blind check; bar ambiguity and the unanswered second condition
CI / Lint / Test / Vet (pull_request) Successful in 23s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:01:31 +02:00
mathiasandClaude Sonnet 5.5 2e9d76dad0 docs(research): first blind check of two-engine reconcile on a real recording (aggregates only)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:01:31 +02:00
mathiasandClaude Sonnet 5.5 7f8e3677b3 docs(adr-034): gemma4-31b slowness explained; reopen the 26B-vs-31B reconciler choice
CI / Lint / Test / Vet (pull_request) Successful in 22s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 07:13:10 +02:00
mathiasandClaude Sonnet 5.5 e3aa661cf2 docs(research): Low Power Mode confirmed as the cause of slow gemma4-31b (22.3 vs 7.9 tok/s decode)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 07:13:10 +02:00
mathiasandClaude Sonnet 5.5 7ae888a468 docs(claude): two-engine reconciliation, and that task check does not gate lint (#41)
CI / Lint / Test / Vet (pull_request) Successful in 21s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:47 +02:00
mathiasandClaude Sonnet 5.5 b9e0dfa1c9 docs(homelab): two-engine mode, its settings and why every model must be local
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:47 +02:00
mathiasandClaude Sonnet 5.5 8dbcb9a9eb docs(data-model): reconciliation column
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:47 +02:00
mathiasandClaude Sonnet 5.5 26cc00c1cc test(acceptance): map the two-engine scenarios to their covering tests
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:37 +02:00
mathiasandClaude Sonnet 5.5 36a80ac4f5 docs(use-cases): scenarios for two-engine reconciliation (ADR-034)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:22 +02:00
mathiasandClaude Sonnet 5.5 2497554683 docs(adr): ADR-034 records the first live smoke, including the wrong call
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:22 +02:00
mathiasandClaude Sonnet 5.5 c3c9b1e48a test(e2e): two engines + local reconciler, live
TAPIR_E2E_RECONCILE=1 runs pianissimo + kb-whisper + gemma4-26b on the audio and
asserts the reconciliation is recorded, every contributor is named in the endpoint, and
the page says what was done. Run 2026-10-01 on 82 s of synthetic Swedish: 3 disputes, all
3 changed; against the known script 2 were right (virket, a missing 'procent') and 1 was
WRONG (Maria -> Vi replaced a correct base reading). A smoke test, not a measurement.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:22 +02:00
mathiasandClaude Sonnet 5.5 ed8ca54674 feat(cmd): buildTranscriber -- the base-engine setting turns on the two-engine ensemble
Default is the single-engine chain exactly as before. With TAPIR_RECONCILE_BASE_MODEL set,
the base engine and the primary run on the same WAV and a LOCAL model (config has already
refused any non-local alias) reconciles them, at temperature 0 with a short reply -- the
conditions its accuracy was measured under. The cloud fallback is unchanged and runs
only when both local engines fail.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:26:02 +02:00
mathiasandClaude Sonnet 5.5 7338d9d603 test(e2e): discard the deferred Close error golangci-lint flags
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:26:02 +02:00
mathiasandClaude Sonnet 5.5 3d755fd57a fix(transcribe): check (discard) the deferred Close errors golangci-lint flags
Found because the project's lint step cannot fail: the Taskfile runs
'golangci-lint run ./... || echo "not installed -- skipping"', so a failing lint
prints that message and exits 0. CI's log showed these findings and still passed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:26:02 +02:00
mathiasandClaude Sonnet 5.5 b492a3144e test(cmd): the base-engine setting turns on the ensemble, and nothing else does (red)
Red: buildTranscriber is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:25:10 +02:00
mathiasandClaude Sonnet 5.5 505670c0e1 feat(llm): WithTemperature, default unchanged
Zero is sent as zero (the field has no omitempty). The reconciler's measured accuracy
was at temperature 0, so it runs there; every other caller keeps 0.2.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:25:10 +02:00
mathiasandClaude Sonnet 5.5 7891675393 test(llm): a temperature option; zero is sent as zero, not 'unset' (red)
The reconciler was measured at temperature 0 but the client hardcodes 0.2. Zero must
be sent explicitly, and the default must not change for existing callers.

Red: WithTemperature is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:54 +02:00
mathiasandClaude Sonnet 5.5 8110fe56ab style(config): gofumpt the new test rows
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:38 +02:00
mathiasandClaude Sonnet 5.5 61fb50ce99 feat(config): two-engine reconciliation settings with a local-only guard (ADR-034)
TAPIR_RECONCILE_BASE_MODEL turns it on and needs TAPIR_RECONCILE_MODEL; every model must
be a local alias (koala/ or iguana/, matched as a PREFIX): a cloud adjudicator is a
start-up error because private meeting content must not reach it; the two engines must
differ; default job timeout 60m. Off by default. Mutation-checked.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:34 +02:00
mathiasandClaude Sonnet 5.5 f25fdaf398 test(config): a cloud alias that merely contains a local host name is still cloud
Mutation-testing showed a substring match instead of a prefix match went unnoticed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:34 +02:00
mathiasandClaude Sonnet 5.5 ad071d44cb test(config): reconciliation settings -- off by default, local only, complete (red)
Off by default; a base engine needs an adjudicator; every model must be a local alias
(koala/ or iguana/): a cloud adjudicator is a start-up error because private meeting
content must not reach it; the base engine must differ from the other engine; default
job timeout 60m (two engines plus adjudication).

Red: the Config fields do not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:20 +02:00
mathiasandClaude Sonnet 5.5 59f2893434 docs(adr): ADR-034 names the settings for what they are
TAPIR_RECONCILE_BASE_MODEL, not 'second model': the engine added to turn the
feature on is the one whose TEXT is the base, and the old name hid that. All three
models must be local aliases; the base must differ from the other engine. ADR is
still Proposed, so this is an in-place correction.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:09 +02:00
mathiasandClaude Sonnet 5.5 6fb7a52980 feat(web): say what reconciliation did, and never claim a second opinion that was not there
CI / Lint / Test / Vet (pull_request) Successful in 21s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Reconciled: the two engines, the deciding model and the exact counts; the contested
spots (changed or uncertain) listed with a minute mark and both readings, the first
50 shown and the rest counted from the exact totals. Single-engine and
model-unavailable results say why. A nil reconciliation renders nothing. Readings are
escaped. Mutation-checked: the cap, the minute rounding and the 'more' count.

uploads.go and the templ pair are one commit: the helpers and the template need each other.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:23:39 +02:00
mathiasandClaude Sonnet 5.5 d6ba11d288 test(web): the page says what reconciliation did, plainly, and never claims more (red)
Reconciled: who the two engines and the deciding model were, and the counts
(differences, settled automatically, decided by the model, uncertain). The contested
spots are listed with a minute mark, both readings and which was used, uncertain ones
marked, readings escaped, and a very long list shortened with its true count. A
single-engine transcript makes no claim of a second opinion; one-engine and
model-unavailable cases say why.

Red: the page renders none of this yet.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:23:00 +02:00
mathiasandClaude Sonnet 5.5 ae4f9fd775 feat(store): persist and read back the reconciliation
CompleteUpload writes it (SQL NULL when there was none, never the JSON text 'null':
the page claims a second opinion only if the column is non-NULL); GetUpload reads it,
List does not. Mutation-checked.

domain.go and the store are one commit: the store tests need the new field.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:22:09 +02:00
mathiasandClaude Sonnet 5.5 e838f6bb6d feat(store): migration 019 -- uploaded_transcripts.reconciliation
JSONB, NULL for a single engine so the page cannot claim a second opinion. On the
already FORCE'd, RLS-scoped table, so the exact-set isolation test is unaffected.
Up and down are one commit: golang-migrate pairs them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:21:52 +02:00
mathiasandClaude Sonnet 5.5 5bd41077af test(store): the reconciliation survives the round trip, and its absence stays absent (red)
Counts and contested spots round-trip; a single-engine transcript has NO
reconciliation (nil, not empty -- the page must not claim a second opinion); a
skipped reconciliation keeps its reason.

Red: Transcription.Reconciliation is stored nowhere and UploadedTranscript has none.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:21:36 +02:00
mathiasandClaude Sonnet 5.5 34bc8dbdee feat(transcribe): Ensemble -- two local engines, a local reconciler, fail-soft at every step
Runs the fast engine then the other from one WAV and reconciles; one engine down
uses the other and says so; the model down gives the base text unreconciled; the
cloud fallback runs only when BOTH local engines fail and is recorded as such; a
cancelled job stops everything and never reaches the cloud. The endpoint lists every
contributor. Mutation-checked, including both cancellation points.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:21:17 +02:00
mathiasandClaude Sonnet 5.5 8778376aa6 test(transcribe): a shutdown in the middle of either engine stops the job
Mutation-testing showed each of the two cancellation checks was redundant with the
other in my first test. Now a cancel during the first engine must not start the second
or the cloud, and a cancel during the second must not be recorded as a single-engine
success ('kb-whisper failed').

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:21:17 +02:00
mathiasandClaude Sonnet 5.5 ad39b6154c test(transcribe): two engines, then a reconciler -- and every way it can fail softly (red)
Both run in order (the fast one first) and are reconciled; identical texts cost no
model call; one engine down uses the other and says so, with the cloud fallback
untouched; only when BOTH local engines fail does the fallback run, and the row says
berget served it; both failing with no fallback reports both errors; a cancelled
job never reaches the cloud; a model that is down yields the base text unreconciled,
not a failed job.

Red: transcribe.Ensemble is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:20:40 +02:00
mathiasandClaude Sonnet 5.5 f25acebe4c feat(reconcile): merge two transcripts with a model choosing between disagreements
CI / Lint / Test / Vet (pull_request) Successful in 20s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Base text stands except where the model picks the alternative's reading; trivial
disputes skip the model; the reading order alternates; the base's context goes to
the model. Fails soft: three consecutive model errors return the base text
unreconciled (none of the partial changes) with the reason; a verdict returned with
an error is ignored; a cancelled job is an error, not 'skipped'. Counts are exact,
the stored list of contested spots is capped at 200 and carries the second each
starts at. Mutation-checked on nine behaviours.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:20:12 +02:00
mathiasandClaude Sonnet 5.5 7f9e967ad0 test(reconcile): a verdict returned with an error is not used
Mutation-testing showed the forced-unsure line was untested because my fake
always returned unsure with its errors.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:20:12 +02:00
mathiasandClaude Sonnet 5.5 0fead2e102 feat(domain): Reconciliation -- how a merged transcript came about (ADR-034)
Status (reconciled / single_engine / skipped), exact counts, and a bounded list of
contested spots (changed or unsure) with a position and both readings.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:19:54 +02:00
mathiasandClaude Sonnet 5.5 a1e16fa658 test(reconcile): fixtures need two shared words between disputes
Disputes separated by a single shared word are merged into one decision by design,
so several of my fixtures were secretly one dispute. Corrected before any
implementation existed; the rule is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:19:22 +02:00
mathiasandClaude Sonnet 5.5 b20558ffa6 test(reconcile): merging two transcripts, and failing soft (red)
Identical texts and trivial disputes cost no model call; an alt verdict takes the
alternative's own words, base/unsure keep the base and only unsure is flagged;
insertions and deletions land in the right place; the reading order alternates; the
model gets the base's context. Three consecutive model errors stop the whole
reconciliation (base text, none of the partial changes, reason recorded) while a
success in between resets the breaker; a cancelled job is an error, not 'skipped'.
Spots carry the second they start at; the stored list is capped, the counts are not.

Red: Reconcile, Options, MaxContested and domain.Reconcile* are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:19:00 +02:00
mathiasandClaude Sonnet 5.5 5e54c9beff feat(reconcile): LLM adjudicator with the measured prompt, plus an order-checked wrapper
Sends the exact wording that scored 92% (gemma4-26b) and 97% (gemma4-31b); maps
1/2 back to base/alt under both orders; an unreadable reply is unsure, a transport
error is an error. OrderChecked asks twice with the order swapped and keeps only
agreement. Mutation-checked: answer mapping, unparseable-as-guess, the word
boundary and the order check each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:18:11 +02:00
mathiasandClaude Sonnet 5.5 2bfe84ee64 test(reconcile): a number or word that merely starts with 1 or 2 is not an answer
Mutation-testing the adjudicator showed dropping the word boundary went unnoticed:
'ANSWER: 12' would be read as 1.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:18:11 +02:00
mathiasandClaude Sonnet 5.5 481321a75c test(reconcile): the adjudicator sends the measured prompt and maps answers back (red)
Readings in the requested order; 1/2 mapped to base/alt under both orders;
UNSURE, an unparseable or out-of-range reply is unsure rather than a guess; an
empty reading is described, not blank; a model error is an error, distinguishable
from 'unsure'. The optional order-checked wrapper asks twice with the order swapped
and keeps only agreement.

Red: Request, Verdict, NewLLMAdjudicator and NewOrderChecked are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:17:42 +02:00
mathiasandClaude Sonnet 5.5 2600e6c45a feat(reconcile): find disputes, resolve trivial ones without a model, give context
Disputes carry both readings with their own spelling; differences separated by
one shared word merge into one decision. Compound splits/joins and one or two
filler particles are resolved without a model; 'nej', numbers and content words
never are. Mutation-checked (merge distance, filler limit, 'nej', compound rule).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:17:21 +02:00
mathiasandClaude Sonnet 5.5 4e43327792 test(reconcile): finding disputes, resolving the trivial ones, and their context (red)
A dispute carries both readings with their own spelling; differences separated by
a single shared word are one decision, two shared words keep them apart; compound
splits/joins and a short run of filler words are resolved without a model, while
numbers, content words, and 'nej' (an answer) are not.

Red: FindDisputes, AutoResolve and ContextAround are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:16:55 +02:00
mathiasandClaude Sonnet 5.5 92a39da45c feat(reconcile): minimal word-level diff (Myers) with a bounded edit distance
Prefix/suffix trimmed first; adjacent deletes and inserts read as one Replace;
beyond 3,000 edits (two texts sharing almost nothing) the middle is one Replace
instead of gigabytes of trace. Mutation-checked: a wrong tie-break, missing
replace coalescing, wrong suffix accounting and a punctuation-bearing key each
turn tests red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:16:31 +02:00
mathiasandClaude Sonnet 5.5 aa5607b002 feat(reconcile): tokenise for word-level comparison
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:16:31 +02:00
mathiasandClaude Sonnet 5.5 984d92e3a9 test(reconcile): word-level diff is minimal, tiles both inputs and is fast (red)
A random-input property test compares the equal runs against a dynamic-programming
LCS: a diff that is valid but not minimal would turn real agreement into disputes
the model is then asked about. A 6,500-word, 380-edit case mirrors a real hour.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:15:44 +02:00
mathiasandClaude Sonnet 5.5 00fd884c6c test(reconcile): tokenising for word-level comparison (red)
Keeps the word as written, keys it without case/punctuation/hyphens/plus, joins a
lone dash or full stop to its neighbour so it is not a word the engines can
disagree about. Red: package reconcile does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:15:44 +02:00
mathiasandClaude Sonnet 5.5 de9871c9c3 docs(adr): ADR-034 transcribe with two engines, reconcile with a local model
CI / Lint / Test / Vet (pull_request) Successful in 29s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Proposed, decider pending. Two local engines on one WAV; word-level diff with
pianissimo as the base (readability and timing, NOT accuracy -- that was a tie);
trivial disputes resolved without a model; the rest adjudicated by a LOCAL model
(alias must start koala/ or iguana/) whose only freedom is choosing between the
two readings. Reconciliation never fails a job; the page says what was done and
lists the uncertain spots. Off by default. Validation: re-upload the real
recording and keep the feature only if the reconciled reading is right >= 85% of
the time and no worse than the better single engine.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:15:12 +02:00
mathiasandClaude Sonnet 5.5 e182a6b07d docs(research): can a local model adjudicate two ASR transcripts?
Eleven local models, 120 calls each, scored against the speaker's blind answers
on 30 disagreements, with pass criteria fixed before the run. gemma4-31b (97%)
and gemma4-26b (92%) pass; nothing else clears accuracy and order-consistency.
Records why 31b is slow (MLX gives the same speed; Low Power Mode is the
unproven suspect) and the caveats: 30 items, one recording, thinking models
under-budgeted. Aggregate numbers only; this repo is public.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:14:27 +02:00
mathiasandClaude Sonnet 5.5 ebc6cc3ff8 docs(research): kb-whisper vs pianissimo on one real hour of Swedish speech
CI / Lint / Test / Vet (pull_request) Successful in 22s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
A near tie on accuracy (blind A/B by the speaker: 17 vs 13 on the 30 spans that
change meaning) and a 6x difference in speed (40 s vs 4m08s for 63.7 min).
Different error profiles: pianissimo better on terms and numbers, kb-whisper on
everyday words. Aggregate numbers only -- no transcript text or domain terms,
this repo is public. Also records the mid-word wrapping bug the comparison found
and its fix.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 19:45:56 +02:00
mathiasandClaude Sonnet 5.5 d11afb5ff1 fix(transcribe): rebuild the transcript from segments so split words are whole
CI / Lint / Test / Vet (pull_request) Successful in 22s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
whisper.cpp wraps at ~60 chars and can wrap inside a word; the response's text
field joins segments with newlines, so chat saw 'nyt' / 't godkännande'. Text
is now rebuilt from the raw segments: concatenated when any segment carries a
leading space (whisper.cpp), joined by a space when none do (berget's sentence
level), whitespace collapsed; the text field is only a fallback.

Checked against the first real one-hour transcript with pianissimo's vocabulary
as an independent yardstick: 36 spans changed, 31 now consist entirely of words
pianissimo also produced, words unknown to pianissimo fell 224 -> 177 and word
agreement rose 92.4% -> 93.1%. Five rejoined spans still contain an unknown
word (names/loanwords or an unlucky glue) -- a residual risk, small.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 19:35:14 +02:00
mathiasandClaude Sonnet 5.5 500e11962b test(transcribe): rebuild Text from segments so a split word is not broken (red)
Found by comparing the first real one-hour transcript against pianissimo:
whisper.cpp wraps at ~60 chars and can wrap INSIDE a word. The continuation
segment has no leading space, the text field joins segments with newlines, and
a segment can be a lone '.'. Measured 2026-10-01: 7% of segments (96/1332) in a
synthetic hour, 42/1302 lines in the real one. My earlier tests used short
sentences and could not see it. berget's segments are whole sentences with no
leading space, so the two backends need different joins.

Red: Text still joins by newline.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 19:34:54 +02:00
mathiasandClaude Sonnet 5.5 206fc2f5ca feat(web): log upload rejections by reason only
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 19:28:10 +02:00
mathiasandClaude Sonnet 5.5 5368d5f8c2 test(web): a rejected upload is logged with its reason, never the filename or title (red)
Rejections are synchronous and leave no row, so an operator could not tell 'the
user never retried' from 'the page refused it' (it happened during the first
real-recording trial). The reason is operational; the filename and title are
the user's private content.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 19:28:06 +02:00
mathiasandClaude Sonnet 5.5 365dff5c85 docs(data-model): had_video
CI / Lint / Test / Vet (pull_request) Successful in 26s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:55 +02:00
mathiasandClaude Sonnet 5.5 adaaa9918f test(acceptance): map the video-track scenarios to their covering tests
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:55 +02:00
mathiasandClaude Sonnet 5.5 364a28eecd docs(use-cases): a video track is accepted and the user is told; no-audio is rejected
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:55 +02:00
mathiasandClaude Sonnet 5.5 1fa7ca3a8c test(e2e): a video file (camera-off Teams meeting) is transcribed and the user is told
Run 2026-10-01 with 82 s of synthetic Swedish muxed with black video (H.264 +
AAC .mp4, built with the real ffmpeg): accepted, had_video=true, transcribed by
iguana/kb-whisper, the page said only its audio was used, and chat answered.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:48 +02:00
mathiasandClaude Sonnet 5.5 cca0a7461b feat(web): tell the user when a video track was ignored, in every state
A Teams meeting with the camera off is still a video file. It is accepted, only
its audio is transcribed, and the page says so in queued, transcribing, done and
failed -- not just at the end.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:14 +02:00
mathiasandClaude Sonnet 5.5 5175496809 test(web): match the notice's real casing
My assertion expected a lowercase 'only'; the sentence begins 'Only'. The user
facing wording is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:14 +02:00
mathiasandClaude Sonnet 5.5 f102199097 feat(uploads): Intake passes whether the file had a video track to the store
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:00 +02:00
mathiasandClaude Sonnet 5.5 01eab1ab6f feat(uploads): a video track is no longer a reason to reject (ADR-033)
Only no-audio, unreadable length and too-long are refused. ReasonVideo is gone.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:00 +02:00
mathiasandClaude Sonnet 5.5 85cf89c86c feat(store): CreateUpload records had_video; Get/List return it
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:50 +02:00
mathiasandClaude Sonnet 5.5 603d02e504 feat(domain): UploadedTranscript.HadVideo
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:40 +02:00
mathiasandClaude Sonnet 5.5 3c4d2880f3 feat(store): migration 018 -- uploaded_transcripts.had_video
Up and down are one commit: golang-migrate pairs them. RLS is unchanged: the
column is on the already-FORCE'd table, so the exact-set isolation test still
holds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:33 +02:00
mathiasandClaude Sonnet 5.5 1157854441 test(cmd): fake store follows the CreateUpload signature (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:33 +02:00
mathiasandClaude Sonnet 5.5 7aec2538be test(web): a video file is accepted and the user is told the video was ignored (red)
Video with no audio is rejected for having none. A file with a video track is
accepted and marked had_video; every state of its page (queued, transcribing,
done, failed) says only its audio was used and the video was discarded, and a
plain audio file shows no such notice.

Red: the had_video column does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:33 +02:00
mathiasandClaude Sonnet 5.5 eb55c7d9d1 test(store): had_video is recorded on creation and survives done/failed (red)
Red: CreateUpload takes no hadVideo and the row has no HadVideo.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:13 +02:00
mathiasandClaude Sonnet 5.5 ab62cdaf1c test(uploads): Accept records whether the file had a video track (red)
Red: Store.CreateUpload does not take hadVideo yet.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:13 +02:00
mathiasandClaude Sonnet 5.5 6cbb7b2df0 test(uploads): a recording with a video track is accepted (red)
A Teams meeting with the camera off is still an .mp4 with a video track.
Video with no audio is now rejected as having no audio.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:04 +02:00
mathiasandClaude Sonnet 5.5 1e192e53ef docs(adr): ADR-033 accepts files with a video track, uses the audio, says so
The first draft rejected video and named #28. The maintainer's first real
recording -- a Teams meeting with the camera off -- is still an .mp4 with a
video track, so that rejected the primary use case. Decision 1 and 4 now:
accept, extract audio only, discard the video, record had_video, and tell the
user. A file with no audio stream is still rejected. Video features stay #28.
In-place amendment: the ADR is still Proposed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:41:49 +02:00
mathiasandClaude Sonnet 5.5 6ebb4e49e9 test(audio): opt-in live ffprobe check on hostile playlists and real audio
CI / Lint / Test / Vet (pull_request) Successful in 40s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Skipped unless TAPIR_LIVE_FFPROBE is set. Synthetic/crafted files only.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:14:29 +02:00
mathiasandClaude Sonnet 5.5 291a360a83 feat(audio): restrict ffmpeg/ffprobe input to local files and audio demuxers
The upload is attacker-supplied and its CONTENT can steer ffmpeg to open other
files or URLs (HLS/concat playlists). Pins -protocol_whitelist file,pipe and
-format_whitelist to the containers real recordings use, before the input
(after -i it would apply to the output).

Verified with the real static ffmpeg 7.0.2 on koala: real AAC .m4a and WAV
still probe/convert; a crafted concat file is rejected with 'Format not on
whitelist'. Note ffmpeg 7.0.2 ALREADY refused the playlists tried without the
whitelist, so the live check cannot prove the whitelist -- the argv unit tests
do; this protects against a different ffmpeg version being put in the image.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:14:29 +02:00
mathiasandClaude Sonnet 5.5 4a0c51d877 test(audio): attacker-supplied input must not steer ffmpeg to other files or URLs (red)
A file's CONTENT can tell ffmpeg to open other paths or URLs (HLS, concat
playlists). ffmpeg 7.0.2 refuses the ones tried, but that is version luck and
the image's ffmpeg is still to be chosen (#34). Pin protocol_whitelist to
file,pipe and format_whitelist to the demuxers real audio uses, before the input.

Red: audio.AllowedDemuxers is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:13:54 +02:00
mathiasandClaude Sonnet 5.5 1c726a810e docs(homelab): upload chat budget is 90000 and the chat model is gemma4-26b
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:08:33 +02:00
mathiasandClaude Sonnet 5.5 77fe6954e5 docs(adr): ADR-033 decision 8 -- chat budget and model from measurement
The proposed 120,000-char budget was a guess; measured, gemma4-26b silently
loses the head at that size and phi4-mini (first in the offered list) 400s on
an hour of speech. Budget is now 90,000, uploads open on gemma4-26b, the
retrieval trigger is concrete, and the iguana contention hazard is recorded.
ADR-033 is still Proposed, so this is an in-place correction, not a supersession.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:08:28 +02:00
mathiasandClaude Sonnet 5.5 4d3e2a471a docs(research): chat over a long transcript -- what the offered models really hold
phi4-mini 400s at ~13k tokens; gemma4-26b is correct to 90,000 chars and
silently loses the HEAD at 120,000 (no error, no truncation flag). The
end-needle test alone passed at 120k -- only a head-needle exposed it.
Records the default-model trap and the iguana contention hazard.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:08:13 +02:00
mathiasandClaude Sonnet 5.5 548bfaee85 test(e2e): use the same wireUploads as serve
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:41 +02:00
mathiasandClaude Sonnet 5.5 ad2e3e0e8e fix(cmd): serve now actually sets the upload chat budget and model
cmdServe assigned Uploads and UploadLimits but not UploadChatMaxChars, so chat
over a long recording would have used the 18,000-char caption default. It now
calls wireUploads, which is tested.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:41 +02:00
mathiasandClaude Sonnet 5.5 e6738bcd3e feat(cmd): wireUploads -- limits, chat budget and chat model into the web App
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:40 +02:00
mathiasandClaude Sonnet 5.5 3bb4e30f4c test(cmd): wireUploads sets the budget and model, and never a typed-nil Intake (red)
I wired Uploads and UploadLimits in cmdServe but not UploadChatMaxChars: chat
over a long recording would have silently used the 18,000-char caption
default. A helper that main and the e2e test both call makes that a tested fact.

Red: wireUploads is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:33 +02:00
mathiasandClaude Sonnet 5.5 6c39ab86cf feat(web): chat over an upload opens with App.UploadChatModel
The Chatter's own default is the summarizer's (koala/phi4-mini, 8,192-token
context), which answers HTTP 400 on an hour of speech. Empty keeps the old
behaviour.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:33 +02:00
mathiasandClaude Sonnet 5.5 4b6ac2a213 test(web): chat over an upload opens with the upload chat model (red)
Red: App.UploadChatModel is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:20 +02:00
mathiasandClaude Sonnet 5.5 af66419a44 feat(config): measured chat budget (90,000) and upload chat model (gemma4-26b)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:20 +02:00
mathiasandClaude Sonnet 5.5 4b09dc8f36 test(config): chat defaults for uploads come from measurement, not a guess (red)
iguana/gemma4-26b answered correctly at 90,000 chars but SILENTLY dropped the
head of a 120,000-char transcript (prompt tokens fell to 16k; 'not in the
text'); koala/phi4-mini has an 8,192-token context and answers HTTP 400 at
~13k tokens. So the budget is 90,000 and the upload chat model defaults to
gemma4-26b. Red: UploadChatModel is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:09 +02:00
mathiasandClaude Sonnet 5.5 8f16ab671a docs(claude): audio upload supersedes the STT deferral for user-supplied audio only
The 'no audio-download + STT' bullet is narrowed to YouTube (ADR-007 stands);
adds the upload packages, the exact-set isolation test and the pending ffmpeg
decision to the build state.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:02:11 +02:00
mathiasandClaude Sonnet 5.5 39aeda7c69 docs(homelab): speech-to-text endpoints, the ingress 413, and the upload knobs
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:02:05 +02:00
mathiasandClaude Sonnet 5.5 0fc25a24ed docs(data-model): uploaded_transcripts, and fix the stale claim that transcripts is RLS'd
Adds the private uploaded_transcripts entity and the exact-set isolation rule
(ADR-033). Also corrects two lists that still named `transcripts` as an RLS'd
table: it has been shared non-RLS since ADR-021 / migration 015, and these
lists sit next to the very boundary that matters.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:02:05 +02:00
mathiasandClaude Sonnet 5.5 07375f75de test(acceptance): map audio_upload.feature scenarios to their covering tests
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:01:38 +02:00
mathiasandClaude Sonnet 5.5 f9d6df9f84 docs(use-cases): audio_upload.feature -- scenarios for #33 (ADR-033)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:01:20 +02:00
mathiasandClaude Sonnet 5.5 3464cdf876 test(e2e): opt-in live upload -> transcript -> grounded chat check
Real ffprobe/ffmpeg, the live gateway and a real chat model, a private
embedded Postgres. Skipped unless TAPIR_E2E_AUDIO is set. Run 2026-10-01 with
82 s of synthetic Swedish as AAC .m4a:
- primary: iguana/kb-whisper, 82.6 s audio, 1,186 chars; asked who owns the
  timber order, the answer named Anders and the 15 November deadline from the
  transcript; the spool directory was empty afterwards.
- primary forced to fail (TAPIR_TRANSCRIBE_MODEL=iguana/does-not-exist): the
  fallback served it and the row recorded endpoint=berget/kb-whisper.
Synthetic audio only; this repo is public.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:01:06 +02:00
mathiasandClaude Sonnet 5.5 4fdb212cef feat(web): chat with an uploaded transcript, reusing the ADR-027 path (ADR-033)
GET/POST /uploads/{id}/chat, mounted only when both an Uploader and a Chatter
are wired. The row comes from the user-scoped private store, so another user's
upload is a 404 before any model is involved. The owner's transcript goes to
the SAME Chatter with App.UploadChatMaxChars instead of the 18,000-char caption
default. The chat panel is shared, not forked: chatView gains PostURL/Subject
(empty = a YouTube video, so existing callers are unchanged). The done card
links to chat only when chat is wired.

Mutation-checked: dropping the upload budget, ignoring the done-status guard,
or scoping the lookup to the wrong user each turn tests red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:58:25 +02:00
mathiasandClaude Sonnet 5.5 a39f58b730 test(web): match the HTML-escaped tail of the inline chat error
templ escapes the apostrophe in "couldn't", so the assertion could never match
the rendered page. A test defect, not a loosening: the failure must still be
shown inline with the conversation kept.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:58:25 +02:00
mathiasandClaude Sonnet 5.5 10ca45fc0a test(web): chat over an uploaded transcript reuses the ADR-027 path (red)
Owner sees the panel posting to /uploads/{id}/chat; an unfinished upload
redirects to its status; ANOTHER USER'S upload is 404 on GET and POST and no
model is ever called with it; the owner's transcript is sent with the upload
budget (not the 18,000-char caption default); an unoffered model falls back; a
model failure is inline and keeps the conversation; routes are absent without a
chat backend; the done card links to chat only when chat is wired.

Red: App.UploadChatMaxChars is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:57:47 +02:00
mathiasandClaude Sonnet 5.5 c72d366935 feat(chat): per-request transcript budget so a long recording is not cut to its head (ADR-033)
chat.Request.MaxChars overrides the Service cap when positive. The 18,000-char
default (~22 min of speech) suits captions; an uploaded hour is ~50,000 chars
and truncation keeps only the head. This is reuse of the existing chat path,
not a fork.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:57:01 +02:00
mathiasandClaude Sonnet 5.5 ec29c5e761 style(chat): gofumpt the new test
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:57:01 +02:00
mathiasandClaude Sonnet 5.5 867afefb08 test(chat): a request may carry its own transcript budget (red)
The 18,000-char default is ~22 min of speech and keeps only the head; an
uploaded hour is ~50,000 chars. Red: Request.MaxChars is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:56:51 +02:00
mathiasandClaude Sonnet 5.5 1c5d85f5c6 feat(cmd): serve mounts audio upload when enabled (ADR-033)
Wires buildUploads into cmdServe: Intake and limits into the web App, the
worker on the server's ctx. Logs the two facts an operator must not miss: the
queue is process-local (single replica), and a configured fallback is a cloud
endpoint that private audio can leave the homelab through.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:56:22 +02:00
mathiasandClaude Sonnet 5.5 9b3175db09 feat(cmd): buildUploads -- spool, sweep, fail interrupted rows, wire the chain (ADR-033)
Returns nils when disabled (never a typed-nil Intake). When enabled: private
0700 spool dir, delete audio left by a crash, fail rows a restart left queued
or transcribing (the audio is not retained, so they cannot resume), then build
the iguana->berget chain, worker and intake. An empty
TAPIR_TRANSCRIBE_FALLBACK_MODEL keeps private audio on the homelab. Enabling
with no gateway fails at start-up, not at the first upload.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:56:21 +02:00
mathiasandClaude Sonnet 5.5 eb9357a5de test(cmd): start-up removes spooled audio left by a crash (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:56:11 +02:00
mathiasandClaude Sonnet 5.5 3e802adade test(cmd): buildUploads wiring -- off by default, sweep first, private spool (red)
Red: buildUploads is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:55:55 +02:00
mathiasandClaude Sonnet 5.5 b8798dbd5d feat(config): upload and transcription settings (ADR-033)
Embedded UploadConfig. Off by default until the image carries ffmpeg; 200 MB /
90 min; iguana primary with a berget fallback that an explicit empty
TAPIR_TRANSCRIBE_FALLBACK_MODEL disables (private audio then never leaves the
homelab); zero or negative limits fail at load instead of rejecting every
upload at runtime.

uploads.go and the two-line hook in config.go are one commit: the tests need both.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:55:32 +02:00
mathiasandClaude Sonnet 5.5 fb521f2b89 test(config): upload and transcription settings (red)
Off by default until the image has ffmpeg; 200 MB / 90 min (the longest length
measured); iguana primary, berget fallback that an explicit empty value
disables; zero or negative limits are refused rather than silently rejecting
every upload.

Red: the Config fields do not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:55:11 +02:00
mathiasandClaude Sonnet 5.5 a8dadb8ca4 feat(uploads): in-process transcription worker, one job at a time (ADR-033)
Queue -> StartTranscribing -> ffmpeg to 16 kHz WAV -> gateway chain ->
CompleteUpload, with a per-job timeout so a hung endpoint ends in failed, not
a spinner. The upload and the WAV are deleted on every path (audio is never
retained). A shutdown mid-job is not a failure: the row is left for the
start-up sweep. Cancellation wins over a non-empty queue, so queued jobs are
never started on shutdown; stop() deletes their audio and refuses new jobs.

Concurrency is 1 on purpose: two simultaneous long jobs on iguana were never
measured, and a Metal OOM there latches whisper (infra#273). The queue is
process-local, so ADR-018's single-replica assumption now has a second
dependent.

Mutation-checked, -race clean: cleanup, shutdown-vs-failure, timeout,
concurrency and stop-cleanup each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:54:52 +02:00
mathiasandClaude Sonnet 5.5 3cfc95640e test(uploads): a queued job at shutdown must never be started; timeout test fails fast
Mutation-testing the worker showed deleting stop()'s cleanup went unnoticed:
Process was incidentally deleting the audio of jobs it should never have
started. The stop test now asserts only the running job was converted (red on
the old code). The timeout test no longer hangs for the 10-minute go test
default when the timeout is removed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:54:52 +02:00
mathiasandClaude Sonnet 5.5 52417e6b47 test(uploads): in-process transcription worker (red)
Success path deletes both the upload and the WAV; failures are typed and still
delete the audio; a shutdown mid-job leaves the row for the start-up sweep
(it is not a transcription failure); a hung endpoint is failed by the job
timeout, never an eternal spinner; strictly one job at a time (two long jobs on
iguana were never measured); stopping deletes queued audio and refuses new jobs.

Red: uploads.Worker is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:47:22 +02:00
mathiasandClaude Sonnet 5.5 4352446f36 feat(audio): ffmpeg adapter -- always transcode to 16 kHz mono PCM (ADR-033)
Replaces whisper-server --convert. A zero exit that wrote nothing is an error.
Also fixes the red test's stub, which wrote an empty file and so (correctly)
tripped the no-output check.

Checked against real ffmpeg on iguana with the same argv: AAC .m4a in,
pcm_s16le 16000 Hz mono WAV out, duration preserved.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:46:34 +02:00
mathiasandClaude Sonnet 5.5 3bc33fc34c test(audio): ffmpeg transcode to 16 kHz mono PCM, always (red)
Red: audio.FFmpeg is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:46:14 +02:00
mathiasandClaude Sonnet 5.5 2227cfff7e test(transcribe): opt-in live check against the real gateway
The httptest fake proves only what we THINK the gateway accepts. Skipped
unless TAPIR_LIVE_WAV and TAPIR_LIVE_GATEWAY_URL are set. Run once on
2026-10-01 with 11 min of synthetic Swedish (21 MB, above the 8-10 MB ingress
cap, via the NodePort): both iguana/kb-whisper (52.8 s) and berget/kb-whisper
(35.4 s) accepted the streamed, computed-Content-Length body and returned
usable verbose_json. Synthetic audio only; this repo is public.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:46:14 +02:00
mathiasandClaude Sonnet 5.5 cae423750c feat(transcribe): gateway client with an iguana->berget fallback chain (ADR-033)
Streams the WAV from disk with a computed Content-Length (no buffering, no
chunking), posts verbose_json with field `file` and language=sv, clamps the
final segment to the audio duration, and treats an empty transcript as a
failure. The chain falls back on ANY primary error, records the serving
endpoint, and never falls back once the context is cancelled.

gateway.go and chain.go are one commit: the chain's tests use the gateway's
StatusError and ErrEmptyTranscript, so neither compiles alone.

Mutation-checked: field name, clamp, cancel-fallback, content length and the
recorded endpoint each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:44:34 +02:00
mathiasandClaude Sonnet 5.5 bb285d8757 test(transcribe): gateway client and iguana->berget chain (red)
Gateway: field is `file`, no Expect header, a computed Content-Length (not
chunked, not buffered -- a 90 min WAV is ~170 MB), verbose_json normalised
with the last segment clamped to the audio duration, errors independent of
the body, an empty transcript is a failure. Chain: ANY primary failure falls
back and the row records the fallback endpoint; a cancelled job never falls
back to the cloud.

Red: package transcribe does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:43:51 +02:00
mathiasandClaude Sonnet 5.5 e12a5e7e89 feat(web): upload routes -- form, multipart intake, honest status (ADR-033)
GET/POST /uploads, GET /uploads/{id}, GET /uploads/{id}/status, mounted only
when an Uploader is wired. The multipart body is streamed to Intake, never
buffered. Rejections are synchronous: 422 (video, too long, unreadable) or 413
with the user-facing message and no row. Another user's id is a 404 that does
not leak the title. Unfinished states poll; done/failed render without
hx-trigger so nothing polls forever (ADR-025). A done transcript names the
endpoint that served it.

handlers.go, uploads.go and the templ pair are one commit: the App fields and
Store methods are what make the handlers compile.

Mutation-checked: dropping the 413 mapping, the not-found mapping, or letting
a done card poll each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:42:58 +02:00
mathiasandClaude Sonnet 5.5 1bc98813fc test(web): read the response body once; match the message's real case
Two defects in my red tests, found when going green: body() drains the
recorder so a second call read empty, and two assertions expected a different
case than the user-facing message. The assertions were not weakened.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:42:40 +02:00
mathiasandClaude Sonnet 5.5 e1ed6ea6f4 test(web): upload routes -- form, multipart POST, rejections, owner-only, honest status (red)
Real Router + real Intake + real Postgres; only ffprobe and the worker are
faked. Covers: the form states the limits up front; accepted audio queues a
row and redirects; video / too-long / too-large are rejected synchronously
(422/422/413) with a message and NO row; a non-owner gets 404 without the
title leaking; unfinished states keep polling and terminal states stop (no
stuck spinner); a done transcript names the endpoint that served it.

Red: App.Uploads / App.UploadLimits are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:41:42 +02:00
mathiasandClaude Sonnet 5.5 f84389028d feat(audio): ffprobe adapter implementing uploads.Prober (ADR-033)
Cover art is not video; non-zero exit is the file's fault (ErrUnreadable),
a missing binary / cancelled ctx / bad output is ours. WaitDelay bounds a
cancelled probe whose child holds the pipe open.

Checked against real ffprobe on iguana: JSON shape matches the fixtures and
/etc/hosts exits 1 with 'Invalid data found when processing input'. The unit
tests drive a stub process; ffprobe is not installed on koala.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:40:51 +02:00
mathiasandClaude Sonnet 5.5 0ab067c510 test(audio): ffprobe adapter -- cover art is not video, exit code vs missing binary (red)
Red: package audio does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:40:29 +02:00
mathiasandClaude Sonnet 5.5 1910d3dfb5 feat(uploads): Intake -- spool, probe, validate, queue; clean up on every failure (ADR-033)
Bounded read (cap+1 byte tells 'at the limit' from 'over'); a file that fails
validation creates no row; a prober fault is our error, not the user's bad
file; a row whose job cannot be submitted is failed, never left queued.

Mutation-checked: cap off-by-one, missing cleanup and deleting the handed-over
file each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:40:13 +02:00
mathiasandClaude Sonnet 5.5 b98e9df462 test(uploads): Accept -- size cap, probe verdicts, cleanup on every failure path (red)
Red: uploads.Job, Intake and ErrUnreadable are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:40:03 +02:00
mathiasandClaude Sonnet 5.5 e3e1f17c40 feat(uploads): validation policy for probed audio uploads (ADR-033)
Video is checked first so it is always refused as video; limits and reasons
are typed so callers and tests branch on Reason, not message text.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:39:51 +02:00
mathiasandClaude Sonnet 5.5 f42b8e23fc test(uploads): validation policy -- video, no-audio, too-long, unreadable (red)
Probe-based, not extension-based: a Teams voice-only export is typically an
.mp4 with no video stream, indistinguishable from video by name (ADR-033).
Red: package uploads does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:39:20 +02:00
mathiasandClaude Sonnet 5.5 836a501576 feat(store): Store API for private uploaded transcripts (ADR-033)
Create/Start/Complete/Fail/Get/List plus the start-up sweep that fails
unfinished rows as interrupted (audio is not retained, so no resume). Every
call goes through withUser AND names user_id in its predicate, so isolation
does not rest on RLS alone and stays testable under the superuser test
connection; a non-owner gets ErrUploadNotFound, never a distinct forbidden.
List drops transcript text.

Mutation-checked: removing the user predicate from Get or from the UPDATEs
turns the cross-user tests red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:38:25 +02:00
mathiasandClaude Sonnet 5.5 c29c0a7127 feat(domain): types for private uploaded transcripts (ADR-033)
Segment, Transcription, UploadStatus, UploadErrorKind and UploadedTranscript.
Error kinds are typed, never a provider's string. UploadedTranscript is a
distinct type from the shared public-caption Transcript on purpose.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:37:39 +02:00
mathiasandClaude Sonnet 5.5 4b557209c8 test(store): Store API for uploaded transcripts (red)
Lifecycle queued -> transcribing -> done|failed, per-row serving endpoint,
segments JSONB round trip, owner-only get/write (not-found, not forbidden),
list without transcript text, and the start-up sweep that fails unfinished
rows as `interrupted` for every user (audio is not retained, so no resume).

Red: CreateUpload and friends are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:37:30 +02:00
mathiasandClaude Sonnet 5.5 e36c4c1e0c feat(store): migration 017 -- private RLS-scoped uploaded_transcripts (ADR-033)
Separate table, not a user_id on the shared transcripts (ADR-021). ENABLE +
FORCE ROW LEVEL SECURITY with the migration-003 policy, user_id NOT NULL
ON DELETE CASCADE, typed error_kind, per-row serving endpoint, JSONB segments.
CHECK constraints: done => content, failed => error_kind.

Up and down are one commit: golang-migrate pairs them.

Mutation-checked: dropping FORCE, the policy, or the cascade each turn a test
red. FORCE is only observable structurally -- the app role is not the table
owner, so no behavioural test can see it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:36:50 +02:00
mathiasandClaude Sonnet 5.5 066ae6966a test(store): isolation tests for the private uploaded_transcripts table (red)
Drives RLS directly as the non-superuser app role: deny-all with no scope,
own-rows-only scoped reads, zero-row cross-user UPDATE/DELETE, rejected
cross-user INSERT, FORCE + NOT NULL user_id, cascade on DeleteUser. Also pins
the EXACT set of public tables with row security off, so a new table added
without RLS fails CI rather than passing it (ADR-033). The existing test only
named transcripts; #33's "the only non-RLS table" is false -- invitations,
user_identities and schema_migrations are off too.

Red: relation "uploaded_transcripts" does not exist. Set test passes today
and guards the migration that follows.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:36:22 +02:00
mathiasandClaude Sonnet 5.5 38b3f1cf33 docs(adr): ADR-033 upload audio -> private Swedish transcript -> chat (#33)
Proposed, decider pending. Supersedes ADR-007's STT deferral for
user-supplied audio only; the YouTube audio-download rejection stands.
Settles #28 decision 1 as a separate RLS-scoped uploaded_transcripts table
and records the measured length ceiling, the ingress 413, and the 18k-char
chat truncation that #33 assumed did not exist. Two DECISION NEEDED items
(ffmpeg in the runtime image; berget fallback for private audio).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:35:04 +02:00
mathiasandClaude Sonnet 5.5 feb108def3 docs(research): measure the audio length ceiling before designing chunking (#33)
No transcription ceiling found up to 90 min: one 174 MB request through the
LiteLLM NodePort transcribed in 440 s. The first real failure is the public
gateway ingress rejecting bodies over ~8-10 MB with 413, not whisper or a
Metal OOM. Also records that srt arrives wrapped in a JSON envelope, that the
two backends differ in timestamp granularity, and that the existing chat path
truncates at 18k chars (~22 min of speech).

Synthetic Swedish TTS fixture only; no real transcript.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:33:34 +02:00
78 changed files with 303 additions and 3694 deletions
-3
View File
@@ -127,9 +127,6 @@ The repo is **green and shipping** — last tag `v0.15.0`. `task check` passes (
implement `uploads.Planner` to declare its own phases. The page tells running / quiet / lost from
two timestamps (`alive_at` heartbeat vs `progress_at`) and never shows a percentage that does not
come from real counts. **Nothing logged may carry transcript text, words from it, or the spool path.**
Errors reach a log, a row or a page as `errkind.Of(err)` (a kind such as `timeout` or `http_502`),
never `err.Error()`: error text can hold a provider's response body, the spool path or transcript
words (ADR-035 decision 8). Local-only for anything that carries transcript text: `config.IsLocalModel`.
`TAPIR_SHOT_DIR=… go test -run TestRenderUploadProgressStatesForReview ./internal/web` renders every
state for a screenshot review.
**Lint caveat:** `task check` does not fail on lint (#41: the Taskfile `|| echo` swallows
+5 -83
View File
@@ -1447,25 +1447,6 @@ an hour of Swedish *was* an unknown here; it is now measured, see decision 8.)
re-uploads. The single-replica assumption gains a second dependent (the semaphore is
process-local) and is recorded here.
*Amended 2026-10-03 (#50, #47).* The process now handles SIGTERM (what Kubernetes sends on
every deploy; only SIGINT was handled, so shutdown code never ran). On shutdown the running job
is recorded `interrupted` at once and the process waits, bounded to 10 s each, for in-flight
requests and for that write; the start-up sweep stays the backstop for a kill. **Known
limitation, deliberately not built: a deploy still loses a job in flight** (finishing it needs
a drain of up to the job time, a ~330 s termination grace and `strategy: Recreate`, i.e.
downtime on every deploy; a 90-minute file takes ~4.5 min and a deploy lands on a running job
perhaps one time in five at current volume). Jobs still queued at shutdown stay `queued` until
the next start's sweep. The user re-uploads, with an honest message.
*Receive limits (#47).* Request bodies are received at most 4 at a time, 2 per user, answered
as busy beyond that. Together with the 20-job queue and the running job that bounds the spool
at about (4 + 20 + 1) x 200 MiB = 5 GiB of the 6 Gi `emptyDir` (infra): raising any of those or
`TAPIR_UPLOAD_MAX_BYTES` must revisit that sum (pinned by `TestNewIntake_Defaults...`, not
enforced against infra). **A slow-drip body holds its slot without a time bound**: the server
sets `ReadHeaderTimeout` only, because a `ReadTimeout` would cut legitimate 200 MiB uploads on
slow links; two slow bodies take a user's two slots and two such users make everyone's upload
busy. It needs a valid session; recorded as a limitation with a follow-up issue.
8. **Chat reuses `chat.Service`; it is not forked.** The uploaded-transcript chat handlers
look the row up through the RLS-scoped store, so a non-owner gets 404 by the same mechanism
as ADR-027's summary gate, **before any model is called**. ADR-027's decisions (ephemeral,
@@ -1480,9 +1461,8 @@ an hour of Swedish *was* an unknown here; it is now measured, see decision 8.)
- **The opening model.** The offered list leads with the summarizer default `koala/phi4-mini`,
whose 8,192-token context returns HTTP 400 on an hour of speech (~13,000 tokens). Chat over
an upload opens on `TAPIR_UPLOAD_CHAT_MODEL` (default `iguana/gemma4-26b`); the switcher
lists **local models only** (amended by ADR-035 after the adversarial review found it listed
the cloud model too: the whole private transcript goes to the one chosen), and picking
`phi4-mini` on a long recording fails loudly and inline. A per-model budget is not built.
still lists every model, and picking `phi4-mini` on a long recording fails loudly and
inline. A per-model budget is not built.
Past ~90,000 chars (≈100 minutes) no offered model can answer by stuffing; see triggers.
9. **Not decided here.** Rich `AVSNITT`/`ALTERNATIV` analysis (gated on #30 — `analyze_srt.py`
@@ -1533,24 +1513,17 @@ deferred. The scope cut — chat first, no rich analysis, no video — is what k
ADR-016 window, it was the interesting thing, not the useful one — remove it.
**Open items this ADR creates.**
- **DECIDED (Mathias, 2026-10-02): option A.** Static ffmpeg and ffprobe 7.0.2 are copied into the
distroless image, pinned by version and sha256 (Dockerfile in this PR; #34). Measured at that time:
**about 160 MB** added, not the 70 MB first estimated (80 MB per binary); the build is **GPLv3**;
the checksum proves the bytes reviewed, not the publisher; upstream's `releases/` URL moves on, so
vendoring the verified tarball is the durable fix (not done). Uploads stay off until the deployment
sets `TAPIR_UPLOADS_ENABLED`. *The question as it was put:* *Options:* (A) static ffmpeg/ffprobe copied
- **DECISION NEEDED — ffmpeg in the runtime image.** *Options:* (A) static ffmpeg/ffprobe copied
into distroless **(recommended — the only option that satisfies "accept m4a/mp3/opus/mp4,
validate duration, reject video"; pure-Go decoders cover wav/mp3 only)**; (B) accept WAV
only, convert client-side (fails "I just want to upload it"); (C) a transcode sidecar
(second deployable, ADR-018's reasoning against). *Blocks:* the Dockerfile slice, not the
tests, which use the port.
- **DECIDED (Mathias, 2026-10-02): option A, enabled by default (#35).** Scope: the audio
transcription fallback only. Reconciliation text and chat over a recording never go to a cloud
model (ADR-034, ADR-035). *The question as it was put:* *Options:* (A) enabled by default as
- **DECISION NEEDED — berget fallback for private audio.** *Options:* (A) enabled by default as
#33 specifies, with the serving endpoint shown **(recommended, matches the issue)**; (B)
disabled by default, a failed iguana job fails honestly and the user retries. *Blocks:*
the production default only; the adapter ships either way.
- ~~Verify Tapir's ingress `proxy-body-size` against the 200 MB cap (`infra`).~~ **Resolved 2026-10-03:** real unauthenticated bodies of 2, 30, 120 and 210 MiB all pass NPM (TLS, piguard) and Traefik to tapir (a 302 to login); there is no body limit in that range. Uploads were enabled in production the same day, local engines only (infra#520).
- Verify Tapir's ingress `proxy-body-size` against the 200 MB cap (`infra`).
- Per-transcript deletion is not built; account deletion cascades. A private-content store
without a delete button is a gap worth a ticket, not a silent omission.
@@ -1669,12 +1642,6 @@ recording.
~30 disputes where the reconciled text differs from pianissimo alone. Keep the feature only if the
reconciled reading is right at least 85% of the time **and** no worse than the better single engine.
If not, switch it off (config) and keep the single-engine path.
**Decision (Mathias, 2026-10-02), on the first result below: keep it as an opt-in feature**, reading
the bar as 95% of decided items, with the second condition to be answered on the next real run.
For deployment the reconciler is `iguana/gemma4-31b` (97% against the speaker's answers, about 22
minutes for the 63.6-minute recording with Low Power Mode off). A cloud reconciler was raised and is
**not adopted**: it would send private transcript text to a cloud API on every upload; recorded as
an evaluation question in #45.
**First result, 2026-10-02, before merge** (`docs/research/reconcile-blind-check-2026-10-02.md`): 30
random replacement spots on the real 63.6-minute recording, judged blind by Opus from text (not by the
speaker, who delegated it): 21 right, 1 worse than pianissimo, 3 neither reading right, 5 undecidable.
@@ -1744,51 +1711,6 @@ wording while only `content` is merged.
a segment edge), they stay the base's and the result says so in the log. This replaces ADR-034
decision 6's "segments stay untouched".
8. **Hardening from the adversarial review** (`skills:pr-gatekeeper`, head `7029dd1`: BLOCK; recorded on
PR #39). Each finding was reproduced by a failing test first.
- **Chat over a recording offers and accepts local models only** (it listed `berget/mistral-small`);
`TAPIR_UPLOAD_CHAT_MODEL` must be local at start-up; with no local model, nothing is sent.
- **A model alias counts as local only if it is one plain name after `koala/` or `iguana/`**
(`koala/../berget/x`, `koala//x`, `koala/x?host=…` were accepted by a bare prefix check).
- **An error reaches a log line, the row or the page as a kind** (`timeout`, `canceled`, `http_502`,
`network`, `error`), never its text, which can hold a provider's response body, the spool path or
a word from the transcript (`internal/errkind`). The stored reconcile reason and an engine's
failure reason are kinds; the page no longer shows provider text.
- **Every local engine call has its own deadline** (`TAPIR_TRANSCRIBE_ENGINE_TIMEOUT`, default 20
minutes: two wedged engines in sequence then leave 20 of the job's 60 minutes for a fallback),
so a wedged engine fails while the job lives and the fallback can run; the job's own
deadline or a cancel still never reaches the cloud.
- **The duration limit is checked against what is converted**, not the file's header: ffmpeg is
capped at the limit plus a minute, a longer result fails the job (`audio_too_long`), and a result
under 80% of the claimed length and more than two minutes short is **transcribed and noted** on
the audio step ("the audio is 20 min long; the file claimed 1 h 19 min"), not refused. The second
adversarial review showed a complete variable-bitrate MP3 whose header over-estimated its length
by 4x was wrongly refused by the first version of this check; a note keeps a truncated file
visible without turning away real recordings.
- **The queue is bounded**, 20 waiting in total and 3 per user, and its room is checked before the
upload is spooled (not after 200 MB and an ffprobe run); a full queue is a 429 with a typed reason,
never a stuck row or a server error.
- **The configured upload chat model is callable.** It is added to the shared chat service's model
list when uploads are on and it is local, because the service re-maps a model it does not offer
(a valid `iguana/gemma4-31b` was being silently swapped for `koala/phi4-mini`). The video chat's own switcher
lists that local model too while uploads are on.
- **A client that hangs up mid-upload is not an error**: the handler logs INFO and writes nothing
(rendering the error page ran a database read on a cancelled context and logged a false ERROR). The
400 "cut off" answer is keyed on a marker for errors from *reading the client's body*
(`uploads.ErrClientRead`), not on `io.ErrUnexpectedEOF`, which is also what a database drop looks like.
- **A note is not hidden**: a finished upload's record opens by default when any step carries a note.
- **Bad input is a 4xx**: a non-UUID id is a 404, a malformed or cut-off multipart body is a 400
whose text is not logged, and a title with control bytes, bidirectional controls or invalid UTF-8 is
cleaned (emoji joiners, Swedish letters and variation selectors are kept), not a 500.
- **The image was built and inspected** (buildah on koala, 2026-10-02): the first build FAILED
(`tar: xz: Cannot exec`, the golang image has no xz); fixed, rebuilt at 180 MB, ffmpeg and ffprobe
run as `nonroot` in distroless. CI does not build the image on a pull request
(`if: github.event_name != 'pull_request'`), so the first CI build of it is the merge to main.
Not done, filed as follow-ups: a vendored ffmpeg tarball, CSRF cookie review, ffprobe CPU and
stream-count limits, mov external references, `alive_at` clock skew, a hung database blocking the
tracker, an unbounded gateway response body, a user-facing notice for a recording that is partial but
inside the tolerance.
**Alternatives rejected.** Server-sent events or a websocket: a long-lived connection through the
ingress, with buffering to get wrong, for a page that changes every few seconds at best; polling is what
the page already does. A percentage from the engine calls: there is no data behind it. A separate
-40
View File
@@ -16,52 +16,12 @@ COPY . .
# no libc. Trim symbols/DWARF to shrink the layer.
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags='-s -w' -o /out/tapir ./cmd/tapir
# ── ffmpeg ──────────────────────────────────────────────────────────────────
# Audio upload probes files with ffprobe and converts them with ffmpeg (ADR-033,
# #34 option A). Static builds, so they run in the distroless image below.
#
# Pinned by version AND sha256: a changed upstream file fails the build instead of
# shipping. The checksum proves these are the bytes that were reviewed (2026-10-02;
# the publisher's md5 matched too). It does NOT prove authenticity: the publisher
# signs nothing we can check. amd64 only (the cluster is one amd64 node).
#
# These builds are GPLv3 (--enable-gpl --enable-version3). The service is private and
# the image is not distributed; the licence text is copied into the image anyway.
#
# The release URL moves on when upstream releases a newer version, so the pinned
# version is tried in releases/ and then old-releases/. If neither has it the build
# fails; vendoring the verified tarball is the durable fix.
FROM golang:1.26 AS ffmpeg
ARG FFMPEG_VERSION=7.0.2
ARG FFMPEG_SHA256=abda8d77ce8309141f83ab8edf0596834087c52467f6badf376a6a2a4c87cf67
# xz is not in the golang image (found by building it: `tar: xz: Cannot exec`).
RUN set -eu; \
apt-get update; \
apt-get install -y --no-install-recommends xz-utils; \
rm -rf /var/lib/apt/lists/*; \
mkdir -p /out; \
for dir in releases old-releases; do \
curl -fsSL -o /out/ffmpeg.tar.xz "https://johnvansickle.com/ffmpeg/${dir}/ffmpeg-${FFMPEG_VERSION}-amd64-static.tar.xz" && break; \
done; \
echo "${FFMPEG_SHA256} /out/ffmpeg.tar.xz" | sha256sum -c -; \
tar -xJf /out/ffmpeg.tar.xz -C /out --strip-components=1 --wildcards '*/ffmpeg' '*/ffprobe' '*/GPLv3.txt'; \
rm /out/ffmpeg.tar.xz
# ── runtime ─────────────────────────────────────────────────────────────────
# distroless static + nonroot: no shell, no package manager, runs as uid 65532.
# ca-certificates are bundled, which the OIDC/HTTPS clients need.
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/tapir /tapir
COPY --from=ffmpeg /out/ffmpeg /out/ffprobe /usr/local/bin/
COPY --from=ffmpeg /out/GPLv3.txt /usr/share/doc/ffmpeg/GPLv3.txt
# Where the service finds them. Uploads stay OFF until TAPIR_UPLOADS_ENABLED is set
# (a deployment decision, not an image one).
ENV TAPIR_FFMPEG_BIN=/usr/local/bin/ffmpeg \
TAPIR_FFPROBE_BIN=/usr/local/bin/ffprobe
EXPOSE 8080
USER nonroot:nonroot
+1 -1
View File
@@ -34,7 +34,7 @@ tasks:
lint:
desc: golangci-lint (installed by CI; skipped locally if absent).
cmds:
- 'if command -v golangci-lint >/dev/null 2>&1; then golangci-lint run ./...; else echo "golangci-lint not installed — skipping (CI installs it)"; fi'
- 'command -v golangci-lint >/dev/null 2>&1 && golangci-lint run ./... || echo "golangci-lint not installed — skipping (CI installs it)"'
test:
desc: Run all tests. -p 1 serializes packages so the embedded-postgres tests
+14 -15
View File
@@ -17,9 +17,9 @@ import (
"errors"
"fmt"
"log/slog"
"net"
"net/http"
"os"
"os/signal"
"sync"
"time"
@@ -41,7 +41,7 @@ func main() {
}
log := slog.New(slog.NewTextHandler(os.Stderr, nil))
ctx, stop := shutdownContext(context.Background())
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()
var err error
@@ -215,11 +215,9 @@ func cmdServe(ctx context.Context, log *slog.Logger) error {
if err != nil {
return err
}
var workerDone chan struct{} // nil without uploads: nothing to wait for at shutdown
wireUploads(app, cfg, intake)
if intake != nil {
workerDone = make(chan struct{})
go func() { worker.Run(ctx); close(workerDone) }()
go worker.Run(ctx)
log.Info("audio upload enabled", "model", cfg.TranscribeModel, "fallback", cfg.TranscribeFallbackModel,
"max_mb", cfg.UploadMaxBytes>>20, "max_minutes", int(cfg.UploadMaxDuration.Minutes()))
log.Warn("audio upload assumes a SINGLE replica: the transcription queue is process-local (ADR-033, ADR-018)")
@@ -352,19 +350,20 @@ func cmdServe(ctx context.Context, log *slog.Logger) error {
}()
}
// Graceful shutdown on signal: stop accepting, drain in-flight requests, and let the
// worker record the job it was running as interrupted (#50).
var extra []*http.Server
// Graceful shutdown on signal: stop accepting, drain in-flight requests.
go func() {
<-ctx.Done()
shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
_ = srv.Shutdown(shutdownCtx)
if metricsSrv != nil {
extra = append(extra, metricsSrv)
}
ln, err := net.Listen("tcp", srv.Addr)
if err != nil {
return fmt.Errorf("serve: listen: %w", err)
_ = metricsSrv.Shutdown(shutdownCtx)
}
}()
log.Info("serving web ui", "addr", cfg.HTTPAddr, "user", cfg.UserID)
if err := serveHTTP(ctx, srv, ln, extra, workerDone, shutdownGrace); err != nil {
return err
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
return fmt.Errorf("serve: %w", err)
}
return nil
}
-21
View File
@@ -134,30 +134,9 @@ func chatModels(cfg config.Config) []string {
add(cfg.SummarizerModel)
add(cfg.FallbackModel)
add(cfg.CloudFallbackModel)
// The model chat over an uploaded recording opens on must be one the shared service can
// call: it re-maps a model it does not offer, which silently swapped a configured 31B for
// phi4-mini, whose context cannot hold an hour of speech. Local only, and only when
// uploads are on, so the video chat is unchanged otherwise (second adversarial review).
if cfg.UploadsEnabled && config.IsLocalModel(cfg.UploadChatModel) {
add(cfg.UploadChatModel)
}
return models
}
// uploadChatModels is the chat switcher for a PRIVATE uploaded recording: the same
// ordered set as chatModels, local models only. The whole transcript (up to 90,000
// characters) is sent to the one chosen, so a cloud model the video chat offers must not
// be offered here; with none local, none is offered (ADR-035 review).
func uploadChatModels(cfg config.Config) []string {
var local []string
for _, m := range chatModels(cfg) {
if config.IsLocalModel(m) {
local = append(local, m)
}
}
return local
}
// buildChat wires the per-video chat service (ADR-027): a Completer factory over
// the SAME LiteLLM gateway the summarizer uses (a different alias per model, not a
// second client config) and the same transcript-truncation budget. It returns nil
-55
View File
@@ -1,55 +0,0 @@
package main
import (
"context"
"errors"
"fmt"
"net"
"net/http"
"os"
"os/signal"
"syscall"
"time"
)
// shutdownGrace bounds each wait at shutdown (in-flight requests, then the worker). Two of
// them stay inside Kubernetes' default 30 s termination grace.
const shutdownGrace = 10 * time.Second
// shutdownContext is cancelled by SIGINT (a terminal) or SIGTERM (what Kubernetes sends
// on every deploy). Handling only SIGINT meant a deploy killed the process before any
// shutdown code ran (#50).
func shutdownContext(parent context.Context) (context.Context, context.CancelFunc) {
return signal.NotifyContext(parent, os.Interrupt, syscall.SIGTERM)
}
// serveHTTP serves srv on ln until it fails or ctx ends, then shuts every server down and
// waits for in-flight requests and for workerDone (nil: no worker) to finish, each for at
// most grace. It must not return earlier: Serve returns http.ErrServerClosed as soon as
// Shutdown STARTS, and returning from main kills the drain and the worker's last write.
func serveHTTP(ctx context.Context, srv *http.Server, ln net.Listener, extra []*http.Server, workerDone <-chan struct{}, grace time.Duration) error {
errc := make(chan error, 1)
go func() { errc <- srv.Serve(ln) }()
select {
case err := <-errc:
if !errors.Is(err, http.ErrServerClosed) {
return fmt.Errorf("serve: %w", err)
}
return nil
case <-ctx.Done():
}
shutdownCtx, cancel := context.WithTimeout(context.Background(), grace)
defer cancel()
for _, s := range append([]*http.Server{srv}, extra...) {
_ = s.Shutdown(shutdownCtx)
}
if workerDone != nil {
select {
case <-workerDone:
case <-time.After(grace):
}
}
return nil
}
-124
View File
@@ -1,124 +0,0 @@
package main
import (
"context"
"net/http"
"net/http/httptest"
"os"
"sync"
"syscall"
"testing"
"time"
)
// Kubernetes stops a pod with SIGTERM; only SIGINT was handled, so a deploy killed the
// process without running any shutdown code (#50).
func TestShutdownContext_IsCancelledBySIGTERMAndSIGINT(t *testing.T) {
for _, sig := range []syscall.Signal{syscall.SIGTERM, syscall.SIGINT} {
t.Run(sig.String(), func(t *testing.T) {
ctx, stop := shutdownContext(context.Background())
defer stop()
if err := syscall.Kill(os.Getpid(), sig); err != nil {
t.Fatalf("send %v: %v", sig, err)
}
select {
case <-ctx.Done():
case <-time.After(2 * time.Second):
t.Fatalf("context not cancelled by %v", sig)
}
})
}
}
// serveHTTP must not return while a request is in flight or the worker is still recording
// its job: Serve returns http.ErrServerClosed as soon as Shutdown STARTS, and returning
// from it let the process exit before either finished (#50). Each half is tested with the
// other already satisfied, so neither can hold the call open on the other's behalf.
func pendingFor(t *testing.T, returned <-chan error, why string) {
t.Helper()
select {
case err := <-returned:
t.Fatalf("serveHTTP returned (%v) %s", err, why)
case <-time.After(150 * time.Millisecond):
}
}
func TestServeHTTP_DrainsAnInFlightRequestBeforeReturning(t *testing.T) {
release := make(chan struct{})
entered := make(chan struct{})
srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
close(entered)
<-release
_, _ = w.Write([]byte("ok"))
}))
srv.Start()
defer srv.Close()
var once sync.Once
releaseHandler := func() { once.Do(func() { close(release) }) }
defer releaseHandler() // runs before srv.Close: a failing test must not hang on the blocked handler
ctx, cancel := context.WithCancel(context.Background())
workerDone := make(chan struct{})
close(workerDone) // the worker is already done: only the request can hold serveHTTP open
returned := make(chan error, 1)
go func() { returned <- serveHTTP(ctx, srv.Config, srv.Listener, nil, workerDone, 5*time.Second) }()
go func() { _, _ = http.Get(srv.URL) }()
<-entered
cancel()
pendingFor(t, returned, "while a request was still in flight")
releaseHandler()
select {
case err := <-returned:
if err != nil {
t.Fatalf("serveHTTP: %v", err)
}
case <-time.After(2 * time.Second):
t.Fatal("serveHTTP did not return once the request finished")
}
}
func TestServeHTTP_WaitsForTheWorkerBeforeReturning(t *testing.T) {
srv := httptest.NewUnstartedServer(http.NotFoundHandler())
srv.Start()
defer srv.Close()
ctx, cancel := context.WithCancel(context.Background())
workerDone := make(chan struct{})
returned := make(chan error, 1)
go func() { returned <- serveHTTP(ctx, srv.Config, srv.Listener, nil, workerDone, 5*time.Second) }() // no request in flight
cancel()
pendingFor(t, returned, "before the worker finished")
close(workerDone)
select {
case err := <-returned:
if err != nil {
t.Fatalf("serveHTTP: %v", err)
}
case <-time.After(2 * time.Second):
t.Fatal("serveHTTP did not return once the worker finished")
}
}
func TestServeHTTP_GivesUpOnAWorkerThatNeverFinishesAfterTheGrace(t *testing.T) {
srv := httptest.NewUnstartedServer(http.NotFoundHandler())
srv.Start()
defer srv.Close()
ctx, cancel := context.WithCancel(context.Background())
returned := make(chan error, 1)
go func() {
returned <- serveHTTP(ctx, srv.Config, srv.Listener, nil, make(chan struct{}), 200*time.Millisecond)
}()
cancel()
select {
case <-returned:
case <-time.After(3 * time.Second):
t.Fatal("a stuck worker must not hold the process past the grace period")
}
}
+4 -16
View File
@@ -55,8 +55,8 @@ func buildUploads(ctx context.Context, cfg config.Config, st uploadsStore, log *
log.Warn("uploads: marked interrupted by a restart (audio is not retained; they must be re-uploaded)", "count", n)
}
worker := uploads.NewWorker(st, newConverter(cfg), buildTranscriber(cfg),
cfg.UploadTmpDir, cfg.TranscribeTimeout, uploads.WithMaxDuration(cfg.UploadMaxDuration))
worker := uploads.NewWorker(st, &audio.FFmpeg{Bin: cfg.FFmpegBin}, buildTranscriber(cfg),
cfg.UploadTmpDir, cfg.TranscribeTimeout)
intake := uploads.NewIntake(st, &audio.FFprobe{Bin: cfg.FFprobeBin}, worker,
uploadLimits(cfg), cfg.UploadTmpDir)
return intake, worker, nil
@@ -91,13 +91,6 @@ func wireUploads(app *web.App, cfg config.Config, in *uploads.Intake) {
app.UploadLimits = uploadLimits(cfg)
app.UploadChatMaxChars = cfg.MaxUploadedTranscriptChars
app.UploadChatModel = cfg.UploadChatModel
app.UploadChatModels = uploadChatModels(cfg)
}
// newConverter is the ffmpeg adapter capped at the configured maximum duration, so a file
// whose header lies about its length cannot become hours of WAV (ADR-035 review).
func newConverter(cfg config.Config) *audio.FFmpeg {
return &audio.FFmpeg{Bin: cfg.FFmpegBin, MaxSeconds: int(cfg.UploadMaxDuration.Seconds())}
}
// adjudicatorTimeout bounds one adjudication call. A call takes ~2 s on gemma4-26b and ~10 s
@@ -114,12 +107,7 @@ func buildTranscriber(cfg config.Config) uploads.Transcriber {
// No overall client timeout: a 90 minute recording takes minutes. The deadline is
// the worker's per-job context.
client := &http.Client{}
// Each LOCAL engine call gets its own deadline: a wedged engine does not answer, and
// without one it used the whole job before the fallback could run.
local := func(model string) transcribe.Backend {
return transcribe.WithTimeout(transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, model, client), cfg.TranscribeEngineTimeout)
}
primary := local(cfg.TranscribeModel)
primary := transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, cfg.TranscribeModel, client)
var fallback transcribe.Backend
if cfg.TranscribeFallbackModel != "" && cfg.TranscribeFallbackModel != cfg.TranscribeModel {
fallback = transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, cfg.TranscribeFallbackModel, client)
@@ -128,7 +116,7 @@ func buildTranscriber(cfg config.Config) uploads.Transcriber {
return transcribe.NewChain(primary, fallback)
}
base := local(cfg.ReconcileBaseModel)
base := transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, cfg.ReconcileBaseModel, client)
// Temperature 0 and a short reply: the adjudicator's accuracy was measured that way.
chat := llm.New(cfg.GatewayURL, cfg.GatewayKey, cfg.ReconcileModel, adjudicatorTimeout,
llm.WithTemperature(0), llm.WithMaxTokens(200))
-85
View File
@@ -1,85 +0,0 @@
package main
import (
"reflect"
"testing"
"git.d-ma.be/mathias/tapir/internal/config"
)
// The chat over an uploaded recording sends the whole private transcript to the model
// the user picks, so the switcher for it may offer local models only (ADR-035 review).
func TestUploadChatModels_NeverOffersACloudModelButKeepsTheLocalOrder(t *testing.T) {
var cfg config.Config
cfg.SummarizerModel = "koala/phi4-mini"
cfg.FallbackModel = "iguana/gemma4-26b"
cfg.CloudFallbackModel = "berget/mistral-small"
got := uploadChatModels(cfg)
want := []string{"koala/phi4-mini", "iguana/gemma4-26b"}
if !reflect.DeepEqual(got, want) {
t.Errorf("uploadChatModels = %v, want %v: the cloud model that the video chat offers must not be offered for a private recording", got, want)
}
}
func TestUploadChatModels_ACloudPrimaryIsDroppedToo(t *testing.T) {
var cfg config.Config
cfg.SummarizerModel = "berget/mistral-small"
cfg.FallbackModel = "iguana/gemma4-26b"
got := uploadChatModels(cfg)
if !reflect.DeepEqual(got, []string{"iguana/gemma4-26b"}) {
t.Errorf("uploadChatModels = %v", got)
}
}
func TestUploadChatModels_NoLocalModelMeansNoneOfferedNotACloudDefault(t *testing.T) {
var cfg config.Config
cfg.SummarizerModel = "berget/mistral-small"
if got := uploadChatModels(cfg); len(got) != 0 {
t.Errorf("uploadChatModels = %v, want none", got)
}
}
// A valid local TAPIR_UPLOAD_CHAT_MODEL that is not in the video chat's chain used to be
// accepted by config and then silently ignored: the shared chat service re-maps a model it
// does not offer, so the user got koala/phi4-mini, whose 8,192-token context cannot hold an
// hour of speech (second adversarial review).
func TestChatModels_AnUploadChatModelOutsideTheChainIsOfferedWhenUploadsAreOn(t *testing.T) {
var cfg config.Config
cfg.SummarizerModel = "koala/phi4-mini"
cfg.FallbackModel = "iguana/gemma4-26b"
cfg.UploadsEnabled = true
cfg.UploadChatModel = "iguana/gemma4-31b"
service := chatModels(cfg)
upload := uploadChatModels(cfg)
if !reflect.DeepEqual(service, []string{"koala/phi4-mini", "iguana/gemma4-26b", "iguana/gemma4-31b"}) {
t.Errorf("chatModels = %v: the shared service must be able to call the configured upload model", service)
}
if !reflect.DeepEqual(upload, service) {
t.Errorf("uploadChatModels = %v, want %v", upload, service)
}
}
func TestChatModels_NothingIsAddedWhenUploadsAreOffOrTheModelIsNotLocal(t *testing.T) {
var cfg config.Config
cfg.SummarizerModel = "koala/phi4-mini"
cfg.UploadChatModel = "iguana/gemma4-31b"
if got := chatModels(cfg); !reflect.DeepEqual(got, []string{"koala/phi4-mini"}) {
t.Errorf("uploads off: chatModels = %v, want the video chain unchanged", got)
}
cfg.UploadsEnabled = true
cfg.UploadChatModel = "berget/mistral-small"
if got := chatModels(cfg); !reflect.DeepEqual(got, []string{"koala/phi4-mini"}) {
t.Errorf("a cloud upload chat model was added to the service: %v", got)
}
}
-26
View File
@@ -1,26 +0,0 @@
package main
import (
"testing"
"time"
"git.d-ma.be/mathias/tapir/internal/config"
)
// The converter is told the configured maximum, so a file whose header lies about its
// length cannot become hours of WAV (adversarial review F2).
func TestNewConverter_IsCappedAtTheConfiguredMaximumDuration(t *testing.T) {
var cfg config.Config
cfg.FFmpegBin = "/usr/local/bin/ffmpeg"
cfg.UploadMaxDuration = 90 * time.Minute
c := newConverter(cfg)
if c.Bin != "/usr/local/bin/ffmpeg" {
t.Errorf("Bin = %q", c.Bin)
}
if c.MaxSeconds != 5400 {
t.Errorf("MaxSeconds = %v, want 5400", c.MaxSeconds)
}
}
+1 -3
View File
@@ -22,15 +22,13 @@ import (
type fakeUploadsStore struct {
sweeps int
sweepErr error
failedKinds []domain.UploadErrorKind
}
func (f *fakeUploadsStore) CreateUpload(context.Context, string, string, bool) (string, error) {
return "id", nil
}
func (f *fakeUploadsStore) FailUpload(_ context.Context, _, _ string, k domain.UploadErrorKind) error {
f.failedKinds = append(f.failedKinds, k)
func (f *fakeUploadsStore) FailUpload(context.Context, string, string, domain.UploadErrorKind) error {
return nil
}
-78
View File
@@ -1,78 +0,0 @@
package main
import (
"context"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// The wiring of the per-engine deadline: a gateway whose primary model never answers
// must still reach the fallback model inside the job's own deadline (adversarial review
// F4). The fake gateway answers per model from the multipart `model` field.
func TestBuildTranscriber_AWedgedPrimaryReachesTheFallbackBeforeTheJobDeadline(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = r.ParseMultipartForm(1 << 20)
if r.FormValue("model") == "iguana/kb-whisper" {
<-r.Context().Done() // wedged: never answers
return
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"text":"hej","duration":1,"segments":[{"start":0,"end":1,"text":"hej"}]}`))
}))
defer srv.Close()
cfg := uploadCfg(t, true)
cfg.GatewayURL = srv.URL
cfg.TranscribeModel = "iguana/kb-whisper"
cfg.TranscribeFallbackModel = "berget/kb-whisper"
cfg.TranscribeEngineTimeout = 150 * time.Millisecond
wav := filepath.Join(t.TempDir(), "a.wav")
require.NoError(t, os.WriteFile(wav, []byte("RIFF....WAVEfmt "), 0o600))
job, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
got, err := buildTranscriber(cfg).Transcribe(job, wav)
require.NoError(t, err, "the wedged primary used the whole job and the fallback never ran")
assert.Equal(t, "berget/kb-whisper", got.Endpoint)
}
func TestBuildTranscriber_AWedgedBaseEngineIsASoftFailureInTheEnsemble(t *testing.T) {
// The same deadline must guard the ensemble's BASE engine, not only the chain's primary
// (second adversarial review: the wiring was tested for the chain alone).
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = r.ParseMultipartForm(1 << 20)
if r.FormValue("model") == "iguana/pianissimo" {
<-r.Context().Done()
return
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"text":"hej","duration":1,"segments":[{"start":0,"end":1,"text":"hej"}]}`))
}))
defer srv.Close()
cfg := uploadCfg(t, true)
cfg.GatewayURL = srv.URL
cfg.TranscribeModel = "iguana/kb-whisper"
cfg.ReconcileBaseModel = "iguana/pianissimo"
cfg.ReconcileModel = "iguana/gemma4-31b"
cfg.TranscribeEngineTimeout = 150 * time.Millisecond
wav := filepath.Join(t.TempDir(), "a.wav")
require.NoError(t, os.WriteFile(wav, []byte("RIFF....WAVEfmt "), 0o600))
job, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
got, err := buildTranscriber(cfg).Transcribe(job, wav)
require.NoError(t, err, "a wedged base engine must not use up the job")
assert.Equal(t, "hej", got.Text)
require.NotNil(t, got.Reconciliation)
assert.Equal(t, "single_engine", string(got.Reconciliation.Status))
assert.Contains(t, got.Reconciliation.Reason, "timeout")
}
-39
View File
@@ -1,39 +0,0 @@
package main
import (
"context"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
// The worker buildUploads returns enforces the configured maximum on what was really
// converted (adversarial review F2). A stub ffmpeg writes a sparse 100-minute WAV however
// short the file claimed to be; the gateway is never reached because the job fails first.
func TestBuildUploads_TheWorkerRefusesAudioLongerThanTheConfiguredMaximum(t *testing.T) {
dir := t.TempDir()
stub := filepath.Join(dir, "ffmpeg")
require.NoError(t, os.WriteFile(stub, []byte("#!/bin/sh\nfor a; do last=\"$a\"; done\ntruncate -s 192000044 \"$last\"\n"), 0o700))
cfg := uploadCfg(t, true)
cfg.FFmpegBin = stub
cfg.UploadMaxDuration = 90 * time.Minute
cfg.TranscribeTimeout = time.Minute
st := &fakeUploadsStore{}
_, w, err := buildUploads(context.Background(), cfg, st, quietLog())
require.NoError(t, err)
src := filepath.Join(cfg.UploadTmpDir, "upload-x")
require.NoError(t, os.WriteFile(src, []byte("audio"), 0o600))
w.Process(context.Background(), uploads.Job{UserID: "u", UploadID: "id", Path: src, Duration: 10 * time.Second})
assert.Equal(t, []domain.UploadErrorKind{domain.UploadErrTooLong}, st.failedKinds,
"a file that claimed 10 s but converted to 100 minutes must fail, not be sent to an engine")
}
+5 -3
View File
@@ -23,13 +23,15 @@ it** — endpoints and aliases drift, and this file is a snapshot (2026-06-06),
`{"error":"failed to process audio"}` string — it has had at least five causes. Tapir's knobs:
`TAPIR_UPLOADS_ENABLED` (off by default), `TAPIR_UPLOAD_MAX_MB` (200), `TAPIR_UPLOAD_MAX_MINUTES`
(90), `TAPIR_TRANSCRIBE_MODEL`, `TAPIR_TRANSCRIBE_FALLBACK_MODEL` (empty disables the cloud
fallback), `TAPIR_TRANSCRIBE_TIMEOUT` (60m, the whole job), `TAPIR_TRANSCRIBE_ENGINE_TIMEOUT` (20m, one engine call: a wedged engine then fails while the job lives and the fallback can run), `TAPIR_UPLOAD_TMP_DIR`, `TAPIR_FFMPEG_BIN`,
fallback), `TAPIR_TRANSCRIBE_TIMEOUT` (30m), `TAPIR_UPLOAD_TMP_DIR`, `TAPIR_FFMPEG_BIN`,
`TAPIR_FFPROBE_BIN`, `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS` (90000 — the measured ceiling of
`iguana/gemma4-26b`, which silently drops the head above it), `TAPIR_UPLOAD_CHAT_MODEL`
(`iguana/gemma4-26b`; must be a local `koala/` or `iguana/` alias, since the whole private transcript is sent to it; `koala/phi4-mini`'s 8,192-token context 400s on an hour of speech). `ffmpeg` + `ffprobe` are in the runtime image (static, pinned by sha256; #34 decided A on 2026-10-02); Tapir's own ingress body limit for a 200 MB upload is unverified (infra).
(`iguana/gemma4-26b`; `koala/phi4-mini`'s 8,192-token context 400s on an hour of speech). Needs `ffmpeg` + `ffprobe`
in the runtime image; Tapir's own ingress body limit for a 200 MB upload is unverified (infra).
**Two-engine mode (ADR-034), off unless `TAPIR_RECONCILE_BASE_MODEL` is set:** it runs
`iguana/pianissimo` (the base text) and `TAPIR_TRANSCRIBE_MODEL` (kb-whisper) on the same audio, then a
LOCAL chat model (`TAPIR_RECONCILE_MODEL`) chooses between their disagreements. **Deployment choice (2026-10-02): `iguana/gemma4-31b`** (97% against a speaker's blind answers; 92% for `iguana/gemma4-26b`). With iguana's macOS **Low Power Mode off** the 31B decodes at 22 tok/s and reconciled a 63.6-minute recording in 21 min 49 s end to end; with it on it ran at 8 tok/s. Check `pmset -g | grep lowpowermode` before blaming a model or an engine for slowness.
LOCAL chat model (`TAPIR_RECONCILE_MODEL`, e.g. `iguana/gemma4-26b`: 92% against a speaker's blind answers
at ~2 s a call; `iguana/gemma4-31b` 97% but ~5x slower) chooses between their disagreements.
`TAPIR_RECONCILE_CHECK_ORDER=true` asks twice per dispute with the order swapped (doubles the time).
Every model must start `koala/` or `iguana/`: a cloud alias is a start-up error. The default job timeout is
60 minutes. `iguana/pianissimo` is KlangAI's Swedish ASR (a Parakeet/FastConformer on MLX, served by a small
@@ -29,7 +29,7 @@ ingress configuration, which lives in `infra`. "About 10 MB" is an inference.
## Method
- **Audio.** No known-good Swedish recording is available that is safe to use: the only real
one is a private recording (#31 custody still open) and was
one (`IMG_1233.wav`) is the private `/bygge` recording (#31 custody still open) and was
deliberately not touched. Instead: macOS `say -v Alva` (Swedish) on iguana, a 1,171-char
five-paragraph meeting script → 82.5 s of 16 kHz mono PCM WAV (`ffmpeg -ar 16000 -ac 1`).
Concatenated ×4, ×8, ×16, ×32, ×44, ×66 with Python `wave` (the issue's own recipe).
@@ -1,88 +0,0 @@
# Sprint: audio upload follow-ups (Sweep Sprint, 2026-10)
Method: Sweep Sprint (phase 1 parallel read-only research per issue: **done**; phase 2 sequential
execution, one issue per commit: **not started, waits for the owner's go**). Closing oath below.
Milestones: [v0.31.1 - security and robustness](https://git.d-ma.be/mathias/tapir/milestone/10),
[v0.32.0 - audio upload follow-ups](https://git.d-ma.be/mathias/tapir/milestone/11).
Context: v0.31.0 shipped audio upload (ADR-033/034/035) and is live in production behind auth at
`/uploads`, with the Berget cloud fallback on for audio (infra#522). A real 63.7 min upload
transcribed in 2m53s with no errors. The phase-1 investigation found two pre-existing or new
weaknesses an authenticated user can reach, so the first release is a hotfix.
## Oath (closing gates, every clause must hold; blocked = descoped and tracked, never "done")
| # | Clause | Evidence |
|---|--------|----------|
| SG1 | Every issue in the sprint is closed with a linked commit, or moved out with a stated reason | issue list of both milestones |
| SG2 | Each fix has a test that was seen red on main first, named after the mechanism it observes | red/green output in the issue |
| SG3 | `task check` AND `golangci-lint run ./...` exit 0 (checked by `$?`, not by grep); `task check` does not gate lint until #41 | gate output |
| SG4 | Control code (#48, #49, the upload cap in #47) has a recorded adversarial review verified by execution, verdict in the issue | gatekeeper verdict comment |
| SG5 | Docs current in the same commit: ADR-033/035 text, feature file, use-case coverage test | diff |
| SG6 | Each release tagged (v0.31.1, v0.32.0), pushed, deployed, and observed live (pod running the new image, no restart loop, a real upload still works) | `kubectl`/log + one upload |
| SG7 | Learnings written to the brain; deferred work filed as issues; decisions that are the owner's recorded as DECISION NEEDED | brain ids + issue numbers |
## Work order
### v0.31.1 (security and robustness; hotfix, do first)
1. **#41 lint gate.** `task check` swallows lint findings (`|| echo`). Make the gate fail on findings,
fix what it surfaces. First, so every later gate means something. (Decision D1.)
2. **#48 cross-origin POST protection.** `http.NewCrossOriginProtection()` around `Router()`.
Red: foreign-origin POST to `/account/delete`, `/uploads`, `/uploads/{id}/chat` is refused.
3. **#49 ffprobe/ffmpeg limits.** `-max_streams 32` + narrower probe output; probe stdout cap;
probe deadline. Three commits.
4. **#47 concurrent-upload cap (remaining item).** Bound the spool: reservation per accepted upload
against the 6Gi emptyDir so concurrent uploads cannot evict the pod.
5. **#50 SIGTERM interim.** Handle SIGTERM; record the interrupted job at once; fix the false
comment. Full drain recorded as a limitation. (Decision D2.)
Release v0.31.1: one adversarial review round over 2-5, then tag, push, observe.
### v0.32.0 (follow-ups)
6. **#52** label a cloud-served transcript in plain words; document the empty-transcript trigger.
7. **#53** bound the LLM client's response body read.
8. **#51** tracker: save timeout, then app-clock heartbeat.
9. **#37** nav link to uploads and per-recording delete (queued/transcribing: refuse until it
ends). (Decision D3.)
10. **#36** chat default model after measurement. (Decision D4.)
11. **#38** measurements (Berget length limit, large models on production iguana). (D5, D6.)
12. **#46** remainder of the hardening list after the items above are split out.
13. **#45** evaluate Berget as the reconciler: a question, not work. (D8.)
## DECISION NEEDED (owner's call; recommendation first)
| # | Question | Recommendation | Blocks |
|---|----------|----------------|--------|
| D1 | #41: should any linter be allowed to opt out of the gate? | No opt-out; fix findings | #41 |
| D2 | #50: full SIGTERM drain, or record as a limitation and ship the interim fix? | Limitation + interim; a drain costs 4-6 h and downtime on every deploy at current volume | #50 |
| D3 | #37: delete of a queued/transcribing recording | Refuse until it ends | #37 |
| D4 | #36: make a 31B model the chat default? | Only after measurement, conditional | #36 |
| D5 | #38: Berget length measurement needs a throwaway key | Yes, throwaway key | #38 |
| D6 | #38: load large models on production iguana | Yes, in a quiet window with the owner present | #38 |
| D7 | #46: where the ffmpeg tarball lives | Do nothing now; file an issue | none |
| D8 | #45: evaluate Berget as the reconciler | No; stay local-only | #45 |
| D9 | Who judges ADR-034's second blind check | The owner; do not reuse the one-off Opus authorisation | ADR-034 |
| D10 | When to enable reconcile in production | Needs `iguana/pianissimo` in the key pin and env vars; owner present | ADR-034 |
## Dropped or reduced, with reasons
- **#46 item 1 (`alis`/`dref` external reference):** checked, not a problem. Under the application's
flags ffmpeg skips the external track and writes a 78-byte WAV.
- **#46 item 2 / #47 item 2:** merged into #49.
- **#47 item 3 (full drain):** reduced to #50 plus a recorded limitation.
- **#46 `Beat` taking the tracker mutex:** incorrect in #46; only `logStill` does. Corrected in #51.
## Human-present items (do not run unattended)
- Production iguana model loads (#38, D6).
- Enabling reconcile in production (D10).
- Any key change or `apply` in infra (infra#523, infra#524 are open and not part of this sprint).
## Release plan
SemVer: v0.31.1 patch (fixes, no new behaviour); v0.32.0 minor (nav link, delete, cloud labelling).
Every push to tapir main redeploys the pod and fails any job in flight: push when no upload is
running (check the pod log first).
-42
View File
@@ -214,45 +214,3 @@ Feature: Upload an audio recording, get a private Swedish transcript, discuss it
When the model changes a word
Then the segment containing it carries the reconciled word
And the segment's timing is unchanged
# ADR-035 decision 8: hardening from the adversarial review
Scenario: Chat about a recording is offered with local models only
When I open the chat for my recording
Then the model list holds only models that run on the homelab
And a cloud model the shared chat service knows is not in it
Scenario: A forged cloud model is never sent the transcript
When a chat request names a cloud model
Then a local model answers instead, and nothing was sent to the cloud one
Scenario: A wedged engine does not use up the whole job
When the primary engine never answers
Then it fails at its own deadline while the job is still alive
And the fallback runs, unless the job itself was cancelled or timed out
Scenario: A file that lies about its length is refused
When a file claims to be short but converts to more than the limit
Then the job fails with a plain explanation and no engine receives the audio
Scenario: A recording much shorter than its file claimed is transcribed and noted
When a file converts to under 80 percent of the length it claimed and more than two minutes short
Then it is transcribed anyway and its audio step says how long the audio really is
And a complete recording whose header over-estimated its length is not refused
Scenario: A full queue is a retry, not an error
When too many recordings are already waiting
Then the upload is answered as busy, its row is failed honestly and its audio is deleted
Scenario: One user cannot fill the queue for everyone
When one user has several recordings waiting
Then further uploads from that user are refused and other users are not affected
Scenario: Errors are shown as kinds, never their text
When a model or an engine fails
Then the log, the stored reason and the page carry a kind such as timeout or http_502
And never the provider's response, a file path or a word from the transcript
Scenario: Input that cannot be right is a client error
When the id is not an id, or a header or title holds control bytes
Then the answer is a 4xx or the title is cleaned, never a server error
+3 -22
View File
@@ -5,7 +5,6 @@ import (
"fmt"
"os"
"os/exec"
"strconv"
"strings"
)
@@ -18,17 +17,8 @@ import (
type FFmpeg struct {
// Bin is the ffmpeg executable ("ffmpeg" resolves via PATH).
Bin string
// MaxSeconds caps the converted length when positive. The duration limit is a claim
// in the file's header and can be lied about (a 2 h file claiming 10 s converted to a
// 120-minute WAV); this bounds what a lie can cost. A minute of slack is added so a
// file at the limit is never cut, and the worker measures the result.
MaxSeconds int
}
// lengthSlackSeconds is added to MaxSeconds so ffmpeg's rounding never clips a file that
// is really at the limit.
const lengthSlackSeconds = 60
// ToWAV writes src, converted, to dst. Video streams are dropped (-vn): by now the
// file has already been probed as audio-only, but a cover-art stream must not end
// up in the WAV.
@@ -39,23 +29,14 @@ func (f *FFmpeg) ToWAV(ctx context.Context, src, dst string) error {
}
// Input restrictions go BEFORE -i: placed after it they would apply to the output.
args := append([]string{"-y", "-v", "error"}, inputRestrictions...)
args = append(args, "-i", src, "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le")
if f.MaxSeconds > 0 {
// After -i: an OUTPUT cap. Before it, it would limit what is read, not written.
args = append(args, "-t", strconv.Itoa(f.MaxSeconds+lengthSlackSeconds))
}
args = append(args, "-f", "wav", dst)
args = append(args, "-i", src, "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le", "-f", "wav", dst)
cmd := exec.CommandContext(ctx, bin, args...)
cmd.WaitDelay = killGrace
// Only a bounded head of ffmpeg's output is kept: a hostile file made it print ~0.4
// bytes per input byte, all of which CombinedOutput would buffer and copy (#49).
out := &headWriter{max: maxStderr}
cmd.Stdout, cmd.Stderr = out, out
if err := cmd.Run(); err != nil {
if out, err := cmd.CombinedOutput(); err != nil {
if ctxErr := ctx.Err(); ctxErr != nil {
return fmt.Errorf("audio: ffmpeg: %w", ctxErr)
}
return fmt.Errorf("audio: ffmpeg: %w: %s", err, strings.TrimSpace(out.buf.String()))
return fmt.Errorf("audio: ffmpeg: %w: %s", err, strings.TrimSpace(string(out)))
}
// A zero exit that wrote nothing is still a failure: never hand whisper a
// missing file and let the error surface three layers later.
-61
View File
@@ -77,64 +77,3 @@ printf data > "$last"`)
assert.Less(t, pw, in, "input options must come BEFORE -i or ffmpeg applies them to the output")
assert.Less(t, fw, in)
}
func TestFFmpegToWAV_PassesAnOutputLengthCapAfterTheInputWhenAMaximumIsSet(t *testing.T) {
// The duration limit is a CLAIM in the file's header; a file that lies about it can
// convert to hours of audio (adversarial review F2: a 2 h file claiming 10 s became a
// 120-minute WAV). The cap is an OUTPUT option, after -i, with a minute of slack.
rec := filepath.Join(t.TempDir(), "argv")
bin := stub(t, `printf '%s\n' "$@" > `+rec+`
for a; do last="$a"; done
printf data > "$last"`)
err := (&audio.FFmpeg{Bin: bin, MaxSeconds: 5400}).ToWAV(context.Background(), "/spool/in", filepath.Join(t.TempDir(), "out.wav"))
require.NoError(t, err)
raw, err := os.ReadFile(rec)
require.NoError(t, err)
joined := " " + strings.Join(strings.Split(strings.TrimSpace(string(raw)), "\n"), " ") + " "
assert.Contains(t, joined, " -t 5460 ", "limit plus a minute of slack")
assert.Less(t, strings.Index(joined, " -i /spool/in "), strings.Index(joined, " -t 5460 "), "-t after -i is an output cap; before it, an input one")
}
func TestFFmpegToWAV_PassesNoLengthCapWhenNoMaximumIsSet(t *testing.T) {
rec := filepath.Join(t.TempDir(), "argv")
bin := stub(t, `printf '%s\n' "$@" > `+rec+`
for a; do last="$a"; done
printf data > "$last"`)
require.NoError(t, (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/spool/in", filepath.Join(t.TempDir(), "out.wav")))
raw, err := os.ReadFile(rec)
require.NoError(t, err)
assert.NotContains(t, " "+strings.ReplaceAll(string(raw), "\n", " ")+" ", " -t ")
}
func TestFFmpegToWAV_CapsTheStreamCountBeforeTheInput(t *testing.T) {
// ffmpeg also allocated ~570 MB opening the 900-stream file (#49).
rec := filepath.Join(t.TempDir(), "argv")
bin := stub(t, `printf '%s\n' "$@" > `+rec+`
for a; do last="$a"; done
printf data > "$last"`)
err := (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/spool/in", filepath.Join(t.TempDir(), "o.wav"))
require.NoError(t, err)
args := argvOf(t, rec)
ms, in := indexOf(args, "-max_streams"), indexOf(args, "-i")
require.GreaterOrEqual(t, ms, 0, "argv: %v", args)
assert.Equal(t, "32", args[ms+1])
assert.Less(t, ms, in, "an input option placed after -i is applied to the output")
}
func TestFFmpegToWAV_ABoundedPartOfAFloodingStderrIsKept(t *testing.T) {
// A hostile mp3 made ffmpeg print ~0.4 bytes of stderr per input byte (83 MB for a
// 209 MB file), all of it buffered, then copied twice into the error (#49 review).
bin := stub(t, `head -c 5000000 /dev/zero | tr '\0' 'e' >&2; exit 1`)
err := (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/spool/in", filepath.Join(t.TempDir(), "o.wav"))
require.Error(t, err)
assert.Less(t, len(err.Error()), 16*1024, "the error carries a bounded head of stderr, not all of it")
assert.Contains(t, err.Error(), "eeee", "and still carries some of it")
}
+9 -85
View File
@@ -4,7 +4,6 @@
package audio
import (
"bytes"
"context"
"encoding/json"
"errors"
@@ -20,51 +19,6 @@ import (
type FFprobe struct {
// Bin is the ffprobe executable ("ffprobe" resolves via PATH).
Bin string
// MaxOutput bounds how much of ffprobe's stdout is read; zero means DefaultMaxOutput.
MaxOutput int
// Timeout bounds one probe; zero means DefaultProbeTimeout. A probe is milliseconds of
// work on a real file, so one that runs this long is the FILE's doing (#49).
Timeout time.Duration
}
// DefaultProbeTimeout is the probe deadline when FFprobe.Timeout is unset.
const DefaultProbeTimeout = 45 * time.Second
// DefaultMaxOutput is far above the narrowed probe output of a legitimate file (a few
// hundred bytes) and far below what a hostile one can make ffprobe print.
const DefaultMaxOutput = 1 << 20
// maxStderr is how much of ffprobe's stderr is kept for the error message.
const maxStderr = 4096
// errOutputCap is the write error that stops ffprobe once it prints past the cap.
var errOutputCap = errors.New("audio: ffprobe output exceeds the cap")
// capWriter collects up to max bytes and then refuses further writes, which closes the
// pipe and ends the process instead of buffering whatever it prints.
type capWriter struct {
buf bytes.Buffer
max int
}
func (w *capWriter) Write(p []byte) (int, error) {
if w.buf.Len()+len(p) > w.max {
return 0, errOutputCap
}
return w.buf.Write(p)
}
// headWriter keeps the first max bytes and silently drops the rest.
type headWriter struct {
buf bytes.Buffer
max int
}
func (w *headWriter) Write(p []byte) (int, error) {
if room := w.max - w.buf.Len(); room > 0 {
w.buf.Write(p[:min(room, len(p))])
}
return len(p), nil
}
// AllowedDemuxers are the input formats an upload may be. The input is
@@ -76,22 +30,9 @@ func (w *headWriter) Write(p []byte) (int, error) {
// formats. "mov" covers .mp4/.m4a/.3gp; "matroska" covers .webm.
const AllowedDemuxers = "wav,mov,mp4,m4a,3gp,mp3,ogg,matroska,webm,flac,aac,opus,aiff,amr,asf"
// MaxStreams bounds how many streams a file may declare. A 1.3 MB file declaring 900 of
// them cost ~570 MB of RSS in each of ffprobe and ffmpeg (#49), against a 512 Mi pod;
// real recordings have a handful.
const MaxStreams = "32"
// inputRestrictions are ffmpeg/ffprobe INPUT options and must precede the input.
// Local files and pipes only: no network, no other protocols, and a bounded stream count.
var inputRestrictions = []string{
"-protocol_whitelist", "file,pipe",
"-format_whitelist", AllowedDemuxers,
"-max_streams", MaxStreams,
}
// probeEntries is exactly what ParseProbe reads. -show_format/-show_streams print every
// tag too, and a 120 MB WAV made 491 MB of JSON that Output() buffers.
const probeEntries = "format=duration:stream=codec_type:stream_disposition=attached_pic"
// Local files and pipes only: no network, no other protocols.
var inputRestrictions = []string{"-protocol_whitelist", "file,pipe", "-format_whitelist", AllowedDemuxers}
// killGrace bounds how long Probe waits for a cancelled ffprobe's pipes to close.
const killGrace = 2 * time.Second
@@ -107,38 +48,21 @@ func (f *FFprobe) Probe(ctx context.Context, path string) (uploads.Probe, error)
bin = "ffprobe"
}
args := append([]string{"-v", "error"}, inputRestrictions...)
args = append(args, "-print_format", "json", "-show_entries", probeEntries, path)
timeout := f.Timeout
if timeout <= 0 {
timeout = DefaultProbeTimeout
}
probeCtx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
cmd := exec.CommandContext(probeCtx, bin, args...)
args = append(args, "-print_format", "json", "-show_format", "-show_streams", path)
cmd := exec.CommandContext(ctx, bin, args...)
cmd.WaitDelay = killGrace
maxOut := f.MaxOutput
if maxOut <= 0 {
maxOut = DefaultMaxOutput
}
stdout, stderr := &capWriter{max: maxOut}, &headWriter{max: maxStderr}
cmd.Stdout, cmd.Stderr = stdout, stderr
if err := cmd.Run(); err != nil {
if ctxErr := ctx.Err(); ctxErr != nil { // the caller gave up: not a verdict on the file
out, err := cmd.Output()
if err != nil {
if ctxErr := ctx.Err(); ctxErr != nil {
return uploads.Probe{}, fmt.Errorf("audio: ffprobe: %w", ctxErr)
}
if probeCtx.Err() != nil { // our own deadline passed while the caller was still waiting
return uploads.Probe{}, fmt.Errorf("%w: ffprobe did not finish within %s", uploads.ErrUnreadable, timeout)
}
if errors.Is(err, errOutputCap) {
return uploads.Probe{}, fmt.Errorf("%w: ffprobe printed more than %d bytes", uploads.ErrUnreadable, maxOut)
}
var exit *exec.ExitError
if errors.As(err, &exit) {
return uploads.Probe{}, fmt.Errorf("%w: ffprobe: %s", uploads.ErrUnreadable, stderr.buf.String())
return uploads.Probe{}, fmt.Errorf("%w: ffprobe: %s", uploads.ErrUnreadable, exit.Stderr)
}
return uploads.Probe{}, fmt.Errorf("audio: run ffprobe: %w", err)
}
p, err := ParseProbe(stdout.buf.Bytes())
p, err := ParseProbe(out)
if err != nil {
return uploads.Probe{}, fmt.Errorf("audio: ffprobe output: %w", err)
}
-75
View File
@@ -184,78 +184,3 @@ func TestAllowedDemuxers_ExcludeThePlaylistFormats(t *testing.T) {
assert.Contains(t, strings.Split(audio.AllowedDemuxers, ","), good)
}
}
func TestFFprobeProbe_CapsTheStreamCountBeforeTheInput(t *testing.T) {
// A 1.3 MB file declaring 900 streams cost ~570 MB of RSS in ffprobe (#49) against
// a 512 Mi pod. -max_streams is an input option: it must precede the path.
bin, argvFile := argvStub(t)
_, err := (&audio.FFprobe{Bin: bin}).Probe(context.Background(), "/spool/upload-1")
require.NoError(t, err)
args := argvOf(t, argvFile)
ms := indexOf(args, "-max_streams")
require.GreaterOrEqual(t, ms, 0, "argv: %v", args)
assert.Equal(t, "32", args[ms+1])
assert.Less(t, ms, len(args)-1, "must come before the path, which stays last")
}
func TestFFprobeProbe_AsksOnlyForTheFieldsParseProbeReads(t *testing.T) {
// -show_format/-show_streams print every tag: a 120 MB WAV produced 491 MB of JSON
// that cmd.Output() buffers in Go (#49).
bin, argvFile := argvStub(t)
_, err := (&audio.FFprobe{Bin: bin}).Probe(context.Background(), "/spool/upload-1")
require.NoError(t, err)
args := argvOf(t, argvFile)
assert.Equal(t, -1, indexOf(args, "-show_format"), "argv: %v", args)
assert.Equal(t, -1, indexOf(args, "-show_streams"), "argv: %v", args)
se := indexOf(args, "-show_entries")
require.GreaterOrEqual(t, se, 0, "argv: %v", args)
assert.Equal(t, "format=duration:stream=codec_type:stream_disposition=attached_pic", args[se+1])
}
func TestFFprobeProbe_OutputPastTheCapIsUnreadableNotBuffered(t *testing.T) {
// ffprobe's stdout is attacker-influenced (tags, 491 MB observed). Past the cap the
// probe is stopped and the file is reported as unreadable (#49).
bin := stub(t, `head -c 200000 /dev/zero | tr '\0' 'a'`)
_, err := (&audio.FFprobe{Bin: bin, MaxOutput: 4096}).Probe(context.Background(), "/x")
assert.ErrorIs(t, err, uploads.ErrUnreadable)
}
func TestFFprobeProbe_OutputWithinTheCapIsParsed(t *testing.T) {
bin := stub(t, `echo '{"streams":[{"codec_type":"audio"}],"format":{"duration":"90"}}'`)
got, err := (&audio.FFprobe{Bin: bin, MaxOutput: 4096}).Probe(context.Background(), "/x")
require.NoError(t, err)
assert.True(t, got.HasAudio)
}
func TestFFprobeProbe_AHungProbeTimesOutOnItsOwnDeadlineAsUnreadableNotAsOurFault(t *testing.T) {
// Accept runs the probe on the request context, which has no deadline of its own;
// a probe that never returns would pin a request (#49). `exec` so the sleep is the
// process itself and not a child holding the pipe open.
bin := stub(t, `exec sleep 30`)
start := time.Now()
_, err := (&audio.FFprobe{Bin: bin, Timeout: 200 * time.Millisecond}).Probe(context.Background(), "/x")
assert.ErrorIs(t, err, uploads.ErrUnreadable)
assert.Less(t, time.Since(start), 3*time.Second)
}
func TestFFprobeProbe_ParentCancellationIsStillNotUnreadableWhenATimeoutIsSet(t *testing.T) {
bin := stub(t, `exec sleep 30`)
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
_, err := (&audio.FFprobe{Bin: bin, Timeout: time.Minute}).Probe(ctx, "/x")
require.Error(t, err)
assert.False(t, errors.Is(err, uploads.ErrUnreadable), "the caller giving up is not a verdict on the file")
assert.ErrorIs(t, err, context.DeadlineExceeded)
}
+1 -24
View File
@@ -55,29 +55,6 @@ func WithMaxTokens(n int) Option {
}
}
// maxErrorBody bounds how much of a failing response ends up in an error message.
const maxErrorBody = 512
// HTTPError is a non-200 answer from the gateway. Its text keeps the form callers that
// log it already rely on, with the body bounded; its Code is what code may branch on and
// what errkind turns into a kind. The body is provider text and can echo the prompt, so
// nothing private may persist or display Error() (ADR-035 review F3).
type HTTPError struct {
Code int
Body string
}
func (e *HTTPError) Error() string {
body := e.Body
if len(body) > maxErrorBody {
body = body[:maxErrorBody] + "…"
}
return fmt.Sprintf("LLM returned %d: %s", e.Code, body)
}
// HTTPStatus implements errkind.HTTPStatuser.
func (e *HTTPError) HTTPStatus() int { return e.Code }
// WithTemperature overrides the sampling temperature. Zero IS a valid value and is sent
// as zero: the reconciler's accuracy was measured at temperature 0 and must run there.
func WithTemperature(t float64) Option {
@@ -189,7 +166,7 @@ func (c *Client) Complete(ctx context.Context, system, user string) (string, err
return "", fmt.Errorf("read response: %w", err)
}
if resp.StatusCode != http.StatusOK {
return "", &HTTPError{Code: resp.StatusCode, Body: string(out)}
return "", fmt.Errorf("LLM returned %d: %s", resp.StatusCode, out)
}
var cr chatResponse
@@ -1,63 +0,0 @@
package llm
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"git.d-ma.be/mathias/tapir/internal/errkind"
)
// A non-200 from the gateway is an HTTP status error: its kind can be logged and stored
// without its body, which can echo the prompt (adversarial review F3).
func failingGateway(t *testing.T, status int, body string) *httptest.Server {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(status)
_, _ = w.Write([]byte(body))
}))
t.Cleanup(srv.Close)
return srv
}
func TestClient_ANon200IsAnHTTPStatusErrorWhoseKindCarriesNoBody(t *testing.T) {
srv := failingGateway(t, http.StatusBadRequest, `{"error":"prompt was: Anders ansvarar för virket"}`)
_, err := New(srv.URL, "", "m", 5*time.Second).Complete(context.Background(), "s", "u")
if err == nil {
t.Fatal("expected an error")
}
if got := errkind.Of(err); got != "http_400" {
t.Errorf("errkind.Of = %q, want http_400", got)
}
var he *HTTPError
if !errors.As(err, &he) || he.Code != http.StatusBadRequest {
t.Errorf("error is not an *HTTPError with the status code: %v", err)
}
}
func TestClient_TheErrorTextIsUnchangedForCallersThatLogIt(t *testing.T) {
srv := failingGateway(t, http.StatusBadGateway, "upstream down")
_, err := New(srv.URL, "", "m", 5*time.Second).Complete(context.Background(), "s", "u")
if err == nil || err.Error() != "LLM returned 502: upstream down" {
t.Errorf("error text = %v: callers that log it already rely on this form", err)
}
}
func TestClient_AnEnormousErrorBodyIsBoundedInTheErrorText(t *testing.T) {
srv := failingGateway(t, http.StatusInternalServerError, strings.Repeat("x", 100_000))
_, err := New(srv.URL, "", "m", 5*time.Second).Complete(context.Background(), "s", "u")
if err == nil || len(err.Error()) > 1024 {
t.Errorf("error text is %d bytes: a gateway's whole body must not become an error message", len(err.Error()))
}
}
@@ -88,16 +88,12 @@ func (s *Store) Heartbeat(ctx context.Context, userID, id string) error {
WHERE id = $1 AND user_id = $2`)
}
// FailUpload moves a queued or transcribing upload to failed with a typed reason. A
// finished upload is left as it was: a shutdown can record a job as failed after its
// transcript was already committed (the commit lands, the caller sees a cancelled
// context), and that must not hide the text. Such a call touches no row and returns
// ErrUploadNotFound.
// FailUpload moves an upload to failed with a typed reason.
func (s *Store) FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error {
return s.updateUpload(ctx, userID, id, "fail upload",
`UPDATE uploaded_transcripts
SET status = 'failed', error_kind = $3, finished_at = NOW()
WHERE id = $1 AND user_id = $2 AND status IN ('queued', 'transcribing')`,
WHERE id = $1 AND user_id = $2`,
string(kind))
}
@@ -1,47 +0,0 @@
package store_test
// A shutdown can record a job as failed after its transcript was already committed (the
// commit lands, the caller sees a cancelled context). FailUpload must not be able to turn a
// finished row into a failed one and hide the text (#50 review).
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
)
func TestFailUpload_LeavesAFinishedUploadAsItWas(t *testing.T) {
s := newStore(t)
ctx := context.Background()
id := startedUpload(t, s)
require.NoError(t, s.StartTranscribing(ctx, userA, id))
require.NoError(t, s.CompleteUpload(ctx, userA, id, domain.Transcription{Text: "synthetic text", Endpoint: "iguana/kb-whisper"}))
_ = s.FailUpload(ctx, userA, id, domain.UploadErrInterrupted)
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, domain.UploadDone, got.Status)
assert.Equal(t, "synthetic text", got.Content)
assert.Empty(t, got.ErrorKind)
}
func TestFailUpload_StillFailsAQueuedOrTranscribingUpload(t *testing.T) {
s := newStore(t)
ctx := context.Background()
queued := startedUpload(t, s)
running, err := s.CreateUpload(ctx, userA, "r", false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, running))
for _, id := range []string{queued, running} {
require.NoError(t, s.FailUpload(ctx, userA, id, domain.UploadErrBusy))
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, domain.UploadFailed, got.Status)
}
}
@@ -242,8 +242,8 @@ func TestUploadLifecycle_ReconciliationRoundTripsWithItsContestedSpots(t *testin
Status: domain.ReconcileDone, Base: "iguana/pianissimo", Alt: "iguana/kb-whisper", Reconciler: "iguana/gemma4-26b",
Disputes: 380, AutoResolved: 90, Adjudicated: 285, Changed: 120, Unsure: 5, Errors: 0,
Contested: []domain.ContestedSpot{
{Seconds: 2210.5, Base: "Axel", Alt: "Alex", Used: "alt"},
{Seconds: 3400, Base: "A+2", Alt: "en A", Used: "base", Unsure: true},
{Seconds: 2210.5, Base: "SEB", Alt: "SCB", Used: "alt"},
{Seconds: 3400, Base: "T+1", Alt: "en T", Used: "base", Unsure: true},
},
}
+2 -3
View File
@@ -6,7 +6,6 @@ import (
"fmt"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/errkind"
"git.d-ma.be/mathias/tapir/internal/progress"
"git.d-ma.be/mathias/tapir/internal/reconcile"
)
@@ -73,9 +72,9 @@ func (e *Ensemble) Transcribe(ctx context.Context, wavPath string) (domain.Trans
}
return out, nil
case berr == nil:
return e.single(b, e.base.Name(), fmt.Sprintf("%s failed (%s)", e.alt.Name(), errkind.Of(aerr))), nil
return e.single(b, e.base.Name(), fmt.Sprintf("%s failed: %v", e.alt.Name(), aerr)), nil
case aerr == nil:
return e.single(a, e.alt.Name(), fmt.Sprintf("%s failed (%s)", e.base.Name(), errkind.Of(berr))), nil
return e.single(a, e.alt.Name(), fmt.Sprintf("%s failed: %v", e.base.Name(), berr)), nil
}
both := errors.Join(fmt.Errorf("%s: %w", e.base.Name(), berr), fmt.Errorf("%s: %w", e.alt.Name(), aerr))
@@ -1,36 +0,0 @@
package transcribe_test
import (
"context"
"errors"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
"git.d-ma.be/mathias/tapir/internal/reconcile"
)
// When one engine is down the page says so and why; the "why" is a kind, not the
// engine's error text, which can carry provider output (adversarial review F3).
func TestEnsemble_AFailedEnginesReasonIsAKindNotItsErrorText(t *testing.T) {
r := newEnsemble("text", "text", nil, errors.New("kb-whisper said: zebrakaka secret words"), false, &verdictAdj{v: reconcile.VerdictAlt})
got, err := r.e.Transcribe(context.Background(), "/a.wav")
require.NoError(t, err)
assert.NotContains(t, got.Reconciliation.Reason, "zebrakaka")
assert.Contains(t, got.Reconciliation.Reason, "iguana/kb-whisper failed", "which engine failed is still said")
}
func TestEnsemble_AnHTTPFailureShowsItsStatusCodeAsTheKind(t *testing.T) {
r := newEnsemble("text", "text", &transcribe.StatusError{Code: 502, Snippet: "zebrakaka"}, nil, false, &verdictAdj{})
got, err := r.e.Transcribe(context.Background(), "/a.wav")
require.NoError(t, err)
assert.Contains(t, got.Reconciliation.Reason, "http_502")
assert.NotContains(t, got.Reconciliation.Reason, "zebrakaka")
}
-3
View File
@@ -31,9 +31,6 @@ type StatusError struct {
Snippet string
}
// HTTPStatus implements errkind.HTTPStatuser: the code, never the snippet.
func (e *StatusError) HTTPStatus() int { return e.Code }
func (e *StatusError) Error() string {
return fmt.Sprintf("transcribe: gateway returned HTTP %d: %s", e.Code, e.Snippet)
}
@@ -3,13 +3,11 @@ package transcribe_test
import (
"bytes"
"context"
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
@@ -17,7 +15,6 @@ import (
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/errkind"
)
// The httptest server stands in for the LiteLLM gateway: it is the process
@@ -233,25 +230,3 @@ func TestGateway_WithNoSegmentsTheTextFieldIsUsedWithWhitespaceCollapsed(t *test
require.NoError(t, err)
assert.Equal(t, "Hej allihopa. Vi börjar.", got.Text)
}
func TestStatusError_ExposesItsCodeAsAKindAndNotItsSnippet(t *testing.T) {
err := fmt.Errorf("transcribe: %w", &transcribe.StatusError{Code: 502, Snippet: "echoed: Anders ansvarar"})
assert.Equal(t, "http_502", errkind.Of(err))
}
func TestGateway_AnErrorBodyIsBoundedToTwoHundredBytesInTheStatusError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
_, _ = w.Write([]byte(strings.Repeat("x", 100_000)))
}))
defer srv.Close()
wav := filepath.Join(t.TempDir(), "a.wav")
require.NoError(t, os.WriteFile(wav, []byte("RIFF....WAVEfmt "), 0o600))
_, err := transcribe.NewGateway(srv.URL, "", "iguana/kb-whisper", srv.Client()).Transcribe(context.Background(), wav)
var se *transcribe.StatusError
require.ErrorAs(t, err, &se)
assert.LessOrEqual(t, len(se.Snippet), 200, "a gateway's whole body must not become part of an error")
}
-33
View File
@@ -1,33 +0,0 @@
package transcribe
import (
"context"
"time"
"git.d-ma.be/mathias/tapir/internal/domain"
)
// WithTimeout gives every call to b its own deadline. A wedged engine does not answer at
// all (iguana after a Metal OOM, infra#273), and with only the job's deadline it used up
// the whole job before a fallback could run -- and Chain then refuses the fallback because
// the job context has expired. With its own deadline the call fails while the job is still
// alive, which is exactly when the fallback, or the other engine, can take over.
//
// A zero or negative d returns b unchanged. The wrapper is invisible: Name is b's.
func WithTimeout(b Backend, d time.Duration) Backend {
if d <= 0 {
return b
}
return timeoutBackend{Backend: b, d: d}
}
type timeoutBackend struct {
Backend
d time.Duration
}
func (t timeoutBackend) Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error) {
ctx, cancel := context.WithTimeout(ctx, t.d)
defer cancel()
return t.Backend.Transcribe(ctx, wavPath)
}
@@ -1,104 +0,0 @@
package transcribe_test
import (
"context"
"errors"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/errkind"
"git.d-ma.be/mathias/tapir/internal/reconcile"
)
// A wedged engine (iguana after a Metal OOM, infra#273) does not answer at all. With only
// the job's deadline, the primary used up the whole job before any fallback could run, and
// the chain then refused the fallback because the job context had expired. Each engine call
// gets its own deadline (adversarial review F4).
type hangingBackend struct{ name string }
func (h hangingBackend) Name() string { return h.name }
func (hangingBackend) Transcribe(ctx context.Context, _ string) (domain.Transcription, error) {
<-ctx.Done()
return domain.Transcription{}, ctx.Err()
}
func TestWithTimeout_AHungBackendFailsAtItsOwnDeadlineWhileTheJobIsStillAlive(t *testing.T) {
job, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
b := transcribe.WithTimeout(hangingBackend{name: "iguana/kb-whisper"}, 40*time.Millisecond)
start := time.Now()
_, err := b.Transcribe(job, "/a.wav")
require.Error(t, err)
assert.Equal(t, "timeout", errkind.Of(err))
assert.Less(t, time.Since(start), 2*time.Second, "it must give up at its own deadline, not wait for the job's")
assert.NoError(t, job.Err(), "the job context is untouched, which is what lets a fallback run")
assert.Equal(t, "iguana/kb-whisper", b.Name(), "the wrapper is invisible in the recorded endpoint")
}
func TestWithTimeout_ASuccessPassesStraightThrough(t *testing.T) {
calls := &[]string{}
inner := &orderedBackend{name: "iguana/kb-whisper", text: "hej", calls: calls}
got, err := transcribe.WithTimeout(inner, time.Minute).Transcribe(context.Background(), "/a.wav")
require.NoError(t, err)
assert.Equal(t, "hej", got.Text)
}
func TestWithTimeout_ZeroMeansNoDeadlineOfItsOwn(t *testing.T) {
inner := &orderedBackend{name: "x", text: "hej", calls: &[]string{}}
assert.Same(t, inner, transcribe.WithTimeout(inner, 0), "a zero timeout must not wrap or change anything")
}
func TestChain_AHungPrimaryReachesTheFallbackWhenItHasItsOwnDeadline(t *testing.T) {
calls := &[]string{}
fb := &orderedBackend{name: "berget/kb-whisper", text: "fallback text", calls: calls}
c := transcribe.NewChain(transcribe.WithTimeout(hangingBackend{name: "iguana/kb-whisper"}, 40*time.Millisecond), fb)
job, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
got, err := c.Transcribe(job, "/a.wav")
require.NoError(t, err, "a wedged primary is exactly what the fallback exists for")
assert.Equal(t, "berget/kb-whisper", got.Endpoint)
}
func TestChain_AJobThatEndedWhileThePrimaryHungStillNeverReachesTheFallback(t *testing.T) {
calls := &[]string{}
fb := &orderedBackend{name: "berget/kb-whisper", text: "fallback text", calls: calls}
c := transcribe.NewChain(transcribe.WithTimeout(hangingBackend{name: "iguana/kb-whisper"}, time.Minute), fb)
job, cancel := context.WithTimeout(context.Background(), 40*time.Millisecond)
defer cancel()
_, err := c.Transcribe(job, "/a.wav")
require.Error(t, err)
assert.Empty(t, *calls, "the job's own deadline or a cancel must still stop the audio from reaching a cloud endpoint")
}
func TestEnsemble_AHungEngineIsASoftFailureAndTheOtherEnginesTextIsUsed(t *testing.T) {
calls := &[]string{}
alt := &orderedBackend{name: "iguana/kb-whisper", text: "text from the other engine", calls: calls}
e := transcribe.NewEnsemble(
transcribe.WithTimeout(hangingBackend{name: "iguana/pianissimo"}, 40*time.Millisecond), alt, nil,
&verdictAdj{v: reconcile.VerdictAlt}, ensembleOpts)
job, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
got, err := e.Transcribe(job, "/a.wav")
require.NoError(t, err)
assert.Equal(t, "text from the other engine", got.Text)
assert.Equal(t, domain.ReconcileSingle, got.Reconciliation.Status)
assert.Contains(t, got.Reconciliation.Reason, "timeout")
assert.False(t, errors.Is(err, context.DeadlineExceeded))
}
+12 -30
View File
@@ -3,7 +3,6 @@ package config
import (
"fmt"
"os"
"regexp"
"strconv"
"strings"
"time"
@@ -28,10 +27,6 @@ type UploadConfig struct {
// TranscribeTimeout bounds one job (conversion + transcription); a 90 minute
// recording took ~7.5 minutes on iguana.
TranscribeTimeout time.Duration
// TranscribeEngineTimeout bounds ONE engine call, so a wedged engine fails while the
// job is still alive and the fallback (or the other engine) can take over. The job
// timeout alone cannot do that (ADR-035 review).
TranscribeEngineTimeout time.Duration
// UploadTmpDir is where uploads are spooled and converted. Audio is deleted
// after transcription, so this holds at most the in-flight files.
UploadTmpDir string
@@ -66,7 +61,6 @@ const (
defaultTranscribeModel = "iguana/kb-whisper"
defaultTranscribeFallbackModel = "berget/kb-whisper"
defaultTranscribeTimeout = 60 * time.Minute
defaultTranscribeEngineTimeout = 20 * time.Minute
defaultMaxUploadedChars = 90000
defaultUploadChatModel = "iguana/gemma4-26b"
)
@@ -112,15 +106,6 @@ func loadUploadConfig() (UploadConfig, error) {
}
u.TranscribeTimeout = timeout
engineTimeout, err := durationOr("TAPIR_TRANSCRIBE_ENGINE_TIMEOUT", defaultTranscribeEngineTimeout)
if err != nil {
return UploadConfig{}, err
}
if engineTimeout <= 0 {
return UploadConfig{}, fmt.Errorf("config: TAPIR_TRANSCRIBE_ENGINE_TIMEOUT must be positive, got %v", engineTimeout)
}
u.TranscribeEngineTimeout = engineTimeout
if v := os.Getenv("TAPIR_RECONCILE_CHECK_ORDER"); v != "" {
on, err := strconv.ParseBool(v)
if err != nil {
@@ -131,9 +116,6 @@ func loadUploadConfig() (UploadConfig, error) {
if err := u.validateReconcile(); err != nil {
return UploadConfig{}, err
}
if !IsLocalModel(u.UploadChatModel) {
return UploadConfig{}, fmt.Errorf("config: TAPIR_UPLOAD_CHAT_MODEL=%q is not a local model (koala/ or iguana/): the whole private transcript is sent to it", u.UploadChatModel)
}
chars, err := intOr("TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS", defaultMaxUploadedChars)
if err != nil {
@@ -159,18 +141,18 @@ func positiveInt(key string, fallback int) (int, error) {
return n, nil
}
// localAlias is the shape of a model alias that runs on the homelab: a local host
// prefix, then one plain name. A bare prefix check let through aliases that are shaped
// like an escape (koala/../berget/x, koala//x, koala/x?host=berget): whether the
// gateway would route them elsewhere is unknown, so they are refused (ADR-035 review).
var localAlias = regexp.MustCompile(`^(koala|iguana)/[A-Za-z0-9][A-Za-z0-9._:-]*$`)
// localPrefixes are the host prefixes of the models that run on the homelab. Private
// meeting content may be sent to these and to nothing else for reconciliation
// (ADR-034): a cloud alias is refused at start-up rather than discovered later.
var localPrefixes = []string{"koala/", "iguana/"}
// IsLocalModel reports whether alias is a local model. Private meeting content may be
// sent to these and to nothing else: reconciliation (ADR-034) and chat over an uploaded
// recording (ADR-035) are both held to it, and a cloud alias is refused at start-up
// rather than discovered later.
func IsLocalModel(alias string) bool {
return localAlias.MatchString(alias) && !strings.Contains(alias, "..")
func isLocalModel(alias string) bool {
for _, p := range localPrefixes {
if strings.HasPrefix(alias, p) {
return true
}
}
return false
}
// validateReconcile checks the two-engine settings. They are inert while no base
@@ -186,7 +168,7 @@ func (u UploadConfig) validateReconcile() error {
{"TAPIR_RECONCILE_BASE_MODEL", u.ReconcileBaseModel},
{"TAPIR_RECONCILE_MODEL", u.ReconcileModel},
} {
if !IsLocalModel(m.alias) {
if !isLocalModel(m.alias) {
return fmt.Errorf("config: %s=%q is not a local model (koala/ or iguana/): private meeting content must not leave the homelab for reconciliation", m.name, m.alias)
}
}
-101
View File
@@ -1,101 +0,0 @@
package config
import (
"testing"
"time"
)
// What counts as a LOCAL model alias, and where that is demanded (ADR-034, ADR-035 review).
// A private transcript may be sent to a local model and to nothing else; the check is a
// guard against a misconfiguration or a crafted alias, so it must be strict about shape,
// not just about a prefix.
func TestIsLocalModel_AcceptsRealLocalAliasesAndNothingShapedLikeAnEscape(t *testing.T) {
tests := []struct {
alias string
want bool
}{
{"koala/phi4-mini", true},
{"iguana/gemma4-26b", true},
{"iguana/gemma4-31b-think", true},
{"koala/qwen36-35b-a3b", true},
{"iguana/kb-whisper", true},
{"iguana/hf.co-model_v1:q4", true},
{"", false},
{"koala/", false},
{"iguana/", false},
{"gemma4-26b", false},
{"berget/mistral-small", false},
{"berget/koala/x", false},
{"xkoala/x", false},
{"Koala/x", false},
{" koala/x", false},
{"koala/x ", false},
{"koala/x y", false},
{"koala/x\n", false},
{"koala/../berget/kb", false},
{"koala//berget/mistral-small", false},
{"koala/x/y", false},
{"koala/x?host=berget", false},
{"koala/x#frag", false},
{"koala/x@evil", false},
{"koalа/x", false}, // Cyrillic а
}
for _, tt := range tests {
t.Run(tt.alias, func(t *testing.T) {
if got := IsLocalModel(tt.alias); got != tt.want {
t.Errorf("IsLocalModel(%q) = %v, want %v", tt.alias, got, tt.want)
}
})
}
}
func TestLoad_UploadChatModelMustBeLocalBecauseTheWholeTranscriptIsSentToIt(t *testing.T) {
for _, alias := range []string{"berget/mistral-small", "gemini/gemini-2.5-pro", "gemma4-26b", "koala/../berget/kb"} {
t.Run(alias, func(t *testing.T) {
clearUploadEnv(t)
setEnv(t, map[string]string{"TAPIR_UPLOAD_CHAT_MODEL": alias})
if _, err := Load(); err == nil {
t.Errorf("Load accepted TAPIR_UPLOAD_CHAT_MODEL=%q: a private transcript of up to 90,000 characters would be sent to it", alias)
}
})
}
}
func TestLoad_ALocalUploadChatModelIsAccepted(t *testing.T) {
clearUploadEnv(t)
setEnv(t, map[string]string{"TAPIR_UPLOAD_CHAT_MODEL": "iguana/gemma4-31b"})
cfg, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
if cfg.UploadChatModel != "iguana/gemma4-31b" {
t.Errorf("UploadChatModel = %q", cfg.UploadChatModel)
}
}
func TestLoad_TheEngineTimeoutDefaultsToTwentyMinutesAndMustBePositive(t *testing.T) {
clearUploadEnv(t)
cfg, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
if cfg.TranscribeEngineTimeout != 20*time.Minute {
t.Errorf("TranscribeEngineTimeout = %v, want 20m: a 90-minute recording takes about 8 minutes on kb-whisper, and two wedged engines in sequence must leave the job time to reach a fallback", cfg.TranscribeEngineTimeout)
}
setEnv(t, map[string]string{"TAPIR_TRANSCRIBE_ENGINE_TIMEOUT": "7m"})
cfg, err = Load()
if err != nil || cfg.TranscribeEngineTimeout != 7*time.Minute {
t.Errorf("override: %v, %v", cfg.TranscribeEngineTimeout, err)
}
for _, bad := range []string{"0", "-1m", "soon"} {
setEnv(t, map[string]string{"TAPIR_TRANSCRIBE_ENGINE_TIMEOUT": bad})
if _, err := Load(); err == nil {
t.Errorf("Load accepted TAPIR_TRANSCRIBE_ENGINE_TIMEOUT=%q", bad)
}
}
}
+1 -1
View File
@@ -12,7 +12,7 @@ func clearUploadEnv(t *testing.T) {
t.Helper()
for _, k := range []string{
"TAPIR_UPLOADS_ENABLED", "TAPIR_UPLOAD_MAX_MB", "TAPIR_UPLOAD_MAX_MINUTES",
"TAPIR_TRANSCRIBE_MODEL", "TAPIR_TRANSCRIBE_FALLBACK_MODEL", "TAPIR_TRANSCRIBE_TIMEOUT", "TAPIR_TRANSCRIBE_ENGINE_TIMEOUT",
"TAPIR_TRANSCRIBE_MODEL", "TAPIR_TRANSCRIBE_FALLBACK_MODEL", "TAPIR_TRANSCRIBE_TIMEOUT",
"TAPIR_UPLOAD_TMP_DIR", "TAPIR_FFMPEG_BIN", "TAPIR_FFPROBE_BIN",
"TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS", "TAPIR_UPLOAD_CHAT_MODEL",
"TAPIR_RECONCILE_BASE_MODEL", "TAPIR_RECONCILE_MODEL", "TAPIR_RECONCILE_CHECK_ORDER",
-11
View File
@@ -203,10 +203,6 @@ type ProgressStep struct {
FinishedAt time.Time `json:"finished_at,omitzero"`
Done int `json:"done,omitempty"`
Total int `json:"total,omitempty"`
// Note is a short plain remark about the phase that the reader should see even though
// it did not fail, for example that the audio is much shorter than the file claimed.
// Durations and counts only: never transcript text, a name or a path.
Note string `json:"note,omitempty"`
}
// UploadStatus is the state of an uploaded-audio transcription job.
@@ -230,13 +226,6 @@ const (
UploadErrInterrupted UploadErrorKind = "interrupted"
// UploadErrTranscriptionFailed: every configured transcription endpoint failed.
UploadErrTranscriptionFailed UploadErrorKind = "transcription_failed"
// UploadErrTooLong: the audio, once converted, is longer than the limit. The limit is
// first checked against the file's header, which a hostile or damaged file can lie
// about (ADR-035 review).
UploadErrTooLong UploadErrorKind = "audio_too_long"
// UploadErrBusy: too many recordings were already waiting, so this one was not
// queued. The user's to retry later (ADR-035 review).
UploadErrBusy UploadErrorKind = "queue_full"
)
// UploadedTranscript is a user's PRIVATE transcript of audio they uploaded. It is
-51
View File
@@ -1,51 +0,0 @@
// Package errkind reduces an error to a short kind that is safe to log, store and show.
//
// An error's own text can hold a provider's response body (a gateway that echoes the
// prompt), a file path (ffmpeg names the spool file), or a word from a private
// transcript. None of that may reach a log line, a database column or a page about a
// private recording (ADR-033: "typed, never a provider's string"; ADR-035). Of returns
// only a fixed vocabulary, so it can never carry any.
package errkind
import (
"context"
"errors"
"fmt"
"net"
)
// HTTPStatuser is implemented by errors that are a non-2xx answer from a service.
type HTTPStatuser interface{ HTTPStatus() int }
// SQLStater is implemented by database errors that carry a SQLSTATE (pgx's PgError does).
// The five-character code is a fixed vocabulary; the error's message is not: Postgres quotes
// the offending value, which for a transcript row is transcript text.
type SQLStater interface{ SQLState() string }
// Of is the kind of err: "timeout", "canceled", "http_<code>", "sql_<state>", "network",
// or "error".
// It is empty for nil.
func Of(err error) string {
if err == nil {
return ""
}
var hs HTTPStatuser
var ss SQLStater
var ne net.Error
switch {
case errors.Is(err, context.DeadlineExceeded):
return "timeout"
case errors.Is(err, context.Canceled):
return "canceled"
case errors.As(err, &hs):
return fmt.Sprintf("http_%d", hs.HTTPStatus())
case errors.As(err, &ss) && ss.SQLState() != "":
return "sql_" + ss.SQLState()
case errors.As(err, &ne):
if ne.Timeout() {
return "timeout"
}
return "network"
}
return "error"
}
-82
View File
@@ -1,82 +0,0 @@
package errkind_test
import (
"context"
"errors"
"fmt"
"net"
"net/url"
"testing"
"github.com/stretchr/testify/assert"
"git.d-ma.be/mathias/tapir/internal/errkind"
)
// Of reduces an error to a short kind that is safe to log, store and show: it never
// carries the error's own text, because that text can hold a provider's response body, a
// file path or a word from a private transcript (ADR-035 review F3, F5).
type statusErr struct{ code int }
func (e statusErr) Error() string {
return fmt.Sprintf("gateway said: %d and echoed the transcript", e.code)
}
func (e statusErr) HTTPStatus() int { return e.code }
type timeoutNet struct{}
func (timeoutNet) Error() string { return "i/o timeout to 10.0.0.1" }
func (timeoutNet) Timeout() bool { return true }
func (timeoutNet) Temporary() bool { return true }
func TestOf_ReducesAnErrorToAKindAndNeverItsText(t *testing.T) {
tests := []struct {
name string
err error
want string
}{
{"nothing", nil, ""},
{"a deadline", context.DeadlineExceeded, "timeout"},
{"a wrapped deadline", fmt.Errorf("transcribe: %w", context.DeadlineExceeded), "timeout"},
{"a cancellation", fmt.Errorf("job: %w", context.Canceled), "canceled"},
{"an HTTP status", statusErr{code: 502}, "http_502"},
{"a wrapped HTTP status", fmt.Errorf("model: %w", statusErr{code: 400}), "http_400"},
{"a network timeout", timeoutNet{}, "timeout"},
{"a network failure", &net.OpError{Op: "dial", Err: errors.New("connection refused")}, "network"},
{"a url error wrapping a refusal", &url.Error{Op: "Post", URL: "http://gateway/secret-path", Err: errors.New("refused")}, "network"},
{"anything else", errors.New("moov atom not found in /spool/upload-555"), "error"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.want, errkind.Of(tt.err))
})
}
}
type sqlErr struct{}
func (sqlErr) Error() string {
return `ERROR: invalid input syntax for type json: "Anders ansvarar för virket"`
}
func (sqlErr) SQLState() string { return "22P02" }
func TestOf_ADatabaseErrorIsReducedToItsSQLStateNeverItsMessage(t *testing.T) {
// A Postgres error can quote the offending value, which for a transcript row is
// transcript text. The five-character SQLSTATE is a fixed vocabulary (fourth review).
err := fmt.Errorf("store: complete upload: %w", sqlErr{})
got := errkind.Of(err)
assert.Equal(t, "sql_22P02", got)
assert.NotContains(t, got, "Anders")
}
func TestOf_AKindCarriesNoWordFromTheError(t *testing.T) {
err := fmt.Errorf("convert /spool/upload-555: %w", errors.New("Anders ansvarar för virket"))
got := errkind.Of(err)
assert.NotContains(t, got, "Anders")
assert.NotContains(t, got, "spool")
}
+1 -16
View File
@@ -12,7 +12,6 @@ import (
"time"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/errkind"
)
// Planned is one phase of a job, declared up front so the page can show what is coming.
@@ -147,7 +146,7 @@ func (t *Tracker) Fail(key string, err error) {
return
}
t.close(s, domain.StepFailed)
t.log.Warn("step failed", "step", key, "elapsed", s.FinishedAt.Sub(s.StartedAt), "kind", errkind.Of(err))
t.log.Warn("step failed", "step", key, "elapsed", s.FinishedAt.Sub(s.StartedAt), "err", err)
t.save()
}
@@ -186,20 +185,6 @@ func (t *Tracker) Finish() {
t.save()
}
// Annotate attaches a short plain note to a phase and saves it at once. It is for a
// remark the reader should see although nothing failed. Durations and counts only: a note
// is shown on a page about a private recording.
func (t *Tracker) Annotate(key, note string) {
t.mu.Lock()
defer t.mu.Unlock()
s := t.find(key)
if s == nil {
return
}
s.Note = note
t.save()
}
// Heartbeat stamps the job's row every beatEvery and logs which phase is still running
// every stillEvery, until ctx ends. It runs on its own, independent of any engine call:
// a heartbeat that stopped means the process is gone, not that a request is slow.
+6 -64
View File
@@ -197,7 +197,7 @@ func TestTracker_FailRunningMarksTheRunningStepFailedAndLeavesTheRestPending(t *
require.Equal(t, domain.StepPending, stateOf(steps, "save"), "what never started stays pending: it shows where the job stopped")
}
func TestTracker_AKeyedFailMarksOneStepFailedLogsItsKindAndTheJobGoesOn(t *testing.T) {
func TestTracker_AKeyedFailMarksOneStepFailedLogsWhyAndTheJobGoesOn(t *testing.T) {
tr, sink, _, buf := newTracker(t)
tr.Start()
tr.Begin("audio", 0)
@@ -209,8 +209,7 @@ func TestTracker_AKeyedFailMarksOneStepFailedLogsItsKindAndTheJobGoesOn(t *testi
require.Equal(t, domain.StepFailed, stateOf(steps, "audio"))
require.Equal(t, domain.StepRunning, stateOf(steps, "decide"), "a soft failure must not stop the next step from running")
require.Contains(t, buf.String(), "level=WARN")
require.NotContains(t, buf.String(), "gateway said no", "an error's text can carry provider output or a path: the log gets its kind")
require.Contains(t, buf.String(), "kind=error")
require.Contains(t, buf.String(), "gateway said no")
require.Contains(t, buf.String(), "step=audio")
}
@@ -260,56 +259,12 @@ func TestTracker_EveryTransitionIsLoggedByTheSameCall(t *testing.T) {
tr.Begin("audio", 0)
clk.tick(3 * time.Second)
tr.Done("audio")
tr.Begin("decide", 5)
clk.tick(2 * time.Second)
tr.FailRunning()
tr.Finish()
out := buf.String()
for _, want := range []string{
`msg="job started"`,
`msg="step started" step=audio`,
`msg="step finished" step=audio elapsed=3s`,
`msg="step started" step=decide total=5`,
`msg="step failed" step=decide elapsed=2s`,
`msg="job finished"`,
} {
require.Contains(t, out, want, "a transition with no log line is invisible to the operator")
}
}
func TestTracker_AKeyedFailIsSavedAtOnceNotOnTheNextTransition(t *testing.T) {
tr, sink, _, _ := newTracker(t)
tr.Start()
tr.Begin("audio", 0)
before := sink.saveCount()
tr.Fail("audio", errors.New("down"))
require.Equal(t, before+1, sink.saveCount(), "a soft-failed phase must show as failed on the page now")
require.Equal(t, domain.StepFailed, stateOf(sink.last(t), "audio"))
}
// retainingSink keeps the very slice it is handed, like a sink that queues it for later.
type retainingSink struct{ got [][]domain.ProgressStep }
func (r *retainingSink) SaveProgress(_ context.Context, steps []domain.ProgressStep, _ time.Time) error {
r.got = append(r.got, steps)
return nil
}
func (r *retainingSink) Beat(context.Context) error { return nil }
func TestTracker_ASnapshotHandedToTheSinkIsNotChangedByLaterTransitions(t *testing.T) {
sink := &retainingSink{}
tr := progress.NewTracker(context.Background(), sink, slog.New(slog.NewTextHandler(&syncBuf{}, nil)), plan(),
progress.WithClock(newClock().now))
tr.Start()
first := sink.got[0]
tr.Begin("audio", 0)
tr.Done("audio")
require.Equal(t, domain.StepPending, first[0].State, "a sink that keeps a snapshot must not see it change under it")
require.Contains(t, out, "step=audio")
require.Contains(t, out, "started")
require.Contains(t, out, "finished")
require.Contains(t, out, "elapsed=3s")
}
func TestTracker_AdvanceIsLoggedAtMostOncePerMinute(t *testing.T) {
@@ -387,16 +342,3 @@ func TestWithAndFrom_ACarriedReporterIsTheOneReturned(t *testing.T) {
require.Equal(t, domain.StepRunning, stateOf(sink.last(t), "audio"))
}
func TestTracker_AnnotateAttachesANoteToAStepAndSavesItAtOnce(t *testing.T) {
tr, sink, _, _ := newTracker(t)
tr.Start()
tr.Begin("audio", 0)
before := sink.saveCount()
tr.Annotate("audio", "20 min of the 79 the file claimed")
require.Equal(t, before+1, sink.saveCount())
require.Equal(t, "20 min of the 79 the file claimed", sink.last(t)[0].Note)
require.NotPanics(t, func() { tr.Annotate("no-such-step", "x") })
}
+7 -7
View File
@@ -32,7 +32,7 @@ func (f *fakeCompleter) Complete(_ context.Context, system, user string) (string
return f.replies[i], nil
}
var req = reconcile.Request{Before: "grannarna ska då", After: "De ska kunna måla", Base: "målade", Alt: "mölade"}
var req = reconcile.Request{Before: "kunderna ska då", After: "De ska kunna betala", Base: "onboardade", Alt: "omvårdade"}
func TestLLMAdjudicator_SendsTheMeasuredPromptWithTheReadingsInTheRequestedOrder(t *testing.T) {
fc := &fakeCompleter{replies: []string{"ANSWER: 1"}}
@@ -45,10 +45,10 @@ func TestLLMAdjudicator_SendsTheMeasuredPromptWithTheReadingsInTheRequestedOrder
assert.Contains(t, fc.systems[0], "two automatic transcriptions of the same Swedish speech")
assert.Contains(t, fc.systems[0], "ANSWER: 1")
u := fc.users[0]
assert.Contains(t, u, "Text before the spot:\ngrannarna ska då")
assert.Contains(t, u, "Text before the spot:\nkunderna ska då")
assert.Contains(t, u, "[[ THE SPOT ]]")
assert.Contains(t, u, "Text after the spot:\nDe ska kunna måla")
assert.Contains(t, u, "Reading 1: målade\nReading 2: mölade", "base first when AltFirst is false")
assert.Contains(t, u, "Text after the spot:\nDe ska kunna betala")
assert.Contains(t, u, "Reading 1: onboardade\nReading 2: omvårdade", "base first when AltFirst is false")
}
func TestLLMAdjudicator_AltFirstSwapsWhichReadingIsNumberOne(t *testing.T) {
@@ -59,7 +59,7 @@ func TestLLMAdjudicator_AltFirstSwapsWhichReadingIsNumberOne(t *testing.T) {
_, err := reconcile.NewLLMAdjudicator(fc).Adjudicate(context.Background(), r)
require.NoError(t, err)
assert.Contains(t, fc.users[0], "Reading 1: mölade\nReading 2: målade")
assert.Contains(t, fc.users[0], "Reading 1: omvårdade\nReading 2: onboardade")
}
func TestLLMAdjudicator_MapsTheAnswerBackToBaseOrAltWhicheverOrderWasShown(t *testing.T) {
@@ -134,8 +134,8 @@ func TestOrderChecked_AsksTwiceWithTheOrderSwappedAndKeepsOnlyAgreement(t *testi
require.NoError(t, err)
assert.Equal(t, tt.want, got)
require.Len(t, fc.users, 2)
assert.True(t, strings.Contains(fc.users[0], "Reading 1: målade"), "first call shows the base first")
assert.True(t, strings.Contains(fc.users[1], "Reading 1: mölade"), "second call shows the alternative first")
assert.True(t, strings.Contains(fc.users[0], "Reading 1: onboardade"), "first call shows the base first")
assert.True(t, strings.Contains(fc.users[1], "Reading 1: omvårdade"), "second call shows the alternative first")
})
}
}
+9 -9
View File
@@ -19,12 +19,12 @@ func TestFindDisputes_NoneWhenTheTextsAgreeWordForWord(t *testing.T) {
}
func TestFindDisputes_OneReplacedWordIsOneDisputeCarryingBothReadings(t *testing.T) {
got := disputes("Vi ska målade grannarna.", "Vi ska mölade grannarna.")
got := disputes("Vi ska onboardade kunderna.", "Vi ska omvårdade kunderna.")
require.Len(t, got, 1)
d := got[0]
assert.Equal(t, "målade", d.Base)
assert.Equal(t, "mölade", d.Alt)
assert.Equal(t, "onboardade", d.Base)
assert.Equal(t, "omvårdade", d.Alt)
assert.Equal(t, 2, d.BaseStart)
assert.Equal(t, 3, d.BaseEnd)
assert.Equal(t, 2, d.AltStart)
@@ -32,11 +32,11 @@ func TestFindDisputes_OneReplacedWordIsOneDisputeCarryingBothReadings(t *testing
}
func TestFindDisputes_ReadingsKeepTheirOwnSpellingAndPunctuation(t *testing.T) {
got := disputes("Det blir A+2, alltså.", "Det blir A 2, alltså.")
got := disputes("Det blir T+1, alltså.", "Det blir T 1, alltså.")
require.Len(t, got, 1)
assert.Equal(t, "A+2,", got[0].Base)
assert.Equal(t, "A 2,", got[0].Alt, "the alternative's words replace the span as the engine wrote them")
assert.Equal(t, "T+1,", got[0].Base)
assert.Equal(t, "T 1,", got[0].Alt, "the alternative's words replace the span as the engine wrote them")
}
func TestFindDisputes_AWordOnlyOneEngineHeardHasAnEmptyOtherSide(t *testing.T) {
@@ -90,12 +90,12 @@ func TestAutoResolve(t *testing.T) {
base, alt string
resolved bool
}{
{"a compound written as two words or one", "i takrenns delen", "i takrennsdelen", true},
{"a compound with a hyphen", "takrenns-delen", "takrenns delen", true},
{"a compound written as two words or one", "i crossborder delen", "i crossborderdelen", true},
{"a compound with a hyphen", "crossborder-delen", "crossborder delen", true},
{"a filler only the alternative heard", "vi ska ju betala", "vi ska betala", true},
{"a filler only the base heard", "vi ska betala", "vi ska ju betala", true},
{"two fillers in a row", "vi ska ja ju betala", "vi ska betala", true},
{"a different word is for the model", "vi ska målade", "vi ska mölade", false},
{"a different word is for the model", "vi ska onboardade", "vi ska omvårdade", false},
{"a different number is for the model", "totalt 1050 kronor", "totalt 2050 kronor", false},
{"a content word only one side heard is for the model", "ett förslag", "ett coolt förslag", false},
{"three filler words are too many to wave through", "vi ska ju då liksom betala", "vi ska betala", false},
+3 -3
View File
@@ -6,7 +6,6 @@ import (
"strings"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/errkind"
"git.d-ma.be/mathias/tapir/internal/progress"
)
@@ -102,8 +101,9 @@ func Reconcile(ctx context.Context, base, alt domain.Transcription, adj Adjudica
rec.Errors++
consecutive++
if consecutive >= breakAfter {
rep.Fail("decide", err)
return skipped(base, rec, fmt.Sprintf("the model failed %d times in a row (%s)", consecutive, errkind.Of(err))), nil
failure := fmt.Errorf("the model failed %d times in a row: %w", consecutive, err)
rep.Fail("decide", failure)
return skipped(base, rec, failure.Error()), nil
}
v = VerdictUnsure
} else {
@@ -48,8 +48,8 @@ func reconcileWith(t *testing.T, ctx context.Context, base, alt string, adj reco
// two compound/filler disputes the model is never asked about, then two it is.
const (
progBase = "i takrenns delen ska vi ju betala totalt 1015 kronor till Axel idag"
progAlt = "i takrennsdelen ska vi betala totalt 1050 kronor till Alex idag"
progBase = "i crossborder delen ska vi ju betala totalt 1015 kronor till SEB idag"
progAlt = "i crossborderdelen ska vi betala totalt 1050 kronor till SCB idag"
)
func TestReconcileProgress_TheDecidingPhaseTotalIsTheModelDecisionsNotTheDisputes(t *testing.T) {
@@ -1,33 +0,0 @@
package reconcile_test
import (
"context"
"errors"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/reconcile"
)
// The reason a reconciliation was skipped is stored and shown on a page about a private
// recording, so it carries a kind, never the model's error text, which can echo the
// prompt (adversarial review F3).
func TestReconcile_TheBreakerReasonCarriesAKindNotTheModelsText(t *testing.T) {
adj := &scripted{fn: func(int, reconcile.Request) (reconcile.Verdict, error) {
return reconcile.VerdictBase, errors.New("gateway echoed the prompt: zebrakaka secret words")
}}
small := opts
small.BreakAfter = 2
out, err := reconcile.Reconcile(context.Background(), tr(progBase), tr(progAlt), adj, small)
require.NoError(t, err)
require.Equal(t, domain.ReconcileSkipped, out.Reconciliation.Status)
assert.NotContains(t, out.Reconciliation.Reason, "zebrakaka", "provider text must not be stored or shown")
assert.Contains(t, out.Reconciliation.Reason, "2 times in a row", "the reason still says what happened")
assert.Contains(t, out.Reconciliation.Reason, "error", "and carries a kind")
}
+3 -3
View File
@@ -31,16 +31,16 @@ func TestReconcileSpots_AKeptSpotAndAChangedSpotAreBothStoredWithTheirVerdict(t
}
func TestReconcileSpots_AutoResolvedDisputesAreNotStored(t *testing.T) {
out := merge(t, tr("i takrenns delen ska vi ju betala"), tr("i takrennsdelen ska vi betala"), always(reconcile.VerdictAlt))
out := merge(t, tr("i crossborder delen ska vi ju betala"), tr("i crossborderdelen ska vi betala"), always(reconcile.VerdictAlt))
assert.Empty(t, out.Reconciliation.Contested, "compounds and fillers were never judged; they are counted, not listed")
}
func TestReconcileSpots_ASpotCarriesTheWordsJustBeforeIt(t *testing.T) {
out := merge(t, tr("grannen målar ett vanligt staket mot gatan"), tr("grannen målar ett KORT staket mot gatan"), always(reconcile.VerdictAlt))
out := merge(t, tr("vi skickar in ett special meddelande till taxsystemet"), tr("vi skickar in ett CAMT meddelande till taxsystemet"), always(reconcile.VerdictAlt))
require.Len(t, out.Reconciliation.Contested, 1)
assert.Equal(t, "grannen målar ett", out.Reconciliation.Contested[0].Before)
assert.Equal(t, "vi skickar in ett", out.Reconciliation.Contested[0].Before)
}
func TestReconcileSpots_TheContextIsAFewWordsNotTheWholeMeeting(t *testing.T) {
+14 -15
View File
@@ -58,10 +58,10 @@ func TestReconcile_IdenticalTextsNeedNoModelAndComeBackUnchanged(t *testing.T) {
func TestReconcile_TrivialDisputesAreResolvedWithoutAskingTheModel(t *testing.T) {
adj := always(reconcile.VerdictAlt)
out := merge(t, tr("i takrenns delen ska vi ju betala"), tr("i takrennsdelen ska vi betala"), adj)
out := merge(t, tr("i crossborder delen ska vi ju betala"), tr("i crossborderdelen ska vi betala"), adj)
assert.Empty(t, adj.calls, "a compound split and a filler particle are not worth a model call")
assert.Equal(t, "i takrenns delen ska vi ju betala", out.Text, "the base reading is kept")
assert.Equal(t, "i crossborder delen ska vi ju betala", out.Text, "the base reading is kept")
assert.Equal(t, 2, out.Reconciliation.Disputes)
assert.Equal(t, 2, out.Reconciliation.AutoResolved)
assert.Zero(t, out.Reconciliation.Adjudicated)
@@ -76,13 +76,13 @@ func TestReconcile_AltVerdictReplacesTheSpanWithTheAlternativesOwnWords(t *testi
}
func TestReconcile_BaseAndUnsureVerdictsKeepTheBaseButOnlyUnsureIsFlagged(t *testing.T) {
base := tr("Totalt 1015 kronor till Axel.")
alt := tr("Totalt 1050 kronor till Alex.")
base := tr("Totalt 1015 kronor till SEB.")
alt := tr("Totalt 1050 kronor till SCB.")
keptBase := merge(t, base, alt, always(reconcile.VerdictBase))
unsure := merge(t, base, alt, always(reconcile.VerdictUnsure))
assert.Equal(t, "Totalt 1015 kronor till Axel.", keptBase.Text)
assert.Equal(t, "Totalt 1015 kronor till SEB.", keptBase.Text)
assert.Zero(t, keptBase.Reconciliation.Unsure)
require.Len(t, keptBase.Reconciliation.Contested, 2,
"every judged spot is stored, kept ones too: without them the reconciler cannot be compared with either engine alone (ADR-035)")
@@ -90,7 +90,7 @@ func TestReconcile_BaseAndUnsureVerdictsKeepTheBaseButOnlyUnsureIsFlagged(t *tes
assert.Equal(t, "base", c.Used)
assert.False(t, c.Unsure, "a confident 'the base is right' is stored but is not an uncertain spot")
}
assert.Equal(t, "Totalt 1015 kronor till Axel.", unsure.Text, "unsure keeps the base")
assert.Equal(t, "Totalt 1015 kronor till SEB.", unsure.Text, "unsure keeps the base")
assert.Equal(t, 2, unsure.Reconciliation.Unsure)
require.Len(t, unsure.Reconciliation.Contested, 2)
assert.True(t, unsure.Reconciliation.Contested[0].Unsure)
@@ -98,12 +98,12 @@ func TestReconcile_BaseAndUnsureVerdictsKeepTheBaseButOnlyUnsureIsFlagged(t *tes
}
func TestReconcile_ChangedSpotsAreRecordedWithBothReadings(t *testing.T) {
out := merge(t, tr("det blir A+2 imorgon"), tr("det blir en A imorgon"), always(reconcile.VerdictAlt))
out := merge(t, tr("det blir T+1 imorgon"), tr("det blir en T imorgon"), always(reconcile.VerdictAlt))
require.Len(t, out.Reconciliation.Contested, 1)
c := out.Reconciliation.Contested[0]
assert.Equal(t, "A+2", c.Base)
assert.Equal(t, "en A", c.Alt)
assert.Equal(t, "T+1", c.Base)
assert.Equal(t, "en T", c.Alt)
assert.Equal(t, "alt", c.Used)
assert.False(t, c.Unsure)
}
@@ -122,16 +122,16 @@ func TestReconcile_AWordOnlyTheBaseHeardCanBeRemoved(t *testing.T) {
func TestReconcile_SeveralDisputesAreAppliedIndependentlyAndInOrder(t *testing.T) {
adj := &scripted{fn: func(i int, r reconcile.Request) (reconcile.Verdict, error) {
if r.Alt == "mölade" || r.Alt == "Alex" {
if r.Alt == "omvårdade" || r.Alt == "SCB" {
return reconcile.VerdictAlt, nil
}
return reconcile.VerdictBase, nil
}}
out := merge(t, tr("kunderna målade nu och sen betalas till Axel via ombud med 1050 kronor"),
tr("kunderna mölade nu och sen betalas till Alex via ombud med 2050 kronor"), adj)
out := merge(t, tr("kunderna onboardade nu och sen betalas till SEB via ombud med 1050 kronor"),
tr("kunderna omvårdade nu och sen betalas till SCB via ombud med 2050 kronor"), adj)
assert.Equal(t, "kunderna mölade nu och sen betalas till Alex via ombud med 1050 kronor", out.Text)
assert.Equal(t, "kunderna omvårdade nu och sen betalas till SCB via ombud med 1050 kronor", out.Text)
assert.Equal(t, 3, out.Reconciliation.Disputes)
assert.Equal(t, 2, out.Reconciliation.Changed)
}
@@ -193,8 +193,7 @@ func TestReconcile_ConsecutiveModelErrorsStopTheWholeReconciliationInsteadOfWait
assert.Len(t, adj.calls, 3, "three consecutive failures and it stops asking")
assert.Equal(t, strings.Join(a, " "), out.Text, "the base text, with none of the partial changes")
assert.Equal(t, domain.ReconcileSkipped, out.Reconciliation.Status)
assert.Contains(t, out.Reconciliation.Reason, "3 times in a row")
assert.NotContains(t, out.Reconciliation.Reason, "gateway down", "the reason is shown on the page: a kind, not the model's own text")
assert.Contains(t, out.Reconciliation.Reason, "gateway down")
}
func TestReconcile_ASuccessBetweenFailuresResetsTheBreaker(t *testing.T) {
+2 -2
View File
@@ -34,9 +34,9 @@ func TestTokenize_KeepsTheWordAsWrittenAndKeysItWithoutCaseOrPunctuation(t *test
}
func TestTokenize_KeysIgnoreHyphensAndPlusSoOnlyWordsDecideEquality(t *testing.T) {
got := reconcile.Tokenize("end-to-end A+2 5 000 kr")
got := reconcile.Tokenize("end-to-end T+1 5 000 kr")
assert.Equal(t, []string{"endtoend", "a2", "5", "000", "kr"}, keys(got))
assert.Equal(t, []string{"endtoend", "t1", "5", "000", "kr"}, keys(got))
}
func TestTokenize_APunctuationOnlyWordJoinsThePreviousWordInsteadOfBecomingOne(t *testing.T) {
+2 -101
View File
@@ -6,8 +6,6 @@ import (
"fmt"
"io"
"os"
"sync"
"time"
"git.d-ma.be/mathias/tapir/internal/domain"
)
@@ -35,9 +33,6 @@ type Job struct {
UserID string
UploadID string
Path string
// Duration is what the file CLAIMED when probed. The worker compares it with what
// was really converted; zero means unknown (ADR-035 review).
Duration time.Duration
}
// Submitter hands accepted jobs to the in-process worker (ADR-033).
@@ -52,53 +47,6 @@ type Intake struct {
sub Submitter
limits Limits
tmpDir string
// MaxReceives and MaxReceivesPerUser bound how many request bodies Accept reads at
// once; zero means the defaults. Without them N parallel POSTs each spool up to
// MaxBytes while the queue still looks empty, and the spool volume fills (#47).
MaxReceives int
MaxReceivesPerUser int
mu sync.Mutex
receives int
byUser map[string]int
}
const (
defaultMaxReceives = 4
defaultMaxReceivesPerUser = 2
)
// acquireReceive reserves a receive slot for userID, or reports there is none. Every
// successful call must be paired with releaseReceive.
func (in *Intake) acquireReceive(userID string) bool {
maxAll, maxUser := in.MaxReceives, in.MaxReceivesPerUser
if maxAll <= 0 {
maxAll = defaultMaxReceives
}
if maxUser <= 0 {
maxUser = defaultMaxReceivesPerUser
}
in.mu.Lock()
defer in.mu.Unlock()
if in.receives >= maxAll || in.byUser[userID] >= maxUser {
return false
}
if in.byUser == nil {
in.byUser = map[string]int{}
}
in.receives++
in.byUser[userID]++
return true
}
func (in *Intake) releaseReceive(userID string) {
in.mu.Lock()
defer in.mu.Unlock()
in.receives--
if in.byUser[userID]--; in.byUser[userID] <= 0 {
delete(in.byUser, userID)
}
}
// NewIntake wires an Intake. tmpDir is where uploads are spooled; it must exist.
@@ -106,47 +54,12 @@ func NewIntake(s Store, p Prober, sub Submitter, l Limits, tmpDir string) *Intak
return &Intake{store: s, prober: p, sub: sub, limits: l, tmpDir: tmpDir}
}
// ErrClientRead marks an error that came from READING the client's body: a connection that
// dropped, a body cut short. It is the sender's to retry, unlike a spool, probe or database
// fault, which is ours. It is a marker on the error, not a sentinel that others may wrap:
// io.ErrUnexpectedEOF alone is also what a database drop looks like.
var ErrClientRead = errors.New("uploads: client body read failed")
// clientReader remembers the last non-EOF error its source returned, so Accept can tell
// a failure reading the client from a failure writing the spool file.
type clientReader struct {
r io.Reader
err error
}
func (c *clientReader) Read(p []byte) (int, error) {
n, err := c.r.Read(p)
if err != nil && !errors.Is(err, io.EOF) {
c.err = err
}
return n, err
}
// busyMessage is what a user is told when the queue is full.
const busyMessage = "Too many recordings are waiting to be transcribed. Please try again in a few minutes."
// Accept spools src to disk (bounded by MaxBytes), probes it, validates it, records
// a queued upload and submits the job. It returns the upload id. A file that fails
// validation yields a *RejectError and creates NO row — rejection is synchronous
// and leaves nothing behind. On success the temp file is handed to the job and the
// worker deletes it; on every failure path Accept deletes it itself.
func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reader) (string, error) {
// Refuse before spooling when the queue has no room, if the submitter can say so: a full
// queue otherwise cost up to the upload cap on disk, an ffprobe run and a failed row for
// every refused attempt.
if rc, ok := in.sub.(interface{ HasRoom(userID string) bool }); ok && !rc.HasRoom(userID) {
return "", &RejectError{Reason: ReasonBusy, Message: busyMessage}
}
// HasRoom cannot see bodies still being received; bound those separately.
if !in.acquireReceive(userID) {
return "", &RejectError{Reason: ReasonBusy, Message: busyMessage}
}
defer in.releaseReceive(userID)
f, err := os.CreateTemp(in.tmpDir, "upload-*")
if err != nil {
return "", fmt.Errorf("uploads: spool: %w", err)
@@ -160,15 +73,11 @@ func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reade
}()
// Read one byte past the cap so "exactly at the limit" and "over" are told apart.
cr := &clientReader{r: io.LimitReader(src, in.limits.MaxBytes+1)}
n, err := io.Copy(f, cr)
n, err := io.Copy(f, io.LimitReader(src, in.limits.MaxBytes+1))
if cerr := f.Close(); err == nil {
err = cerr
}
if err != nil {
if cr.err != nil && errors.Is(err, cr.err) {
return "", fmt.Errorf("%w: %w", ErrClientRead, err)
}
return "", fmt.Errorf("uploads: receive: %w", err)
}
if n > in.limits.MaxBytes {
@@ -194,15 +103,7 @@ func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reade
if err != nil {
return "", fmt.Errorf("uploads: create: %w", err)
}
if err := in.sub.Submit(Job{UserID: userID, UploadID: id, Path: path, Duration: probe.Duration}); err != nil {
if errors.Is(err, ErrQueueFull) {
// The user's to retry, not a server fault: refuse it as such, and do not leave
// the row queued -- that is the stuck spinner.
if ferr := in.store.FailUpload(ctx, userID, id, domain.UploadErrBusy); ferr != nil {
return "", fmt.Errorf("uploads: submit: %w (and failing the row: %v)", err, ferr)
}
return "", &RejectError{Reason: ReasonBusy, Message: busyMessage}
}
if err := in.sub.Submit(Job{UserID: userID, UploadID: id, Path: path}); err != nil {
// A row that will never run must not sit queued: that is the stuck spinner.
if ferr := in.store.FailUpload(ctx, userID, id, domain.UploadErrInterrupted); ferr != nil {
return "", fmt.Errorf("uploads: submit: %w (and failing the row: %v)", err, ferr)
-96
View File
@@ -1,96 +0,0 @@
package uploads_test
import (
"context"
"errors"
"fmt"
"io"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
// A full queue is the user's to retry, not a server fault: a typed rejection, the row
// failed honestly, the audio removed (adversarial review F6).
func TestAccept_AFullQueueIsABusyRejectionNotAServerErrorAndLeavesNothingBehind(t *testing.T) {
r := newRig(t)
r.sub.err = uploads.ErrQueueFull
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
var rej *uploads.RejectError
require.True(t, errors.As(err, &rej), "a full queue must be a rejection the web layer can answer with 429: %v", err)
assert.Equal(t, uploads.ReasonBusy, rej.Reason)
assert.Contains(t, rej.Message, "waiting")
assert.Equal(t, domain.UploadErrBusy, r.st.failed["up-1"], "the row must not sit queued: that is the stuck spinner")
assert.Empty(t, r.tmpFiles(t), "the audio is deleted")
}
// roomSubmitter is a Submitter that can say, before anything is spooled, that there is no room.
type roomSubmitter struct {
fakeSubmitter
room bool
}
func (r *roomSubmitter) HasRoom(string) bool { return r.room }
func TestAccept_ANoRoomQueueIsRefusedBeforeAnythingIsSpooledProbedOrStored(t *testing.T) {
// Checking only at Submit meant 200 MB was spooled and ffprobe run for every refused
// attempt, and a failed row was left behind each time (second adversarial review).
r := newRig(t)
full := &roomSubmitter{room: false}
in := uploads.NewIntake(r.st, r.pr, full, uploads.Limits{MaxBytes: 1024, MaxDuration: 90 * time.Minute}, r.tmp)
_, err := in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
var rej *uploads.RejectError
require.True(t, errors.As(err, &rej))
assert.Equal(t, uploads.ReasonBusy, rej.Reason)
assert.Empty(t, r.st.created, "no row for an upload that was never accepted")
assert.Empty(t, r.pr.paths, "ffprobe was never run on it")
assert.Empty(t, r.tmpFiles(t), "nothing was spooled")
assert.Empty(t, full.jobs)
}
func TestAccept_ASubmitterThatCannotSayStillWorksAsBefore(t *testing.T) {
r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
require.NoError(t, err, "a Submitter without HasRoom is accepted: the check at Submit remains")
}
// brokenBody is a client body that fails mid-read, like a connection that dropped.
type brokenBody struct{ n int }
func (b *brokenBody) Read(p []byte) (int, error) {
if b.n > 0 {
b.n = 0
return copy(p, "partial audio"), nil
}
return 0, io.ErrUnexpectedEOF
}
func TestAccept_AnErrorReadingTheClientsBodyIsMarkedAsTheClientsAndAWrappedSubmitErrorIsNot(t *testing.T) {
r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "t", &brokenBody{n: 1})
require.Error(t, err)
assert.ErrorIs(t, err, uploads.ErrClientRead, "a dropped client body is the sender's to retry")
assert.Empty(t, r.tmpFiles(t))
// A fault that is ours must NOT carry the marker, even if it wraps the same sentinel.
r2 := newRig(t)
r2.sub.err = fmt.Errorf("db: %w", io.ErrUnexpectedEOF)
_, err = r2.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
require.Error(t, err)
assert.NotErrorIs(t, err, uploads.ErrClientRead)
}
-131
View File
@@ -1,131 +0,0 @@
package uploads_test
// Observes how many request bodies Accept will read at once (#47). HasRoom and the
// per-user queue cap act before and after the receive; N parallel POSTs all pass
// HasRoom while the queue is still empty, so the receive itself is bounded.
import (
"context"
"errors"
"io"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
// slowReceive starts an Accept whose body stays open until the returned writer is closed,
// and waits until it holds a spool file, which proves it is inside the receive.
type slowReceive struct {
w *io.PipeWriter
done chan error
finished bool
}
func startSlowReceive(t *testing.T, r *rig, in *uploads.Intake, user string, wantFiles int) *slowReceive {
t.Helper()
pr, pw := io.Pipe()
s := &slowReceive{w: pw, done: make(chan error, 1)}
go func() {
_, err := in.Accept(context.Background(), user, "t", pr)
s.done <- err
}()
require.Eventually(t, func() bool { return len(r.tmpFiles(t)) == wantFiles }, 2*time.Second, 5*time.Millisecond,
"the receive for %s should have started", user)
// An abandoned receive ends in a read error, so it never reaches the rig's fakes, which
// are not safe for concurrent use. A receive that finish() already ended is a no-op.
t.Cleanup(func() {
if s.finished {
return
}
_ = pw.CloseWithError(errors.New("test over"))
select {
case <-s.done:
case <-time.After(2 * time.Second):
}
})
return s
}
func (s *slowReceive) finish(t *testing.T) error {
t.Helper()
_, _ = s.w.Write([]byte("data"))
require.NoError(t, s.w.Close())
s.finished = true
select {
case err := <-s.done:
return err
case <-time.After(2 * time.Second):
t.Fatal("Accept did not return after its body ended")
return nil
}
}
func requireBusy(t *testing.T, err error) {
t.Helper()
var rej *uploads.RejectError
require.True(t, errors.As(err, &rej), "want a busy rejection, got %v", err)
assert.Equal(t, uploads.ReasonBusy, rej.Reason)
}
func TestAccept_APerUserLimitOnConcurrentReceivesRefusesTheNextBodyBeforeSpoolingIt(t *testing.T) {
r := newRig(t)
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
in.MaxReceivesPerUser = 2
in.MaxReceives = 10
startSlowReceive(t, r, in, "user-1", 1)
startSlowReceive(t, r, in, "user-1", 2)
_, err := in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
requireBusy(t, err)
assert.Len(t, r.tmpFiles(t), 2, "the refused body was never spooled")
// Another user is not blocked by user-1's two receives.
other := startSlowReceive(t, r, in, "user-2", 3)
require.NoError(t, other.finish(t))
}
func TestAccept_AnOverallLimitOnConcurrentReceivesRefusesTheNextUser(t *testing.T) {
r := newRig(t)
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
in.MaxReceivesPerUser = 5
in.MaxReceives = 3
for i, u := range []string{"a", "b", "c"} {
startSlowReceive(t, r, in, u, i+1)
}
_, err := in.Accept(context.Background(), "d", "t", strings.NewReader("data"))
requireBusy(t, err)
assert.Len(t, r.tmpFiles(t), 3)
}
func TestAccept_AFinishedReceiveFreesItsSlot(t *testing.T) {
r := newRig(t)
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
in.MaxReceivesPerUser = 1
in.MaxReceives = 5
first := startSlowReceive(t, r, in, "user-1", 1)
require.NoError(t, first.finish(t))
_, err := in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
require.NoError(t, err, "the slot is released when Accept returns")
}
func TestAccept_AFailedReceiveFreesItsSlot(t *testing.T) {
r := newRig(t)
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
in.MaxReceivesPerUser = 1
in.MaxReceives = 5
_, err := in.Accept(context.Background(), "user-1", "t", &brokenBody{n: 1})
require.ErrorIs(t, err, uploads.ErrClientRead)
_, err = in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
require.NoError(t, err, "a dropped client must not leak a slot")
}
-31
View File
@@ -1,31 +0,0 @@
package uploads_test
// The limits production runs on are the DEFAULTS: nothing outside tests sets
// MaxReceives or MaxReceivesPerUser. Pinned here with the literal numbers, so changing a
// default is a visible decision (#47 review). With 4 receives, 20 queued and 1 running at
// the 200 MiB cap, the spool needs ~5 GiB of the 6 Gi emptyDir (infra); raising any of
// them or TAPIR_UPLOAD_MAX_BYTES must revisit that arithmetic.
import (
"context"
"strings"
"testing"
"time"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
func TestNewIntake_DefaultsAllowTwoReceivesPerUserAndFourOverall(t *testing.T) {
r := newRig(t)
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
startSlowReceive(t, r, in, "a", 1)
startSlowReceive(t, r, in, "a", 2)
_, err := in.Accept(context.Background(), "a", "t", strings.NewReader("x"))
requireBusy(t, err) // a third body from one user
startSlowReceive(t, r, in, "b", 3)
startSlowReceive(t, r, in, "b", 4)
_, err = in.Accept(context.Background(), "c", "t", strings.NewReader("x"))
requireBusy(t, err) // a fifth body overall
}
+1 -2
View File
@@ -106,8 +106,7 @@ func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
assert.Equal(t, audio, r.pr.seen, "the prober must see the complete uploaded bytes, on disk")
require.Len(t, r.sub.jobs, 1)
job := r.sub.jobs[0]
assert.Equal(t, uploads.Job{UserID: "user-1", UploadID: "up-1", Path: job.Path, Duration: 10 * time.Minute}, job,
"the job carries what the file CLAIMED, so the worker can compare it with what was converted")
assert.Equal(t, uploads.Job{UserID: "user-1", UploadID: "up-1", Path: job.Path}, job)
assert.Equal(t, filepath.Dir(job.Path), r.tmp, "the temp file lives in the configured directory")
got, err := os.ReadFile(job.Path)
require.NoError(t, err)
-2
View File
@@ -34,8 +34,6 @@ const (
ReasonTooLong Reason = "too_long"
ReasonTooLarge Reason = "too_large"
ReasonUnreadable Reason = "unreadable"
// ReasonBusy: the queue is full. Not about the file: the user retries later.
ReasonBusy Reason = "busy"
)
// RejectError is a synchronous, user-presentable refusal. Message is written for
+23 -136
View File
@@ -10,18 +10,12 @@ import (
"time"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/errkind"
"git.d-ma.be/mathias/tapir/internal/progress"
)
// ErrStopped is returned by Submit once the worker has stopped.
var ErrStopped = errors.New("uploads: worker stopped")
// ErrQueueFull is returned by Submit when the queue, or one user's share of it, is at
// its cap. Every queued job holds up to the upload cap on disk until its turn, so an
// unbounded queue is a disk-filling lever (ADR-035 review).
var ErrQueueFull = errors.New("uploads: queue full")
// Converter turns the uploaded file into the 16 kHz mono PCM WAV whisper wants.
type Converter interface {
ToWAV(ctx context.Context, src, dst string) error
@@ -59,24 +53,10 @@ func WithHeartbeat(beat, still time.Duration) WorkerOption {
return func(w *Worker) { w.beatEvery, w.stillEvery = beat, still }
}
// WithQueueLimits bounds how many jobs may WAIT: in total, and per user. A job being
// processed is not waiting. The defaults are 20 and 3 (one instance, one job at a time).
func WithQueueLimits(total, perUser int) WorkerOption {
return func(w *Worker) { w.maxQueued, w.maxPerUser = total, perUser }
}
// WithMaxDuration makes the worker measure the converted audio and fail a job whose real
// length exceeds the limit, or falls far short of what the file claimed. Zero (the
// default) measures nothing. The limit is first checked against the file's header, which
// can lie (ADR-035 review).
func WithMaxDuration(d time.Duration) WorkerOption { return func(w *Worker) { w.maxDuration = d } }
// WithLogger replaces slog.Default() as the worker's logger.
func WithLogger(l *slog.Logger) WorkerOption { return func(w *Worker) { w.log = l } }
const (
defaultMaxQueued = 20
defaultMaxPerUser = 3
defaultBeatEvery = 10 * time.Second
defaultStillEvery = 5 * time.Minute
)
@@ -107,8 +87,6 @@ type Worker struct {
log *slog.Logger
beatEvery, stillEvery time.Duration
maxDuration time.Duration
maxQueued, maxPerUser int
mu sync.Mutex
queue []Job
@@ -122,7 +100,6 @@ func NewWorker(s WorkerStore, c Converter, t Transcriber, tmpDir string, timeout
w := &Worker{
store: s, conv: c, tr: t, tmpDir: tmpDir, timeout: timeout, wake: make(chan struct{}, 1),
beatEvery: defaultBeatEvery, stillEvery: defaultStillEvery,
maxQueued: defaultMaxQueued, maxPerUser: defaultMaxPerUser,
}
for _, o := range opts {
o(w)
@@ -130,24 +107,6 @@ func NewWorker(s WorkerStore, c Converter, t Transcriber, tmpDir string, timeout
return w
}
// HasRoom reports whether a job from this user would be accepted right now. It lets the
// intake refuse BEFORE spooling up to the upload cap and running ffprobe; Submit still
// enforces the same limits atomically, so a race costs at worst one refused submit.
func (w *Worker) HasRoom(userID string) bool {
w.mu.Lock()
defer w.mu.Unlock()
if w.stopped || len(w.queue) >= w.maxQueued {
return false
}
mine := 0
for _, q := range w.queue {
if q.UserID == userID {
mine++
}
}
return mine < w.maxPerUser
}
// Submit queues a job. It never blocks. It satisfies Submitter.
func (w *Worker) Submit(j Job) error {
w.mu.Lock()
@@ -155,18 +114,6 @@ func (w *Worker) Submit(j Job) error {
if w.stopped {
return ErrStopped
}
if len(w.queue) >= w.maxQueued {
return ErrQueueFull
}
mine := 0
for _, q := range w.queue {
if q.UserID == j.UserID {
mine++
}
}
if mine >= w.maxPerUser {
return ErrQueueFull
}
w.queue = append(w.queue, j)
select {
case w.wake <- struct{}{}:
@@ -176,10 +123,9 @@ func (w *Worker) Submit(j Job) error {
}
// Run processes queued jobs until ctx ends, then deletes the audio of anything
// still queued and refuses new jobs. The job running at shutdown is recorded interrupted
// by Process; the rows of jobs still queued stay queued and are failed as interrupted by
// the start-up sweep (store.FailInterruptedUploads): the audio is not retained, so they
// cannot resume.
// still queued and refuses new jobs. The rows of those jobs stay queued or
// transcribing and are failed as interrupted by the start-up sweep
// (store.FailInterruptedUploads): the audio is not retained, so they cannot resume.
func (w *Worker) Run(ctx context.Context) {
defer w.stop()
for {
@@ -228,7 +174,7 @@ func (w *Worker) stop() {
// Process runs one job: row -> transcribing, convert, transcribe, store. The
// uploaded file and the converted WAV are deleted on every path — the audio is
// never retained (ADR-033). A failure is recorded with a TYPED kind; a shutdown
// mid-job is recorded as interrupted at once (see fail).
// mid-job is not a failure and leaves the row for the start-up sweep.
//
// Every phase is reported through one progress.Tracker, which saves it to the row for
// the page and logs it, and a heartbeat stamps the row on its own while the job runs
@@ -243,8 +189,8 @@ func (w *Worker) Process(ctx context.Context, job Job) {
start := time.Now()
if err := w.store.StartTranscribing(ctx, job.UserID, job.UploadID); err != nil {
log.Error("upload: start transcribing", "kind", errkind.Of(err))
w.fail(ctx, job, log, domain.UploadErrTranscriptionFailed)
log.Error("upload: start transcribing", "err", err)
w.fail(ctx, job, log)
return
}
@@ -270,18 +216,18 @@ func (w *Worker) Process(ctx context.Context, job Job) {
defer func() { stopBeat(); <-hbDone }() // no heartbeat outlives its job
// abort records where the job stopped. A shutdown (the parent context ended) is not
// a failure of the step: it stays as it was, and fail records the row as interrupted.
abort := func(kind domain.UploadErrorKind) {
// a failure: the row is left for the start-up sweep, so the step stays as it was.
abort := func() {
if ctx.Err() == nil {
tr.FailRunning()
}
w.fail(ctx, job, log, kind)
w.fail(ctx, job, log)
}
wav, err := w.tempWAV()
if err != nil {
log.Error("upload: temp wav", "kind", errkind.Of(err))
abort(domain.UploadErrTranscriptionFailed)
log.Error("upload: temp wav", "err", err)
abort()
return
}
defer func() { _ = os.Remove(wav) }()
@@ -290,22 +236,10 @@ func (w *Worker) Process(ctx context.Context, job Job) {
defer cancel()
tr.Begin("audio", 0)
if err := w.conv.ToWAV(jobCtx, job.Path, wav); err != nil {
log.Error("upload: convert", "kind", errkind.Of(err))
abort(domain.UploadErrTranscriptionFailed)
log.Error("upload: convert", "err", err)
abort()
return
}
got, tooLong := w.audioLengthProblem(wav)
if tooLong {
abort(domain.UploadErrTooLong)
log.Error("upload: converted audio is over the limit", "audio_s", int(got.Seconds()), "limit_s", int(w.maxDuration.Seconds()))
return
}
if w.maxDuration > 0 && job.Duration > 0 {
if note := shortfallNote(got, job.Duration); note != "" {
log.Warn("upload: audio is much shorter than the file claimed", "audio_s", int(got.Seconds()), "claimed_s", int(job.Duration.Seconds()))
tr.Annotate("audio", note)
}
}
tr.Done("audio")
if !planned {
@@ -313,8 +247,8 @@ func (w *Worker) Process(ctx context.Context, job Job) {
}
t, err := w.tr.Transcribe(progress.With(jobCtx, tr), wav)
if err != nil {
log.Error("upload: transcribe", "kind", errkind.Of(err), "elapsed", time.Since(start).Round(time.Second))
abort(domain.UploadErrTranscriptionFailed)
log.Error("upload: transcribe", "err", err, "elapsed", time.Since(start).Round(time.Second))
abort()
return
}
if !planned {
@@ -323,8 +257,8 @@ func (w *Worker) Process(ctx context.Context, job Job) {
tr.Begin("save", 0)
if err := w.store.CompleteUpload(ctx, job.UserID, job.UploadID, t); err != nil {
log.Error("upload: store transcript", "kind", errkind.Of(err))
abort(domain.UploadErrTranscriptionFailed)
log.Error("upload: store transcript", "err", err)
abort()
return
}
tr.Done("save")
@@ -333,22 +267,14 @@ func (w *Worker) Process(ctx context.Context, job Job) {
"segments", len(t.Segments), "elapsed", time.Since(start).Round(time.Second))
}
// interruptedWriteTimeout bounds the one write a dying process makes to record its job.
const interruptedWriteTimeout = 3 * time.Second
// fail records a transcription failure. When the PARENT context ended the process is
// shutting down: the job is recorded interrupted at once, on a short context of its own,
// because the process may exit before any other pod's start-up sweep would do it (#50).
// The sweep remains the backstop for a process that is killed outright.
func (w *Worker) fail(ctx context.Context, job Job, log *slog.Logger, kind domain.UploadErrorKind) {
// fail records a transcription failure — unless the PARENT context ended, which
// is a shutdown, not a failure: the row is left for the start-up sweep.
func (w *Worker) fail(ctx context.Context, job Job, log *slog.Logger) {
if ctx.Err() != nil {
kind = domain.UploadErrInterrupted
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(context.WithoutCancel(ctx), interruptedWriteTimeout)
defer cancel()
return
}
if err := w.store.FailUpload(ctx, job.UserID, job.UploadID, kind); err != nil {
log.Error("upload: record failure", "kind", errkind.Of(err))
if err := w.store.FailUpload(ctx, job.UserID, job.UploadID, domain.UploadErrTranscriptionFailed); err != nil {
log.Error("upload: record failure", "err", err)
}
}
@@ -364,42 +290,3 @@ func (w *Worker) tempWAV() (string, error) {
}
return name, nil
}
const (
// tooLongSlack is how far over the limit the converted audio may be before the claim
// in the file's header is called a lie: ffmpeg rounds.
tooLongSlack = 30 * time.Second
// A file counts as truncated when it converts to under 80% of the length it claimed
// AND more than two minutes short. Probe estimates for variable-bitrate files are
// imprecise; refusing real recordings would be worse than the problem.
incompleteBelow = 0.8
incompleteShortfall = 2 * time.Minute
)
// audioLengthProblem measures the converted WAV (16 kHz mono 16-bit PCM is 32,000 bytes a
// second after a 44-byte header) against the limit. It returns the real length and whether
// that is over the limit by more than the tolerance.
func (w *Worker) audioLengthProblem(wav string) (got time.Duration, tooLong bool) {
if w.maxDuration <= 0 {
return 0, false
}
info, err := os.Stat(wav)
if err != nil {
return 0, false
}
got = time.Duration(float64(max(info.Size()-44, 0)) / 32000 * float64(time.Second))
return got, got > w.maxDuration+tooLongSlack
}
// shortfallNote is a remark for the audio step when the converted audio is much shorter
// than the file claimed, or "" when it is not. It is a NOTE and never a failure: a
// container header can over-estimate a complete file (a variable-bitrate MP3 without a Xing
// header claimed 79 minutes for 20 real ones), so refusing would turn away real
// recordings, while saying nothing about a truncated one would hide a partial transcript.
func shortfallNote(got, claimed time.Duration) string {
if float64(got) >= incompleteBelow*float64(claimed) || claimed-got <= incompleteShortfall {
return ""
}
return fmt.Sprintf("The audio is %s long; the file claimed %s. If the recording was cut off, upload the complete one.",
FormatDuration(got), FormatDuration(claimed))
}
-187
View File
@@ -1,187 +0,0 @@
package uploads_test
import (
"context"
"os"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
// The duration limit is a claim in the file's header. The worker measures what was
// really converted and refuses both a recording that is longer than the limit (the
// header lied low) and one that ends far earlier than the file claimed (it was truncated
// and would otherwise yield a partial transcript with no warning) -- adversarial review F2.
// sizedConverter writes a (sparse) 16 kHz mono PCM WAV of the given length, without
// actually allocating it: 32,000 bytes per second plus a 44-byte header.
type sizedConverter struct{ seconds float64 }
func (c sizedConverter) ToWAV(_ context.Context, _, dst string) error {
f, err := os.Create(dst)
if err != nil {
return err
}
defer func() { _ = f.Close() }()
return f.Truncate(44 + int64(c.seconds*32000))
}
func durationWorker(t *testing.T, r *wrig, wavSeconds float64, max time.Duration) *uploads.Worker {
t.Helper()
opts := []uploads.WorkerOption{}
if max > 0 {
opts = append(opts, uploads.WithMaxDuration(max))
}
return uploads.NewWorker(r.st, sizedConverter{seconds: wavSeconds}, r.tr, r.tmp, time.Minute, opts...)
}
func jobWith(r *wrig, t *testing.T, claimed time.Duration) uploads.Job {
j := r.spool(t, "a")
j.Duration = claimed
return j
}
func TestWorkerDuration_AConvertedLengthOverTheLimitFailsTheJobBeforeAnyEngineSeesIt(t *testing.T) {
r := newWRig(t, time.Minute)
w := durationWorker(t, r, 100*60, 90*time.Minute)
w.Process(context.Background(), jobWith(r, t, 10*time.Second))
_, failed := r.st.snapshot()
assert.Equal(t, domain.UploadErrTooLong, failed["a"], "a header that claimed 10 s must not let 100 minutes through")
assert.Empty(t, r.tr.paths, "no engine may be sent audio that is over the limit")
assert.Empty(t, r.leftovers(t), "the audio is still deleted")
}
func TestWorkerDuration_JustOverTheLimitWithinTheSlackStillTranscribes(t *testing.T) {
r := newWRig(t, time.Minute)
w := durationWorker(t, r, 90*60+20, 90*time.Minute)
w.Process(context.Background(), jobWith(r, t, 90*time.Minute))
completed, failed := r.st.snapshot()
assert.Empty(t, failed)
assert.Contains(t, completed, "a", "ffmpeg rounds: 20 s over a 90-minute limit is not a lie")
}
func audioStepNote(r *wrig) string {
for _, st := range r.st.lastSteps() {
if st.Key == "audio" {
return st.Note
}
}
return ""
}
func TestWorkerDuration_AFileThatEndsFarEarlierThanItClaimedIsStillTranscribedAndSaysSo(t *testing.T) {
// A container header can over-estimate a complete file (VBR MP3 without a Xing header
// claimed 79 minutes for 20 real ones). Refusing it was wrong; staying silent about a
// truncated one is wrong too. So it is transcribed, and the audio step carries a note.
r := newWRig(t, time.Minute)
w := durationWorker(t, r, 20*60, 90*time.Minute)
w.Process(context.Background(), jobWith(r, t, 79*time.Minute))
completed, failed := r.st.snapshot()
assert.Empty(t, failed, "a complete recording with a bad header estimate must not be refused")
assert.Contains(t, completed, "a")
note := audioStepNote(r)
assert.Contains(t, note, "20 min", "what was really transcribed")
assert.Contains(t, note, "1 h 19 min", "what the file claimed")
assert.Contains(t, note, "cut off", "and what to do if that is not a surprise")
}
func TestWorkerDuration_TheShortfallNoteIsPinnedAtEightyPercentAndTwoMinutes(t *testing.T) {
tests := []struct {
name string
claimed time.Duration
wav float64
noted bool
}{
{"79 percent of an hour", 60 * time.Minute, 0.79 * 3600, true},
{"exactly 80 percent of an hour", 60 * time.Minute, 2880, false},
{"one second under 80 percent of an hour", 60 * time.Minute, 2879, true},
{"81 percent of an hour", 60 * time.Minute, 0.81 * 3600, false},
{"exactly two minutes short and under 80 percent", 6 * time.Minute, 4 * 60, false},
{"two minutes and a second short and under 80 percent", 6 * time.Minute, 4*60 - 1, true},
{"unknown claimed length", 0, 20 * 60, false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
r := newWRig(t, time.Minute)
w := durationWorker(t, r, tt.wav, 90*time.Minute)
w.Process(context.Background(), jobWith(r, t, tt.claimed))
assert.Equal(t, tt.noted, audioStepNote(r) != "")
completed, _ := r.st.snapshot()
assert.Contains(t, completed, "a", "a note never stops the job")
})
}
}
func TestWorkerDuration_TheTooLongToleranceIsExactlyThirtySeconds(t *testing.T) {
tests := []struct {
name string
wav float64
fail bool
}{
{"at the limit plus 30 s", 90*60 + 30, false},
{"at the limit plus 31 s", 90*60 + 31, true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
r := newWRig(t, time.Minute)
w := durationWorker(t, r, tt.wav, 90*time.Minute)
w.Process(context.Background(), jobWith(r, t, 90*time.Minute))
_, failed := r.st.snapshot()
if tt.fail {
assert.Equal(t, domain.UploadErrTooLong, failed["a"])
} else {
assert.Empty(t, failed)
}
})
}
}
func TestWorkerDuration_AnEstimateThatIsMerelyALittleOffIsNotTruncation(t *testing.T) {
tests := []struct {
name string
claimed time.Duration
wav float64
}{
{"five percent short", 60 * time.Minute, 57 * 60},
{"under two minutes short of a long file", 60 * time.Minute, 59 * 60},
{"unknown claimed length", 0, 20 * 60},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
r := newWRig(t, time.Minute)
w := durationWorker(t, r, tt.wav, 90*time.Minute)
w.Process(context.Background(), jobWith(r, t, tt.claimed))
completed, failed := r.st.snapshot()
assert.Empty(t, failed)
assert.Contains(t, completed, "a")
})
}
}
func TestWorkerDuration_WithNoMaximumConfiguredNothingIsMeasured(t *testing.T) {
r := newWRig(t, time.Minute)
w := durationWorker(t, r, 100*60, 0)
w.Process(context.Background(), jobWith(r, t, 0))
completed, _ := r.st.snapshot()
assert.Contains(t, completed, "a")
require.NotNil(t, w)
}
+1 -51
View File
@@ -146,7 +146,7 @@ func TestWorkerProgress_TheHeartbeatBeatsWhileAnEngineCallIsStuckAndStopsWhenThe
assert.Equal(t, after, r.st.beatCount(), "no heartbeat may outlive its job")
}
func TestWorkerProgress_OnASuccessfulJobTheLogNamesEveryPhaseInOrderAndNeverTheTranscriptOrThePath(t *testing.T) {
func TestWorkerProgress_TheLogNamesEveryPhaseInOrderAndNeverTheTranscriptOrThePath(t *testing.T) {
r := newWRig(t, time.Minute)
r.tr.result = domain.Transcription{Text: "zebrakaka secret meeting words", Endpoint: "iguana/kb-whisper", AudioSeconds: 3}
buf := &syncLog{}
@@ -162,54 +162,4 @@ func TestWorkerProgress_OnASuccessfulJobTheLogNamesEveryPhaseInOrderAndNeverTheT
assert.Contains(t, out, "upload=a", "every line says which upload it is about")
assert.NotContains(t, out, "zebrakaka", "no transcript text in the log")
assert.NotContains(t, out, r.tmp, "no spool path, which carries the upload's identity, in the log")
assert.Contains(t, out, "upload transcribed", "the job's own summary line")
assert.Contains(t, out, "endpoint=iguana/kb-whisper", "which endpoint served it is a quality fact")
}
func TestWorkerProgress_AFailureLogsAKindNeverTheErrorTextOrThePath(t *testing.T) {
tests := []struct {
name string
setup func(*wrig)
line string
}{
{"conversion fails and names the spool file", func(r *wrig) {
r.conv.err = errors.New("Error opening input file " + r.tmp + "/upload-a: Invalid data zebrakaka")
}, "upload: convert"},
{"every engine fails and the gateway echoed a word", func(r *wrig) {
r.tr.err = errors.New("gateway echoed: zebrakaka secret words")
}, "upload: transcribe"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
r := newWRig(t, time.Minute)
tt.setup(r)
buf := &syncLog{}
w := newProgressWorker(t, r, r.tr, uploads.WithLogger(slog.New(slog.NewTextHandler(buf, nil))))
w.Process(context.Background(), r.spool(t, "a"))
out := buf.String()
assert.NotContains(t, out, "zebrakaka", "provider text or file contents in the log")
assert.NotContains(t, out, r.tmp, "the spool path in the log")
assert.Contains(t, out, tt.line, "the failure must be logged at all, not only scrubbed")
assert.Contains(t, out, "kind=error")
assert.Contains(t, out, "upload=a")
})
}
}
func TestWorkerProgress_AStoreFailureLogsItsSQLStateNeverTheValueItQuotes(t *testing.T) {
// A Postgres error can quote the offending value; for a transcript row that is the
// transcript (fourth adversarial review). The log gets the SQLSTATE.
r := newWRig(t, time.Minute)
r.st.completeErr = dbErr{}
buf := &syncLog{}
w := newProgressWorker(t, r, r.tr, uploads.WithLogger(slog.New(slog.NewTextHandler(buf, nil))))
w.Process(context.Background(), r.spool(t, "a"))
out := buf.String()
assert.NotContains(t, out, "zebrakaka")
assert.Contains(t, out, "upload: store transcript")
assert.Contains(t, out, "kind=sql_22P02")
}
-108
View File
@@ -1,108 +0,0 @@
package uploads_test
import (
"context"
"errors"
"strconv"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
// The queue was append-only: 100,000 submits with the worker idle all succeeded, and each
// queued job holds up to the upload cap on disk until its turn (adversarial review F6).
func queueWorker(t *testing.T, total, perUser int) *uploads.Worker {
t.Helper()
r := newWRig(t, time.Minute)
return uploads.NewWorker(r.st, r.conv, r.tr, r.tmp, time.Minute, uploads.WithQueueLimits(total, perUser))
}
func submit(w *uploads.Worker, user string, n int) error {
return w.Submit(uploads.Job{UserID: user, UploadID: user + "-" + strconv.Itoa(n), Path: "/nonexistent"})
}
func TestWorkerSubmit_RefusesBeyondTheTotalQueueCap(t *testing.T) {
w := queueWorker(t, 3, 10)
for i := 0; i < 3; i++ {
require.NoError(t, submit(w, "u"+strconv.Itoa(i), i))
}
err := submit(w, "u9", 9)
assert.ErrorIs(t, err, uploads.ErrQueueFull)
}
func TestWorkerSubmit_OneUserCannotFillTheQueueForEveryoneElse(t *testing.T) {
w := queueWorker(t, 100, 2)
require.NoError(t, submit(w, "alice", 1))
require.NoError(t, submit(w, "alice", 2))
errAlice := submit(w, "alice", 3)
errBob := submit(w, "bob", 1)
assert.True(t, errors.Is(errAlice, uploads.ErrQueueFull), "a third waiting upload from one user is refused")
assert.NoError(t, errBob, "another user is not affected by alice's queue")
}
func TestWorkerSubmit_ThereIsADefaultCapSoANeglectedWorkerCannotBeFilledForever(t *testing.T) {
r := newWRig(t, time.Minute)
w := uploads.NewWorker(r.st, r.conv, r.tr, r.tmp, time.Minute)
refused := 0
for i := 0; i < 1000; i++ {
if err := submit(w, "u"+strconv.Itoa(i), i); errors.Is(err, uploads.ErrQueueFull) {
refused++
}
}
assert.Positive(t, refused, "with the defaults, 1,000 submits to an idle worker must not all be accepted")
}
func TestWorkerSubmit_AJobThatHasLeftTheQueueFreesItsPlace(t *testing.T) {
r := newWRig(t, time.Minute)
w := uploads.NewWorker(r.st, r.conv, r.tr, r.tmp, time.Minute, uploads.WithQueueLimits(1, 1))
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { defer close(done); w.Run(ctx) }()
defer func() { cancel(); <-done }()
job := r.spool(t, "first")
require.NoError(t, w.Submit(job))
require.Eventually(t, func() bool { c, _ := r.st.snapshot(); _, ok := c["first"]; return ok },
5*time.Second, 5*time.Millisecond, "the first job never completed")
assert.NoError(t, w.Submit(r.spool(t, "second")), "the cap counts jobs waiting, not jobs ever submitted")
}
func TestWorkerHasRoom_SaysWhetherASubmitWouldBeAccepted(t *testing.T) {
w := queueWorker(t, 3, 2)
assert.True(t, w.HasRoom("alice"))
require.NoError(t, submit(w, "alice", 1))
require.NoError(t, submit(w, "alice", 2))
assert.False(t, w.HasRoom("alice"), "alice is at her share")
assert.True(t, w.HasRoom("bob"), "bob is not")
require.NoError(t, submit(w, "bob", 1))
assert.False(t, w.HasRoom("carol"), "the queue as a whole is at its cap")
}
func TestWorkerHasRoom_AgreesWithSubmitAtEveryStepAndIsFalseOnceStopped(t *testing.T) {
w := queueWorker(t, 2, 1)
for i, user := range []string{"a", "a", "b", "c"} {
room := w.HasRoom(user)
err := submit(w, user, i)
assert.Equal(t, room, err == nil, "step %d (%s): HasRoom=%v but Submit returned %v", i, user, room, err)
}
r := newWRig(t, time.Minute)
stopped := uploads.NewWorker(r.st, r.conv, r.tr, r.tmp, time.Minute)
ctx, cancel := context.WithCancel(context.Background())
cancel()
stopped.Run(ctx)
assert.False(t, stopped.HasRoom("a"), "a worker that has stopped has no room")
}
+4 -26
View File
@@ -23,8 +23,6 @@ type wStore struct {
completed map[string]domain.Transcription
failed map[string]domain.UploadErrorKind
startErr error
// completeErr makes CompleteUpload fail, as a database outage would.
completeErr error
saves [][]domain.ProgressStep
beats int
// beatDelay makes each heartbeat write slow, so one is still in flight when a job
@@ -46,17 +44,11 @@ func (s *wStore) StartTranscribing(_ context.Context, _, id string) error {
func (s *wStore) CompleteUpload(_ context.Context, _, id string, t domain.Transcription) error {
s.mu.Lock()
defer s.mu.Unlock()
if s.completeErr != nil {
return s.completeErr
}
s.completed[id] = t
return nil
}
func (s *wStore) FailUpload(ctx context.Context, _, id string, k domain.UploadErrorKind) error {
if err := ctx.Err(); err != nil { // like the real store: a cancelled context cannot write
return err
}
func (s *wStore) FailUpload(_ context.Context, _, id string, k domain.UploadErrorKind) error {
s.mu.Lock()
defer s.mu.Unlock()
s.failed[id] = k
@@ -216,10 +208,6 @@ func TestWorkerProcess_FailuresAreTypedAndStillDeleteTheAudio(t *testing.T) {
}{
{"conversion fails", func(r *wrig) { r.conv.err = errors.New("moov atom not found") }},
{"every transcription endpoint fails", func(r *wrig) { r.tr.err = errors.New("iguana down; berget down") }},
{"storing the finished transcript fails", func(r *wrig) { r.st.completeErr = errors.New("db down") }},
{"the temporary WAV cannot be created", func(r *wrig) {
r.w = uploads.NewWorker(r.st, r.conv, r.tr, filepath.Join(r.tmp, "missing"), time.Minute)
}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
@@ -245,9 +233,7 @@ func TestWorkerProcess_FailuresAreTypedAndStillDeleteTheAudio(t *testing.T) {
})
}
func TestWorkerProcess_ShutdownMidJobRecordsTheRowInterruptedAtOnce(t *testing.T) {
// SIGTERM on a deploy used to leave the row "transcribing" until the NEXT pod's
// start-up sweep; the process may exit before any sweep runs (#50).
func TestWorkerProcess_ShutdownMidJobLeavesTheRowForTheStartupSweep(t *testing.T) {
r := newWRig(t, time.Minute)
r.tr.block = make(chan struct{})
ctx, cancel := context.WithCancel(context.Background())
@@ -260,8 +246,8 @@ func TestWorkerProcess_ShutdownMidJobRecordsTheRowInterruptedAtOnce(t *testing.T
<-done
_, f := r.st.snapshot()
assert.Equal(t, domain.UploadErrInterrupted, f["a"], "the row is recorded interrupted by the dying process, not left for a sweep")
assert.Empty(t, r.leftovers(t), "and the audio is still deleted")
assert.Empty(t, f, "a shutdown is not a transcription failure: the row stays for FailInterruptedUploads to mark interrupted")
assert.Empty(t, r.leftovers(t), "but the audio is still deleted")
}
func TestWorkerProcess_AJobThatOutlivesItsTimeoutIsFailedNotStuck(t *testing.T) {
@@ -329,11 +315,3 @@ func TestWorkerRun_StoppingDeletesQueuedAudioAndRefusesNewJobs(t *testing.T) {
assert.Empty(t, r.leftovers(t), "audio of queued jobs is deleted on stop: the rows are swept to interrupted at next start")
assert.ErrorIs(t, r.w.Submit(r.spool(t, "c")), uploads.ErrStopped)
}
// dbErr is an error shaped like pgx's: a message that quotes a value, and an SQLSTATE.
type dbErr struct{}
func (dbErr) Error() string {
return `ERROR: invalid input syntax for type json: "zebrakaka secret words"`
}
func (dbErr) SQLState() string { return "22P02" }
-137
View File
@@ -1,137 +0,0 @@
package web_test
// Observes the cross-origin gate around Router (#48): SameSite=Lax does not stop a POST
// from a sibling *.d-ma.be origin, so the router itself must refuse a non-safe request
// whose Origin or Sec-Fetch-Site says it did not come from this site's own pages.
import (
"context"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// foreign is a header set a browser sends for a form post made by another page.
var foreign = map[string]map[string]string{
"sibling subdomain, Sec-Fetch-Site": {"Origin": "https://evil.d-ma.be", "Sec-Fetch-Site": "same-site"},
"other site, Sec-Fetch-Site": {"Origin": "https://evil.example", "Sec-Fetch-Site": "cross-site"},
"sibling subdomain, Origin only": {"Origin": "https://evil.d-ma.be"},
"opaque origin": {"Origin": "null"},
}
// own is a header set for the app's own pages (or a non-browser client): must pass.
var own = map[string]map[string]string{
"same-origin Sec-Fetch-Site": {"Sec-Fetch-Site": "same-origin", "Origin": "http://example.com"},
"matching Origin only": {"Origin": "http://example.com"},
"no browser headers": {},
}
func withHeaders(req *http.Request, h map[string]string) *http.Request {
for k, v := range h {
req.Header.Set(k, v)
}
return req
}
func userCount(t *testing.T) int {
t.Helper()
var n int
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT count(*) FROM users WHERE id = $1`, userID).Scan(&n))
return n
}
func TestCrossOrigin_ARefusedDeleteAccountPostLeavesTheAccountInPlace(t *testing.T) {
for name, h := range foreign {
t.Run(name, func(t *testing.T) {
app, _ := newAccountApp(t)
resetDB(t, rawPool(t))
seedConnection(t, app, "youtube", "ada@channel", "ref")
rec := do(t, app, withHeaders(httptest.NewRequest(http.MethodPost, "/account/delete", nil), h))
assert.Equal(t, http.StatusForbidden, rec.Code)
assert.Equal(t, 1, userCount(t), "the account must survive a refused post")
})
}
}
func TestCrossOrigin_ASameOriginOrHeaderlessDeleteAccountPostIsServed(t *testing.T) {
for name, h := range own {
t.Run(name, func(t *testing.T) {
app, _ := newAccountApp(t)
resetDB(t, rawPool(t))
seedConnection(t, app, "youtube", "ada@channel", "ref")
rec := do(t, app, withHeaders(httptest.NewRequest(http.MethodPost, "/account/delete", nil), h))
assert.Equal(t, http.StatusSeeOther, rec.Code)
assert.Equal(t, "/auth/logout", rec.Header().Get("Location"))
})
}
}
func TestCrossOrigin_ARefusedUploadPostSubmitsNoJob(t *testing.T) {
for name, h := range foreign {
t.Run(name, func(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
rec := do(t, app, withHeaders(uploadReq(t, "m", "m.wav", []byte("RIFF....WAVE")), h))
assert.Equal(t, http.StatusForbidden, rec.Code)
assert.Empty(t, sub.jobs, "a refused post must not enqueue transcription")
})
}
}
func TestCrossOrigin_ARefusedUploadChatPostCallsNoModel(t *testing.T) {
for name, h := range foreign {
t.Run(name, func(t *testing.T) {
app, fc := newUploadChatApp(t)
id := seedUpload(t, userID, "Roadmap sync", "done", meetingText)
req := chatPost("/uploads/"+id+"/chat", url.Values{"message": {"hej"}})
rec := do(t, app, withHeaders(req, h))
assert.Equal(t, http.StatusForbidden, rec.Code)
assert.Zero(t, fc.callCount)
})
}
}
func TestCrossOrigin_AForeignPostIsRefusedOnEachListedStateChangingPath(t *testing.T) {
routes := []string{
"/account/disconnect/youtube",
"/account/summarize-mode",
"/register",
"/paste",
"/v/abc/action",
"/v/abc/summarize",
"/v/abc/retry-now",
"/v/abc/chat",
}
for _, path := range routes {
t.Run(path, func(t *testing.T) {
app := newApp(t)
req := httptest.NewRequest(http.MethodPost, path, strings.NewReader(""))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
rec := do(t, app, withHeaders(req, foreign["sibling subdomain, Sec-Fetch-Site"]))
assert.Equal(t, http.StatusForbidden, rec.Code)
})
}
}
// A login redirect comes back to /auth/callback as a cross-site GET navigation, and
// health probes carry no headers: safe methods must never be refused on origin grounds.
func TestCrossOrigin_ACrossSiteGETIsNotRefused(t *testing.T) {
app := newApp(t)
for _, path := range []string{"/healthz", "/welcome"} {
req := httptest.NewRequest(http.MethodGet, path, nil)
req.Header.Set("Sec-Fetch-Site", "cross-site")
assert.Equal(t, http.StatusOK, do(t, app, req).Code, path)
}
}
+1 -11
View File
@@ -122,11 +122,6 @@ type App struct {
// summarizer's (koala/phi4-mini, 8,192-token context) and cannot hold an hour of
// speech.
UploadChatModel string
// UploadChatModels is the set of models chat over an uploaded recording offers and
// accepts. The whole private transcript goes to the one chosen, so it must hold
// LOCAL models only, and the caller decides that. Nil or empty fails closed: no
// model is offered and none is called (ADR-035 review).
UploadChatModels []string
// Processing tracks in-flight immediate summarizations so the status endpoint
// shows the animation until the summary lands. The zero value is ready to use.
Processing ProcessingSet
@@ -229,12 +224,7 @@ func (a *App) Router() http.Handler {
// resolved user_id. /register lives inside the auth guard but is exempt from
// the registration gate (you must be able to reach it before you have a user).
root.Handle("/", a.Auth.Middleware(a.registrationGate(app)))
// The session cookie is SameSite=Lax, which still travels on a POST made from a
// sibling *.d-ma.be page (same site, different origin). Refuse any non-safe request
// whose Origin or Sec-Fetch-Site says it did not come from this origin (#48). It wraps
// everything, so a route added later is covered without remembering to ask.
return http.NewCrossOriginProtection().Handler(root)
return root
}
// handleWelcome renders the public landing page (/welcome). It is mounted outside
+11 -140
View File
@@ -8,10 +8,8 @@ import (
"math"
"net/http"
"path/filepath"
"regexp"
"strings"
"time"
"unicode"
"unicode/utf8"
"github.com/a-h/templ"
@@ -19,7 +17,6 @@ import (
"git.d-ma.be/mathias/tapir/internal/adapters/chat"
"git.d-ma.be/mathias/tapir/internal/adapters/store"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/errkind"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
@@ -94,32 +91,13 @@ func (a *App) handleUploadPost(w http.ResponseWriter, r *http.Request) {
break
}
if err != nil {
var tooBig *http.MaxBytesError
if a.abandoned(w, r) {
return
}
if !errors.As(err, &tooBig) {
// A malformed multipart body (a header with a control byte, a truncated
// part) is the sender's to fix: a 400, not a server error, and the error
// text is never logged: it quotes the headers, which carry the file name.
a.logger().Info("upload rejected", "reason", "malformed")
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "That upload could not be read. Please try again.")
return
}
a.uploadFailure(w, r, userID, err)
return
}
switch part.FormName() {
case "title":
b, err := io.ReadAll(io.LimitReader(part, maxTitleRunes*utf8.UTFMax))
if err != nil {
if a.abandoned(w, r) {
return
}
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "That upload could not be read. Please try again.")
return
}
title = cleanTitle(string(b))
b, _ := io.ReadAll(io.LimitReader(part, maxTitleRunes*utf8.UTFMax))
title = clipRunes(strings.TrimSpace(string(b)), maxTitleRunes)
case "file":
if title == "" {
title = titleFromFilename(part.FileName())
@@ -136,27 +114,10 @@ func (a *App) handleUploadPost(w http.ResponseWriter, r *http.Request) {
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "Choose an audio file to upload.")
}
// abandoned reports, logs at INFO and answers 499, because the client has already gone: its request
// context is cancelled, so there is nobody to answer and rendering a page would run its
// database read on a dead context and log a false ERROR. A hang-up mid-upload is routine.
func (a *App) abandoned(w http.ResponseWriter, r *http.Request) bool {
if r.Context().Err() == nil {
return false
}
a.logger().Info("upload abandoned", "reason", "client_gone")
// 499 (the "client closed request" convention): a client that half-closed after sending
// can still read, and an unwritten status would reach it as a false, empty 200.
w.WriteHeader(499)
return true
}
// uploadFailure maps an Accept (or multipart) error to a response: a validation
// rejection is the user's to fix (422, or 413 for size) and carries its own
// message; anything else is ours.
func (a *App) uploadFailure(w http.ResponseWriter, r *http.Request, userID string, err error) {
if a.abandoned(w, r) {
return
}
var rej *uploads.RejectError
var tooBig *http.MaxBytesError
switch {
@@ -164,26 +125,16 @@ func (a *App) uploadFailure(w http.ResponseWriter, r *http.Request, userID strin
// Logged by reason only: the filename and title are private content.
a.logger().Info("upload rejected", "reason", string(rej.Reason))
status := http.StatusUnprocessableEntity
switch rej.Reason {
case uploads.ReasonTooLarge:
if rej.Reason == uploads.ReasonTooLarge {
status = http.StatusRequestEntityTooLarge
case uploads.ReasonBusy:
status = http.StatusTooManyRequests
}
a.renderUploadsPage(w, r, userID, status, rej.Message)
case errors.As(err, &tooBig):
a.logger().Info("upload rejected", "reason", string(uploads.ReasonTooLarge))
a.renderUploadsPage(w, r, userID, http.StatusRequestEntityTooLarge,
"This file is larger than the upload limit.")
case errors.Is(err, uploads.ErrClientRead):
// An error reading the CLIENT's body: theirs to retry, not a server fault. Only that
// marker counts: io.ErrUnexpectedEOF alone is also what a database drop looks like.
a.logger().Info("upload rejected", "reason", "truncated")
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "The upload was cut off before it finished. Please try again.")
default:
// Not err: it can carry the file's name, a spool path or a database value.
a.logger().Error("upload failed", "kind", errkind.Of(err))
http.Error(w, "internal error", http.StatusInternalServerError)
a.serverError(w, r, "upload", err)
}
}
@@ -214,14 +165,7 @@ func (a *App) loadOwnedUpload(w http.ResponseWriter, r *http.Request) (*domain.U
if !ok {
return nil, false
}
id := r.PathValue("id")
if !uuidShape.MatchString(id) {
// Not an id at all: the same 404 as an unknown or foreign one, never a database
// error that surfaces as a 500 (and no oracle about which ids exist).
http.NotFound(w, r)
return nil, false
}
u, err := a.Store.GetUpload(r.Context(), userID, id)
u, err := a.Store.GetUpload(r.Context(), userID, r.PathValue("id"))
if errors.Is(err, store.ErrUploadNotFound) {
http.NotFound(w, r)
return nil, false
@@ -236,49 +180,11 @@ func (a *App) loadOwnedUpload(w http.ResponseWriter, r *http.Request) (*domain.U
// titleFromFilename is the default title: the file's name without its extension.
func titleFromFilename(name string) string {
base := filepath.Base(name)
t := cleanTitle(strings.TrimSuffix(base, filepath.Ext(base)))
t := strings.TrimSuffix(base, filepath.Ext(base))
if t == "" || t == "." {
return "Untitled recording"
}
return t
}
// hasNote reports whether any phase carries a note. A note is the only warning for a
// recording that is shorter than its file claimed, so a record that has one is shown open.
func hasNote(steps []domain.ProgressStep) bool {
for _, s := range steps {
if s.Note != "" {
return true
}
}
return false
}
// uuidShape is the form of an upload id. Anything else cannot be one, so it is a 404.
var uuidShape = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
// isBidiControl reports the invisible characters that change text direction. Emoji joiners
// (ZWJ, ZWNJ) and variation selectors are not among them and are kept.
func isBidiControl(r rune) bool {
return r >= 0x202A && r <= 0x202E || r >= 0x2066 && r <= 0x2069 || r == 0x200E || r == 0x200F || r == 0x061C
}
// cleanTitle makes a title safe to store and show: invalid UTF-8 dropped, control
// characters (NUL, newlines, escapes) turned into spaces, runs of space collapsed, trimmed
// and clipped. A title is cosmetic; bad bytes in it must never fail an upload, and Postgres
// refuses NUL in text.
func cleanTitle(s string) string {
s = strings.ToValidUTF8(s, "")
s = strings.Map(func(r rune) rune {
switch {
case isBidiControl(r):
return -1 // dropped: U+202E would reverse the rest of the title on its owner's page
case unicode.IsControl(r):
return ' '
}
return r
}, s)
return clipRunes(strings.Join(strings.Fields(s), " "), maxTitleRunes)
return clipRunes(t, maxTitleRunes)
}
func clipRunes(s string, n int) string {
@@ -294,10 +200,6 @@ func uploadFailureMessage(k domain.UploadErrorKind) string {
switch k {
case domain.UploadErrInterrupted:
return "The server restarted before this recording was transcribed. Please upload it again."
case domain.UploadErrBusy:
return "Too many recordings were waiting to be transcribed when this one was uploaded. Please upload it again in a few minutes."
case domain.UploadErrTooLong:
return "This recording is longer than the limit once its audio is read (the file's header said otherwise). Please upload a shorter recording."
case domain.UploadErrTranscriptionFailed:
return "Transcription failed on every available service. Please upload it again later."
default:
@@ -318,8 +220,7 @@ func (a *App) handleUploadChat(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/uploads/"+u.ID, http.StatusSeeOther)
return
}
model, _ := a.resolveUploadModel("")
a.render(w, r, UploadChatPage(*u, a.uploadChatView(*u, model, nil, false, "")))
a.render(w, r, UploadChatPage(*u, a.uploadChatView(*u, a.Chat.DefaultModel(a.UploadChatModel), nil, false, "")))
}
// handleUploadChatMessage answers one question against the owner's uploaded
@@ -339,13 +240,7 @@ func (a *App) handleUploadChatMessage(w http.ResponseWriter, r *http.Request) {
http.Error(w, "bad form", http.StatusBadRequest)
return
}
model, ok := a.resolveUploadModel(r.FormValue("model"))
if !ok {
a.logger().Error("upload chat: no local model is configured; nothing was sent", "upload", u.ID)
view := a.uploadChatView(*u, "", nil, false, "No local model is available for chat about a recording.")
a.render(w, r, UploadChatPage(*u, view))
return
}
model := a.resolveModel(r.FormValue("model"), a.UploadChatModel)
history := parseHistory(r.Form["hq"], r.Form["ha"])
question := strings.TrimSpace(r.FormValue("question"))
@@ -359,7 +254,7 @@ func (a *App) handleUploadChatMessage(w http.ResponseWriter, r *http.Request) {
MaxChars: a.UploadChatMaxChars,
})
if err != nil {
a.logger().Error("upload chat answer", "upload", u.ID, "model", model, "kind", errkind.Of(err))
a.logger().Error("upload chat answer", "upload", u.ID, "model", model, "err", err)
view.Error = "That model couldn't answer just now. Try again, or switch models."
} else {
view.History = appendTurn(history, chat.Turn{Question: question, Answer: reply.Answer})
@@ -378,7 +273,7 @@ func (a *App) uploadChatView(u domain.UploadedTranscript, model string, history
VideoID: u.ID,
Title: u.Title,
Available: true,
Models: a.UploadChatModels,
Models: a.Chat.Models(),
Selected: model,
History: history,
Truncated: truncated,
@@ -506,27 +401,3 @@ func minutesAgo(d time.Duration) string {
}
return "under a minute"
}
// resolveUploadModel picks the model for chat over an uploaded recording from the
// offered LOCAL set only: the posted one when it is in the set, else the configured
// default when it is, else the first. ok is false when no model is offered, in which
// case nothing may be sent anywhere (the whole private transcript is the payload).
func (a *App) resolveUploadModel(posted string) (model string, ok bool) {
offered := func(m string) bool {
for _, o := range a.UploadChatModels {
if o == m {
return true
}
}
return false
}
switch {
case posted != "" && offered(posted):
return posted, true
case a.UploadChatModel != "" && offered(a.UploadChatModel):
return a.UploadChatModel, true
case len(a.UploadChatModels) > 0:
return a.UploadChatModels[0], true
}
return "", false
}
+1 -4
View File
@@ -68,7 +68,7 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
</p>
@reconciliationNote(u.Reconciliation)
if len(u.Progress) > 0 {
<details class="what-was-done" if hasNote(u.Progress) { open }>
<details class="what-was-done">
<summary>What was done</summary>
@progressSteps(u.Progress, now, false)
</details>
@@ -218,9 +218,6 @@ templ progressSteps(steps []domain.ProgressStep, now time.Time, live bool) {
if t := stepTime(s, now); t != "" {
<span class="step-time">{ t }</span>
}
if s.Note != "" {
<span class="step-note">{ s.Note }</span>
}
if s.State == domain.StepRunning && s.Total > 0 {
<progress class="step-bar" value={ strconv.Itoa(s.Done) } max={ strconv.Itoa(s.Total) }></progress>
<span class="step-count">{ strconv.Itoa(s.Done) } of { strconv.Itoa(s.Total) }</span>
@@ -1,93 +0,0 @@
package web_test
import (
"context"
"fmt"
"log/slog"
"net/http"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// Input that cannot be right is the caller's to fix: a 4xx, never a 500 (adversarial
// review F7). No cross-tenant oracle either way: unknown and foreign ids look the same.
func TestUploadRoutes_ANonUUIDIdIs404NotA500(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
for _, path := range []string{"/uploads/not-a-uuid", "/uploads/not-a-uuid/status", "/uploads/not-a-uuid/chat", "/uploads/%00/status", "/uploads/1/status"} {
t.Run(path, func(t *testing.T) {
rec := do(t, app, get(path))
assert.Equal(t, http.StatusNotFound, rec.Code)
})
}
}
func TestUploadPost_ATitleWithControlCharactersOrInvalidUTF8IsCleanedNotA500(t *testing.T) {
tests := []struct{ name, title, filename string }{
{"a NUL in the title", "Sprint\x00review", "a.m4a"},
{"invalid UTF-8 in the title", "Sprint \xff\xfe review", "a.m4a"},
{"a newline in the title", "Sprint\nreview", "a.m4a"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
rec := do(t, app, uploadReq(t, tt.title, tt.filename, []byte("synthetic audio bytes")))
require.Equal(t, http.StatusSeeOther, rec.Code, "bad bytes in a cosmetic title must not fail the upload: %s", body(t, rec))
require.Len(t, sub.jobs, 1)
var title string
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT title FROM uploaded_transcripts WHERE id = $1`, sub.jobs[0].UploadID).Scan(&title))
assert.NotContains(t, title, "\x00")
assert.NotContains(t, title, "\n")
assert.True(t, len(title) > 0, "a title is still shown")
})
}
}
func TestUploadPost_AMalformedMultipartHeaderIsA400AndItsTextIsNotLogged(t *testing.T) {
// A control byte in the part's file name makes the multipart body malformed. It is the
// sender's to fix, and the parser's error quotes the header, which carries the name.
app, sub := newUploadApp(t, audioOK, 200<<20)
buf := &lockedBuf{}
app.Log = slog.New(slog.NewTextHandler(buf, nil))
rec := do(t, app, uploadReq(t, "", "zebrakaka\x00weekly.m4a", []byte("x")))
assert.Equal(t, http.StatusBadRequest, rec.Code)
assert.Empty(t, sub.jobs)
assert.NotContains(t, buf.String(), "zebrakaka", "a file name is private content")
}
func TestUploadPost_EveryBidirectionalControlCharacterIsStrippedFromATitle(t *testing.T) {
// U+202E reverses the text that follows it, so a title could render reversed on its
// owner's own page (second adversarial review). Every one of the twelve is pinned.
controls := []rune{0x202A, 0x202B, 0x202C, 0x202D, 0x202E, 0x2066, 0x2067, 0x2068, 0x2069, 0x200E, 0x200F, 0x061C}
for _, c := range controls {
t.Run(fmt.Sprintf("U+%04X", c), func(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
rec := do(t, app, uploadReq(t, "Sprint"+string(c)+"review", "a.m4a", []byte("x")))
require.Equal(t, http.StatusSeeOther, rec.Code)
var title string
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT title FROM uploaded_transcripts WHERE id = $1`, sub.jobs[0].UploadID).Scan(&title))
assert.Equal(t, "Sprintreview", title, "dropped, not replaced by a space: a space would leave a visible gap")
})
}
}
func TestUploadPost_SwedishLettersAndEmojiSurviveTheTitleCleaning(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
// Includes a ZWJ sequence (woman + ZWJ + laptop) and a ZWNJ: joiners are not direction controls.
rec := do(t, app, uploadReq(t, "Möte åäö 🎙️ 👩\u200d💻 a\u200cb", "a.m4a", []byte("x")))
require.Equal(t, http.StatusSeeOther, rec.Code)
var title string
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT title FROM uploaded_transcripts WHERE id = $1`, sub.jobs[0].UploadID).Scan(&title))
assert.Equal(t, "Möte åäö 🎙️ 👩\u200d💻 a\u200cb", title, "joiners and variation selectors belong to the text")
}
-31
View File
@@ -1,31 +0,0 @@
package web_test
import (
"net/http"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
func TestUploadPost_ABusyRejectionIs429WithItsMessage(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
app.Uploads = failingUploader{err: &uploads.RejectError{Reason: uploads.ReasonBusy, Message: "Too many recordings are waiting. Try again in a few minutes."}}
rec := do(t, app, uploadReq(t, "t", "a.m4a", []byte("x")))
require.Equal(t, http.StatusTooManyRequests, rec.Code, "the user's to retry: not a 500, not a 422")
assert.Contains(t, body(t, rec), "Too many recordings are waiting")
}
func TestUploadStatus_ABusyFailureIsExplainedInWords(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "failed", "queue_full")
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.Contains(t, b, "waiting")
assert.NotContains(t, b, "This upload failed. Please upload it again.")
}
@@ -31,7 +31,6 @@ func newUploadChatApp(t *testing.T) (*web.App, *fakeChatter) {
}
app.Chat = fc
app.UploadChatMaxChars = 120000
app.UploadChatModels = fc.models
return app, fc
}
@@ -1,73 +0,0 @@
package web_test
import (
"net/http"
"net/url"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/chat"
"git.d-ma.be/mathias/tapir/internal/web"
)
// The whole private transcript goes to the model chosen in the form, so the upload chat
// offers and accepts local models only, even when the shared chat service knows a cloud
// one (ADR-035 review). The cloud model here is only a name; no network is involved.
func newUploadChatAppWithCloud(t *testing.T) (*web.App, *fakeChatter) {
t.Helper()
app, _ := newUploadApp(t, audioOK, 200<<20)
fc := &fakeChatter{
models: []string{"koala/phi4-mini", "iguana/gemma4-26b", "berget/mistral-small"},
reply: chat.Reply{Answer: "Anders owns the timber order."},
}
app.Chat = fc
app.UploadChatMaxChars = 120000
app.UploadChatModels = []string{"koala/phi4-mini", "iguana/gemma4-26b"}
return app, fc
}
func TestUploadChatPage_DoesNotOfferACloudModelEvenThoughTheServiceKnowsOne(t *testing.T) {
app, _ := newUploadChatAppWithCloud(t)
id := seedUpload(t, userID, "Roadmap sync", "done", meetingText)
b := body(t, do(t, app, get("/uploads/"+id+"/chat")))
assert.Contains(t, b, "iguana/gemma4-26b")
assert.NotContains(t, b, "berget/mistral-small", "the switcher must not list a model the transcript must not be sent to")
}
func TestUploadChatPost_AForgedCloudModelIsReplacedByALocalOneBeforeTheTranscriptIsSent(t *testing.T) {
app, fc := newUploadChatAppWithCloud(t)
id := seedUpload(t, userID, "Roadmap sync", "done", meetingText)
rec := do(t, app, chatPost("/uploads/"+id+"/chat", url.Values{"question": {"Who owns timber?"}, "model": {"berget/mistral-small"}}))
require.Equal(t, http.StatusOK, rec.Code)
require.Equal(t, 1, fc.callCount)
assert.NotEqual(t, "berget/mistral-small", fc.gotReq.Model, "a cloud model reached the chat service with the private transcript")
assert.Contains(t, []string{"koala/phi4-mini", "iguana/gemma4-26b"}, fc.gotReq.Model)
}
func TestUploadChatPost_ALocalModelThatIsOfferedIsHonoured(t *testing.T) {
app, fc := newUploadChatAppWithCloud(t)
id := seedUpload(t, userID, "Roadmap sync", "done", meetingText)
rec := do(t, app, chatPost("/uploads/"+id+"/chat", url.Values{"question": {"q"}, "model": {"koala/phi4-mini"}}))
require.Equal(t, http.StatusOK, rec.Code)
assert.Equal(t, "koala/phi4-mini", fc.gotReq.Model)
}
func TestUploadChat_WithNoLocalModelConfiguredNothingIsSentAnywhere(t *testing.T) {
app, fc := newUploadChatAppWithCloud(t)
app.UploadChatModels = nil
id := seedUpload(t, userID, "Roadmap sync", "done", meetingText)
rec := do(t, app, chatPost("/uploads/"+id+"/chat", url.Values{"question": {"q"}, "model": {"berget/mistral-small"}}))
require.Equal(t, http.StatusOK, rec.Code)
assert.Zero(t, fc.callCount, "failing closed: no local model offered means no model is called")
}
@@ -1,124 +0,0 @@
package web_test
import (
"context"
"fmt"
"io"
"log/slog"
"net"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// A client that hangs up in the middle of an upload is routine, not a server fault. The
// in-process recorder cannot see it (its context never cancels), so this talks to a real
// listener (third adversarial review: over a socket every such case logged an ERROR and
// half-close returned a 500, because the error page was rendered on a cancelled context).
// body positions at which the client can hang up: inside the title field, inside the file
// part's own headers, and mid-file. Every one must be handled alike (fourth adversarial
// review: the title branch had no guard and still logged an ERROR).
const (
inTitle = "in the title"
inHeaders = "in the file part's headers"
midFile = "mid-file"
)
func partialUpload(t *testing.T, addr, where string, hangUp func(*net.TCPConn)) {
t.Helper()
c, err := net.Dial("tcp", addr)
require.NoError(t, err)
tc := c.(*net.TCPConn)
const boundary = "xBOUNDARYx"
var body string
switch where {
case inTitle:
body = "--" + boundary + "\r\nContent-Disposition: form-data; name=\"title\"\r\n\r\nhalf a tit"
case inHeaders:
body = "--" + boundary + "\r\nContent-Disposition: form-data; name=\"file\"; filena"
default:
body = "--" + boundary + "\r\nContent-Disposition: form-data; name=\"file\"; filename=\"a.m4a\"\r\nContent-Type: audio/mp4\r\n\r\n" + strings.Repeat("a", 4000)
}
_, err = fmt.Fprintf(tc, "POST /uploads HTTP/1.1\r\nHost: x\r\nContent-Type: multipart/form-data; boundary=%s\r\nContent-Length: 200000\r\n\r\n%s", boundary, body)
require.NoError(t, err)
time.Sleep(50 * time.Millisecond) // let the server start reading
hangUp(tc)
}
func TestUploadPost_AClientThatHangsUpAnywhereInTheBodyIsLoggedAtInfoAndNeverAsAnError(t *testing.T) {
hangUps := map[string]func(*net.TCPConn){
"a half close": func(c *net.TCPConn) { _ = c.CloseWrite(); time.Sleep(100 * time.Millisecond); _ = c.Close() },
"a full close": func(c *net.TCPConn) { _ = c.Close() },
"a reset (RST)": func(c *net.TCPConn) { _ = c.SetLinger(0); _ = c.Close() },
}
for _, where := range []string{inTitle, inHeaders, midFile} {
for name, hangUp := range hangUps {
t.Run(where+", "+name, func(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
buf := &lockedBuf{}
app.Log = slog.New(slog.NewTextHandler(buf, nil))
srv := httptest.NewServer(app.Router())
defer srv.Close()
partialUpload(t, strings.TrimPrefix(srv.URL, "http://"), where, hangUp)
require.Eventually(t, func() bool {
out := buf.String()
return strings.Contains(out, "upload abandoned") || strings.Contains(out, "upload rejected") || strings.Contains(out, "level=ERROR")
}, 3*time.Second, 20*time.Millisecond, "the server never finished handling the aborted upload")
time.Sleep(100 * time.Millisecond)
out := buf.String()
assert.NotContains(t, out, "level=ERROR", "a client hanging up is not a server fault:\n%s", out)
assert.Empty(t, sub.jobs)
})
}
}
}
// goneMidAccept is an Uploader during whose Accept the client disappears: the request's
// context is cancelled, then Accept fails the way a read from a dead connection does.
type goneMidAccept struct{ cancel context.CancelFunc }
func (g goneMidAccept) Accept(context.Context, string, string, io.Reader) (string, error) {
g.cancel()
return "", fmt.Errorf("receive: %w", io.ErrClosedPipe)
}
func TestUploadPost_AnAbandonedRequestGetsA499NotAnImplicitEmpty200(t *testing.T) {
// If the context is cancelled but the connection can still be written to (a client that
// half-closed after sending), a missing status becomes 200 with an empty body: a false
// success (fourth adversarial review).
app, _ := newUploadApp(t, audioOK, 200<<20)
buf := &lockedBuf{}
app.Log = slog.New(slog.NewTextHandler(buf, nil))
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
app.Uploads = goneMidAccept{cancel: cancel}
rec := do(t, app, uploadReq(t, "t", "a.m4a", []byte("x")).WithContext(ctx))
assert.Equal(t, 499, rec.Code, "never an implicit 200 for an upload that was not accepted")
assert.Contains(t, buf.String(), "upload abandoned")
assert.NotContains(t, buf.String(), "level=ERROR")
}
func TestUploadPost_ADatabaseDropThatLooksLikeAnEOFIsStillAServerError(t *testing.T) {
// pgx turns a connection that drops mid-message into io.ErrUnexpectedEOF. Mapping that
// sentinel to "your upload was cut off" (a 400) would hide a real fault: only an error
// from reading the CLIENT's body is theirs (third adversarial review).
app, _ := newUploadApp(t, audioOK, 200<<20)
buf := &lockedBuf{}
app.Log = slog.New(slog.NewTextHandler(buf, nil))
app.Uploads = failingUploader{err: fmt.Errorf("failed to receive message: %w", io.ErrUnexpectedEOF)}
rec := do(t, app, uploadReq(t, "t", "a.m4a", []byte("x")))
assert.Equal(t, http.StatusInternalServerError, rec.Code)
assert.Contains(t, buf.String(), "level=ERROR")
}
+7 -24
View File
@@ -132,7 +132,7 @@ func TestUploadPost_AcceptedAudioQueuesARowAndRedirectsToIt(t *testing.T) {
assert.Equal(t, "Sprint review", title)
assert.Equal(t, "/uploads/"+id, rec.Header().Get("Location"))
require.Len(t, sub.jobs, 1)
assert.Equal(t, uploads.Job{UserID: userID, UploadID: id, Path: sub.jobs[0].Path, Duration: audioOK.Duration}, sub.jobs[0])
assert.Equal(t, uploads.Job{UserID: userID, UploadID: id, Path: sub.jobs[0].Path}, sub.jobs[0])
spooled, err := os.ReadFile(sub.jobs[0].Path)
require.NoError(t, err)
assert.Equal(t, "synthetic audio bytes", string(spooled))
@@ -366,8 +366,8 @@ func setReconciliation(t *testing.T, id, json string) {
const reconciledJSON = `{"status":"reconciled","base":"iguana/pianissimo","alt":"iguana/kb-whisper","reconciler":"iguana/gemma4-26b",
"disputes":380,"auto_resolved":90,"adjudicated":285,"changed":120,"unsure":5,"errors":0,
"contested":[
{"seconds":2210.5,"base":"Axel","alt":"Alex","used":"alt","unsure":false},
{"seconds":3400,"base":"A+2","alt":"en A","used":"base","unsure":true}]}`
{"seconds":2210.5,"base":"SEB","alt":"SCB","used":"alt","unsure":false},
{"seconds":3400,"base":"T+1","alt":"en T","used":"base","unsure":true}]}`
func TestUploadStatus_ReconciledSaysWhatWasDoneInPlainWords(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
@@ -394,11 +394,11 @@ func TestUploadStatus_ListsTheContestedSpotsWithAMinuteMarkBothReadingsAndWhichW
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.Contains(t, b, "~37 min", "2210.5 s")
assert.Contains(t, b, "Axel")
assert.Contains(t, b, "Alex")
assert.Contains(t, b, "SEB")
assert.Contains(t, b, "SCB")
assert.Contains(t, b, "~57 min", "3400 s")
assert.Contains(t, b, "A+2")
assert.Contains(t, b, "en A")
assert.Contains(t, b, "T+1")
assert.Contains(t, b, "en T")
assert.Contains(t, b, "uncertain", "an unsure spot is marked so the reader knows to check it")
}
@@ -460,20 +460,3 @@ func TestUploadStatus_AVeryLongListOfContestedSpotsIsShortenedWithItsTrueCount(t
assert.NotContains(t, b, ">b119<", "the page does not render every one")
assert.Contains(t, b, "70 more", "120 stored, 50 shown")
}
func TestUploadStatus_ALengthFailureIsExplainedInWordsAndNotAsAGenericError(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
tests := []struct{ kind, want string }{
{"audio_too_long", "longer than the limit"},
}
for _, tt := range tests {
t.Run(tt.kind, func(t *testing.T) {
id := seedUpload(t, userID, "m-"+tt.kind, "failed", tt.kind)
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.Contains(t, b, tt.want)
assert.NotContains(t, b, "This upload failed. Please upload it again.")
})
}
}
@@ -1,72 +0,0 @@
package web_test
import (
"bytes"
"context"
"errors"
"io"
"log/slog"
"net/http"
"net/url"
"sync"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// Error paths of the upload routes log a kind, never the error's text, which can carry
// a file name, a path, a database value or a provider's output (adversarial review F5).
type lockedBuf struct {
mu sync.Mutex
b bytes.Buffer
}
func (l *lockedBuf) Write(p []byte) (int, error) {
l.mu.Lock()
defer l.mu.Unlock()
return l.b.Write(p)
}
func (l *lockedBuf) String() string {
l.mu.Lock()
defer l.mu.Unlock()
return l.b.String()
}
type failingUploader struct{ err error }
func (f failingUploader) Accept(context.Context, string, string, io.Reader) (string, error) {
return "", f.err
}
func TestUploadPost_AnUnexpectedFailureLogsAKindAndNotItsText(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
buf := &lockedBuf{}
app.Log = slog.New(slog.NewTextHandler(buf, nil))
app.Uploads = failingUploader{err: errors.New(`ffprobe: Error opening input file /spool/upload-555 for "zebrakaka-meeting.m4a"`)}
rec := do(t, app, uploadReq(t, "t", "zebrakaka-meeting.m4a", []byte("x")))
assert.Equal(t, http.StatusInternalServerError, rec.Code)
out := buf.String()
assert.NotContains(t, out, "zebrakaka", "a file name is private content")
assert.NotContains(t, out, "/spool/", "a spool path carries the upload's identity")
assert.Contains(t, out, "kind=error")
}
func TestUploadChatPost_AModelFailureLogsAKindAndNotItsText(t *testing.T) {
app, fc := newUploadChatApp(t)
buf := &lockedBuf{}
app.Log = slog.New(slog.NewTextHandler(buf, nil))
fc.err = errors.New("LLM returned 400: the prompt was zebrakaka secret words")
id := seedUpload(t, userID, "m", "done", meetingText)
rec := do(t, app, chatPost("/uploads/"+id+"/chat", url.Values{"question": {"q"}}))
require.Equal(t, http.StatusOK, rec.Code)
out := buf.String()
assert.NotContains(t, out, "zebrakaka")
assert.Contains(t, out, "kind=error")
}
+6 -49
View File
@@ -229,59 +229,16 @@ func TestUploadProgress_ADoneJobKeepsItsRecordOfWhatWasDoneAndHowLongEachTook(t
assert.NotContains(t, b, "spinner")
}
// stepsBlock is the progress list of a page: from its opening tag to its closing one.
func stepsBlock(t *testing.T, html string) string {
t.Helper()
start := strings.Index(html, `<ol class="steps"`)
require.GreaterOrEqual(t, start, 0, "no progress list in:\n%s", html)
end := strings.Index(html[start:], "</ol>")
require.GreaterOrEqual(t, end, 0)
return html[start : start+end+len("</ol>")]
}
func TestUploadProgress_TheFullPageCarriesTheVerySameProgressListAsTheFragment(t *testing.T) {
func TestUploadProgress_TheFullPageRendersTheSameDisplayAsTheFragment(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
fixedClock(app)
id := seedUpload(t, userID, "m", "transcribing", "")
seedProgress(t, id, twoEngineRun(), progressNow.Add(-10*time.Second), progressNow.Add(-4*time.Second))
full := body(t, do(t, app, get("/uploads/"+id)))
fragment := statusBody(t, app, id)
rec := do(t, app, get("/uploads/"+id))
assert.Equal(t, stepsBlock(t, fragment), stepsBlock(t, full),
"the page a user opens and the fragment HTMX swaps in must show the same list, or the display jumps on the first poll")
}
func TestUploadProgress_AStepNoteIsShownDirectlyUnderItsStepAndEscaped(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
fixedClock(app)
id := seedUpload(t, userID, "m", "transcribing", "")
steps := twoEngineRun()
steps[1].Note = "The audio is 20 min long; the file claimed 1 h 19 min. <b>cut off</b>?"
seedProgress(t, id, steps, progressNow.Add(-10*time.Second), progressNow.Add(-4*time.Second))
b := statusBody(t, app, id)
audio := strings.Index(b, "Preparing the audio")
note := strings.Index(b, "The audio is 20 min long")
next := strings.Index(b, "Transcribing with iguana/pianissimo")
require.True(t, audio >= 0 && note > audio && next > note, "the note must sit between its own step and the next one:\n%s", b)
assert.Contains(t, b, "step-note")
assert.NotContains(t, b, "<b>cut off</b>", "a note is words, never markup")
}
func TestUploadProgress_ADonePageShowsItsRecordOpenWhenAStepHasANote(t *testing.T) {
// A truncated recording must not look like a clean one: the note that replaced the
// refusal is the only warning, so it cannot sit inside a collapsed list (third review).
app, _ := newUploadApp(t, audioOK, 200<<20)
fixedClock(app)
noted := seedUpload(t, userID, "noted", "done", "Synthetic transcript text.")
steps := []domain.ProgressStep{step("audio", "Preparing the audio", domain.StepDone, 20*time.Minute, 40*time.Second, 0, 0)}
steps[0].Note = "The audio is 20 min long; the file claimed 1 h 19 min."
seedProgress(t, noted, steps, progressNow.Add(-time.Minute), progressNow.Add(-time.Minute))
clean := seedUpload(t, userID, "clean", "done", "Synthetic transcript text.")
seedProgress(t, clean, []domain.ProgressStep{step("audio", "Preparing the audio", domain.StepDone, 20*time.Minute, 40*time.Second, 0, 0)}, progressNow.Add(-time.Minute), progressNow.Add(-time.Minute))
assert.Contains(t, statusBody(t, app, noted), `<details class="what-was-done" open>`)
assert.NotContains(t, statusBody(t, app, clean), `<details class="what-was-done" open>`, "a clean record stays collapsed")
require.Equal(t, 200, rec.Code)
b := body(t, rec)
assert.Contains(t, b, "Transcribing with iguana/kb-whisper", "the page a user opens, not only the polled fragment, shows the phases")
assert.Contains(t, b, "3m 12s")
}
-4
View File
@@ -47,9 +47,6 @@ func TestRenderUploadProgressStatesForReview(t *testing.T) {
step("save", "Saving the transcript", domain.StepDone, time.Minute, time.Second, 0, 0),
}
noted := twoEngineRun()
noted[1].Note = "The audio is 20 min long; the file claimed 1 h 19 min. If the recording was cut off, upload the complete one."
cases := []struct {
name, status, content string
steps []domain.ProgressStep
@@ -61,7 +58,6 @@ func TestRenderUploadProgressStatesForReview(t *testing.T) {
{"lost", "transcribing", "", twoEngineRun(), 3 * time.Minute, 3 * time.Minute},
{"failed", "failed", "transcription_failed", failed, 2 * time.Minute, 2 * time.Minute},
{"done", "done", "Synthetic transcript text for the review.", done, time.Minute, time.Minute},
{"noted", "transcribing", "", noted, 10 * time.Second, 4 * time.Second},
{"queued", "queued", "", nil, 0, 0},
}
for _, c := range cases {
+4 -4
View File
@@ -62,11 +62,11 @@ func TestUploadStatus_ASpotShowsTheWordsJustBeforeIt(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript.")
setReconciliation(t, id, `{"status":"reconciled","base":"a","alt":"b","reconciler":"c","disputes":1,"changed":1,
"contested":[{"seconds":2640,"base":"vanligt","alt":"KORT","used":"alt","unsure":false,"before":"grannen målar ett"}]}`)
"contested":[{"seconds":2640,"base":"special","alt":"CAMT","used":"alt","unsure":false,"before":"vi skickar in ett"}]}`)
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.Contains(t, b, "grannen målar ett", "the words before the spot are how a reader finds it in the audio")
assert.Contains(t, b, "vanligt")
assert.Contains(t, b, "KORT")
assert.Contains(t, b, "vi skickar in ett", "the words before the spot are how a reader finds it in the audio")
assert.Contains(t, b, "special")
assert.Contains(t, b, "CAMT")
}
+115 -144
View File
@@ -312,17 +312,7 @@ func uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
return templ_7745c5c3_Err
}
if len(u.Progress) > 0 {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 21, "<details class=\"what-was-done\"")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
if hasNote(u.Progress) {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 22, " open")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 23, "><summary>What was done</summary>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 21, "<details class=\"what-was-done\"><summary>What was done</summary>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -330,13 +320,13 @@ func uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 24, "</details> ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 22, "</details> ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
if chatEnabled {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 25, "<p><a class=\"btn\" href=\"")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 23, "<p><a class=\"btn\" href=\"")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -349,12 +339,12 @@ func uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 26, "\">Ask about this recording</a></p>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 24, "\">Ask about this recording</a></p>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 27, "<details open><summary>Transcript</summary><pre class=\"transcript\">")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 25, "<details open><summary>Transcript</summary><pre class=\"transcript\">")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -367,12 +357,12 @@ func uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 28, "</pre></details></div>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 26, "</pre></details></div>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
case domain.UploadFailed:
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 29, "<div class=\"upload-status upload-failed flash flash-error\" role=\"alert\">")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 27, "<div class=\"upload-status upload-failed flash flash-error\" role=\"alert\">")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -380,7 +370,7 @@ func uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 30, "<p>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 28, "<p>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -393,12 +383,12 @@ func uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 31, "</p>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 29, "</p>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
if len(u.Progress) > 0 {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 32, "<details class=\"what-was-done\" open><summary>Where it stopped</summary>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 30, "<details class=\"what-was-done\" open><summary>Where it stopped</summary>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -406,17 +396,17 @@ func uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 33, "</details>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 31, "</details>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 34, "</div>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 32, "</div>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
default:
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 35, "<div class=\"upload-status upload-working\" role=\"status\" hx-get=\"")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 33, "<div class=\"upload-status upload-working\" role=\"status\" hx-get=\"")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -429,7 +419,7 @@ func uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 36, "\" hx-trigger=\"")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 34, "\" hx-trigger=\"")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -442,7 +432,7 @@ func uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 37, "\" hx-swap=\"outerHTML\">")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 35, "\" hx-swap=\"outerHTML\">")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -457,7 +447,7 @@ func uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 38, " ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 36, " ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -466,18 +456,18 @@ func uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
return templ_7745c5c3_Err
}
} else {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 39, "<p>Transcribing… a long recording takes a few minutes.</p>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 37, "<p>Transcribing… a long recording takes a few minutes.</p>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
} else {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 40, "<p>Queued — waiting for its turn to be transcribed.</p>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 38, "<p>Queued — waiting for its turn to be transcribed.</p>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 41, "</div>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 39, "</div>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -521,7 +511,7 @@ func UploadChatPage(u domain.UploadedTranscript, v chatView) templ.Component {
}()
}
ctx = templ.InitializeContext(ctx)
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 42, "<article class=\"account\"><h1>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 40, "<article class=\"account\"><h1>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -534,7 +524,7 @@ func UploadChatPage(u domain.UploadedTranscript, v chatView) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 43, "</h1><section id=\"chat-section\" class=\"chat-dock chat-dock-open\"><h2 class=\"chat-heading\">Ask about this recording</h2><p class=\"chat-scope muted\">Answers come only from this recording's transcript, which is private to you.</p>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 41, "</h1><section id=\"chat-section\" class=\"chat-dock chat-dock-open\"><h2 class=\"chat-heading\">Ask about this recording</h2><p class=\"chat-scope muted\">Answers come only from this recording's transcript, which is private to you.</p>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -542,7 +532,7 @@ func UploadChatPage(u domain.UploadedTranscript, v chatView) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 44, "</section><p><a href=\"")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 42, "</section><p><a href=\"")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -555,7 +545,7 @@ func UploadChatPage(u domain.UploadedTranscript, v chatView) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 45, "\">Back to the transcript</a></p></article>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 43, "\">Back to the transcript</a></p></article>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -595,7 +585,7 @@ func videoNotice(hadVideo bool) templ.Component {
}
ctx = templ.ClearChildren(ctx)
if hadVideo {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 46, "<p class=\"muted video-notice\">This file contained a video track. Only its audio was used; the video was discarded and nothing from it was kept.</p>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 44, "<p class=\"muted video-notice\">This file contained a video track. Only its audio was used; the video was discarded and nothing from it was kept.</p>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -632,7 +622,7 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
if r != nil {
switch r.Status {
case domain.ReconcileDone:
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 47, "<div class=\"reconciliation\"><p>Reconciled from two transcripts (")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 45, "<div class=\"reconciliation\"><p>Reconciled from two transcripts (")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -645,7 +635,7 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 48, " and ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 46, " and ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -658,7 +648,7 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 49, "): ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 47, "): ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -671,7 +661,7 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 50, " differences, ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 48, " differences, ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -684,7 +674,7 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 51, " settled automatically, ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 49, " settled automatically, ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -697,7 +687,7 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 52, " decided by the model (")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 50, " decided by the model (")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -710,7 +700,7 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 53, "), ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 51, "), ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -723,17 +713,17 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 54, " uncertain.</p>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 52, " uncertain.</p>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
if len(shownSpots(r)) > 0 {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 55, "<details><summary>Spots to check against the audio</summary><ul>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 53, "<details><summary>Spots to check against the audio</summary><ul>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
for _, c := range shownSpots(r) {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 56, "<li>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 54, "<li>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -746,12 +736,12 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 57, ": ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 55, ": ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
if c.Before != "" {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 58, "<span class=\"muted\">…")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 56, "<span class=\"muted\">…")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -764,7 +754,7 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 59, "</span> ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 57, "</span> ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -778,7 +768,7 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 60, " / ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 58, " / ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -791,12 +781,12 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 61, " — ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 59, " — ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
if c.Unsure {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 62, "<span class=\"chip chip-warn\">uncertain</span> kept ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 60, "<span class=\"chip chip-warn\">uncertain</span> kept ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -810,7 +800,7 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
return templ_7745c5c3_Err
}
} else {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 63, "<span class=\"chip\">changed</span> used ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 61, "<span class=\"chip\">changed</span> used ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -824,17 +814,17 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 64, "</li>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 62, "</li>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 65, "</ul>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 63, "</ul>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
if moreSpots(r) > 0 {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 66, "<p class=\"muted\">and ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 64, "<p class=\"muted\">and ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -847,22 +837,22 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 67, " more not listed</p>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 65, " more not listed</p>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 68, "</details>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 66, "</details>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 69, "</div>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 67, "</div>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
case domain.ReconcileSingle:
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 70, "<p class=\"muted reconciliation\">Only one engine produced this transcript (")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 68, "<p class=\"muted reconciliation\">Only one engine produced this transcript (")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -875,12 +865,12 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 71, "), so there was no second opinion to reconcile it with.</p>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 69, "), so there was no second opinion to reconcile it with.</p>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
case domain.ReconcileSkipped:
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 72, "<p class=\"muted reconciliation\">Two transcripts were made but could not be reconciled (")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 70, "<p class=\"muted reconciliation\">Two transcripts were made but could not be reconciled (")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -893,7 +883,7 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 73, "); the text shown is ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 71, "); the text shown is ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -906,7 +896,7 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 74, "'s, unreconciled.</p>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 72, "'s, unreconciled.</p>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -942,7 +932,7 @@ func progressSteps(steps []domain.ProgressStep, now time.Time, live bool) templ.
templ_7745c5c3_Var43 = templ.NopComponent
}
ctx = templ.ClearChildren(ctx)
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 75, "<ol class=\"steps\" aria-label=\"Progress\">")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 73, "<ol class=\"steps\" aria-label=\"Progress\">")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -952,7 +942,7 @@ func progressSteps(steps []domain.ProgressStep, now time.Time, live bool) templ.
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 76, "<li class=\"")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 74, "<li class=\"")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -965,32 +955,32 @@ func progressSteps(steps []domain.ProgressStep, now time.Time, live bool) templ.
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 77, "\"")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 75, "\"")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
if s.State == domain.StepRunning {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 78, " aria-current=\"step\"")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 76, " aria-current=\"step\"")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 79, ">")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 77, ">")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
if s.State == domain.StepRunning && live {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 80, "<span class=\"step-mark\" aria-hidden=\"true\"><span class=\"spinner\"></span></span> ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 78, "<span class=\"step-mark\" aria-hidden=\"true\"><span class=\"spinner\"></span></span> ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
} else if s.State == domain.StepRunning {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 81, "<span class=\"step-mark step-mark-warn\" aria-hidden=\"true\">!</span> ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 79, "<span class=\"step-mark step-mark-warn\" aria-hidden=\"true\">!</span> ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
} else {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 82, "<span class=\"step-mark\" aria-hidden=\"true\">")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 80, "<span class=\"step-mark\" aria-hidden=\"true\">")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -1003,12 +993,12 @@ func progressSteps(steps []domain.ProgressStep, now time.Time, live bool) templ.
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 83, "</span> ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 81, "</span> ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 84, "<span class=\"step-label\">")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 82, "<span class=\"step-label\">")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -1021,12 +1011,12 @@ func progressSteps(steps []domain.ProgressStep, now time.Time, live bool) templ.
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 85, "</span> ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 83, "</span> ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
if t := stepTime(s, now); t != "" {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 86, "<span class=\"step-time\">")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 84, "<span class=\"step-time\">")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -1039,94 +1029,75 @@ func progressSteps(steps []domain.ProgressStep, now time.Time, live bool) templ.
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 87, "</span> ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
if s.Note != "" {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 88, "<span class=\"step-note\">")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
var templ_7745c5c3_Var49 string
templ_7745c5c3_Var49, templ_7745c5c3_Err = templ.JoinStringErrs(s.Note)
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 222, Col: 37}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var49))
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 89, "</span> ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 85, "</span> ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
if s.State == domain.StepRunning && s.Total > 0 {
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 90, "<progress class=\"step-bar\" value=\"")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 86, "<progress class=\"step-bar\" value=\"")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
var templ_7745c5c3_Var49 string
templ_7745c5c3_Var49, templ_7745c5c3_Err = templ.ResolveAttributeValue(strconv.Itoa(s.Done))
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 222, Col: 60}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var49)
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 87, "\" max=\"")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
var templ_7745c5c3_Var50 string
templ_7745c5c3_Var50, templ_7745c5c3_Err = templ.ResolveAttributeValue(strconv.Itoa(s.Done))
templ_7745c5c3_Var50, templ_7745c5c3_Err = templ.ResolveAttributeValue(strconv.Itoa(s.Total))
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 225, Col: 60}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 222, Col: 90}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var50)
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 91, "\" max=\"")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 88, "\"></progress> <span class=\"step-count\">")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
var templ_7745c5c3_Var51 string
templ_7745c5c3_Var51, templ_7745c5c3_Err = templ.ResolveAttributeValue(strconv.Itoa(s.Total))
templ_7745c5c3_Var51, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(s.Done))
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 225, Col: 90}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 223, Col: 52}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var51)
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var51))
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 92, "\"></progress> <span class=\"step-count\">")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 89, " of ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
var templ_7745c5c3_Var52 string
templ_7745c5c3_Var52, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(s.Done))
templ_7745c5c3_Var52, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(s.Total))
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 226, Col: 52}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 223, Col: 81}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var52))
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 93, " of ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
var templ_7745c5c3_Var53 string
templ_7745c5c3_Var53, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(s.Total))
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 226, Col: 81}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var53))
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 94, "</span>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 90, "</span>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 95, "</li>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 91, "</li>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 96, "</ol>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 92, "</ol>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
@@ -1152,45 +1123,45 @@ func healthNote(h jobHealth, silent time.Duration) templ.Component {
}()
}
ctx = templ.InitializeContext(ctx)
templ_7745c5c3_Var54 := templ.GetChildren(ctx)
if templ_7745c5c3_Var54 == nil {
templ_7745c5c3_Var54 = templ.NopComponent
templ_7745c5c3_Var53 := templ.GetChildren(ctx)
if templ_7745c5c3_Var53 == nil {
templ_7745c5c3_Var53 = templ.NopComponent
}
ctx = templ.ClearChildren(ctx)
switch h {
case healthLost:
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 97, "<p class=\"step-health step-health-lost\" role=\"alert\">The service stopped reporting ")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 93, "<p class=\"step-health step-health-lost\" role=\"alert\">The service stopped reporting ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
var templ_7745c5c3_Var54 string
templ_7745c5c3_Var54, templ_7745c5c3_Err = templ.JoinStringErrs(minutesAgo(silent))
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 236, Col: 54}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var54))
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 94, " ago. It may have restarted or crashed. This page keeps checking; if nothing changes, upload the recording again.</p>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
case healthQuiet:
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 95, "<p class=\"step-health step-health-quiet\">Nothing new for ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
var templ_7745c5c3_Var55 string
templ_7745c5c3_Var55, templ_7745c5c3_Err = templ.JoinStringErrs(minutesAgo(silent))
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 239, Col: 54}
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 241, Col: 40}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var55))
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 98, " ago. It may have restarted or crashed. This page keeps checking; if nothing changes, upload the recording again.</p>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
case healthQuiet:
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 99, "<p class=\"step-health step-health-quiet\">Nothing new for ")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
var templ_7745c5c3_Var56 string
templ_7745c5c3_Var56, templ_7745c5c3_Err = templ.JoinStringErrs(minutesAgo(silent))
if templ_7745c5c3_Err != nil {
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 244, Col: 40}
}
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var56))
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 100, ": the model may be slow. The service is still running, and the job gives up on its own if it takes too long.</p>")
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 96, ": the model may be slow. The service is still running, and the job gives up on its own if it takes too long.</p>")
if templ_7745c5c3_Err != nil {
return templ_7745c5c3_Err
}
-1
View File
@@ -780,7 +780,6 @@ a.btn, a.btn:visited { color: var(--accent-fg); }
.step-skipped { opacity: .7; }
.step-pending { opacity: .75; }
.step-bar { flex: none; width: calc(100% - 1.6rem); margin: 0 0 0 1.6rem; height: .5rem; accent-color: var(--accent); }
.step-note { flex: none; width: calc(100% - 1.6rem); margin-left: 1.6rem; font-weight: 400; color: var(--muted); font-family: system-ui, -apple-system, sans-serif; font-size: .82rem; }
.step-count { flex: none; width: calc(100% - 1.6rem); margin-left: 1.6rem; font-weight: 400; color: var(--muted); }
.step-running .step-mark-warn { color: var(--err-fg); }
.spinner { display: inline-block; vertical-align: -.1em; width: .8rem; height: .8rem; box-sizing: border-box; border: 2px solid var(--line); border-top-color: var(--accent); border-radius: 50%; animation: tapir-spin 1s linear infinite; }
+1 -10
View File
@@ -143,17 +143,8 @@ var scenarioCoverage = map[string]string{
"A failed job shows where it stopped": "TestUploadProgress_AFailedJobShowsWhichPhaseFailedAndWhatNeverStarted",
"A finished job keeps its record of what was done": "TestUploadProgress_ADoneJobKeepsItsRecordOfWhatWasDoneAndHowLongEachTook",
"The heartbeat runs on its own while an engine call is stuck": "TestWorkerProgress_TheHeartbeatBeatsWhileAnEngineCallIsStuckAndStopsWhenTheJobEnds",
"The log names every phase and never the transcript": "TestWorkerProgress_OnASuccessfulJobTheLogNamesEveryPhaseInOrderAndNeverTheTranscriptOrThePath",
"The log names every phase and never the transcript": "TestWorkerProgress_TheLogNamesEveryPhaseInOrderAndNeverTheTranscriptOrThePath",
"Every judged spot is stored but only the ones to check are listed": "TestUploadStatus_OnlyChangedAndUncertainSpotsAreListedNotTheKeptOnes",
"Chat about a recording is offered with local models only": "TestUploadChatPage_DoesNotOfferACloudModelEvenThoughTheServiceKnowsOne",
"A forged cloud model is never sent the transcript": "TestUploadChatPost_AForgedCloudModelIsReplacedByALocalOneBeforeTheTranscriptIsSent",
"A wedged engine does not use up the whole job": "TestChain_AHungPrimaryReachesTheFallbackWhenItHasItsOwnDeadline",
"A file that lies about its length is refused": "TestWorkerDuration_AConvertedLengthOverTheLimitFailsTheJobBeforeAnyEngineSeesIt",
"A recording much shorter than its file claimed is transcribed and noted": "TestWorkerDuration_AFileThatEndsFarEarlierThanItClaimedIsStillTranscribedAndSaysSo",
"A full queue is a retry, not an error": "TestAccept_AFullQueueIsABusyRejectionNotAServerErrorAndLeavesNothingBehind",
"One user cannot fill the queue for everyone": "TestWorkerSubmit_OneUserCannotFillTheQueueForEveryoneElse",
"Errors are shown as kinds, never their text": "TestReconcile_TheBreakerReasonCarriesAKindNotTheModelsText",
"Input that cannot be right is a client error": "TestUploadRoutes_ANonUUIDIdIs404NotA500",
"The saved segments carry the merged words": "TestReconcileSegments_AChangedWordReplacesTheBaseWordInItsOwnSegmentOnly",
// summarize_new_video.feature
-143
View File
@@ -1,143 +0,0 @@
package lintgate
// Observes the exit status of `task lint` (the lint step of `task check`), not the lint
// rules themselves. It runs the repo's real Taskfile in a scratch module so a change to the
// lint command that stops it failing on findings (#41) turns this red.
import (
"bytes"
"errors"
"os"
"os/exec"
"path/filepath"
"regexp"
"strings"
"testing"
)
const scratchWithFinding = `package main
import "os"
func main() { os.Remove("x") }
`
func scratchModule(t *testing.T) string {
t.Helper()
taskfile, err := os.ReadFile(filepath.Join("..", "..", "Taskfile.yml"))
if err != nil {
t.Fatalf("read Taskfile.yml: %v", err)
}
dir := t.TempDir()
for name, body := range map[string]string{
"Taskfile.yml": string(taskfile),
"go.mod": "module scratch\n\ngo 1.21\n",
"main.go": scratchWithFinding,
} {
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil {
t.Fatalf("write %s: %v", name, err)
}
}
return dir
}
// ansi matches the colour escape sequences task prints when it thinks it has a terminal.
var ansi = regexp.MustCompile("\x1b\\[[0-9;]*m")
// hasSkipNotice reports whether the skip notice was printed as output. Task echoes the command
// line first, and that line contains the same words, so a substring match would be vacuous.
func hasSkipNotice(out string) bool {
for _, line := range strings.Split(ansi.ReplaceAllString(out, ""), "\n") {
if strings.HasPrefix(line, "golangci-lint not installed") {
return true
}
}
return false
}
// runLint runs `task lint` in dir with the given PATH and returns its combined output and exit code.
func runLint(t *testing.T, dir, path string) (string, int) {
t.Helper()
return runTask(t, dir, path, "lint")
}
func runTask(t *testing.T, dir, path string, args ...string) (string, int) {
t.Helper()
taskBin, err := exec.LookPath("task")
if err != nil {
t.Skip("task not installed")
}
cmd := exec.Command(taskBin, args...)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "PATH="+path)
var out bytes.Buffer
cmd.Stdout, cmd.Stderr = &out, &out
err = cmd.Run()
var exitErr *exec.ExitError
switch {
case err == nil:
return out.String(), 0
case errors.As(err, &exitErr):
return out.String(), exitErr.ExitCode()
default:
t.Fatalf("run task lint: %v", err)
return "", -1
}
}
func TestTaskLint_ExitsNonZeroWhenGolangciLintReportsAFinding(t *testing.T) {
if _, err := exec.LookPath("golangci-lint"); err != nil {
t.Skip("golangci-lint not installed")
}
out, code := runLint(t, scratchModule(t), os.Getenv("PATH"))
if code == 0 {
t.Fatalf("task lint exited 0 on a module with an unchecked error; output:\n%s", out)
}
if hasSkipNotice(out) {
t.Fatalf("task lint blamed a missing binary for a lint failure; output:\n%s", out)
}
if !strings.Contains(out, "errcheck") {
t.Fatalf("a non-zero exit without the finding means the linter crashed, not that it found something; output:\n%s", out)
}
}
// The lint step only gates anything if `task check` runs it.
func TestTaskCheck_RunsTheLintStep(t *testing.T) {
out, code := runTask(t, scratchModule(t), os.Getenv("PATH"), "--dry", "check")
if code != 0 {
t.Fatalf("task --dry check exited %d; output:\n%s", code, out)
}
if !strings.Contains(out, "task: [lint]") {
t.Fatalf("task check no longer runs the lint step; output:\n%s", out)
}
}
func TestTaskLint_SkipsWithANoticeWhenGolangciLintIsAbsent(t *testing.T) {
taskBin, err := exec.LookPath("task")
if err != nil {
t.Skip("task not installed")
}
bin := t.TempDir() // a PATH holding task and nothing else
if err := os.Symlink(taskBin, filepath.Join(bin, "task")); err != nil {
t.Fatalf("symlink task: %v", err)
}
out, code := runLint(t, scratchModule(t), bin)
if code != 0 {
t.Fatalf("task lint exited %d without golangci-lint on PATH; output:\n%s", code, out)
}
if !hasSkipNotice(out) {
t.Fatalf("expected the skip notice; output:\n%s", out)
}
}
// CI runs task with colour on, so the notice line starts with an escape sequence; a bare
// prefix match missed it there while passing on a terminal-less local run.
func TestHasSkipNotice_SeesTheNoticeBehindColourCodes(t *testing.T) {
out := "\x1b[32mtask: [lint] ...\n\x1b[0mgolangci-lint not installed — skipping (CI installs it)\n"
if !hasSkipNotice(out) {
t.Fatalf("notice not found in coloured output: %q", out)
}
if hasSkipNotice("\x1b[32mtask: [lint] if command -v golangci-lint ...; echo \"golangci-lint not installed\"\n") {
t.Fatal("the echoed command line must not count as the notice")
}
}