Compare commits

...
72 Commits
Author SHA1 Message Date
mathias 813e679f93 fix(uploads): bound the silence in an upload body with a re-armed read deadline (#56) (#63)
CI / Deploy via GitOps (push) Successful in 6s
CI / Lint / Test / Vet (push) Successful in 32s
CI / Build & Import (push) Successful in 11s
Closes #56

Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-05 04:59:29 +00:00
mathias b9643f5fb7 Merge pull request 'feat(uploads): choose the language of a recording; Swedish default, Other = auto (#61 interim)' (#62) from sprint/language into main
CI / Deploy via GitOps (push) Successful in 5s
CI / Lint / Test / Vet (push) Successful in 34s
CI / Build & Import (push) Successful in 14s
2026-10-05 04:59:03 +00:00
mathiasandClaude Opus 5.5 cace7d179d fix(deps): bump x/text past GO-2026-5970
CI / Lint / Test / Vet (push) Successful in 30s
CI / Build & Import (push) Successful in 11s
CI / Deploy via GitOps (push) Skipped
govulncheck reported GO-2026-5970 as called via golang.org/x/text v0.31.0
(reached through the DB driver). Bump to v0.39.0; x/sync moves to v0.21.0
as its required minimum. govulncheck ./...: 0 vulnerabilities.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 22:52:55 +02:00
mathiasandClaude Opus 5.5 550764d43c fix(oidc): retry issuer discovery with bounded back-off at start-up
A 503 from the issuer's discovery endpoint made `serve` exit and the pod
restart. New now retries discovery 5 times (2+4+8+16 s back-off, ~30 s)
before failing as before; a cancelled ctx ends the back-off at once.
WithDiscoveryRetry lets tests shrink the schedule.

Closes #59

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 22:52:26 +02:00
mathiasandClaude Sonnet 5.5 f704258fdf test(transcribe): the wrapper does not restart a cancelled job; PlanFor of an empty or unknown language is pinned (#61)
CI / Lint / Test / Vet (pull_request) Successful in 30s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Low findings of the PR #62 re-review: the WithTimeout test's comment claimed cancellation
travels on the context but only the language was asserted (the WithoutCancel mutant survived),
and PlanFor('') and PlanFor(unknown) were untested.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-04 22:48:29 +02:00
mathiasandClaude Sonnet 5.5 e3288d0654 feat(web): the language is named on the recording's page in every state; the feature says what Other really does (#61)
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
CI / Lint / Test / Vet (pull_request) Successful in 37s
The label moves from the done fragment to the page header, so queued, transcribing and
failed recordings show it too. The scenario no longer implies a general engine exists. Found
by the PR #62 review.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-04 22:41:42 +02:00
mathiasandClaude Sonnet 5.5 e0c53f8719 feat(uploads): an auto-language job is planned without the engines it never runs (#61)
PlanFor(lang) on the Ensemble and a LanguagePlanner the worker prefers: the page lists
the other engine (and the fallback if configured), not the base engine, the comparison and
the decisions as coming. Found by the PR #62 review.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-04 22:41:21 +02:00
mathiasandClaude Sonnet 5.5 0828f0debf test(web): the language refusal asserts its message, not a word the form always contains (#61)
Found by the PR #62 review: the message could be replaced with 'x' and the test stayed green.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-04 22:40:47 +02:00
mathiasandClaude Sonnet 5.5 7282893877 test(transcribe): WithTimeout hands the engine the job's language, as on the production path (#61)
The wrapper sits in front of both local engines; a fresh context there would send an auto
upload as language=sv and un-cancel the job, and no test noticed. Found by the PR #62 review.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-04 22:40:35 +02:00
mathiasandClaude Sonnet 5.5 cc6fe7083d refactor(web): drop a length check the closed-set parse already makes (#61)
CI / Lint / Test / Vet (pull_request) Successful in 36s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
The read is capped at 17 bytes and no valid language is longer than 4, so the parse
refuses anything the length check would; the mutant that removed it survived.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-04 22:31:03 +02:00
mathiasandClaude Sonnet 5.5 0338b18a17 feat(uploads): choose the language of a recording; record it, show it, carry it to the engines (#61)
Form field (Swedish preselected, Other = auto) read before the file part and refused
before anything is spooled if outside the closed set; stored on the row (migration 021,
CHECK constraint, default sv for history); carried on the Job and the worker's context;
shown on the status page. Feature scenarios and the coverage map are updated, and
ADR-033/034 amended.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-04 22:29:58 +02:00
mathiasandClaude Sonnet 5.5 ed9d76b0cf feat(transcribe): a non-Swedish upload skips the Swedish-only base engine and is not reconciled (#61)
The other engine transcribes alone, then the cloud fallback if it fails and the job is
still live. A cancelled job never reaches the fallback (mutation-checked).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-04 22:26:37 +02:00
mathiasandClaude Sonnet 5.5 0b807c6bcd feat(transcribe): the gateway sends language=sv, omits it for auto, and refuses anything else (#61)
The request used to hard-code language=sv for every upload, so a recording in another
language was decoded as Swedish. The language now comes from the context; no language in
the context is still sv.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-04 22:26:13 +02:00
mathiasandClaude Sonnet 5.5 8e4872d2e7 feat(domain): Language, a closed set carried to the engines on the context (#61)
sv and auto; an empty form value is sv, anything else is refused. LanguageFrom defaults to
sv so a caller that sets nothing behaves as before.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-04 22:25:51 +02:00
mathiasandClaude Sonnet 5.5 4affeb12e8 docs: ADR-034 second blind check protocol; D8 decided (Berget allowed, per-recording local-only opt-out first, #60); #37 verified
CI / Lint / Test / Vet (push) Successful in 33s
CI / Build & Import (push) Successful in 10s
CI / Deploy via GitOps (push) Successful in 5s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-04 08:47:10 +02:00
mathiasandClaude Sonnet 5.5 460e07719d docs(sprint): #50 observed live (old pod Completed); #37 and #52 wait for an authenticated look
CI / Build & Import (push) Successful in 10s
CI / Lint / Test / Vet (push) Successful in 32s
CI / Deploy via GitOps (push) Successful in 4s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-04 00:41:07 +02:00
mathiasandClaude Sonnet 5.5 7b7da650d4 docs(sprint): v0.31.1 and v0.32.0 shipped; what is observed live and what waits for an authenticated look
CI / Deploy via GitOps (push) Successful in 5s
CI / Lint / Test / Vet (push) Successful in 31s
CI / Build & Import (push) Successful in 10s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-04 00:34:59 +02:00
mathias 85f4cc3236 Merge pull request 'v0.32.0: per-recording delete and nav link, cloud-served note, tracker hardening, bounded LLM reads (#37 #51 #52 #53)' (#58) from sprint/v032 into main
CI / Deploy via GitOps (push) Successful in 5s
CI / Lint / Test / Vet (push) Successful in 34s
CI / Build & Import (push) Successful in 11s
2026-10-03 22:31:26 +00:00
mathiasandClaude Sonnet 5.5 41ab1f5d50 fix(web): take the reconciled endpoint apart exactly, so a connector word used as a model name is still judged (#52)
CI / Lint / Test / Vet (pull_request) Successful in 35s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Skipping 'by', 'reconciled' and '+' as words everywhere let an alias with one of those
names pass as local. The two-engine shape is now split on its own separators and each alias
judged whole; a reconciler that is not local is pinned too. Hardening suggested by the third
PR #58 review.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-04 00:28:09 +02:00
mathiasandClaude Sonnet 5.5 67bd0350a3 fix(web): judge every word of a reconciled endpoint that is not a connector, so an unprefixed alias is not called local (#52)
CI / Lint / Test / Vet (pull_request) Successful in 33s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
The previous version kept only words containing '/' and dropped the rest, so a bare
TAPIR_TRANSCRIBE_MODEL alias in the composite produced a false 'all local' result. Found
by the PR #58 re-review; red tests for a bare alias, an alias with a space and a split
'berget / x' first.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-04 00:22:41 +02:00
mathiasandClaude Sonnet 5.5 c24573bd9f docs: state what the save timeout and a racing delete actually guarantee (#51, #37)
CI / Lint / Test / Vet (pull_request) Successful in 29s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
The tracker comment said 'never a stalled job'; each transition can wait up to the timeout.
The delete refusal is judged at the moment of the attempt. Found by the PR #58 review.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:32:14 +02:00
mathiasandClaude Sonnet 5.5 fc1e8e7ba8 test(web): a foreign post to the recording delete route leaves the recording, on an app that mounts uploads (#37, #48)
The earlier entry posted to an app without uploads, where the route does not exist, and
stayed green with the route removed. This one seeds a finished row and goes red without the
gate. Found by the PR #58 review.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:32:14 +02:00
mathiasandClaude Sonnet 5.5 4a0226d8bd test(llm): the response cap is pinned at its boundary and as the production default (#53)
Only the 8-byte override was exercised; >= vs >, a huge default and a non-positive option
all stayed green. Also says why the error-body read ignores its error. Found by the PR #58
review.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:31:30 +02:00
mathiasandClaude Sonnet 5.5 2e54d80a78 test(uploads): jobSink forwards the tracker's timestamp to the store (#51)
The worker fake ignored the heartbeat time, so the store-boundary half of the app-clock
fix had no assertion. Found by the PR #58 review.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:31:14 +02:00
mathiasandClaude Sonnet 5.5 01c1606caa test(progress): the default save and heartbeat deadline is pinned, not only the override (#51)
Every earlier test passed WithSaveTimeout; the worker passes nothing, so the 5 s default
was the only production value and a flip to 0, 1ms or 1h stayed green. Found by the PR #58
review; each flip is now red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:30:58 +02:00
mathiasandClaude Sonnet 5.5 9aa01468ce fix(web): judge each alias of a reconciled endpoint, so an all-local transcript is not called cloud-served (#52)
The two-engine path writes '<base> + <alt>, reconciled by <model>'. IsLocalModel is a
single-alias match, so the composite was 'not local' and the note claimed the audio left
the homelab, naming a local host as the provider. Found by the PR #58 review.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:30:44 +02:00
mathiasandClaude Sonnet 5.5 25628a9994 test(web): the per-recording delete route is covered by the cross-origin gate (#37, #48)
CI / Lint / Test / Vet (pull_request) Successful in 30s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:24 +02:00
mathiasandClaude Sonnet 5.5 a6691ac46b docs: per-recording delete and the recordings link are built; feature scenarios and coverage map (#37)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathiasandClaude Sonnet 5.5 c909b4e775 feat(web): confirm-then-delete control on a finished or failed recording (#37)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathiasandClaude Sonnet 5.5 f309a39027 feat(web): POST /uploads/{id}/delete, owner-scoped and refused while the recording is active (#37)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathiasandClaude Sonnet 5.5 89611164e3 test(web): per-recording delete is owner-scoped, refused while active, offered only when finished (#37), red
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathiasandClaude Sonnet 5.5 93860210ef feat(store): DeleteUpload removes a finished upload of the owner (#37)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathiasandClaude Sonnet 5.5 8509004eb7 test(store): an owner can delete a finished upload; an active or foreign one is refused (#37), red
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathiasandClaude Sonnet 5.5 453464a22b feat(web): the home list links to recordings when uploads are mounted (#37)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathiasandClaude Sonnet 5.5 4ef3e85d16 test(web): the home list links to recordings only when uploads are mounted (#37), red
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathiasandClaude Sonnet 5.5 b8af7f32c7 fix(progress): stamp the heartbeat with the application clock the page compares it with (#51)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathiasandClaude Sonnet 5.5 813192e5ba test: heartbeats are stamped with the application clock, not the database's NOW() (#51), red
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathiasandClaude Sonnet 5.5 2ec1557a8c fix(progress): bound every save and heartbeat write so a hung database cannot stall the job (#51)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathiasandClaude Sonnet 5.5 68815755e8 test(progress): a hung save or beat must not stall the reporter or the heartbeat (#51), red
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathiasandClaude Sonnet 5.5 ba18668796 docs(feature): cloud-served transcripts say the audio left the homelab; empty transcript also triggers the fallback (#52)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathiasandClaude Sonnet 5.5 a65f2f2492 feat(web): say in plain words when a recording's audio left the homelab (#52)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathiasandClaude Sonnet 5.5 a43ee19505 feat(web): helper naming the cloud provider of a transcript, one definition of local (#52)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathiasandClaude Sonnet 5.5 4301e691f5 test(web): a cloud-served transcript says the audio left the homelab (#52), red
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathiasandClaude Sonnet 5.5 61d1ff2255 fix(llm): read a bounded response body, 8 MiB on success and the error bound on failure (#53)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathiasandClaude Sonnet 5.5 9cc954df40 test(llm): the client reads a bounded response body, success and error (#53), red
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:22:17 +02:00
mathias e037c6b683 Merge pull request 'v0.31.1: cross-origin POST protection, ffprobe/ffmpeg limits, receive cap, SIGTERM, lint gate (#41 #47 #48 #49 #50)' (#55) from sprint/audio-followups into main
CI / Deploy via GitOps (push) Successful in 6s
CI / Lint / Test / Vet (push) Successful in 30s
CI / Build & Import (push) Successful in 10s
2026-10-03 19:19:55 +00:00
mathiasandClaude Sonnet 5.5 5baffd0189 fix(test): strip colour codes before looking for the lint skip notice (CI run 8937)
CI / Lint / Test / Vet (pull_request) Successful in 30s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:18:05 +02:00
mathiasandClaude Sonnet 5.5 228bcc77da test(lint): the skip notice is found behind CI's colour codes, red
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:18:01 +02:00
mathiasandClaude Sonnet 5.5 4ef446548d docs(adr): spool arithmetic and the unbounded slow-drip limitation for the receive cap (#47)
CI / Lint / Test / Vet (pull_request) Failing after 30s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:12:48 +02:00
mathiasandClaude Sonnet 5.5 145949875e fix(audio): keep a bounded head of ffmpeg's output instead of buffering all of it (#49)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:12:48 +02:00
mathiasandClaude Sonnet 5.5 82aeac7c19 fix(store): FailUpload only fails a queued or transcribing upload (review of #55)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:12:48 +02:00
mathiasandClaude Sonnet 5.5 1b4747ed03 test: FailUpload must not overwrite a finished upload; ffmpeg stderr is bounded, red (review of #55)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:12:34 +02:00
mathiasandClaude Sonnet 5.5 9939afb90c test: pin the production receive limits, split the vacuous drain test, harden the lint-gate test (review of #55)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:12:34 +02:00
mathiasandClaude Sonnet 5.5 3e69c00a64 docs(adr): ADR-033 amended for SIGTERM handling and the recorded deploy limitation (#50, #47)
CI / Lint / Test / Vet (pull_request) Failing after 31s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:00:55 +02:00
mathiasandClaude Sonnet 5.5 253aa567aa fix(serve): handle SIGTERM and wait for in-flight requests and the worker before exiting (#50)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:00:55 +02:00
mathiasandClaude Sonnet 5.5 5644dc3fc8 fix(uploads): record a job interrupted by shutdown at once, on its own short context (#50)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 21:00:55 +02:00
mathiasandClaude Sonnet 5.5 7df1de6628 test(serve): shutdown waits for in-flight requests and the worker, bounded (#50), red
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 20:59:57 +02:00
mathiasandClaude Sonnet 5.5 7e920102f6 test: SIGTERM cancels the process context; a shutdown mid-job records the row interrupted at once (#50), red
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 20:59:40 +02:00
mathiasandClaude Sonnet 5.5 480c7227b5 fix(uploads): bound concurrent request-body receives, 4 total and 2 per user (#47)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 20:59:09 +02:00
mathiasandClaude Sonnet 5.5 0eb3be9631 test(uploads): concurrent receives are bounded in total and per user (#47), red
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 20:57:31 +02:00
mathiasandClaude Sonnet 5.5 e8c5ac50bf fix(audio): give ffprobe its own 45s deadline, a timeout is the file's fault (#49)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 20:56:58 +02:00
mathiasandClaude Sonnet 5.5 6101dd4e2f test(audio): a hung ffprobe times out on its own deadline as unreadable (#49), red
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 20:56:47 +02:00
mathiasandClaude Sonnet 5.5 089444c19f fix(audio): stop reading ffprobe output past a cap and call the file unreadable (#49)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 20:56:47 +02:00
mathiasandClaude Sonnet 5.5 d44db88787 test(audio): ffprobe output past a cap is unreadable, not buffered (#49), red
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 20:56:37 +02:00
mathiasandClaude Sonnet 5.5 1f8bd849fe fix(audio): cap ffprobe/ffmpeg streams at 32 and narrow the probe output (#49)
A 900-stream file cost ~570 MB RSS in each tool; with the cap the same file fails
in ~18 MB and is reported as unreadable.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 20:56:28 +02:00
mathiasandClaude Sonnet 5.5 7b19801bfa test(audio): ffprobe/ffmpeg must cap streams, ffprobe must narrow its output (#49), red
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 20:55:54 +02:00
mathiasandClaude Sonnet 5.5 1260500d32 fix(web): refuse cross-origin non-safe requests at the router (#48)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 20:55:19 +02:00
mathiasandClaude Sonnet 5.5 10b4c43bd2 test(web): foreign-origin POSTs must be refused by the router (#48), red
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 20:55:00 +02:00
mathiasandClaude Sonnet 5.5 f1f8b6d02e fix(task): lint step fails on findings instead of printing 'not installed' (#41)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 20:54:20 +02:00
mathiasandClaude Sonnet 5.5 0d4c5c9d1d test(lint): task lint must exit non-zero on a finding (#41), red
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 20:53:50 +02:00
mathiasandClaude Sonnet 5.5 24d65b2085 docs(sprint): audio upload follow-ups sweep sprint plan, oath and decision list
CI / Lint / Test / Vet (push) Successful in 27s
CI / Build & Import (push) Successful in 22s
CI / Deploy via GitOps (push) Successful in 5s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 20:50:34 +02:00
mathiasandClaude Sonnet 5.5 df7bd459ed docs(adr): the ingress body-size item is resolved, and uploads are on in production (infra#520)
CI / Build & Import (push) Successful in 13s
CI / Deploy via GitOps (push) Successful in 5s
CI / Lint / Test / Vet (push) Successful in 36s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-03 16:29:32 +02:00
69 changed files with 4073 additions and 905 deletions
+47 -4
View File
@@ -1434,7 +1434,12 @@ an hour of Swedish *was* an unknown here; it is now measured, see decision 8.)
empty disables), recorded per row, and called out in Open items rather than defaulted
silently.
- Swedish only (`language=sv`). Language detection and routing (#28 decision 2) are not
touched.
touched. **Amended 2026-10-04 (#61, interim):** the language is now a per-upload choice on
the form, `sv` (preselected, and what an upload that names none gets) or `auto` ("Other":
`language` is omitted so the engine detects it), a closed set held by the form, the request
code and a CHECK constraint on `uploaded_transcripts.language` (migration 021). Automatic
detection and a general-purpose engine for other languages are still not built (#61); an
`auto` upload goes to the same configured engine, which is not tuned for it.
- `-H 'Expect:'`, multipart field `file` — encoded in the adapter and its tests, not rediscovered.
7. **Jobs run in-process, one at a time (ADR-018 stands).** Transcription is minutes (~7.5 min
@@ -1447,6 +1452,34 @@ an hour of Swedish *was* an unknown here; it is now measured, see decision 8.)
re-uploads. The single-replica assumption gains a second dependent (the semaphore is
process-local) and is recorded here.
*Amended 2026-10-03 (#50, #47).* The process now handles SIGTERM (what Kubernetes sends on
every deploy; only SIGINT was handled, so shutdown code never ran). On shutdown the running job
is recorded `interrupted` at once and the process waits, bounded to 10 s each, for in-flight
requests and for that write; the start-up sweep stays the backstop for a kill. **Known
limitation, deliberately not built: a deploy still loses a job in flight** (finishing it needs
a drain of up to the job time, a ~330 s termination grace and `strategy: Recreate`, i.e.
downtime on every deploy; a 90-minute file takes ~4.5 min and a deploy lands on a running job
perhaps one time in five at current volume). Jobs still queued at shutdown stay `queued` until
the next start's sweep. The user re-uploads, with an honest message.
*Receive limits (#47).* Request bodies are received at most 4 at a time, 2 per user, answered
as busy beyond that. Together with the 20-job queue and the running job that bounds the spool
at about (4 + 20 + 1) x 200 MiB = 5 GiB of the 6 Gi `emptyDir` (infra): raising any of those or
`TAPIR_UPLOAD_MAX_BYTES` must revisit that sum (pinned by `TestNewIntake_Defaults...`, not
enforced against infra). **A body is bounded by its silence, not its length (#56, amended
2026-10-04)**: a `ReadTimeout` would cut legitimate 200 MiB uploads on slow links, so instead
every read of the upload body re-arms the connection's read deadline 60 s ahead
(`App.UploadIdleTimeout`). A body that stops delivering bytes fails its read, Accept returns
and the slot is released; one that keeps moving is never cut. The handler never clears the
deadline: an armed deadline also bounds net/http's drain of an unread remainder before it
replies, and net/http resets it itself before its background read. A deadline still armed
while the file is probed (the multipart part can end before the body's last bytes arrive)
costs at most keep-alive reuse of that connection, not the upload. The 60 s default is the
deployed value (cmd/tapir does not set it) and is pinned by a test. It needs `http.ResponseController` to reach the connection, so the
metrics middleware's writer implements `Unwrap`; where a deadline cannot be set the handler
logs `upload idle bound unavailable` rather than silently running unbounded. Still not
bounded: a body that drips one byte per 59 s holds its slot for as long as it keeps that up.
8. **Chat reuses `chat.Service`; it is not forked.** The uploaded-transcript chat handlers
look the row up through the RLS-scoped store, so a non-owner gets 404 by the same mechanism
as ADR-027's summary gate, **before any model is called**. ADR-027's decisions (ephemeral,
@@ -1531,9 +1564,15 @@ deferred. The scope cut — chat first, no rich analysis, no video — is what k
#33 specifies, with the serving endpoint shown **(recommended, matches the issue)**; (B)
disabled by default, a failed iguana job fails honestly and the user retries. *Blocks:*
the production default only; the adapter ships either way.
- Verify Tapir's ingress `proxy-body-size` against the 200 MB cap (`infra`).
- Per-transcript deletion is not built; account deletion cascades. A private-content store
without a delete button is a gap worth a ticket, not a silent omission.
- ~~Verify Tapir's ingress `proxy-body-size` against the 200 MB cap (`infra`).~~ **Resolved 2026-10-03:** real unauthenticated bodies of 2, 30, 120 and 210 MiB all pass NPM (TLS, piguard) and Traefik to tapir (a 302 to login); there is no body limit in that range. Uploads were enabled in production the same day, local engines only (infra#520).
- ~~Per-transcript deletion is not built; account deletion cascades.~~ **Built 2026-10-03
(#37):** `POST /uploads/{id}/delete`, owner-scoped (a non-owner gets 404 by the same RLS
mechanism), confirm-then-delete on the recording's page. A recording that is **queued or
transcribing is refused (409) until it ends**, because the worker would otherwise write to a
row that is gone; a job whose service was lost stays undeletable until the next start's sweep
marks it failed. The refusal reflects the row's state when the delete was attempted: a
recording that finishes in the same instant still gets a 409 once, and a retry deletes it.
The home list now links to `/uploads` when uploads are mounted.
**Reversibility.** Additive: one table (down-migration drops it), one adapter, one upload
surface, one image change. Removing the upload route removes the feature; `transcripts`, `videos`
@@ -1620,6 +1659,10 @@ on the slow numbers and is reopened: 31B scored 97% against 92%.
job timeout rises from 30 to 60 minutes: ~4 min of kb-whisper and ~40 s of pianissimo per hour of
audio, plus the adjudication.
**Amended 2026-10-04 (#61, interim):** reconcile is for Swedish. A recording whose language is
`auto` is never sent to the base engine (it is Swedish-only) and is not reconciled: the other
engine transcribes alone, then the cloud fallback if that fails and the job is live, and the
page does not claim a second opinion.
8. **Not decided here.** A cloud adjudicator (rejected: private content); tuning the prompt; making
`gemma4-31b` the default; a "reconciling" progress state (the page says "transcribing" for the whole
job); feeding the uncertain spots to chat; whether pianissimo should also be a middle tier in the
+1 -1
View File
@@ -34,7 +34,7 @@ tasks:
lint:
desc: golangci-lint (installed by CI; skipped locally if absent).
cmds:
- 'command -v golangci-lint >/dev/null 2>&1 && golangci-lint run ./... || echo "golangci-lint not installed — skipping (CI installs it)"'
- 'if command -v golangci-lint >/dev/null 2>&1; then golangci-lint run ./...; else echo "golangci-lint not installed — skipping (CI installs it)"; fi'
test:
desc: Run all tests. -p 1 serializes packages so the embedded-postgres tests
+17 -16
View File
@@ -17,9 +17,9 @@ import (
"errors"
"fmt"
"log/slog"
"net"
"net/http"
"os"
"os/signal"
"sync"
"time"
@@ -41,7 +41,7 @@ func main() {
}
log := slog.New(slog.NewTextHandler(os.Stderr, nil))
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
ctx, stop := shutdownContext(context.Background())
defer stop()
var err error
@@ -215,9 +215,11 @@ func cmdServe(ctx context.Context, log *slog.Logger) error {
if err != nil {
return err
}
var workerDone chan struct{} // nil without uploads: nothing to wait for at shutdown
wireUploads(app, cfg, intake)
if intake != nil {
go worker.Run(ctx)
workerDone = make(chan struct{})
go func() { worker.Run(ctx); close(workerDone) }()
log.Info("audio upload enabled", "model", cfg.TranscribeModel, "fallback", cfg.TranscribeFallbackModel,
"max_mb", cfg.UploadMaxBytes>>20, "max_minutes", int(cfg.UploadMaxDuration.Minutes()))
log.Warn("audio upload assumes a SINGLE replica: the transcription queue is process-local (ADR-033, ADR-018)")
@@ -350,20 +352,19 @@ func cmdServe(ctx context.Context, log *slog.Logger) error {
}()
}
// Graceful shutdown on signal: stop accepting, drain in-flight requests.
go func() {
<-ctx.Done()
shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
_ = srv.Shutdown(shutdownCtx)
if metricsSrv != nil {
_ = metricsSrv.Shutdown(shutdownCtx)
}
}()
// Graceful shutdown on signal: stop accepting, drain in-flight requests, and let the
// worker record the job it was running as interrupted (#50).
var extra []*http.Server
if metricsSrv != nil {
extra = append(extra, metricsSrv)
}
ln, err := net.Listen("tcp", srv.Addr)
if err != nil {
return fmt.Errorf("serve: listen: %w", err)
}
log.Info("serving web ui", "addr", cfg.HTTPAddr, "user", cfg.UserID)
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
return fmt.Errorf("serve: %w", err)
if err := serveHTTP(ctx, srv, ln, extra, workerDone, shutdownGrace); err != nil {
return err
}
return nil
}
+55
View File
@@ -0,0 +1,55 @@
package main
import (
"context"
"errors"
"fmt"
"net"
"net/http"
"os"
"os/signal"
"syscall"
"time"
)
// shutdownGrace bounds each wait at shutdown (in-flight requests, then the worker). Two of
// them stay inside Kubernetes' default 30 s termination grace.
const shutdownGrace = 10 * time.Second
// shutdownContext is cancelled by SIGINT (a terminal) or SIGTERM (what Kubernetes sends
// on every deploy). Handling only SIGINT meant a deploy killed the process before any
// shutdown code ran (#50).
func shutdownContext(parent context.Context) (context.Context, context.CancelFunc) {
return signal.NotifyContext(parent, os.Interrupt, syscall.SIGTERM)
}
// serveHTTP serves srv on ln until it fails or ctx ends, then shuts every server down and
// waits for in-flight requests and for workerDone (nil: no worker) to finish, each for at
// most grace. It must not return earlier: Serve returns http.ErrServerClosed as soon as
// Shutdown STARTS, and returning from main kills the drain and the worker's last write.
func serveHTTP(ctx context.Context, srv *http.Server, ln net.Listener, extra []*http.Server, workerDone <-chan struct{}, grace time.Duration) error {
errc := make(chan error, 1)
go func() { errc <- srv.Serve(ln) }()
select {
case err := <-errc:
if !errors.Is(err, http.ErrServerClosed) {
return fmt.Errorf("serve: %w", err)
}
return nil
case <-ctx.Done():
}
shutdownCtx, cancel := context.WithTimeout(context.Background(), grace)
defer cancel()
for _, s := range append([]*http.Server{srv}, extra...) {
_ = s.Shutdown(shutdownCtx)
}
if workerDone != nil {
select {
case <-workerDone:
case <-time.After(grace):
}
}
return nil
}
+124
View File
@@ -0,0 +1,124 @@
package main
import (
"context"
"net/http"
"net/http/httptest"
"os"
"sync"
"syscall"
"testing"
"time"
)
// Kubernetes stops a pod with SIGTERM; only SIGINT was handled, so a deploy killed the
// process without running any shutdown code (#50).
func TestShutdownContext_IsCancelledBySIGTERMAndSIGINT(t *testing.T) {
for _, sig := range []syscall.Signal{syscall.SIGTERM, syscall.SIGINT} {
t.Run(sig.String(), func(t *testing.T) {
ctx, stop := shutdownContext(context.Background())
defer stop()
if err := syscall.Kill(os.Getpid(), sig); err != nil {
t.Fatalf("send %v: %v", sig, err)
}
select {
case <-ctx.Done():
case <-time.After(2 * time.Second):
t.Fatalf("context not cancelled by %v", sig)
}
})
}
}
// serveHTTP must not return while a request is in flight or the worker is still recording
// its job: Serve returns http.ErrServerClosed as soon as Shutdown STARTS, and returning
// from it let the process exit before either finished (#50). Each half is tested with the
// other already satisfied, so neither can hold the call open on the other's behalf.
func pendingFor(t *testing.T, returned <-chan error, why string) {
t.Helper()
select {
case err := <-returned:
t.Fatalf("serveHTTP returned (%v) %s", err, why)
case <-time.After(150 * time.Millisecond):
}
}
func TestServeHTTP_DrainsAnInFlightRequestBeforeReturning(t *testing.T) {
release := make(chan struct{})
entered := make(chan struct{})
srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
close(entered)
<-release
_, _ = w.Write([]byte("ok"))
}))
srv.Start()
defer srv.Close()
var once sync.Once
releaseHandler := func() { once.Do(func() { close(release) }) }
defer releaseHandler() // runs before srv.Close: a failing test must not hang on the blocked handler
ctx, cancel := context.WithCancel(context.Background())
workerDone := make(chan struct{})
close(workerDone) // the worker is already done: only the request can hold serveHTTP open
returned := make(chan error, 1)
go func() { returned <- serveHTTP(ctx, srv.Config, srv.Listener, nil, workerDone, 5*time.Second) }()
go func() { _, _ = http.Get(srv.URL) }()
<-entered
cancel()
pendingFor(t, returned, "while a request was still in flight")
releaseHandler()
select {
case err := <-returned:
if err != nil {
t.Fatalf("serveHTTP: %v", err)
}
case <-time.After(2 * time.Second):
t.Fatal("serveHTTP did not return once the request finished")
}
}
func TestServeHTTP_WaitsForTheWorkerBeforeReturning(t *testing.T) {
srv := httptest.NewUnstartedServer(http.NotFoundHandler())
srv.Start()
defer srv.Close()
ctx, cancel := context.WithCancel(context.Background())
workerDone := make(chan struct{})
returned := make(chan error, 1)
go func() { returned <- serveHTTP(ctx, srv.Config, srv.Listener, nil, workerDone, 5*time.Second) }() // no request in flight
cancel()
pendingFor(t, returned, "before the worker finished")
close(workerDone)
select {
case err := <-returned:
if err != nil {
t.Fatalf("serveHTTP: %v", err)
}
case <-time.After(2 * time.Second):
t.Fatal("serveHTTP did not return once the worker finished")
}
}
func TestServeHTTP_GivesUpOnAWorkerThatNeverFinishesAfterTheGrace(t *testing.T) {
srv := httptest.NewUnstartedServer(http.NotFoundHandler())
srv.Start()
defer srv.Close()
ctx, cancel := context.WithCancel(context.Background())
returned := make(chan error, 1)
go func() {
returned <- serveHTTP(ctx, srv.Config, srv.Listener, nil, make(chan struct{}), 200*time.Millisecond)
}()
cancel()
select {
case <-returned:
case <-time.After(3 * time.Second):
t.Fatal("a stuck worker must not hold the process past the grace period")
}
}
+2 -2
View File
@@ -25,7 +25,7 @@ type fakeUploadsStore struct {
failedKinds []domain.UploadErrorKind
}
func (f *fakeUploadsStore) CreateUpload(context.Context, string, string, bool) (string, error) {
func (f *fakeUploadsStore) CreateUpload(context.Context, string, string, domain.Language, bool) (string, error) {
return "id", nil
}
@@ -44,7 +44,7 @@ func (f *fakeUploadsStore) SaveProgress(context.Context, string, string, []domai
return nil
}
func (f *fakeUploadsStore) Heartbeat(context.Context, string, string) error { return nil }
func (f *fakeUploadsStore) Heartbeat(context.Context, string, string, time.Time) error { return nil }
func (f *fakeUploadsStore) FailInterruptedUploads(context.Context) (int, error) {
f.sweeps++
+2 -1
View File
@@ -14,7 +14,8 @@ it** — endpoints and aliases drift, and this file is a snapshot (2026-06-06),
not LiteLLM. Use `30401` for off-cluster LiteLLM. `confirm` resolved.
- **Speech-to-text (ADR-033):** the same gateway serves `POST /v1/audio/transcriptions` for
`iguana/kb-whisper` (primary, local) and `berget/kb-whisper` (fallback, EU cloud — private audio
leaves the homelab if it is used). **Call it via the NodePort / in-cluster address above, NOT
leaves the homelab if it is used: when the primary fails **or returns an empty transcript**,
e.g. a silent or music-only file; the transcript page then says so, #52). **Call it via the NodePort / in-cluster address above, NOT
`https://llm-api.d-ma.be`:** that public ingress (openresty) returns `413` for bodies over
roughly 8–10 MB (~5 minutes of 16 kHz WAV), while the NodePort took 174 MB (90 minutes) in one
request (measured 2026-10-01, `docs/research/audio-length-ceiling-2026-10-01.md`). Send multipart
@@ -0,0 +1,69 @@
# ADR-034 second blind check: protocol (ready to run; tapir#33, D9, D10)
This repo is public and the recordings are private, so this file holds the method only. The
sheet, the key and the filled-in verdicts stay outside the repo, and the trial data is deleted
after the check. Results go back here as aggregate numbers, as in
`reconcile-blind-check-2026-10-02.md`.
## What the check decides
ADR-034 (two engines, a local model reconciles their disagreements) is `Proposed` and off in
production. Two conditions were set for keeping it. The first run answered the first one on one
reading only and did not answer the second:
1. **The reconciled reading is right at least 85% of the time.** Open flaw in the bar: it did not
say how ties count. The first run gave 95% (decided items only) or 70% (all 30), so it was met on
one reading and missed on the other.
2. **It is no worse than the better single engine.** Not answered: before ADR-035 the spots the
model *kept* as the base were not stored, so kb-whisper alone could not be compared.
ADR-035 now stores every judged spot (`reconciliation.contested`: both readings, which one was
used, unsure or not, a few words of left context, a time). That makes condition 2 measurable.
## Rules to fix BEFORE looking at any result (the owner confirms or changes them)
| Rule | Proposed |
|---|---|
| Who judges | The speaker, hearing the audio where the text cannot settle it (D9). Not Opus: it cannot hear, and its agreement with the speaker was only shown on items picked as decidable. |
| Sample | 30 spots drawn at random with a fixed seed, from ALL judged spots (changed and kept-base alike), not only the changed ones. Insertions and deletions that cannot be placed are listed separately and not sampled. |
| Verdict per spot | `x`, `y` (the two readings, shown in random order, key kept apart), `neither`, or `cannot tell`. |
| Condition 1 | Reconciled is right on at least 85% of the spots where the judge could tell (`cannot tell` excluded, reported, and `neither` counts as not right). Also report all-30. Pass needs the decided-only figure at 85% or more AND the all-30 figure above 70%. |
| Condition 2 | Over the same spots, the reconciled text is right at least as often as the better single engine (base engine alone is right where the base reading is right; the other engine alone where the alt reading is right). Ties favour the single engine. |
| If it fails | Switch reconcile off by configuration; the single-engine path is unchanged (ADR-034 reversibility). |
Why a fixed all-30 floor: the literal 70% last time sat well under the bar, and "decided only" can
hide a model that mostly lands on spelling differences.
## What a run needs (D10: the owner is present)
- The same real recording as before (about 60 minutes). Only the owner has it.
- A **throwaway** tapir instance with its own Postgres, not the production pod, so production's
key pin and deployment are untouched. Settings: `TAPIR_RECONCILE_BASE_MODEL=iguana/pianissimo`,
`TAPIR_TRANSCRIBE_MODEL=iguana/kb-whisper`, `TAPIR_RECONCILE_MODEL=iguana/gemma4-31b`, cloud
fallbacks empty. Job time was 21 min 49 s per 63.6 min last run.
- iguana with Low Power Mode off and gemma4-31b able to load (the production-iguana model load
that needs a quiet window; D6).
## Steps
1. Start the throwaway instance, upload the recording, wait for `done` (about 22 minutes).
2. Read `reconciliation.contested` from the throwaway database. From it build, **outside the repo**:
`sheet` (spot number, time, left context, reading x, reading y, random order) and `key` (which
of x/y is the base, which is the other engine, which one the reconciler used).
3. The speaker fills in the sheet (about 20 to 30 minutes for 30 spots, audio open at the times).
The agent does not see the key until the sheet is returned.
4. Score with the rules above. Write the aggregate numbers into a dated note next to this one,
and amend ADR-034's validation section (kept, or switched off, and the tie rule used).
5. Delete the throwaway instance, its database and the sheet, as last time.
6. Only if the check passes and the owner wants it in production: enabling reconcile there is
the separate D10 step (it needs `iguana/pianissimo` and the reconciler added to the tapir key
pin in the infra repo plus the env vars on the deployment, `risk/medium`, owner present).
## Open and owner-only
- Confirm or change the rules table above before the run.
- The **tool that builds the sheet and key** is the one piece not in the repo (last time it was a
one-off script). It can be built test-first on request; until then the agent builds it ad hoc
outside the repo.
- A Berget adjudicator (D8, tapir#45) is a different question with its own measurement and is
not part of this check: this check judges the local setup that ADR-034 describes.
@@ -0,0 +1,115 @@
# Sprint: audio upload follow-ups (Sweep Sprint, 2026-10)
Method: Sweep Sprint (phase 1 parallel read-only research per issue: **done**; phase 2 sequential
execution, one issue per commit: **v0.31.1 and v0.32.0 shipped 2026-10-03/04, the owner-held items
below remain open**). Closing oath below.
Milestones: [v0.31.1 - security and robustness](https://git.d-ma.be/mathias/tapir/milestone/10),
[v0.32.0 - audio upload follow-ups](https://git.d-ma.be/mathias/tapir/milestone/11).
Context: v0.31.0 shipped audio upload (ADR-033/034/035) and is live in production behind auth at
`/uploads`, with the Berget cloud fallback on for audio (infra#522). A real 63.7 min upload
transcribed in 2m53s with no errors. The phase-1 investigation found two pre-existing or new
weaknesses an authenticated user can reach, so the first release is a hotfix.
## Oath (closing gates, every clause must hold; blocked = descoped and tracked, never "done")
| # | Clause | Evidence |
|---|--------|----------|
| SG1 | Every issue in the sprint is closed with a linked commit, or moved out with a stated reason | issue list of both milestones |
| SG2 | Each fix has a test that was seen red on main first, named after the mechanism it observes | red/green output in the issue |
| SG3 | `task check` AND `golangci-lint run ./...` exit 0 (checked by `$?`, not by grep); `task check` does not gate lint until #41 | gate output |
| SG4 | Control code (#48, #49, the upload cap in #47) has a recorded adversarial review verified by execution, verdict in the issue | gatekeeper verdict comment |
| SG5 | Docs current in the same commit: ADR-033/035 text, feature file, use-case coverage test | diff |
| SG6 | Each release tagged (v0.31.1, v0.32.0), pushed, deployed, and observed live (pod running the new image, no restart loop, a real upload still works) | `kubectl`/log + one upload |
| SG7 | Learnings written to the brain; deferred work filed as issues; decisions that are the owner's recorded as DECISION NEEDED | brain ids + issue numbers |
## Work order
### v0.31.1 (security and robustness; hotfix, do first)
1. **#41 lint gate.** `task check` swallows lint findings (`|| echo`). Make the gate fail on findings,
fix what it surfaces. First, so every later gate means something. (Decision D1.)
2. **#48 cross-origin POST protection.** `http.NewCrossOriginProtection()` around `Router()`.
Red: foreign-origin POST to `/account/delete`, `/uploads`, `/uploads/{id}/chat` is refused.
3. **#49 ffprobe/ffmpeg limits.** `-max_streams 32` + narrower probe output; probe stdout cap;
probe deadline. Three commits.
4. **#47 concurrent-upload cap (remaining item).** Bound the spool: reservation per accepted upload
against the 6Gi emptyDir so concurrent uploads cannot evict the pod.
5. **#50 SIGTERM interim.** Handle SIGTERM; record the interrupted job at once; fix the false
comment. Full drain recorded as a limitation. (Decision D2.)
Release v0.31.1: one adversarial review round over 2-5, then tag, push, observe.
### v0.32.0 (follow-ups)
6. **#52** label a cloud-served transcript in plain words; document the empty-transcript trigger.
7. **#53** bound the LLM client's response body read.
8. **#51** tracker: save timeout, then app-clock heartbeat.
9. **#37** nav link to uploads and per-recording delete (queued/transcribing: refuse until it
ends). (Decision D3.)
10. **#36** chat default model after measurement. (Decision D4.)
11. **#38** measurements (Berget length limit, large models on production iguana). (D5, D6.)
12. **#46** remainder of the hardening list after the items above are split out.
13. **#45** evaluate Berget as the reconciler: a question, not work. (D8.)
## DECISION NEEDED (owner's call; recommendation first)
| # | Question | Recommendation | Blocks |
|---|----------|----------------|--------|
| D1 | #41: should any linter be allowed to opt out of the gate? | No opt-out; fix findings | #41 |
| D2 | #50: full SIGTERM drain, or record as a limitation and ship the interim fix? | Limitation + interim; a drain costs 4-6 h and downtime on every deploy at current volume | #50 |
| D3 | #37: delete of a queued/transcribing recording | Refuse until it ends | #37 |
| D4 | #36: make a 31B model the chat default? | Only after measurement, conditional | #36 |
| D5 | #38: Berget length measurement needs a throwaway key | Yes, throwaway key | #38 |
| D6 | #38: load large models on production iguana | Yes, in a quiet window with the owner present | #38 |
| D7 | #46: where the ffmpeg tarball lives | Do nothing now; file an issue | none |
| D8 | #45: evaluate Berget as the reconciler | **Decided 2026-10-04 by the owner: yes, Berget may be used, and each user must be able to keep a recording on local models only** (reverses the recommendation). Opt-out first: #60; #45 stays as the model measurement | #60, #45 |
| D9 | Who judges ADR-034's second blind check | The owner; do not reuse the one-off Opus authorisation. Protocol: `docs/research/reconcile-second-check-protocol.md` | ADR-034 |
| D10 | When to enable reconcile in production | Needs `iguana/pianissimo` in the key pin and env vars; owner present | ADR-034 |
## Dropped or reduced, with reasons
- **#46 item 1 (`alis`/`dref` external reference):** checked, not a problem. Under the application's
flags ffmpeg skips the external track and writes a 78-byte WAV.
- **#46 item 2 / #47 item 2:** merged into #49.
- **#47 item 3 (full drain):** reduced to #50 plus a recorded limitation.
- **#46 `Beat` taking the tracker mutex:** incorrect in #46; only `logStill` does. Corrected in #51.
## Human-present items (do not run unattended)
- Production iguana model loads (#38, D6).
- Enabling reconcile in production (D10).
- Any key change or `apply` in infra (infra#523, infra#524 are open and not part of this sprint).
## Release plan
SemVer: v0.31.1 patch (fixes, no new behaviour); v0.32.0 minor (nav link, delete, cloud labelling).
Every push to tapir main redeploys the pod and fails any job in flight: push when no upload is
running (check the pod log first).
## Status (2026-10-04)
**Shipped and deployed:** v0.31.1 (PR #55, merge e037c6b) and v0.32.0 (PR #58, merge 85f4cc3; pod
tapir-597dbfccc-ng78q 1/1, 0 restarts, image 85f4cc3). Each went through an adversarial review
round by agent with the verdicts recorded on the PR (v0.31.1: BLOCK then APPROVE; v0.32.0: BLOCK,
BLOCK, APPROVE, APPROVE; every block was a real defect, fixed test-first).
**Live evidence:** a foreign-origin `POST` to `/account/delete` and to `/uploads/{id}/delete` is
refused (403) on production, a same-origin one reaches the login redirect (302).
**Not observed live (needs an authenticated session, so it is the owner's look; closed on word or screenshot):**
- #52, cloud side: a transcript served by the Berget fallback names `berget`; a reconciled all-local
one shows no note. The local side was observed on 2026-10-04 (a new clip served by iguana: chip
`iguana/…`, no cloud note). The cloud side is carried as an acceptance item in #60.
**Observed live since:** #37 (owner, 2026-10-04: link, confirm-then-delete, no control while transcribing; closed). #50: at the 2026-10-04 rollout the old pod went `Terminating` then `Completed`
(not `Error`), 0 restarts; #50 closed. No job was in flight, so a drained job is still unobserved
(a recorded limitation, D2).
**Still open (owner's call, milestone 11):** #36, #38, #45 (now the Berget adjudicator measurement), #60 (per-recording local-only choice, 4 sub-decisions), #46 remainder, #56 (slow-drip bodies hold
a receive slot), #57 (shutdown logging, second signal). Decisions D4-D6, D9, D10 above are unchanged; D8 is decided (above).
**Deferred and filed from the reviews:** #56, #57, #59 (start-up exits when the identity provider's OIDC
discovery returns 503 (seen once at 19:40 on 2026-10-03; the pod restarted itself and recovered)).
+35
View File
@@ -240,6 +240,24 @@ Feature: Upload an audio recording, get a private Swedish transcript, discuss it
Then it is transcribed anyway and its audio step says how long the audio really is
And a complete recording whose header over-estimated its length is not refused
Scenario: A transcript made by the cloud fallback says the audio left the homelab
When the transcript was produced by an endpoint that is not local
Then its page says in plain words that the audio left the homelab and names the provider
And a locally transcribed recording carries no such note
Scenario: I can delete a finished recording
When I confirm deleting a done or failed recording
Then the recording and its transcript are gone and I am back on my list
And another user's recording is a 404 and is untouched
Scenario: A recording still being transcribed cannot be deleted
When I try to delete a recording that is queued or transcribing
Then I am told it is still being transcribed and nothing is removed
Scenario: The home list links to my recordings when uploads are on
Given audio upload is enabled
Then the home list links to the recordings page
Scenario: A full queue is a retry, not an error
When too many recordings are already waiting
Then the upload is answered as busy, its row is failed honestly and its audio is deleted
@@ -256,3 +274,20 @@ Feature: Upload an audio recording, get a private Swedish transcript, discuss it
Scenario: Input that cannot be right is a client error
When the id is not an id, or a header or title holds control bytes
Then the answer is a 4xx or the title is cleaned, never a server error
Scenario: I choose the language of a recording
When I upload a recording and choose Swedish or Other on the form
Then the recording is transcribed as that language and its page says which
Scenario: Swedish stays the default
When an upload names no language
Then it is transcribed as Swedish, as before the choice existed
Scenario: A language outside the list is refused
When an upload names a language the form does not offer
Then it is answered 400 before anything is stored or sent to an engine
Scenario: A recording in another language never reaches the Swedish-only engine
When I choose Other for a recording
Then the language is left for the engine to detect and the Swedish-only base engine and the reconcile are not used
And the engine that does run is the configured one, which is not tuned for other languages
+2 -2
View File
@@ -33,9 +33,9 @@ require (
github.com/rogpeppe/go-internal v1.15.0 // indirect
github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect
golang.org/x/sync v0.18.0 // indirect
golang.org/x/sync v0.21.0 // indirect
golang.org/x/sys v0.41.0 // indirect
golang.org/x/text v0.31.0 // indirect
golang.org/x/text v0.39.0 // indirect
google.golang.org/protobuf v1.36.8 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
+4 -4
View File
@@ -113,12 +113,12 @@ go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=
go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
golang.org/x/sync v0.18.0 h1:kr88TuHDroi+UVf+0hZnirlk8o8T+4MrK6mr60WkH/I=
golang.org/x/sync v0.18.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/text v0.31.0 h1:aC8ghyu4JhP8VojJ2lEHBnochRno1sgL6nEi9WGFGMM=
golang.org/x/text v0.31.0/go.mod h1:tKRAlv61yKIjGGHX/4tP1LTbc13YSec1pxVEWXzfoeM=
golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus=
golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc=
+6 -2
View File
@@ -47,11 +47,15 @@ func (f *FFmpeg) ToWAV(ctx context.Context, src, dst string) error {
args = append(args, "-f", "wav", dst)
cmd := exec.CommandContext(ctx, bin, args...)
cmd.WaitDelay = killGrace
if out, err := cmd.CombinedOutput(); err != nil {
// Only a bounded head of ffmpeg's output is kept: a hostile file made it print ~0.4
// bytes per input byte, all of which CombinedOutput would buffer and copy (#49).
out := &headWriter{max: maxStderr}
cmd.Stdout, cmd.Stderr = out, out
if err := cmd.Run(); err != nil {
if ctxErr := ctx.Err(); ctxErr != nil {
return fmt.Errorf("audio: ffmpeg: %w", ctxErr)
}
return fmt.Errorf("audio: ffmpeg: %w: %s", err, strings.TrimSpace(string(out)))
return fmt.Errorf("audio: ffmpeg: %w: %s", err, strings.TrimSpace(out.buf.String()))
}
// A zero exit that wrote nothing is still a failure: never hand whisper a
// missing file and let the error surface three layers later.
+29
View File
@@ -109,3 +109,32 @@ printf data > "$last"`)
require.NoError(t, err)
assert.NotContains(t, " "+strings.ReplaceAll(string(raw), "\n", " ")+" ", " -t ")
}
func TestFFmpegToWAV_CapsTheStreamCountBeforeTheInput(t *testing.T) {
// ffmpeg also allocated ~570 MB opening the 900-stream file (#49).
rec := filepath.Join(t.TempDir(), "argv")
bin := stub(t, `printf '%s\n' "$@" > `+rec+`
for a; do last="$a"; done
printf data > "$last"`)
err := (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/spool/in", filepath.Join(t.TempDir(), "o.wav"))
require.NoError(t, err)
args := argvOf(t, rec)
ms, in := indexOf(args, "-max_streams"), indexOf(args, "-i")
require.GreaterOrEqual(t, ms, 0, "argv: %v", args)
assert.Equal(t, "32", args[ms+1])
assert.Less(t, ms, in, "an input option placed after -i is applied to the output")
}
func TestFFmpegToWAV_ABoundedPartOfAFloodingStderrIsKept(t *testing.T) {
// A hostile mp3 made ffmpeg print ~0.4 bytes of stderr per input byte (83 MB for a
// 209 MB file), all of it buffered, then copied twice into the error (#49 review).
bin := stub(t, `head -c 5000000 /dev/zero | tr '\0' 'e' >&2; exit 1`)
err := (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/spool/in", filepath.Join(t.TempDir(), "o.wav"))
require.Error(t, err)
assert.Less(t, len(err.Error()), 16*1024, "the error carries a bounded head of stderr, not all of it")
assert.Contains(t, err.Error(), "eeee", "and still carries some of it")
}
+85 -9
View File
@@ -4,6 +4,7 @@
package audio
import (
"bytes"
"context"
"encoding/json"
"errors"
@@ -19,6 +20,51 @@ import (
type FFprobe struct {
// Bin is the ffprobe executable ("ffprobe" resolves via PATH).
Bin string
// MaxOutput bounds how much of ffprobe's stdout is read; zero means DefaultMaxOutput.
MaxOutput int
// Timeout bounds one probe; zero means DefaultProbeTimeout. A probe is milliseconds of
// work on a real file, so one that runs this long is the FILE's doing (#49).
Timeout time.Duration
}
// DefaultProbeTimeout is the probe deadline when FFprobe.Timeout is unset.
const DefaultProbeTimeout = 45 * time.Second
// DefaultMaxOutput is far above the narrowed probe output of a legitimate file (a few
// hundred bytes) and far below what a hostile one can make ffprobe print.
const DefaultMaxOutput = 1 << 20
// maxStderr is how much of ffprobe's stderr is kept for the error message.
const maxStderr = 4096
// errOutputCap is the write error that stops ffprobe once it prints past the cap.
var errOutputCap = errors.New("audio: ffprobe output exceeds the cap")
// capWriter collects up to max bytes and then refuses further writes, which closes the
// pipe and ends the process instead of buffering whatever it prints.
type capWriter struct {
buf bytes.Buffer
max int
}
func (w *capWriter) Write(p []byte) (int, error) {
if w.buf.Len()+len(p) > w.max {
return 0, errOutputCap
}
return w.buf.Write(p)
}
// headWriter keeps the first max bytes and silently drops the rest.
type headWriter struct {
buf bytes.Buffer
max int
}
func (w *headWriter) Write(p []byte) (int, error) {
if room := w.max - w.buf.Len(); room > 0 {
w.buf.Write(p[:min(room, len(p))])
}
return len(p), nil
}
// AllowedDemuxers are the input formats an upload may be. The input is
@@ -30,9 +76,22 @@ type FFprobe struct {
// formats. "mov" covers .mp4/.m4a/.3gp; "matroska" covers .webm.
const AllowedDemuxers = "wav,mov,mp4,m4a,3gp,mp3,ogg,matroska,webm,flac,aac,opus,aiff,amr,asf"
// MaxStreams bounds how many streams a file may declare. A 1.3 MB file declaring 900 of
// them cost ~570 MB of RSS in each of ffprobe and ffmpeg (#49), against a 512 Mi pod;
// real recordings have a handful.
const MaxStreams = "32"
// inputRestrictions are ffmpeg/ffprobe INPUT options and must precede the input.
// Local files and pipes only: no network, no other protocols.
var inputRestrictions = []string{"-protocol_whitelist", "file,pipe", "-format_whitelist", AllowedDemuxers}
// Local files and pipes only: no network, no other protocols, and a bounded stream count.
var inputRestrictions = []string{
"-protocol_whitelist", "file,pipe",
"-format_whitelist", AllowedDemuxers,
"-max_streams", MaxStreams,
}
// probeEntries is exactly what ParseProbe reads. -show_format/-show_streams print every
// tag too, and a 120 MB WAV made 491 MB of JSON that Output() buffers.
const probeEntries = "format=duration:stream=codec_type:stream_disposition=attached_pic"
// killGrace bounds how long Probe waits for a cancelled ffprobe's pipes to close.
const killGrace = 2 * time.Second
@@ -48,21 +107,38 @@ func (f *FFprobe) Probe(ctx context.Context, path string) (uploads.Probe, error)
bin = "ffprobe"
}
args := append([]string{"-v", "error"}, inputRestrictions...)
args = append(args, "-print_format", "json", "-show_format", "-show_streams", path)
cmd := exec.CommandContext(ctx, bin, args...)
args = append(args, "-print_format", "json", "-show_entries", probeEntries, path)
timeout := f.Timeout
if timeout <= 0 {
timeout = DefaultProbeTimeout
}
probeCtx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
cmd := exec.CommandContext(probeCtx, bin, args...)
cmd.WaitDelay = killGrace
out, err := cmd.Output()
if err != nil {
if ctxErr := ctx.Err(); ctxErr != nil {
maxOut := f.MaxOutput
if maxOut <= 0 {
maxOut = DefaultMaxOutput
}
stdout, stderr := &capWriter{max: maxOut}, &headWriter{max: maxStderr}
cmd.Stdout, cmd.Stderr = stdout, stderr
if err := cmd.Run(); err != nil {
if ctxErr := ctx.Err(); ctxErr != nil { // the caller gave up: not a verdict on the file
return uploads.Probe{}, fmt.Errorf("audio: ffprobe: %w", ctxErr)
}
if probeCtx.Err() != nil { // our own deadline passed while the caller was still waiting
return uploads.Probe{}, fmt.Errorf("%w: ffprobe did not finish within %s", uploads.ErrUnreadable, timeout)
}
if errors.Is(err, errOutputCap) {
return uploads.Probe{}, fmt.Errorf("%w: ffprobe printed more than %d bytes", uploads.ErrUnreadable, maxOut)
}
var exit *exec.ExitError
if errors.As(err, &exit) {
return uploads.Probe{}, fmt.Errorf("%w: ffprobe: %s", uploads.ErrUnreadable, exit.Stderr)
return uploads.Probe{}, fmt.Errorf("%w: ffprobe: %s", uploads.ErrUnreadable, stderr.buf.String())
}
return uploads.Probe{}, fmt.Errorf("audio: run ffprobe: %w", err)
}
p, err := ParseProbe(out)
p, err := ParseProbe(stdout.buf.Bytes())
if err != nil {
return uploads.Probe{}, fmt.Errorf("audio: ffprobe output: %w", err)
}
+75
View File
@@ -184,3 +184,78 @@ func TestAllowedDemuxers_ExcludeThePlaylistFormats(t *testing.T) {
assert.Contains(t, strings.Split(audio.AllowedDemuxers, ","), good)
}
}
func TestFFprobeProbe_CapsTheStreamCountBeforeTheInput(t *testing.T) {
// A 1.3 MB file declaring 900 streams cost ~570 MB of RSS in ffprobe (#49) against
// a 512 Mi pod. -max_streams is an input option: it must precede the path.
bin, argvFile := argvStub(t)
_, err := (&audio.FFprobe{Bin: bin}).Probe(context.Background(), "/spool/upload-1")
require.NoError(t, err)
args := argvOf(t, argvFile)
ms := indexOf(args, "-max_streams")
require.GreaterOrEqual(t, ms, 0, "argv: %v", args)
assert.Equal(t, "32", args[ms+1])
assert.Less(t, ms, len(args)-1, "must come before the path, which stays last")
}
func TestFFprobeProbe_AsksOnlyForTheFieldsParseProbeReads(t *testing.T) {
// -show_format/-show_streams print every tag: a 120 MB WAV produced 491 MB of JSON
// that cmd.Output() buffers in Go (#49).
bin, argvFile := argvStub(t)
_, err := (&audio.FFprobe{Bin: bin}).Probe(context.Background(), "/spool/upload-1")
require.NoError(t, err)
args := argvOf(t, argvFile)
assert.Equal(t, -1, indexOf(args, "-show_format"), "argv: %v", args)
assert.Equal(t, -1, indexOf(args, "-show_streams"), "argv: %v", args)
se := indexOf(args, "-show_entries")
require.GreaterOrEqual(t, se, 0, "argv: %v", args)
assert.Equal(t, "format=duration:stream=codec_type:stream_disposition=attached_pic", args[se+1])
}
func TestFFprobeProbe_OutputPastTheCapIsUnreadableNotBuffered(t *testing.T) {
// ffprobe's stdout is attacker-influenced (tags, 491 MB observed). Past the cap the
// probe is stopped and the file is reported as unreadable (#49).
bin := stub(t, `head -c 200000 /dev/zero | tr '\0' 'a'`)
_, err := (&audio.FFprobe{Bin: bin, MaxOutput: 4096}).Probe(context.Background(), "/x")
assert.ErrorIs(t, err, uploads.ErrUnreadable)
}
func TestFFprobeProbe_OutputWithinTheCapIsParsed(t *testing.T) {
bin := stub(t, `echo '{"streams":[{"codec_type":"audio"}],"format":{"duration":"90"}}'`)
got, err := (&audio.FFprobe{Bin: bin, MaxOutput: 4096}).Probe(context.Background(), "/x")
require.NoError(t, err)
assert.True(t, got.HasAudio)
}
func TestFFprobeProbe_AHungProbeTimesOutOnItsOwnDeadlineAsUnreadableNotAsOurFault(t *testing.T) {
// Accept runs the probe on the request context, which has no deadline of its own;
// a probe that never returns would pin a request (#49). `exec` so the sleep is the
// process itself and not a child holding the pipe open.
bin := stub(t, `exec sleep 30`)
start := time.Now()
_, err := (&audio.FFprobe{Bin: bin, Timeout: 200 * time.Millisecond}).Probe(context.Background(), "/x")
assert.ErrorIs(t, err, uploads.ErrUnreadable)
assert.Less(t, time.Since(start), 3*time.Second)
}
func TestFFprobeProbe_ParentCancellationIsStillNotUnreadableWhenATimeoutIsSet(t *testing.T) {
bin := stub(t, `exec sleep 30`)
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
_, err := (&audio.FFprobe{Bin: bin, Timeout: time.Minute}).Probe(ctx, "/x")
require.Error(t, err)
assert.False(t, errors.Is(err, uploads.ErrUnreadable), "the caller giving up is not a verdict on the file")
assert.ErrorIs(t, err, context.DeadlineExceeded)
}
+156
View File
@@ -0,0 +1,156 @@
package llm
// Observes how much of a gateway response the client will read. The body is read before
// the status is looked at, so a misbehaving gateway or proxy that streams forever was
// bounded only by the client's overall timeout (#53).
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"time"
)
// endlessGateway answers with the given status and streams filler until the client hangs
// up; wrote reports how many bytes it managed to send.
func endlessGateway(t *testing.T, status int, wrote *atomic.Int64) *httptest.Server {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(status)
chunk := []byte(strings.Repeat("a", 4096))
for {
n, err := w.Write(chunk)
wrote.Add(int64(n))
if err != nil {
return
}
if f, ok := w.(http.Flusher); ok {
f.Flush()
}
}
}))
t.Cleanup(srv.Close)
return srv
}
func TestClient_ASuccessBodyLargerThanTheCapIsAnErrorAndTheClientStopsReading(t *testing.T) {
var wrote atomic.Int64
srv := endlessGateway(t, http.StatusOK, &wrote)
c := New(srv.URL, "", "m", 30*time.Second, WithMaxResponseBytes(8192))
start := time.Now()
_, err := c.Complete(context.Background(), "s", "u")
if err == nil || !strings.Contains(err.Error(), "exceeds") {
t.Fatalf("want a response-too-large error, got %v", err)
}
if time.Since(start) > 5*time.Second {
t.Fatalf("the client kept reading: took %s", time.Since(start))
}
// Hanging up lets the server's writes fail; allow what the transport had already buffered.
time.Sleep(100 * time.Millisecond)
if got := wrote.Load(); got > 8192+1<<20 {
t.Fatalf("server managed to send %d bytes after a cap of 8192", got)
}
}
func TestClient_AFailingBodyIsReadOnlyUpToTheErrorBound(t *testing.T) {
var wrote atomic.Int64
srv := endlessGateway(t, http.StatusBadGateway, &wrote)
c := New(srv.URL, "", "m", 30*time.Second)
start := time.Now()
_, err := c.Complete(context.Background(), "s", "u")
he, ok := err.(*HTTPError)
if !ok {
t.Fatalf("want *HTTPError, got %T: %v", err, err)
}
if he.Code != http.StatusBadGateway {
t.Fatalf("code %d", he.Code)
}
if len(he.Body) > maxErrorBody+1 {
t.Fatalf("kept %d bytes of an error body, bound is %d", len(he.Body), maxErrorBody)
}
if time.Since(start) > 5*time.Second {
t.Fatalf("the client kept reading an endless error body: took %s", time.Since(start))
}
}
// sizedGateway answers 200 with a valid completion padded with trailing whitespace to
// exactly size bytes, so the boundary of the cap can be probed to the byte.
func sizedGateway(t *testing.T, size int) *httptest.Server {
t.Helper()
const base = `{"choices":[{"message":{"content":"ok"}}]}`
payload := base + strings.Repeat(" ", size-len(base))
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(payload))
}))
t.Cleanup(srv.Close)
return srv
}
func TestClient_ABodyOfExactlyTheCapIsReadAndOneByteMoreIsRefused(t *testing.T) {
const capBytes = 4096
for _, tt := range []struct {
name string
size int
wantErr bool
}{
{"one under", capBytes - 1, false},
{"exactly the cap", capBytes, false},
{"one over", capBytes + 1, true},
} {
t.Run(tt.name, func(t *testing.T) {
c := New(sizedGateway(t, tt.size).URL, "", "m", 5*time.Second, WithMaxResponseBytes(capBytes))
got, err := c.Complete(context.Background(), "s", "u")
if tt.wantErr {
if err == nil || !strings.Contains(err.Error(), "exceeds") {
t.Fatalf("want a response-too-large error, got %q, %v", got, err)
}
return
}
if err != nil || got != "ok" {
t.Fatalf("want ok, got %q, %v", got, err)
}
})
}
}
// New is what production calls: no option. The default is therefore the live bound.
func TestClient_TheDefaultCapIsEightMiBWhenNoOptionIsGiven(t *testing.T) {
for _, tt := range []struct {
name string
size int
wantErr bool
}{
{"exactly 8 MiB", 8 << 20, false},
{"8 MiB plus one", 8<<20 + 1, true},
} {
t.Run(tt.name, func(t *testing.T) {
c := New(sizedGateway(t, tt.size).URL, "", "m", 10*time.Second)
_, err := c.Complete(context.Background(), "s", "u")
if (err != nil) != tt.wantErr {
t.Fatalf("size %d: err = %v, wantErr %v", tt.size, err, tt.wantErr)
}
})
}
}
func TestClient_ANonPositiveCapOptionKeepsTheDefault(t *testing.T) {
for _, n := range []int64{0, -1} {
c := New(sizedGateway(t, 1024).URL, "", "m", 5*time.Second, WithMaxResponseBytes(n))
if got, err := c.Complete(context.Background(), "s", "u"); err != nil || got != "ok" {
t.Fatalf("WithMaxResponseBytes(%d): want ok from the default cap, got %q, %v", n, got, err)
}
}
}
+25 -3
View File
@@ -37,6 +37,21 @@ type Client struct {
temp float64
httpClient *http.Client
usageHook func(model string, prompt, completion int)
maxBody int64
}
// defaultMaxResponseBytes bounds a success body. A completion is a few KB of text; this
// leaves room for a long one and none for a gateway that streams forever (#53).
const defaultMaxResponseBytes = 8 << 20
// WithMaxResponseBytes overrides the bound on a success response body. A non-positive n is
// ignored (keeps the default).
func WithMaxResponseBytes(n int64) Option {
return func(c *Client) {
if n > 0 {
c.maxBody = n
}
}
}
// Option configures a Client at construction. Variadic so the existing 4-arg
@@ -101,6 +116,7 @@ func New(baseURL, apiKey, model string, timeout time.Duration, opts ...Option) *
maxTokens: defaultMaxTokens,
temp: defaultTemperature,
httpClient: &http.Client{Timeout: timeout},
maxBody: defaultMaxResponseBytes,
}
for _, opt := range opts {
opt(c)
@@ -184,12 +200,18 @@ func (c *Client) Complete(ctx context.Context, system, user string) (string, err
}
defer resp.Body.Close() //nolint:errcheck
out, err := io.ReadAll(resp.Body)
if resp.StatusCode != http.StatusOK {
// Only the bounded head is ever used (HTTPError.Error truncates to it), so read no more.
// A read error here only shortens a diagnostic; the status code below is what callers act on.
head, _ := io.ReadAll(io.LimitReader(resp.Body, maxErrorBody+1))
return "", &HTTPError{Code: resp.StatusCode, Body: string(head)}
}
out, err := io.ReadAll(io.LimitReader(resp.Body, c.maxBody+1))
if err != nil {
return "", fmt.Errorf("read response: %w", err)
}
if resp.StatusCode != http.StatusOK {
return "", &HTTPError{Code: resp.StatusCode, Body: string(out)}
if int64(len(out)) > c.maxBody {
return "", fmt.Errorf("LLM response exceeds %d bytes", c.maxBody)
}
var cr chatResponse
@@ -0,0 +1,2 @@
ALTER TABLE uploaded_transcripts
DROP COLUMN language;
@@ -0,0 +1,13 @@
-- Migration 021: the language an upload is transcribed as (tapir#61).
--
-- 'sv' Swedish: the engines are sent language=sv. Every row before this migration was
-- transcribed that way (the request hard-coded it), so the default is true history.
-- 'auto' Other: the engine detects the language and the Swedish-only engine is not used.
--
-- A closed set held by the table as well as by the form and the request code, so a second
-- writer or a bug cannot store a value the engines are never meant to receive. The column
-- sits on the already-FORCE'd, RLS-scoped uploaded_transcripts, so the exact-set isolation
-- test is unaffected.
ALTER TABLE uploaded_transcripts
ADD COLUMN language TEXT NOT NULL DEFAULT 'sv'
CONSTRAINT uploaded_transcripts_language_check CHECK (language IN ('sv', 'auto'));
+51 -12
View File
@@ -17,6 +17,10 @@ import (
// nothing about whether someone else's upload exists (ADR-033).
var ErrUploadNotFound = errors.New("store: uploaded transcript not found")
// ErrUploadInProgress is returned by DeleteUpload for an upload that is still queued or
// transcribing: the worker would write its result to a row that no longer exists.
var ErrUploadInProgress = errors.New("store: uploaded transcript is still being processed")
// Uploaded transcripts are PRIVATE per-user content, stored in their own
// RLS-scoped table (migration 017) — never in the shared `transcripts` (ADR-021).
// Every method goes through withUser, so row-level security confines it to the
@@ -27,12 +31,12 @@ var ErrUploadNotFound = errors.New("store: uploaded transcript not found")
// CreateUpload records a new upload in the queued state and returns its id.
// hadVideo records that the file carried a video track (only its audio is used);
// it is kept for the life of the row so the page can say so.
func (s *Store) CreateUpload(ctx context.Context, userID, title string, hadVideo bool) (string, error) {
func (s *Store) CreateUpload(ctx context.Context, userID, title string, lang domain.Language, hadVideo bool) (string, error) {
var id string
if err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
return tx.QueryRow(ctx,
`INSERT INTO uploaded_transcripts (user_id, title, had_video) VALUES ($1, $2, $3) RETURNING id`,
userID, title, hadVideo).Scan(&id)
`INSERT INTO uploaded_transcripts (user_id, title, language, had_video) VALUES ($1, $2, $3, $4) RETURNING id`,
userID, title, string(lang), hadVideo).Scan(&id)
}); err != nil {
return "", fmt.Errorf("store: create upload: %w", err)
}
@@ -80,23 +84,56 @@ func (s *Store) SaveProgress(ctx context.Context, userID, id string, steps []dom
WHERE id = $1 AND user_id = $2`, b, at)
}
// Heartbeat stamps the upload as alive now. It does not touch progress_at: alive says
// the service is still reporting, not that work is moving (ADR-035).
func (s *Store) Heartbeat(ctx context.Context, userID, id string) error {
// Heartbeat stamps the upload as alive at the given time. It does not touch progress_at:
// alive says the service is still reporting, not that work is moving (ADR-035). The time
// comes from the application clock, the one progress_at and the page's health check use,
// so a database clock that disagrees cannot make a healthy job look lost (#51).
func (s *Store) Heartbeat(ctx context.Context, userID, id string, at time.Time) error {
return s.updateUpload(ctx, userID, id, "heartbeat",
`UPDATE uploaded_transcripts SET alive_at = NOW()
WHERE id = $1 AND user_id = $2`)
`UPDATE uploaded_transcripts SET alive_at = $3
WHERE id = $1 AND user_id = $2`, at)
}
// FailUpload moves an upload to failed with a typed reason.
// FailUpload moves a queued or transcribing upload to failed with a typed reason. A
// finished upload is left as it was: a shutdown can record a job as failed after its
// transcript was already committed (the commit lands, the caller sees a cancelled
// context), and that must not hide the text. Such a call touches no row and returns
// ErrUploadNotFound.
func (s *Store) FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error {
return s.updateUpload(ctx, userID, id, "fail upload",
`UPDATE uploaded_transcripts
SET status = 'failed', error_kind = $3, finished_at = NOW()
WHERE id = $1 AND user_id = $2`,
WHERE id = $1 AND user_id = $2 AND status IN ('queued', 'transcribing')`,
string(kind))
}
// DeleteUpload removes one of the caller's finished (done or failed) uploads. An upload
// still queued or transcribing is refused with ErrUploadInProgress and left untouched; one
// that is not the caller's is ErrUploadNotFound, as for every other write (#37).
func (s *Store) DeleteUpload(ctx context.Context, userID, id string) error {
return s.withUser(ctx, userID, func(tx pgx.Tx) error {
ct, err := tx.Exec(ctx,
`DELETE FROM uploaded_transcripts
WHERE id = $1 AND user_id = $2 AND status IN ('done', 'failed')`, id, userID)
if err != nil {
return fmt.Errorf("store: delete upload: %w", err)
}
if ct.RowsAffected() > 0 {
return nil
}
var exists bool
if err := tx.QueryRow(ctx,
`SELECT EXISTS (SELECT 1 FROM uploaded_transcripts WHERE id = $1 AND user_id = $2)`,
id, userID).Scan(&exists); err != nil {
return fmt.Errorf("store: delete upload: %w", err)
}
if exists {
return ErrUploadInProgress
}
return ErrUploadNotFound
})
}
// updateUpload runs one UPDATE keyed on ($1 = id, $2 = userID) plus extra args,
// and maps "no row touched" to ErrUploadNotFound — which is what a write to another
// user's upload looks like under RLS (the row is invisible, not forbidden).
@@ -184,7 +221,7 @@ func (s *Store) FailInterruptedUploads(ctx context.Context) (int, error) {
return total, nil
}
const uploadSelect = `SELECT id, user_id, title, status, content, segments, endpoint,
const uploadSelect = `SELECT id, user_id, title, language, status, content, segments, endpoint,
audio_seconds, error_kind, had_video, reconciliation, progress, progress_at, alive_at,
created_at, finished_at FROM uploaded_transcripts`
@@ -197,11 +234,13 @@ func scanUpload(row pgx.Row, u *domain.UploadedTranscript, withText bool) error
seconds *float64
finished, progressAt, aliveAt *time.Time
)
if err := row.Scan(&u.ID, &u.UserID, &u.Title, &status, &content, &segments, &endpoint,
var language string
if err := row.Scan(&u.ID, &u.UserID, &u.Title, &language, &status, &content, &segments, &endpoint,
&seconds, &errKind, &u.HadVideo, &reconciliation, &progress, &progressAt, &aliveAt,
&u.CreatedAt, &finished); err != nil {
return err
}
u.Language = domain.Language(language)
u.Status = domain.UploadStatus(deref(status))
u.Endpoint = deref(endpoint)
u.ErrorKind = domain.UploadErrorKind(deref(errKind))
@@ -0,0 +1,68 @@
package store_test
// Deleting one recording (#37). A finished or failed upload can be removed by its owner;
// one that is queued or transcribing is refused until it ends, because the worker would
// otherwise write its result to a row that is gone (D3 in the sprint plan).
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/store"
"git.d-ma.be/mathias/tapir/internal/domain"
)
func TestDeleteUpload_RemovesAFinishedOrFailedUploadOfTheOwner(t *testing.T) {
for name, finish := range map[string]func(*store.Store, string) error{
"done": func(s *store.Store, id string) error {
return s.CompleteUpload(context.Background(), userA, id, domain.Transcription{Text: "synthetic", Endpoint: "iguana/kb-whisper"})
},
"failed": func(s *store.Store, id string) error {
return s.FailUpload(context.Background(), userA, id, domain.UploadErrTranscriptionFailed)
},
} {
t.Run(name, func(t *testing.T) {
s := newStore(t)
id := startedUpload(t, s)
require.NoError(t, s.StartTranscribing(context.Background(), userA, id))
require.NoError(t, finish(s, id))
require.NoError(t, s.DeleteUpload(context.Background(), userA, id))
_, err := s.GetUpload(context.Background(), userA, id)
assert.ErrorIs(t, err, store.ErrUploadNotFound)
})
}
}
func TestDeleteUpload_RefusesAnUploadThatIsStillQueuedOrTranscribingAndKeepsTheRow(t *testing.T) {
s := newStore(t)
ctx := context.Background()
queued := startedUpload(t, s)
running, err := s.CreateUpload(ctx, userA, "running", domain.LangSwedish, false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, running))
for _, id := range []string{queued, running} {
assert.ErrorIs(t, s.DeleteUpload(ctx, userA, id), store.ErrUploadInProgress)
_, err := s.GetUpload(ctx, userA, id)
assert.NoError(t, err, "a refused delete must leave the row")
}
}
func TestDeleteUpload_AnotherUsersUploadIsNotFoundAndSurvives(t *testing.T) {
s := newStore(t)
ctx := context.Background()
id := startedUpload(t, s)
require.NoError(t, s.StartTranscribing(ctx, userA, id))
require.NoError(t, s.FailUpload(ctx, userA, id, domain.UploadErrTranscriptionFailed))
assert.ErrorIs(t, s.DeleteUpload(ctx, userB, id), store.ErrUploadNotFound,
"existence is not disclosed to a non-owner")
_, err := s.GetUpload(ctx, userA, id)
assert.NoError(t, err, "B's delete must not have removed A's row")
}
@@ -0,0 +1,47 @@
package store_test
// A shutdown can record a job as failed after its transcript was already committed (the
// commit lands, the caller sees a cancelled context). FailUpload must not be able to turn a
// finished row into a failed one and hide the text (#50 review).
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
)
func TestFailUpload_LeavesAFinishedUploadAsItWas(t *testing.T) {
s := newStore(t)
ctx := context.Background()
id := startedUpload(t, s)
require.NoError(t, s.StartTranscribing(ctx, userA, id))
require.NoError(t, s.CompleteUpload(ctx, userA, id, domain.Transcription{Text: "synthetic text", Endpoint: "iguana/kb-whisper"}))
_ = s.FailUpload(ctx, userA, id, domain.UploadErrInterrupted)
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, domain.UploadDone, got.Status)
assert.Equal(t, "synthetic text", got.Content)
assert.Empty(t, got.ErrorKind)
}
func TestFailUpload_StillFailsAQueuedOrTranscribingUpload(t *testing.T) {
s := newStore(t)
ctx := context.Background()
queued := startedUpload(t, s)
running, err := s.CreateUpload(ctx, userA, "r", domain.LangSwedish, false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, running))
for _, id := range []string{queued, running} {
require.NoError(t, s.FailUpload(ctx, userA, id, domain.UploadErrBusy))
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, domain.UploadFailed, got.Status)
}
}
@@ -20,7 +20,7 @@ import (
func startedUpload(t *testing.T, s *store.Store) string {
t.Helper()
seedUsers(t, userA, userB)
id, err := s.CreateUpload(context.Background(), userA, "m", false)
id, err := s.CreateUpload(context.Background(), userA, "m", domain.LangSwedish, false)
require.NoError(t, err)
return id
}
@@ -68,13 +68,13 @@ func TestUploadProgress_AHeartbeatMovesAliveAtAndNotProgressAt(t *testing.T) {
ctx := context.Background()
at := time.Date(2026, 10, 2, 8, 30, 0, 0, time.UTC)
require.NoError(t, s.SaveProgress(ctx, userA, id, []domain.ProgressStep{{Key: "audio", Label: "a", State: domain.StepRunning}}, at))
before := time.Now().Add(-time.Second)
beatAt := at.Add(90 * time.Second) // deliberately not the database's NOW()
require.NoError(t, s.Heartbeat(ctx, userA, id))
require.NoError(t, s.Heartbeat(ctx, userA, id, beatAt))
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.True(t, got.AliveAt.After(before), "the heartbeat must stamp the row with now")
assert.True(t, got.AliveAt.Equal(beatAt), "the heartbeat is stamped with the application's clock, the same one progress_at uses, not the database's NOW() (#51)")
assert.True(t, got.ProgressAt.Equal(at), "alive is not progress: a heartbeat must not move progress_at")
}
@@ -99,7 +99,7 @@ func TestUploadProgress_ByAnotherUserChangesNothing(t *testing.T) {
ctx := context.Background()
errSave := s.SaveProgress(ctx, userB, id, []domain.ProgressStep{{Key: "x", Label: "X", State: domain.StepDone}}, time.Now())
errBeat := s.Heartbeat(ctx, userB, id)
errBeat := s.Heartbeat(ctx, userB, id, time.Now())
assert.ErrorIs(t, errSave, store.ErrUploadNotFound)
assert.ErrorIs(t, errBeat, store.ErrUploadNotFound)
@@ -33,7 +33,7 @@ func TestCreateUpload_StartsQueuedAndIsOwnedByTheCreator(t *testing.T) {
seedUsers(t, userA)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "Teams export", false)
id, err := s.CreateUpload(ctx, userA, "Teams export", domain.LangSwedish, false)
require.NoError(t, err)
got, err := s.GetUpload(ctx, userA, id)
@@ -50,7 +50,7 @@ func TestUploadLifecycle_CompletedKeepsTextSegmentsEndpointAndDuration(t *testin
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", false)
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id))
@@ -84,7 +84,7 @@ func TestUploadLifecycle_FailedRecordsTypedKindAndNoContent(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", false)
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id))
@@ -102,7 +102,7 @@ func TestGetUpload_AnotherUsersIDIsNotFound(t *testing.T) {
s := newStore(t)
seedUsers(t, userA, userB)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "A's meeting", false)
id, err := s.CreateUpload(ctx, userA, "A's meeting", domain.LangSwedish, false)
require.NoError(t, err)
_, err = s.GetUpload(ctx, userB, id)
@@ -115,7 +115,7 @@ func TestUploadWrites_ByAnotherUserChangeNothing(t *testing.T) {
s := newStore(t)
seedUsers(t, userA, userB)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "A's meeting", false)
id, err := s.CreateUpload(ctx, userA, "A's meeting", domain.LangSwedish, false)
require.NoError(t, err)
assert.ErrorIs(t, s.StartTranscribing(ctx, userB, id), store.ErrUploadNotFound)
@@ -132,11 +132,11 @@ func TestListUploads_ReturnsOnlyTheCallersNewestFirstWithoutTranscriptText(t *te
s := newStore(t)
seedUsers(t, userA, userB)
ctx := context.Background()
first, err := s.CreateUpload(ctx, userA, "first", false)
first, err := s.CreateUpload(ctx, userA, "first", domain.LangSwedish, false)
require.NoError(t, err)
second, err := s.CreateUpload(ctx, userA, "second", false)
second, err := s.CreateUpload(ctx, userA, "second", domain.LangSwedish, false)
require.NoError(t, err)
_, err = s.CreateUpload(ctx, userB, "B's", false)
_, err = s.CreateUpload(ctx, userB, "B's", domain.LangSwedish, false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, first))
require.NoError(t, s.CompleteUpload(ctx, userA, first, domain.Transcription{Text: "long text body", Endpoint: "e"}))
@@ -160,12 +160,12 @@ func TestFailInterruptedUploads_FailsOnlyUnfinishedRowsForEveryUser(t *testing.T
_, err := super.Exec(ctx, `INSERT INTO user_identities (dex_subject, user_id) VALUES ($1, $2)`, subject, uid)
require.NoError(t, err)
}
queued, err := s.CreateUpload(ctx, userA, "queued", false)
queued, err := s.CreateUpload(ctx, userA, "queued", domain.LangSwedish, false)
require.NoError(t, err)
running, err := s.CreateUpload(ctx, userB, "running", false)
running, err := s.CreateUpload(ctx, userB, "running", domain.LangSwedish, false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userB, running))
done, err := s.CreateUpload(ctx, userA, "done", false)
done, err := s.CreateUpload(ctx, userA, "done", domain.LangSwedish, false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, done))
require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "kept", Endpoint: "e"}))
@@ -191,9 +191,9 @@ func TestCreateUpload_RecordsWhetherTheFileHadAVideoTrack(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
withVideo, err := s.CreateUpload(ctx, userA, "camera off", true)
withVideo, err := s.CreateUpload(ctx, userA, "camera off", domain.LangSwedish, true)
require.NoError(t, err)
audioOnly, err := s.CreateUpload(ctx, userA, "voice only", false)
audioOnly, err := s.CreateUpload(ctx, userA, "voice only", domain.LangSwedish, false)
require.NoError(t, err)
gotVideo, err := s.GetUpload(ctx, userA, withVideo)
@@ -216,9 +216,9 @@ func TestUploadHadVideo_SurvivesCompletionAndFailure(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
done, err := s.CreateUpload(ctx, userA, "d", true)
done, err := s.CreateUpload(ctx, userA, "d", domain.LangSwedish, true)
require.NoError(t, err)
failed, err := s.CreateUpload(ctx, userA, "f", true)
failed, err := s.CreateUpload(ctx, userA, "f", domain.LangSwedish, true)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, done))
require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "x", Endpoint: "e"}))
@@ -235,7 +235,7 @@ func TestUploadLifecycle_ReconciliationRoundTripsWithItsContestedSpots(t *testin
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", false)
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id))
want := &domain.Reconciliation{
@@ -258,7 +258,7 @@ func TestUploadLifecycle_ASingleEngineTranscriptHasNoReconciliation(t *testing.T
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", false)
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id))
require.NoError(t, s.CompleteUpload(ctx, userA, id, domain.Transcription{Text: "x", Endpoint: "e"}))
@@ -273,7 +273,7 @@ func TestUploadLifecycle_ASkippedReconciliationKeepsItsReason(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", false)
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id))
rec := &domain.Reconciliation{Status: domain.ReconcileSkipped, Reason: "the model failed 3 times in a row: gateway down", Errors: 3}
@@ -284,3 +284,55 @@ func TestUploadLifecycle_ASkippedReconciliationKeepsItsReason(t *testing.T) {
require.NoError(t, err)
assert.Equal(t, rec, got.Reconciliation)
}
func TestCreateUpload_RecordsTheLanguageTheUserChoseAndTheListCarriesIt(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
swedish, err := s.CreateUpload(ctx, userA, "sv", domain.LangSwedish, false)
require.NoError(t, err)
auto, err := s.CreateUpload(ctx, userA, "auto", domain.LangAuto, false)
require.NoError(t, err)
gotSv, err := s.GetUpload(ctx, userA, swedish)
require.NoError(t, err)
gotAuto, err := s.GetUpload(ctx, userA, auto)
require.NoError(t, err)
listed, err := s.ListUploads(ctx, userA)
require.NoError(t, err)
assert.Equal(t, domain.LangSwedish, gotSv.Language)
assert.Equal(t, domain.LangAuto, gotAuto.Language)
byID := map[string]domain.Language{}
for _, u := range listed {
byID[u.ID] = u.Language
}
assert.Equal(t, map[string]domain.Language{swedish: domain.LangSwedish, auto: domain.LangAuto}, byID)
}
// The closed set is also held by the table, so a bug or a second writer cannot store a
// value the engines are never meant to receive (tapir#61).
func TestUploadedTranscripts_TheTableRefusesALanguageOutsideTheClosedSet(t *testing.T) {
newStore(t)
seedUsers(t, userA)
_, err := rawPool(t).Exec(context.Background(),
`INSERT INTO uploaded_transcripts (user_id, title, language) VALUES ($1, 'x', 'en')`, userA)
require.Error(t, err)
assert.Contains(t, err.Error(), "language", "the CHECK constraint names the column")
}
// A row from before the column existed was transcribed as Swedish; the default says so.
func TestUploadedTranscripts_ARowInsertedWithoutALanguageIsSwedish(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
var id string
require.NoError(t, rawPool(t).QueryRow(context.Background(),
`INSERT INTO uploaded_transcripts (user_id, title) VALUES ($1, 'old') RETURNING id`, userA).Scan(&id))
got, err := s.GetUpload(context.Background(), userA, id)
require.NoError(t, err)
assert.Equal(t, domain.LangSwedish, got.Language)
}
+43 -8
View File
@@ -32,14 +32,24 @@ func NewEnsemble(base, alt, fallback Backend, adj reconcile.Adjudicator, opts re
return &Ensemble{base: base, alt: alt, fallback: fallback, adj: adj, opts: opts}
}
// Plan implements uploads.Planner: the phases this ensemble reports, so the page can
// show them before they run. The fallback is a phase only when one is configured.
func (e *Ensemble) Plan() []progress.Planned {
plan := []progress.Planned{
{Key: "engine_a", Label: "Transcribing with " + e.base.Name()},
{Key: "engine_b", Label: "Transcribing with " + e.alt.Name()},
{Key: "compare", Label: "Comparing the two transcripts"},
{Key: "decide", Label: "Deciding the differences"},
// Plan implements uploads.Planner: the phases of a Swedish job, the two-engine one. The
// fallback is a phase only when one is configured.
func (e *Ensemble) Plan() []progress.Planned { return e.PlanFor(domain.LangSwedish) }
// PlanFor implements uploads.LanguagePlanner. A recording that is not Swedish never runs the
// base engine, the comparison or the decisions (transcribeOther), so the page lists only
// what can run: the other engine, then the fallback if one is configured.
func (e *Ensemble) PlanFor(lang domain.Language) []progress.Planned {
var plan []progress.Planned
if lang == domain.LangSwedish || lang == "" {
plan = []progress.Planned{
{Key: "engine_a", Label: "Transcribing with " + e.base.Name()},
{Key: "engine_b", Label: "Transcribing with " + e.alt.Name()},
{Key: "compare", Label: "Comparing the two transcripts"},
{Key: "decide", Label: "Deciding the differences"},
}
} else {
plan = []progress.Planned{{Key: "engine_b", Label: "Transcribing with " + e.alt.Name()}}
}
if e.fallback != nil {
plan = append(plan, progress.Planned{Key: "fallback", Label: "Cloud fallback (" + e.fallback.Name() + ")"})
@@ -51,6 +61,9 @@ func (e *Ensemble) Plan() []progress.Planned {
// progress.Reporter in ctx (a no-op when nobody listens).
func (e *Ensemble) Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error) {
rep := progress.From(ctx)
if domain.LanguageFrom(ctx) != domain.LangSwedish {
return e.transcribeOther(ctx, rep, wavPath)
}
b, berr := e.run(ctx, rep, "engine_a", e.base, wavPath)
if err := ctx.Err(); err != nil {
return domain.Transcription{}, err
@@ -89,6 +102,28 @@ func (e *Ensemble) Transcribe(ctx context.Context, wavPath string) (domain.Trans
return e.single(f, e.fallback.Name(), "both local engines failed"), nil
}
// transcribeOther handles a recording that is not Swedish (tapir#61). The base engine is
// Swedish-only, so it never receives the audio, and with one reading there is nothing to
// reconcile: the other engine transcribes alone, then the cloud fallback if it fails and the
// job is still live. The phases that did not run are shown as skipped by the tracker.
func (e *Ensemble) transcribeOther(ctx context.Context, rep progress.Reporter, wavPath string) (domain.Transcription, error) {
t, err := e.run(ctx, rep, "engine_b", e.alt, wavPath)
if err == nil {
t.Endpoint = e.alt.Name()
return t, nil
}
if e.fallback == nil || ctx.Err() != nil {
return domain.Transcription{}, fmt.Errorf("%s: %w", e.alt.Name(), err)
}
f, ferr := e.run(ctx, rep, "fallback", e.fallback, wavPath)
if ferr != nil {
return domain.Transcription{}, errors.Join(
fmt.Errorf("%s: %w", e.alt.Name(), err), fmt.Errorf("%s: %w", e.fallback.Name(), ferr))
}
f.Endpoint = e.fallback.Name()
return f, nil
}
// run is one engine call, reported as a phase. A job that was stopped is not a broken
// engine, so a cancelled context is never reported as a failed phase.
func (e *Ensemble) run(ctx context.Context, rep progress.Reporter, key string, be Backend, wav string) (domain.Transcription, error) {
@@ -160,3 +160,25 @@ func TestEnsembleProgress_TheReconcileCountsAreNotedForTheOperator(t *testing.T)
assert.Equal(t, 0, n["unsure"])
assert.Equal(t, 0, n["errors"])
}
// An auto-language upload never runs the base engine, the comparison or the decisions, so the
// page must not list them as coming (tapir#61, PR #62 review).
func TestEnsemblePlanFor_AnAutoLanguageListsOnlyThePhasesThatCanRun(t *testing.T) {
with := newEnsemble("a", "a", nil, nil, true, &verdictAdj{}).e
without := newEnsemble("a", "a", nil, nil, false, &verdictAdj{}).e
keys := func(p []progress.Planned) []string {
var k []string
for _, s := range p {
k = append(k, s.Key)
}
return k
}
assert.Equal(t, []string{"engine_b", "fallback"}, keys(with.PlanFor(domain.LangAuto)))
assert.Equal(t, []string{"engine_b"}, keys(without.PlanFor(domain.LangAuto)), "no fallback configured: none promised")
assert.Equal(t, with.Plan(), with.PlanFor(domain.LangSwedish), "Swedish is the plan it always was")
assert.Contains(t, with.PlanFor(domain.LangAuto)[0].Label, "iguana/kb-whisper")
assert.Equal(t, with.Plan(), with.PlanFor(""), "a job queued before the choice existed is Swedish")
assert.Equal(t, with.PlanFor(domain.LangAuto), with.PlanFor(domain.Language("xx")),
"an unknown language is planned as it is run: not the Swedish phases")
}
@@ -204,3 +204,52 @@ func TestEnsemble_AShutdownDuringTheSecondEngineIsAnErrorNotASingleEngineSuccess
assert.ErrorIs(t, err, context.Canceled, "a cancelled job must not be recorded as 'kb-whisper failed'")
assert.Empty(t, got.Text)
}
// The base engine is Swedish-only. A recording that is not Swedish must never reach it, and
// there is no second opinion to reconcile: the other engine transcribes alone (tapir#61).
func autoCtx() context.Context { return domain.WithLanguage(context.Background(), domain.LangAuto) }
func TestEnsemble_AnAutoLanguageUploadNeverReachesTheBaseEngineAndIsNotReconciled(t *testing.T) {
adj := &verdictAdj{v: reconcile.VerdictAlt}
r := newEnsemble("base text", "alt text", nil, nil, true, adj)
got, err := r.e.Transcribe(autoCtx(), "/a.wav")
require.NoError(t, err)
assert.Equal(t, []string{"iguana/kb-whisper"}, *r.calls, "only the other engine is called")
assert.Zero(t, adj.calls, "nothing is adjudicated")
assert.Equal(t, "alt text", got.Text)
assert.Equal(t, "iguana/kb-whisper", got.Endpoint)
assert.Nil(t, got.Reconciliation, "reconciliation does not apply, so none is recorded")
}
func TestEnsemble_AnAutoLanguageUploadFallsBackWhenItsEngineFailsAndStillSkipsTheBaseEngine(t *testing.T) {
r := newEnsemble("base text", "alt text", nil, errors.New("engine down"), true, &verdictAdj{})
got, err := r.e.Transcribe(autoCtx(), "/a.wav")
require.NoError(t, err)
assert.Equal(t, []string{"iguana/kb-whisper", "berget/kb-whisper"}, *r.calls)
assert.Equal(t, "berget/kb-whisper", got.Endpoint)
}
func TestEnsemble_AnAutoLanguageUploadWithNoFallbackFailsWithoutTouchingTheBaseEngine(t *testing.T) {
r := newEnsemble("base text", "alt text", nil, errors.New("engine down"), false, &verdictAdj{})
_, err := r.e.Transcribe(autoCtx(), "/a.wav")
require.Error(t, err)
assert.Equal(t, []string{"iguana/kb-whisper"}, *r.calls)
}
func TestEnsemble_ACancelledAutoLanguageJobDoesNotReachTheFallback(t *testing.T) {
r := newEnsemble("base text", "alt text", nil, errors.New("engine down"), true, &verdictAdj{})
ctx, cancel := context.WithCancel(autoCtx())
cancel()
_, err := r.e.Transcribe(ctx, "/a.wav")
require.Error(t, err)
assert.NotContains(t, *r.calls, "berget/kb-whisper", "a stopped job must not ship its audio to a cloud endpoint")
}
+12 -1
View File
@@ -71,9 +71,20 @@ func (g *Gateway) Transcribe(ctx context.Context, path string) (domain.Transcrip
return domain.Transcription{}, fmt.Errorf("transcribe: stat audio: %w", err)
}
// The language is one request field and a closed set: sv is sent, auto sends nothing so
// the engine detects it, and anything else never leaves this process (tapir#61).
fields := [][2]string{{"model", g.model}, {"response_format", "verbose_json"}}
switch lang := domain.LanguageFrom(ctx); lang {
case domain.LangSwedish:
fields = append(fields, [2]string{"language", string(lang)})
case domain.LangAuto:
default:
return domain.Transcription{}, fmt.Errorf("transcribe: language %q is not one tapir sends", lang)
}
var prefix bytes.Buffer
mw := multipart.NewWriter(&prefix)
for _, kv := range [][2]string{{"model", g.model}, {"response_format", "verbose_json"}, {"language", "sv"}} {
for _, kv := range fields {
if err := mw.WriteField(kv[0], kv[1]); err != nil {
return domain.Transcription{}, fmt.Errorf("transcribe: build request: %w", err)
}
@@ -255,3 +255,48 @@ func TestGateway_AnErrorBodyIsBoundedToTwoHundredBytesInTheStatusError(t *testin
require.ErrorAs(t, err, &se)
assert.LessOrEqual(t, len(se.Snippet), 200, "a gateway's whole body must not become part of an error")
}
// The language travels on the context (tapir#61); the request field is the only place it
// takes effect, so these observe the request the gateway fake received.
func TestGateway_SendsLanguageSvWhenTheContextCarriesNoneAndForSwedish(t *testing.T) {
for name, ctx := range map[string]context.Context{
"no language in the context": context.Background(),
"swedish": domain.WithLanguage(context.Background(), domain.LangSwedish),
} {
t.Run(name, func(t *testing.T) {
var got seen
srv := serve(t, 200, verboseJSON, &got)
g := transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client())
_, err := g.Transcribe(ctx, wavFile(t, "RIFF-synthetic-pcm"))
require.NoError(t, err)
assert.Equal(t, "sv", got.fields["language"])
})
}
}
func TestGateway_OmitsTheLanguageFieldForAnAutoContextSoTheEngineDetectsIt(t *testing.T) {
var got seen
srv := serve(t, 200, verboseJSON, &got)
g := transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client())
_, err := g.Transcribe(domain.WithLanguage(context.Background(), domain.LangAuto), wavFile(t, "RIFF-synthetic-pcm"))
require.NoError(t, err)
_, present := got.fields["language"]
assert.False(t, present, "an auto upload must not be forced to Swedish")
assert.Equal(t, "iguana/kb-whisper", got.fields["model"], "the rest of the request is unchanged")
}
func TestGateway_ALanguageOutsideTheClosedSetIsRefusedBeforeAnythingIsSent(t *testing.T) {
var got seen
srv := serve(t, 200, verboseJSON, &got)
g := transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client())
_, err := g.Transcribe(domain.WithLanguage(context.Background(), domain.Language("en")), wavFile(t, "RIFF-synthetic-pcm"))
require.Error(t, err)
assert.Empty(t, got.method, "no request may reach the engine with a language the closed set does not hold")
}
@@ -102,3 +102,45 @@ func TestEnsemble_AHungEngineIsASoftFailureAndTheOtherEnginesTextIsUsed(t *testi
assert.Contains(t, got.Reconciliation.Reason, "timeout")
assert.False(t, errors.Is(err, context.DeadlineExceeded))
}
// WithTimeout sits in front of both local engines in production (cmd/tapir/uploads.go). It
// must hand the engine the job's context, not a fresh one: the language and the
// cancellation both travel on it (tapir#61).
func TestWithTimeout_TheEngineStillReceivesTheLanguageOfTheJob(t *testing.T) {
for name, tt := range map[string]struct {
ctx context.Context
wantField bool
}{
"auto leaves the field out": {domain.WithLanguage(context.Background(), domain.LangAuto), false},
"swedish sends sv": {domain.WithLanguage(context.Background(), domain.LangSwedish), true},
} {
t.Run(name, func(t *testing.T) {
var got seen
srv := serve(t, 200, verboseJSON, &got)
b := transcribe.WithTimeout(transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client()), time.Minute)
_, err := b.Transcribe(tt.ctx, wavFile(t, "RIFF-synthetic-pcm"))
require.NoError(t, err)
lang, present := got.fields["language"]
assert.Equal(t, tt.wantField, present)
if tt.wantField {
assert.Equal(t, "sv", lang)
}
})
}
}
func TestWithTimeout_ACancelledJobIsNotRestartedByTheWrapper(t *testing.T) {
var got seen
srv := serve(t, 200, verboseJSON, &got)
b := transcribe.WithTimeout(transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client()), time.Minute)
ctx, cancel := context.WithCancel(context.Background())
cancel()
_, err := b.Transcribe(ctx, wavFile(t, "RIFF-synthetic-pcm"))
require.Error(t, err)
assert.Empty(t, got.method, "a cancelled job must not send its audio anywhere through the wrapper")
}
+6 -3
View File
@@ -243,9 +243,12 @@ const (
// never stored in, or merged with, the shared public-caption Transcript (ADR-021).
// Content and Segments are empty until Status is UploadDone.
type UploadedTranscript struct {
ID string
UserID string
Title string
ID string
UserID string
Title string
// Language is what the upload was (or will be) transcribed as: sv, or auto for
// "Other" (tapir#61).
Language Language
Status UploadStatus
Content string
Segments []Segment
+47
View File
@@ -0,0 +1,47 @@
package domain
import "context"
// Language is the spoken language an upload is transcribed as. It is chosen by the user on
// the upload form and reaches the engines as one request field, so it is a closed set,
// never free text (tapir#61).
type Language string
const (
// LangSwedish sends language=sv: the Swedish engines' own language, and what every
// upload was transcribed as before the choice existed.
LangSwedish Language = "sv"
// LangAuto sends no language, so the engine detects it. It is what "Other" on the form
// means: the Swedish-only engines are not used for it.
LangAuto Language = "auto"
)
// ParseLanguage reads a form value. An empty value is Swedish (a client that predates the
// field); anything that is not exactly a known value is refused rather than normalised.
func ParseLanguage(s string) (Language, bool) {
switch Language(s) {
case "", LangSwedish:
return LangSwedish, true
case LangAuto:
return LangAuto, true
}
return "", false
}
type languageKey struct{}
// WithLanguage returns a context that carries the language of the job it serves. The
// Transcriber interface does not change: engines read it with LanguageFrom, the same way
// they read the progress Reporter (ADR-035).
func WithLanguage(ctx context.Context, l Language) context.Context {
return context.WithValue(ctx, languageKey{}, l)
}
// LanguageFrom returns the language carried by ctx, or Swedish when it carries none, so a
// caller that sets nothing behaves as it always did.
func LanguageFrom(ctx context.Context) Language {
if l, ok := ctx.Value(languageKey{}).(Language); ok && l != "" {
return l
}
return LangSwedish
}
+41
View File
@@ -0,0 +1,41 @@
package domain_test
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"git.d-ma.be/mathias/tapir/internal/domain"
)
func TestParseLanguage_KnownValuesAndEmptyAreAcceptedEverythingElseIsNot(t *testing.T) {
tests := []struct {
in string
want domain.Language
wantOK bool
}{
{"sv", domain.LangSwedish, true},
{"auto", domain.LangAuto, true},
{"", domain.LangSwedish, true}, // a client that sends no field keeps today's behaviour
{"SV", "", false}, // exact match: the value reaches a request field
{" sv", "", false},
{"sv\n", "", false},
{"en", "", false}, // a specific other language is not offered yet
{"sv,en", "", false},
{"../sv", "", false},
}
for _, tt := range tests {
t.Run(tt.in, func(t *testing.T) {
got, ok := domain.ParseLanguage(tt.in)
assert.Equal(t, tt.wantOK, ok)
assert.Equal(t, tt.want, got)
})
}
}
func TestLanguageFrom_IsSwedishWhenTheContextCarriesNoneAndOtherwiseWhatWasSet(t *testing.T) {
assert.Equal(t, domain.LangSwedish, domain.LanguageFrom(context.Background()))
ctx := domain.WithLanguage(context.Background(), domain.LangAuto)
assert.Equal(t, domain.LangAuto, domain.LanguageFrom(ctx))
}
+5
View File
@@ -125,6 +125,11 @@ type statusWriter struct {
wroteHeader bool
}
// Unwrap exposes the underlying writer to http.ResponseController, so a handler behind this
// middleware can still set a connection read deadline (#56: without it the upload idle bound
// was silently unavailable in production).
func (s *statusWriter) Unwrap() http.ResponseWriter { return s.ResponseWriter }
func (s *statusWriter) WriteHeader(code int) {
if !s.wroteHeader {
s.code = code
+22 -6
View File
@@ -37,12 +37,13 @@ type Reporter interface {
// is for people, so a failed write is logged and never stops the job.
type Sink interface {
SaveProgress(ctx context.Context, steps []domain.ProgressStep, at time.Time) error
Beat(ctx context.Context) error
Beat(ctx context.Context, at time.Time) error
}
const (
defaultMinSave = 2 * time.Second
logAdvanceEach = time.Minute
defaultMinSave = 2 * time.Second
defaultSaveTimeout = 5 * time.Second
logAdvanceEach = time.Minute
)
// Option tunes a Tracker.
@@ -55,6 +56,14 @@ func WithClock(now func() time.Time) Option { return func(t *Tracker) { t.now =
// Done, Fail and Finish always save.
func WithMinSave(d time.Duration) Option { return func(t *Tracker) { t.minSave = d } }
// WithSaveTimeout bounds one save or heartbeat write. Progress is cosmetic, so a database
// that stops answering costs a dropped write rather than an unbounded stall. It is still a
// delay: saves run under the tracker's lock, which Begin and Advance (called by the engines)
// need, so each transition can wait up to this long, and a hung database slows a job by
// that much per call, out of the job's own time budget (#51). A dropped final save is not
// retried; the row keeps its last saved step.
func WithSaveTimeout(d time.Duration) Option { return func(t *Tracker) { t.saveTimeout = d } }
// Tracker implements Reporter for one job.
type Tracker struct {
ctx context.Context
@@ -63,6 +72,8 @@ type Tracker struct {
now func() time.Time
minSave time.Duration
saveTimeout time.Duration
mu sync.Mutex
steps []domain.ProgressStep
lastAdvanceSave time.Time
@@ -72,7 +83,7 @@ type Tracker struct {
// NewTracker plans a job's phases. ctx is used for the saves, so it should outlive the
// job's own timeout: a job that timed out still gets its failure recorded.
func NewTracker(ctx context.Context, sink Sink, log *slog.Logger, plan []Planned, opts ...Option) *Tracker {
t := &Tracker{ctx: ctx, sink: sink, log: log, now: time.Now, minSave: defaultMinSave}
t := &Tracker{ctx: ctx, sink: sink, log: log, now: time.Now, minSave: defaultMinSave, saveTimeout: defaultSaveTimeout}
for _, o := range opts {
o(t)
}
@@ -212,7 +223,10 @@ func (t *Tracker) Heartbeat(ctx context.Context, beatEvery, stillEvery time.Dura
case <-ctx.Done():
return
case <-beat.C:
if err := t.sink.Beat(ctx); err != nil && ctx.Err() == nil {
beatCtx, cancel := context.WithTimeout(ctx, t.saveTimeout)
err := t.sink.Beat(beatCtx, t.now())
cancel()
if err != nil && ctx.Err() == nil {
t.log.Warn("progress: heartbeat not saved", "err", err)
}
case <-still.C:
@@ -254,7 +268,9 @@ func (t *Tracker) find(key string) *domain.ProgressStep {
// dropped: progress is cosmetic and must never fail the job it describes.
func (t *Tracker) save() {
snapshot := append([]domain.ProgressStep(nil), t.steps...)
if err := t.sink.SaveProgress(t.ctx, snapshot, t.now()); err != nil && t.ctx.Err() == nil {
ctx, cancel := context.WithTimeout(t.ctx, t.saveTimeout)
defer cancel()
if err := t.sink.SaveProgress(ctx, snapshot, t.now()); err != nil && t.ctx.Err() == nil {
t.log.Warn("progress: not saved", "err", err)
}
}
+154
View File
@@ -0,0 +1,154 @@
package progress_test
// Observes what a database that stops answering does to the job it reports on (#51):
// progress is cosmetic, so a save that never returns must not stall the engine calls that
// report through the tracker, nor the heartbeat that says the process is alive.
import (
"bytes"
"context"
"log/slog"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/progress"
)
// stallSink never answers until its context ends, like a database connection that hung.
type stallSink struct{ beats atomic.Int32 }
func (s *stallSink) SaveProgress(ctx context.Context, _ []domain.ProgressStep, _ time.Time) error {
<-ctx.Done()
return ctx.Err()
}
func (s *stallSink) Beat(ctx context.Context, _ time.Time) error {
s.beats.Add(1)
<-ctx.Done()
return ctx.Err()
}
type lockedBuf struct {
mu sync.Mutex
b bytes.Buffer
}
func (l *lockedBuf) Write(p []byte) (int, error) {
l.mu.Lock()
defer l.mu.Unlock()
return l.b.Write(p)
}
func (l *lockedBuf) String() string {
l.mu.Lock()
defer l.mu.Unlock()
return l.b.String()
}
func stallTracker(sink progress.Sink, buf *lockedBuf) *progress.Tracker {
log := slog.New(slog.NewTextHandler(buf, nil))
return progress.NewTracker(context.Background(), sink, log,
[]progress.Planned{{Key: "a", Label: "A"}}, progress.WithSaveTimeout(50*time.Millisecond))
}
func TestTracker_AHungSaveDoesNotBlockTheReporterPastItsTimeout(t *testing.T) {
buf := &lockedBuf{}
tr := stallTracker(&stallSink{}, buf)
done := make(chan struct{})
go func() { tr.Begin("a", 0); tr.Advance("a", 1); tr.Done("a"); close(done) }()
select {
case <-done:
case <-time.After(3 * time.Second):
t.Fatal("the reporter was blocked by a save that never returned")
}
require.Contains(t, buf.String(), "progress: not saved", "a dropped save is logged")
}
func TestTracker_AHungBeatDoesNotStopTheHeartbeat(t *testing.T) {
sink := &stallSink{}
tr := stallTracker(sink, &lockedBuf{})
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go tr.Heartbeat(ctx, 5*time.Millisecond, time.Hour)
require.Eventually(t, func() bool { return sink.beats.Load() >= 3 }, 3*time.Second, 5*time.Millisecond,
"each beat is abandoned at its timeout and the next one is tried")
}
func TestTracker_TheStillRunningLineIsNotHeldUpBehindAHungSave(t *testing.T) {
buf := &lockedBuf{}
tr := stallTracker(&stallSink{}, buf)
go tr.Begin("a", 0) // holds the tracker's lock inside a save that hangs until its timeout
time.Sleep(10 * time.Millisecond)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go tr.Heartbeat(ctx, time.Hour, 5*time.Millisecond)
require.Eventually(t, func() bool { return strings.Contains(buf.String(), "still running") }, 3*time.Second, 5*time.Millisecond)
}
// deadlineSink records how long each write was given, and answers at once.
type deadlineSink struct {
mu sync.Mutex
saves []time.Duration
beats []time.Duration
noDLOK bool
}
func (d *deadlineSink) record(ctx context.Context, into *[]time.Duration) {
d.mu.Lock()
defer d.mu.Unlock()
dl, ok := ctx.Deadline()
if !ok {
d.noDLOK = true
return
}
*into = append(*into, time.Until(dl))
}
func (d *deadlineSink) SaveProgress(ctx context.Context, _ []domain.ProgressStep, _ time.Time) error {
d.record(ctx, &d.saves)
return nil
}
func (d *deadlineSink) Beat(ctx context.Context, _ time.Time) error {
d.record(ctx, &d.beats)
return nil
}
// The production value is the one the worker gets: it builds a Tracker without options.
// A zero or tiny default would drop every write, a huge one would bound nothing.
func TestTracker_TheDefaultSaveAndBeatDeadlineIsAFewSecondsAway(t *testing.T) {
sink := &deadlineSink{}
tr := progress.NewTracker(context.Background(), sink, slog.New(slog.NewTextHandler(&lockedBuf{}, nil)),
[]progress.Planned{{Key: "a", Label: "A"}})
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go tr.Heartbeat(ctx, 5*time.Millisecond, time.Hour)
tr.Begin("a", 0)
require.Eventually(t, func() bool {
sink.mu.Lock()
defer sink.mu.Unlock()
return len(sink.beats) > 0
}, 3*time.Second, 5*time.Millisecond)
sink.mu.Lock()
defer sink.mu.Unlock()
require.False(t, sink.noDLOK, "a write was given no deadline at all")
require.NotEmpty(t, sink.saves)
for _, d := range append(append([]time.Duration{}, sink.saves...), sink.beats...) {
require.Greater(t, d, time.Second, "the deadline is already (nearly) spent when the write starts")
require.LessOrEqual(t, d, 5*time.Second, "the default bound is 5 s")
}
}
+26 -7
View File
@@ -17,11 +17,12 @@ import (
// fakeSink records every saved snapshot and every heartbeat.
type fakeSink struct {
mu sync.Mutex
saves [][]domain.ProgressStep
at []time.Time
beats int
err error
mu sync.Mutex
saves [][]domain.ProgressStep
at []time.Time
beats int
beatAt []time.Time
err error
}
func (f *fakeSink) SaveProgress(_ context.Context, steps []domain.ProgressStep, at time.Time) error {
@@ -32,10 +33,11 @@ func (f *fakeSink) SaveProgress(_ context.Context, steps []domain.ProgressStep,
return f.err
}
func (f *fakeSink) Beat(context.Context) error {
func (f *fakeSink) Beat(_ context.Context, at time.Time) error {
f.mu.Lock()
defer f.mu.Unlock()
f.beats++
f.beatAt = append(f.beatAt, at)
return f.err
}
@@ -297,7 +299,7 @@ func (r *retainingSink) SaveProgress(_ context.Context, steps []domain.ProgressS
r.got = append(r.got, steps)
return nil
}
func (r *retainingSink) Beat(context.Context) error { return nil }
func (r *retainingSink) Beat(context.Context, time.Time) error { return nil }
func TestTracker_ASnapshotHandedToTheSinkIsNotChangedByLaterTransitions(t *testing.T) {
sink := &retainingSink{}
@@ -400,3 +402,20 @@ func TestTracker_AnnotateAttachesANoteToAStepAndSavesItAtOnce(t *testing.T) {
require.Equal(t, "20 min of the 79 the file claimed", sink.last(t)[0].Note)
require.NotPanics(t, func() { tr.Annotate("no-such-step", "x") })
}
func TestTracker_BeatIsStampedWithTheTrackersClock(t *testing.T) {
// alive_at used the database's NOW() while the page compared it with the application
// clock; a skewed database clock showed a healthy job as lost (#51).
tr, sink, clk, _ := newTracker(t)
clk.tick(42 * time.Second)
want := clk.now()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go tr.Heartbeat(ctx, 5*time.Millisecond, time.Hour)
require.Eventually(t, func() bool { return sink.beatCount() >= 1 }, 2*time.Second, 5*time.Millisecond)
sink.mu.Lock()
defer sink.mu.Unlock()
require.True(t, sink.beatAt[0].Equal(want), "beat stamped %v, the tracker's clock says %v", sink.beatAt[0], want)
}
+60 -4
View File
@@ -6,6 +6,7 @@ import (
"fmt"
"io"
"os"
"sync"
"time"
"git.d-ma.be/mathias/tapir/internal/domain"
@@ -24,7 +25,7 @@ type Prober interface {
// Store is the slice of the private transcript store the intake needs.
type Store interface {
CreateUpload(ctx context.Context, userID, title string, hadVideo bool) (string, error)
CreateUpload(ctx context.Context, userID, title string, lang domain.Language, hadVideo bool) (string, error)
FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error
}
@@ -34,6 +35,9 @@ type Job struct {
UserID string
UploadID string
Path string
// Language is what the user chose for this recording; empty means Swedish (a job
// queued before the choice existed).
Language domain.Language
// Duration is what the file CLAIMED when probed. The worker compares it with what
// was really converted; zero means unknown (ADR-035 review).
Duration time.Duration
@@ -51,6 +55,53 @@ type Intake struct {
sub Submitter
limits Limits
tmpDir string
// MaxReceives and MaxReceivesPerUser bound how many request bodies Accept reads at
// once; zero means the defaults. Without them N parallel POSTs each spool up to
// MaxBytes while the queue still looks empty, and the spool volume fills (#47).
MaxReceives int
MaxReceivesPerUser int
mu sync.Mutex
receives int
byUser map[string]int
}
const (
defaultMaxReceives = 4
defaultMaxReceivesPerUser = 2
)
// acquireReceive reserves a receive slot for userID, or reports there is none. Every
// successful call must be paired with releaseReceive.
func (in *Intake) acquireReceive(userID string) bool {
maxAll, maxUser := in.MaxReceives, in.MaxReceivesPerUser
if maxAll <= 0 {
maxAll = defaultMaxReceives
}
if maxUser <= 0 {
maxUser = defaultMaxReceivesPerUser
}
in.mu.Lock()
defer in.mu.Unlock()
if in.receives >= maxAll || in.byUser[userID] >= maxUser {
return false
}
if in.byUser == nil {
in.byUser = map[string]int{}
}
in.receives++
in.byUser[userID]++
return true
}
func (in *Intake) releaseReceive(userID string) {
in.mu.Lock()
defer in.mu.Unlock()
in.receives--
if in.byUser[userID]--; in.byUser[userID] <= 0 {
delete(in.byUser, userID)
}
}
// NewIntake wires an Intake. tmpDir is where uploads are spooled; it must exist.
@@ -87,13 +138,18 @@ const busyMessage = "Too many recordings are waiting to be transcribed. Please t
// validation yields a *RejectError and creates NO row — rejection is synchronous
// and leaves nothing behind. On success the temp file is handed to the job and the
// worker deletes it; on every failure path Accept deletes it itself.
func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reader) (string, error) {
func (in *Intake) Accept(ctx context.Context, userID, title string, lang domain.Language, src io.Reader) (string, error) {
// Refuse before spooling when the queue has no room, if the submitter can say so: a full
// queue otherwise cost up to the upload cap on disk, an ffprobe run and a failed row for
// every refused attempt.
if rc, ok := in.sub.(interface{ HasRoom(userID string) bool }); ok && !rc.HasRoom(userID) {
return "", &RejectError{Reason: ReasonBusy, Message: busyMessage}
}
// HasRoom cannot see bodies still being received; bound those separately.
if !in.acquireReceive(userID) {
return "", &RejectError{Reason: ReasonBusy, Message: busyMessage}
}
defer in.releaseReceive(userID)
f, err := os.CreateTemp(in.tmpDir, "upload-*")
if err != nil {
return "", fmt.Errorf("uploads: spool: %w", err)
@@ -137,11 +193,11 @@ func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reade
return "", err
}
id, err := in.store.CreateUpload(ctx, userID, title, probe.HasVideo)
id, err := in.store.CreateUpload(ctx, userID, title, lang, probe.HasVideo)
if err != nil {
return "", fmt.Errorf("uploads: create: %w", err)
}
if err := in.sub.Submit(Job{UserID: userID, UploadID: id, Path: path, Duration: probe.Duration}); err != nil {
if err := in.sub.Submit(Job{UserID: userID, UploadID: id, Path: path, Language: lang, Duration: probe.Duration}); err != nil {
if errors.Is(err, ErrQueueFull) {
// The user's to retry, not a server fault: refuse it as such, and do not leave
// the row queued -- that is the stuck spinner.
+5 -5
View File
@@ -23,7 +23,7 @@ func TestAccept_AFullQueueIsABusyRejectionNotAServerErrorAndLeavesNothingBehind(
r := newRig(t)
r.sub.err = uploads.ErrQueueFull
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
var rej *uploads.RejectError
require.True(t, errors.As(err, &rej), "a full queue must be a rejection the web layer can answer with 429: %v", err)
@@ -48,7 +48,7 @@ func TestAccept_ANoRoomQueueIsRefusedBeforeAnythingIsSpooledProbedOrStored(t *te
full := &roomSubmitter{room: false}
in := uploads.NewIntake(r.st, r.pr, full, uploads.Limits{MaxBytes: 1024, MaxDuration: 90 * time.Minute}, r.tmp)
_, err := in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
_, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
var rej *uploads.RejectError
require.True(t, errors.As(err, &rej))
@@ -62,7 +62,7 @@ func TestAccept_ANoRoomQueueIsRefusedBeforeAnythingIsSpooledProbedOrStored(t *te
func TestAccept_ASubmitterThatCannotSayStillWorksAsBefore(t *testing.T) {
r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
require.NoError(t, err, "a Submitter without HasRoom is accepted: the check at Submit remains")
}
@@ -81,7 +81,7 @@ func (b *brokenBody) Read(p []byte) (int, error) {
func TestAccept_AnErrorReadingTheClientsBodyIsMarkedAsTheClientsAndAWrappedSubmitErrorIsNot(t *testing.T) {
r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "t", &brokenBody{n: 1})
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, &brokenBody{n: 1})
require.Error(t, err)
assert.ErrorIs(t, err, uploads.ErrClientRead, "a dropped client body is the sender's to retry")
@@ -90,7 +90,7 @@ func TestAccept_AnErrorReadingTheClientsBodyIsMarkedAsTheClientsAndAWrappedSubmi
// A fault that is ours must NOT carry the marker, even if it wraps the same sentinel.
r2 := newRig(t)
r2.sub.err = fmt.Errorf("db: %w", io.ErrUnexpectedEOF)
_, err = r2.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
_, err = r2.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
require.Error(t, err)
assert.NotErrorIs(t, err, uploads.ErrClientRead)
}
+132
View File
@@ -0,0 +1,132 @@
package uploads_test
// Observes how many request bodies Accept will read at once (#47). HasRoom and the
// per-user queue cap act before and after the receive; N parallel POSTs all pass
// HasRoom while the queue is still empty, so the receive itself is bounded.
import (
"context"
"errors"
"io"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
// slowReceive starts an Accept whose body stays open until the returned writer is closed,
// and waits until it holds a spool file, which proves it is inside the receive.
type slowReceive struct {
w *io.PipeWriter
done chan error
finished bool
}
func startSlowReceive(t *testing.T, r *rig, in *uploads.Intake, user string, wantFiles int) *slowReceive {
t.Helper()
pr, pw := io.Pipe()
s := &slowReceive{w: pw, done: make(chan error, 1)}
go func() {
_, err := in.Accept(context.Background(), user, "t", domain.LangSwedish, pr)
s.done <- err
}()
require.Eventually(t, func() bool { return len(r.tmpFiles(t)) == wantFiles }, 2*time.Second, 5*time.Millisecond,
"the receive for %s should have started", user)
// An abandoned receive ends in a read error, so it never reaches the rig's fakes, which
// are not safe for concurrent use. A receive that finish() already ended is a no-op.
t.Cleanup(func() {
if s.finished {
return
}
_ = pw.CloseWithError(errors.New("test over"))
select {
case <-s.done:
case <-time.After(2 * time.Second):
}
})
return s
}
func (s *slowReceive) finish(t *testing.T) error {
t.Helper()
_, _ = s.w.Write([]byte("data"))
require.NoError(t, s.w.Close())
s.finished = true
select {
case err := <-s.done:
return err
case <-time.After(2 * time.Second):
t.Fatal("Accept did not return after its body ended")
return nil
}
}
func requireBusy(t *testing.T, err error) {
t.Helper()
var rej *uploads.RejectError
require.True(t, errors.As(err, &rej), "want a busy rejection, got %v", err)
assert.Equal(t, uploads.ReasonBusy, rej.Reason)
}
func TestAccept_APerUserLimitOnConcurrentReceivesRefusesTheNextBodyBeforeSpoolingIt(t *testing.T) {
r := newRig(t)
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
in.MaxReceivesPerUser = 2
in.MaxReceives = 10
startSlowReceive(t, r, in, "user-1", 1)
startSlowReceive(t, r, in, "user-1", 2)
_, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
requireBusy(t, err)
assert.Len(t, r.tmpFiles(t), 2, "the refused body was never spooled")
// Another user is not blocked by user-1's two receives.
other := startSlowReceive(t, r, in, "user-2", 3)
require.NoError(t, other.finish(t))
}
func TestAccept_AnOverallLimitOnConcurrentReceivesRefusesTheNextUser(t *testing.T) {
r := newRig(t)
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
in.MaxReceivesPerUser = 5
in.MaxReceives = 3
for i, u := range []string{"a", "b", "c"} {
startSlowReceive(t, r, in, u, i+1)
}
_, err := in.Accept(context.Background(), "d", "t", domain.LangSwedish, strings.NewReader("data"))
requireBusy(t, err)
assert.Len(t, r.tmpFiles(t), 3)
}
func TestAccept_AFinishedReceiveFreesItsSlot(t *testing.T) {
r := newRig(t)
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
in.MaxReceivesPerUser = 1
in.MaxReceives = 5
first := startSlowReceive(t, r, in, "user-1", 1)
require.NoError(t, first.finish(t))
_, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
require.NoError(t, err, "the slot is released when Accept returns")
}
func TestAccept_AFailedReceiveFreesItsSlot(t *testing.T) {
r := newRig(t)
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
in.MaxReceivesPerUser = 1
in.MaxReceives = 5
_, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, &brokenBody{n: 1})
require.ErrorIs(t, err, uploads.ErrClientRead)
_, err = in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
require.NoError(t, err, "a dropped client must not leak a slot")
}
+32
View File
@@ -0,0 +1,32 @@
package uploads_test
// The limits production runs on are the DEFAULTS: nothing outside tests sets
// MaxReceives or MaxReceivesPerUser. Pinned here with the literal numbers, so changing a
// default is a visible decision (#47 review). With 4 receives, 20 queued and 1 running at
// the 200 MiB cap, the spool needs ~5 GiB of the 6 Gi emptyDir (infra); raising any of
// them or TAPIR_UPLOAD_MAX_BYTES must revisit that arithmetic.
import (
"context"
"strings"
"testing"
"time"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
func TestNewIntake_DefaultsAllowTwoReceivesPerUserAndFourOverall(t *testing.T) {
r := newRig(t)
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
startSlowReceive(t, r, in, "a", 1)
startSlowReceive(t, r, in, "a", 2)
_, err := in.Accept(context.Background(), "a", "t", domain.LangSwedish, strings.NewReader("x"))
requireBusy(t, err) // a third body from one user
startSlowReceive(t, r, in, "b", 3)
startSlowReceive(t, r, in, "b", 4)
_, err = in.Accept(context.Background(), "c", "t", domain.LangSwedish, strings.NewReader("x"))
requireBusy(t, err) // a fifth body overall
}
+24 -11
View File
@@ -21,13 +21,15 @@ import (
type fakeStore struct {
created []string // titles
langs []domain.Language
hadVideo []bool
failed map[string]domain.UploadErrorKind
nextID string
}
func (f *fakeStore) CreateUpload(_ context.Context, _, title string, hadVideo bool) (string, error) {
func (f *fakeStore) CreateUpload(_ context.Context, _, title string, lang domain.Language, hadVideo bool) (string, error) {
f.created = append(f.created, title)
f.langs = append(f.langs, lang)
f.hadVideo = append(f.hadVideo, hadVideo)
return f.nextID, nil
}
@@ -98,7 +100,7 @@ func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
r := newRig(t)
audio := bytes.Repeat([]byte("a"), 500)
id, err := r.in.Accept(context.Background(), "user-1", "Teams export", bytes.NewReader(audio))
id, err := r.in.Accept(context.Background(), "user-1", "Teams export", domain.LangSwedish, bytes.NewReader(audio))
require.NoError(t, err)
assert.Equal(t, "up-1", id)
@@ -106,7 +108,7 @@ func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
assert.Equal(t, audio, r.pr.seen, "the prober must see the complete uploaded bytes, on disk")
require.Len(t, r.sub.jobs, 1)
job := r.sub.jobs[0]
assert.Equal(t, uploads.Job{UserID: "user-1", UploadID: "up-1", Path: job.Path, Duration: 10 * time.Minute}, job,
assert.Equal(t, uploads.Job{UserID: "user-1", UploadID: "up-1", Path: job.Path, Duration: 10 * time.Minute, Language: domain.LangSwedish}, job,
"the job carries what the file CLAIMED, so the worker can compare it with what was converted")
assert.Equal(t, filepath.Dir(job.Path), r.tmp, "the temp file lives in the configured directory")
got, err := os.ReadFile(job.Path)
@@ -117,7 +119,7 @@ func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
func TestAccept_OversizeBodyIsRejectedAndLeavesNothingBehind(t *testing.T) {
r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "big", strings.NewReader(strings.Repeat("a", 1025)))
_, err := r.in.Accept(context.Background(), "user-1", "big", domain.LangSwedish, strings.NewReader(strings.Repeat("a", 1025)))
var rej *uploads.RejectError
require.True(t, errors.As(err, &rej))
@@ -132,7 +134,7 @@ func TestAccept_OversizeBodyIsRejectedAndLeavesNothingBehind(t *testing.T) {
func TestAccept_BodyExactlyAtTheLimitIsAccepted(t *testing.T) {
r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "edge", strings.NewReader(strings.Repeat("a", 1024)))
_, err := r.in.Accept(context.Background(), "user-1", "edge", domain.LangSwedish, strings.NewReader(strings.Repeat("a", 1024)))
require.NoError(t, err)
}
@@ -152,7 +154,7 @@ func TestAccept_ProbeVerdictRejectsWithoutRowAndRemovesTheFile(t *testing.T) {
r := newRig(t)
r.pr.probe = tt.probe
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
var rej *uploads.RejectError
require.True(t, errors.As(err, &rej))
@@ -169,7 +171,7 @@ func TestAccept_UnreadableFileIsARejectionButAProberFaultIsAnError(t *testing.T)
r := newRig(t)
r.pr.err = uploads.ErrUnreadable
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("not media"))
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("not media"))
var rej *uploads.RejectError
require.True(t, errors.As(err, &rej), "the user's file is the problem: tell them")
@@ -180,7 +182,7 @@ func TestAccept_UnreadableFileIsARejectionButAProberFaultIsAnError(t *testing.T)
r := newRig(t)
r.pr.err = errors.New("exec: ffprobe: executable file not found")
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
require.Error(t, err)
var rej *uploads.RejectError
@@ -194,7 +196,7 @@ func TestAccept_ClientDisconnectMidUploadLeavesNothingBehind(t *testing.T) {
r := newRig(t)
body := io.MultiReader(strings.NewReader("partial"), iotest.ErrReader(errors.New("client went away")))
_, err := r.in.Accept(context.Background(), "user-1", "t", body)
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, body)
require.Error(t, err)
assert.Empty(t, r.pr.paths)
@@ -206,7 +208,7 @@ func TestAccept_SubmitFailureFailsTheRowHonestlyAndRemovesTheFile(t *testing.T)
r := newRig(t)
r.sub.err = errors.New("worker stopped")
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
require.Error(t, err)
assert.Equal(t, domain.UploadErrInterrupted, r.st.failed["up-1"],
@@ -228,7 +230,7 @@ func TestAccept_AFileWithAVideoTrackIsAcceptedAndRecordsThatItHadOne(t *testing.
r := newRig(t)
r.pr.probe = tt.probe
id, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
id, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
require.NoError(t, err)
assert.Equal(t, "up-1", id)
@@ -238,3 +240,14 @@ func TestAccept_AFileWithAVideoTrackIsAcceptedAndRecordsThatItHadOne(t *testing.
})
}
}
func TestAccept_TheChosenLanguageIsStoredAndHandedToTheJob(t *testing.T) {
r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangAuto, strings.NewReader("data"))
require.NoError(t, err)
assert.Equal(t, []domain.Language{domain.LangAuto}, r.st.langs, "the row records what the user chose")
require.Len(t, r.sub.jobs, 1)
assert.Equal(t, domain.LangAuto, r.sub.jobs[0].Language, "the worker is told, so the engines are")
}
+48
View File
@@ -0,0 +1,48 @@
package uploads
// Observes what jobSink hands the store: the heartbeat's timestamp is the one the tracker
// stamped (its application clock), not a second reading taken here (#51).
import (
"context"
"testing"
"time"
"git.d-ma.be/mathias/tapir/internal/domain"
)
type atStore struct{ beatAt, saveAt time.Time }
func (*atStore) StartTranscribing(context.Context, string, string) error { return nil }
func (*atStore) CompleteUpload(context.Context, string, string, domain.Transcription) error {
return nil
}
func (*atStore) FailUpload(context.Context, string, string, domain.UploadErrorKind) error { return nil }
func (s *atStore) SaveProgress(_ context.Context, _, _ string, _ []domain.ProgressStep, at time.Time) error {
s.saveAt = at
return nil
}
func (s *atStore) Heartbeat(_ context.Context, _, _ string, at time.Time) error {
s.beatAt = at
return nil
}
func TestJobSink_ForwardsTheTrackersTimestampToTheStoreUnchanged(t *testing.T) {
st := &atStore{}
sink := jobSink{st: st, userID: "u", id: "i"}
stamp := time.Date(2020, 1, 2, 3, 4, 5, 0, time.UTC)
if err := sink.Beat(context.Background(), stamp); err != nil {
t.Fatal(err)
}
if err := sink.SaveProgress(context.Background(), nil, stamp); err != nil {
t.Fatal(err)
}
if !st.beatAt.Equal(stamp) {
t.Errorf("heartbeat reached the store stamped %v, want the tracker's %v", st.beatAt, stamp)
}
if !st.saveAt.Equal(stamp) {
t.Errorf("progress save reached the store stamped %v, want the tracker's %v", st.saveAt, stamp)
}
}
+36 -12
View File
@@ -33,6 +33,13 @@ type Transcriber interface {
Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error)
}
// LanguagePlanner is a Planner whose phases depend on the language of the job: a recording
// that is not Swedish skips the second engine and the reconcile, and the page should not list
// them as coming (tapir#61). The worker prefers it over Planner.
type LanguagePlanner interface {
PlanFor(lang domain.Language) []progress.Planned
}
// Planner is an optional extra on a Transcriber: it declares its own phases (for
// example one per engine) so the page can show them. A Transcriber that does not
// implement it gets a single "transcribing" phase from the worker. Whatever a Planner
@@ -47,7 +54,7 @@ type WorkerStore interface {
CompleteUpload(ctx context.Context, userID, id string, t domain.Transcription) error
FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error
SaveProgress(ctx context.Context, userID, id string, steps []domain.ProgressStep, at time.Time) error
Heartbeat(ctx context.Context, userID, id string) error
Heartbeat(ctx context.Context, userID, id string, at time.Time) error
}
// WorkerOption tunes a Worker.
@@ -91,7 +98,9 @@ func (s jobSink) SaveProgress(ctx context.Context, steps []domain.ProgressStep,
return s.st.SaveProgress(ctx, s.userID, s.id, steps, at)
}
func (s jobSink) Beat(ctx context.Context) error { return s.st.Heartbeat(ctx, s.userID, s.id) }
func (s jobSink) Beat(ctx context.Context, at time.Time) error {
return s.st.Heartbeat(ctx, s.userID, s.id, at)
}
// Worker transcribes accepted uploads in-process, ONE AT A TIME (ADR-033). It is
// not a second deployable: it shares the web process's lifetime, which is why it
@@ -176,9 +185,10 @@ func (w *Worker) Submit(j Job) error {
}
// Run processes queued jobs until ctx ends, then deletes the audio of anything
// still queued and refuses new jobs. The rows of those jobs stay queued or
// transcribing and are failed as interrupted by the start-up sweep
// (store.FailInterruptedUploads): the audio is not retained, so they cannot resume.
// still queued and refuses new jobs. The job running at shutdown is recorded interrupted
// by Process; the rows of jobs still queued stay queued and are failed as interrupted by
// the start-up sweep (store.FailInterruptedUploads): the audio is not retained, so they
// cannot resume.
func (w *Worker) Run(ctx context.Context) {
defer w.stop()
for {
@@ -227,7 +237,7 @@ func (w *Worker) stop() {
// Process runs one job: row -> transcribing, convert, transcribe, store. The
// uploaded file and the converted WAV are deleted on every path — the audio is
// never retained (ADR-033). A failure is recorded with a TYPED kind; a shutdown
// mid-job is not a failure and leaves the row for the start-up sweep.
// mid-job is recorded as interrupted at once (see fail).
//
// Every phase is reported through one progress.Tracker, which saves it to the row for
// the page and logs it, and a heartbeat stamps the row on its own while the job runs
@@ -248,8 +258,14 @@ func (w *Worker) Process(ctx context.Context, job Job) {
}
plan := []progress.Planned{{Key: "received", Label: "Received"}, {Key: "audio", Label: "Preparing the audio"}}
lang := job.Language
if lang == "" {
lang = domain.LangSwedish // queued before the choice existed
}
planner, planned := w.tr.(Planner)
if planned {
if lp, ok := w.tr.(LanguagePlanner); ok {
plan = append(plan, lp.PlanFor(lang)...)
} else if planned {
plan = append(plan, planner.Plan()...)
} else {
plan = append(plan, progress.Planned{Key: "transcribe", Label: "Transcribing"})
@@ -269,7 +285,7 @@ func (w *Worker) Process(ctx context.Context, job Job) {
defer func() { stopBeat(); <-hbDone }() // no heartbeat outlives its job
// abort records where the job stopped. A shutdown (the parent context ended) is not
// a failure: the row is left for the start-up sweep, so the step stays as it was.
// a failure of the step: it stays as it was, and fail records the row as interrupted.
abort := func(kind domain.UploadErrorKind) {
if ctx.Err() == nil {
tr.FailRunning()
@@ -310,7 +326,7 @@ func (w *Worker) Process(ctx context.Context, job Job) {
if !planned {
tr.Begin("transcribe", 0)
}
t, err := w.tr.Transcribe(progress.With(jobCtx, tr), wav)
t, err := w.tr.Transcribe(domain.WithLanguage(progress.With(jobCtx, tr), lang), wav)
if err != nil {
log.Error("upload: transcribe", "kind", errkind.Of(err), "elapsed", time.Since(start).Round(time.Second))
abort(domain.UploadErrTranscriptionFailed)
@@ -332,11 +348,19 @@ func (w *Worker) Process(ctx context.Context, job Job) {
"segments", len(t.Segments), "elapsed", time.Since(start).Round(time.Second))
}
// fail records a transcription failure — unless the PARENT context ended, which
// is a shutdown, not a failure: the row is left for the start-up sweep.
// interruptedWriteTimeout bounds the one write a dying process makes to record its job.
const interruptedWriteTimeout = 3 * time.Second
// fail records a transcription failure. When the PARENT context ended the process is
// shutting down: the job is recorded interrupted at once, on a short context of its own,
// because the process may exit before any other pod's start-up sweep would do it (#50).
// The sweep remains the backstop for a process that is killed outright.
func (w *Worker) fail(ctx context.Context, job Job, log *slog.Logger, kind domain.UploadErrorKind) {
if ctx.Err() != nil {
return
kind = domain.UploadErrInterrupted
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(context.WithoutCancel(ctx), interruptedWriteTimeout)
defer cancel()
}
if err := w.store.FailUpload(ctx, job.UserID, job.UploadID, kind); err != nil {
log.Error("upload: record failure", "kind", errkind.Of(err))
+44
View File
@@ -213,3 +213,47 @@ func TestWorkerProgress_AStoreFailureLogsItsSQLStateNeverTheValueItQuotes(t *tes
assert.Contains(t, out, "upload: store transcript")
assert.Contains(t, out, "kind=sql_22P02")
}
// langPlannedTr plans per language, like the Ensemble: an auto job has no base engine.
type langPlannedTr struct{ *fakeTranscriber }
func (langPlannedTr) Plan() []progress.Planned { return plannedTr{}.Plan() }
func (langPlannedTr) PlanFor(l domain.Language) []progress.Planned {
if l == domain.LangAuto {
return []progress.Planned{{Key: "engine_b", Label: "Transcribing with B"}}
}
return plannedTr{}.Plan()
}
func (p langPlannedTr) Transcribe(ctx context.Context, wav string) (domain.Transcription, error) {
r := progress.From(ctx)
if domain.LanguageFrom(ctx) != domain.LangAuto {
r.Begin("engine_a", 0)
r.Done("engine_a")
}
r.Begin("engine_b", 0)
r.Done("engine_b")
return p.fakeTranscriber.Transcribe(ctx, wav)
}
func TestWorkerProgress_AJobGetsThePlanOfItsLanguage(t *testing.T) {
for name, tt := range map[string]struct {
lang domain.Language
want []string
}{
"auto": {domain.LangAuto, []string{"received:done", "audio:done", "engine_b:done", "save:done"}},
"swedish": {domain.LangSwedish, []string{"received:done", "audio:done", "engine_a:done", "engine_b:done", "save:done"}},
"no language (queued before the choice existed)": {"", []string{"received:done", "audio:done", "engine_a:done", "engine_b:done", "save:done"}},
} {
t.Run(name, func(t *testing.T) {
r := newWRig(t, time.Minute)
w := newProgressWorker(t, r, langPlannedTr{r.tr})
job := r.spool(t, "a")
job.Language = tt.lang
w.Process(context.Background(), job)
assert.Equal(t, tt.want, keysAndStates(r.st.lastSteps()))
})
}
}
+33 -5
View File
@@ -53,7 +53,10 @@ func (s *wStore) CompleteUpload(_ context.Context, _, id string, t domain.Transc
return nil
}
func (s *wStore) FailUpload(_ context.Context, _, id string, k domain.UploadErrorKind) error {
func (s *wStore) FailUpload(ctx context.Context, _, id string, k domain.UploadErrorKind) error {
if err := ctx.Err(); err != nil { // like the real store: a cancelled context cannot write
return err
}
s.mu.Lock()
defer s.mu.Unlock()
s.failed[id] = k
@@ -67,7 +70,7 @@ func (s *wStore) SaveProgress(_ context.Context, _, _ string, steps []domain.Pro
return nil
}
func (s *wStore) Heartbeat(context.Context, string, string) error {
func (s *wStore) Heartbeat(context.Context, string, string, time.Time) error {
time.Sleep(s.beatDelay)
s.mu.Lock()
defer s.mu.Unlock()
@@ -123,6 +126,7 @@ type fakeTranscriber struct {
result domain.Transcription
err error
paths []string
langs []domain.Language // what each call's context carried
// block, when non-nil, holds every call until it is closed or ctx ends.
block chan struct{}
running atomic.Int32
@@ -131,6 +135,7 @@ type fakeTranscriber struct {
func (f *fakeTranscriber) Transcribe(ctx context.Context, wav string) (domain.Transcription, error) {
f.paths = append(f.paths, wav)
f.langs = append(f.langs, domain.LanguageFrom(ctx))
n := f.running.Add(1)
defer f.running.Add(-1)
for {
@@ -242,7 +247,9 @@ func TestWorkerProcess_FailuresAreTypedAndStillDeleteTheAudio(t *testing.T) {
})
}
func TestWorkerProcess_ShutdownMidJobLeavesTheRowForTheStartupSweep(t *testing.T) {
func TestWorkerProcess_ShutdownMidJobRecordsTheRowInterruptedAtOnce(t *testing.T) {
// SIGTERM on a deploy used to leave the row "transcribing" until the NEXT pod's
// start-up sweep; the process may exit before any sweep runs (#50).
r := newWRig(t, time.Minute)
r.tr.block = make(chan struct{})
ctx, cancel := context.WithCancel(context.Background())
@@ -255,8 +262,8 @@ func TestWorkerProcess_ShutdownMidJobLeavesTheRowForTheStartupSweep(t *testing.T
<-done
_, f := r.st.snapshot()
assert.Empty(t, f, "a shutdown is not a transcription failure: the row stays for FailInterruptedUploads to mark interrupted")
assert.Empty(t, r.leftovers(t), "but the audio is still deleted")
assert.Equal(t, domain.UploadErrInterrupted, f["a"], "the row is recorded interrupted by the dying process, not left for a sweep")
assert.Empty(t, r.leftovers(t), "and the audio is still deleted")
}
func TestWorkerProcess_AJobThatOutlivesItsTimeoutIsFailedNotStuck(t *testing.T) {
@@ -332,3 +339,24 @@ func (dbErr) Error() string {
return `ERROR: invalid input syntax for type json: "zebrakaka secret words"`
}
func (dbErr) SQLState() string { return "22P02" }
func TestWorker_TheTranscriberSeesTheLanguageOfTheJobOnItsContext(t *testing.T) {
for name, tt := range map[string]struct {
job domain.Language
want domain.Language
}{
"auto": {domain.LangAuto, domain.LangAuto},
"swedish": {domain.LangSwedish, domain.LangSwedish},
"a job with no language": {"", domain.LangSwedish}, // queued before the field existed
} {
t.Run(name, func(t *testing.T) {
r := newWRig(t, time.Minute)
job := r.spool(t, "a")
job.Language = tt.job
r.w.Process(context.Background(), job)
assert.Equal(t, []domain.Language{tt.want}, r.tr.langs)
})
}
}
+151
View File
@@ -0,0 +1,151 @@
package web_test
// Observes the cross-origin gate around Router (#48): SameSite=Lax does not stop a POST
// from a sibling *.d-ma.be origin, so the router itself must refuse a non-safe request
// whose Origin or Sec-Fetch-Site says it did not come from this site's own pages.
import (
"context"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// foreign is a header set a browser sends for a form post made by another page.
var foreign = map[string]map[string]string{
"sibling subdomain, Sec-Fetch-Site": {"Origin": "https://evil.d-ma.be", "Sec-Fetch-Site": "same-site"},
"other site, Sec-Fetch-Site": {"Origin": "https://evil.example", "Sec-Fetch-Site": "cross-site"},
"sibling subdomain, Origin only": {"Origin": "https://evil.d-ma.be"},
"opaque origin": {"Origin": "null"},
}
// own is a header set for the app's own pages (or a non-browser client): must pass.
var own = map[string]map[string]string{
"same-origin Sec-Fetch-Site": {"Sec-Fetch-Site": "same-origin", "Origin": "http://example.com"},
"matching Origin only": {"Origin": "http://example.com"},
"no browser headers": {},
}
func withHeaders(req *http.Request, h map[string]string) *http.Request {
for k, v := range h {
req.Header.Set(k, v)
}
return req
}
func userCount(t *testing.T) int {
t.Helper()
var n int
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT count(*) FROM users WHERE id = $1`, userID).Scan(&n))
return n
}
func TestCrossOrigin_ARefusedDeleteAccountPostLeavesTheAccountInPlace(t *testing.T) {
for name, h := range foreign {
t.Run(name, func(t *testing.T) {
app, _ := newAccountApp(t)
resetDB(t, rawPool(t))
seedConnection(t, app, "youtube", "ada@channel", "ref")
rec := do(t, app, withHeaders(httptest.NewRequest(http.MethodPost, "/account/delete", nil), h))
assert.Equal(t, http.StatusForbidden, rec.Code)
assert.Equal(t, 1, userCount(t), "the account must survive a refused post")
})
}
}
func TestCrossOrigin_ASameOriginOrHeaderlessDeleteAccountPostIsServed(t *testing.T) {
for name, h := range own {
t.Run(name, func(t *testing.T) {
app, _ := newAccountApp(t)
resetDB(t, rawPool(t))
seedConnection(t, app, "youtube", "ada@channel", "ref")
rec := do(t, app, withHeaders(httptest.NewRequest(http.MethodPost, "/account/delete", nil), h))
assert.Equal(t, http.StatusSeeOther, rec.Code)
assert.Equal(t, "/auth/logout", rec.Header().Get("Location"))
})
}
}
func TestCrossOrigin_ARefusedUploadPostSubmitsNoJob(t *testing.T) {
for name, h := range foreign {
t.Run(name, func(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
rec := do(t, app, withHeaders(uploadReq(t, "m", "m.wav", []byte("RIFF....WAVE")), h))
assert.Equal(t, http.StatusForbidden, rec.Code)
assert.Empty(t, sub.jobs, "a refused post must not enqueue transcription")
})
}
}
func TestCrossOrigin_ARefusedUploadChatPostCallsNoModel(t *testing.T) {
for name, h := range foreign {
t.Run(name, func(t *testing.T) {
app, fc := newUploadChatApp(t)
id := seedUpload(t, userID, "Roadmap sync", "done", meetingText)
req := chatPost("/uploads/"+id+"/chat", url.Values{"message": {"hej"}})
rec := do(t, app, withHeaders(req, h))
assert.Equal(t, http.StatusForbidden, rec.Code)
assert.Zero(t, fc.callCount)
})
}
}
func TestCrossOrigin_AForeignPostIsRefusedOnEachListedStateChangingPath(t *testing.T) {
routes := []string{
"/account/disconnect/youtube",
"/account/summarize-mode",
"/register",
"/paste",
"/v/abc/action",
"/v/abc/summarize",
"/v/abc/retry-now",
"/v/abc/chat",
}
for _, path := range routes {
t.Run(path, func(t *testing.T) {
app := newApp(t)
req := httptest.NewRequest(http.MethodPost, path, strings.NewReader(""))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
rec := do(t, app, withHeaders(req, foreign["sibling subdomain, Sec-Fetch-Site"]))
assert.Equal(t, http.StatusForbidden, rec.Code)
})
}
}
func TestCrossOrigin_ARefusedRecordingDeletePostLeavesTheRecordingInPlace(t *testing.T) {
for name, h := range foreign {
t.Run(name, func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
rec := do(t, app, withHeaders(httptest.NewRequest(http.MethodPost, "/uploads/"+id+"/delete", nil), h))
assert.Equal(t, http.StatusForbidden, rec.Code)
assert.True(t, uploadExists(t, id), "a refused post must not delete the recording")
})
}
}
// A login redirect comes back to /auth/callback as a cross-site GET navigation, and
// health probes carry no headers: safe methods must never be refused on origin grounds.
func TestCrossOrigin_ACrossSiteGETIsNotRefused(t *testing.T) {
app := newApp(t)
for _, path := range []string{"/healthz", "/welcome"} {
req := httptest.NewRequest(http.MethodGet, path, nil)
req.Header.Set("Sec-Fetch-Site", "cross-site")
assert.Equal(t, http.StatusOK, do(t, app, req).Code, path)
}
}
+11
View File
@@ -0,0 +1,11 @@
package web
import "time"
// DefaultUploadIdleTimeout exposes the production idle bound (#56) to the black-box tests:
// cmd/tapir never sets App.UploadIdleTimeout, so this constant IS the deployed value.
const DefaultUploadIdleTimeout = defaultUploadIdleTimeout
// UploadIdleTimeoutForTest exposes the bound an App resolves to (#56), so a test can pin the
// value the zero-value App -- production's -- actually uses rather than only the constant.
func (a *App) UploadIdleTimeoutForTest() time.Duration { return a.uploadIdleTimeout() }
+16 -2
View File
@@ -59,6 +59,9 @@ type Store interface {
// non-owner's id is ErrUploadNotFound.
GetUpload(ctx context.Context, userID, id string) (*domain.UploadedTranscript, error)
ListUploads(ctx context.Context, userID string) ([]domain.UploadedTranscript, error)
// DeleteUpload removes a finished upload; ErrUploadInProgress while it is still being
// processed, ErrUploadNotFound for a non-owner.
DeleteUpload(ctx context.Context, userID, id string) error
// SetTranscriptStatus clears or updates a video's transcript backoff state.
// Used by handleRetryNow to clear rate_limited_at before immediate processing.
@@ -113,6 +116,11 @@ type App struct {
// front rather than discovered by failing.
Uploads Uploader
UploadLimits uploads.Limits
// UploadIdleTimeout is how long an upload body may go without delivering a byte
// before the connection read is cut off and its receive slot freed (#56). Zero
// means defaultUploadIdleTimeout. It bounds silence, not the whole body, so a
// large upload on a slow but moving link still completes.
UploadIdleTimeout time.Duration
// UploadChatMaxChars is the transcript budget for chat over an uploaded
// recording. The caption default (18,000 chars, ~22 minutes of speech) would
// keep only the first third of an hour-long meeting.
@@ -200,6 +208,7 @@ func (a *App) Router() http.Handler {
app.HandleFunc("POST /uploads", a.handleUploadPost)
app.HandleFunc("GET /uploads/{id}", a.handleUploadDetail)
app.HandleFunc("GET /uploads/{id}/status", a.handleUploadStatus)
app.HandleFunc("POST /uploads/{id}/delete", a.handleUploadDelete)
// Chat over the uploaded transcript reuses the ADR-027 Chatter; mounted only
// when one is wired.
if a.Chat != nil {
@@ -229,7 +238,12 @@ func (a *App) Router() http.Handler {
// resolved user_id. /register lives inside the auth guard but is exempt from
// the registration gate (you must be able to reach it before you have a user).
root.Handle("/", a.Auth.Middleware(a.registrationGate(app)))
return root
// The session cookie is SameSite=Lax, which still travels on a POST made from a
// sibling *.d-ma.be page (same site, different origin). Refuse any non-safe request
// whose Origin or Sec-Fetch-Site says it did not come from this origin (#48). It wraps
// everything, so a route added later is covered without remembering to ask.
return http.NewCrossOriginProtection().Handler(root)
}
// handleWelcome renders the public landing page (/welcome). It is mounted outside
@@ -304,7 +318,7 @@ func (a *App) handleList(w http.ResponseWriter, r *http.Request) {
a.render(w, r, summaryList(buckets, hasConnected, autoSummarize))
return
}
a.render(w, r, ListPage(buckets, f, stats, takeFlash(w, r), hasConnected, channels, autoSummarize))
a.render(w, r, ListPage(buckets, f, stats, takeFlash(w, r), hasConnected, channels, autoSummarize, a.Uploads != nil))
}
// handleDetail renders one summary in full (highlights, takeaways, action group).
+61
View File
@@ -0,0 +1,61 @@
package oidc_test
import (
"context"
"errors"
"testing"
"time"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/web/oidc"
)
func testConfig(issuer string) oidc.Config {
return oidc.Config{
Issuer: issuer,
ClientID: testClientID,
ClientSecret: "test-client-secret",
RedirectURL: "http://tapir.test/auth/callback",
SessionSecret: "test-session-secret-please-change",
}
}
// #59: a discovery 503 at start-up is retried instead of failing New.
func TestNewRetriesDiscoveryAfter503UntilIssuerAnswers(t *testing.T) {
f := newFakeIssuer(t)
f.unavailableFor = 2
auth, err := oidc.New(context.Background(), testConfig(f.server.URL),
oidc.WithDiscoveryRetry(5, time.Millisecond))
require.NoError(t, err)
require.NotNil(t, auth)
require.Equal(t, int32(3), f.discoveryHits.Load())
}
func TestNewReturnsDiscoveryErrorAfterAttemptsExhausted(t *testing.T) {
f := newFakeIssuer(t)
f.unavailableFor = 1000
_, err := oidc.New(context.Background(), testConfig(f.server.URL),
oidc.WithDiscoveryRetry(3, time.Millisecond))
require.ErrorContains(t, err, "503")
require.Equal(t, int32(3), f.discoveryHits.Load())
}
func TestNewStopsDiscoveryBackoffWhenContextCancelled(t *testing.T) {
f := newFakeIssuer(t)
f.unavailableFor = 1000
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
defer cancel()
start := time.Now()
_, err := oidc.New(ctx, testConfig(f.server.URL),
oidc.WithDiscoveryRetry(5, time.Hour))
require.True(t, errors.Is(err, context.DeadlineExceeded), "err = %v", err)
require.Less(t, time.Since(start), 5*time.Second)
require.Equal(t, int32(1), f.discoveryHits.Load())
}
+70 -20
View File
@@ -57,6 +57,10 @@ const (
pendingTTL = 10 * time.Minute
sessionCookie = "tapir_session"
loginPath = "/auth/login"
// Discovery retry (#59): 5 tries, 2+4+8+16 s back-off ≈ 30 s, then fail.
defaultDiscoveryAttempts = 5
defaultDiscoveryDelay = 2 * time.Second
)
// DexAuth implements web.Auth against Dex.
@@ -70,6 +74,9 @@ type DexAuth struct {
sessionTTL time.Duration
now func() time.Time
insecure bool // tests only: drop the Secure attribute so cookies flow over http
discoveryAttempts int
discoveryDelay time.Duration
}
var _ web.Auth = (*DexAuth)(nil)
@@ -96,14 +103,55 @@ func WithSessionTTL(ttl time.Duration) Option {
}
}
// WithDiscoveryRetry overrides the discovery attempt count and the initial
// back-off delay (doubled after each failed attempt).
func WithDiscoveryRetry(attempts int, delay time.Duration) Option {
return func(d *DexAuth) {
if attempts > 0 {
d.discoveryAttempts = attempts
}
if delay > 0 {
d.discoveryDelay = delay
}
}
}
// WithInsecureCookies drops the Secure cookie attribute. TEST ONLY — it lets
// the session cookie travel over plain http; never use it in production.
func WithInsecureCookies() Option {
return func(d *DexAuth) { d.insecure = true }
}
// New discovers the issuer and returns a configured DexAuth. ctx bounds the
// discovery request only.
// discover runs OIDC discovery up to attempts times, doubling delay between
// tries (#59: an issuer briefly unavailable behind its proxy answered 503 and
// serve exited). A cancelled ctx ends the back-off immediately with ctx's error.
func discover(ctx context.Context, issuer string, attempts int, delay time.Duration) (*oidc.Provider, error) {
var lastErr error
for i := range attempts {
if i > 0 {
t := time.NewTimer(delay)
select {
case <-ctx.Done():
t.Stop()
return nil, fmt.Errorf("after %d attempts (last: %v): %w", i, lastErr, ctx.Err())
case <-t.C:
}
delay *= 2
}
p, err := oidc.NewProvider(ctx, issuer)
if err == nil {
return p, nil
}
lastErr = err
if ctx.Err() != nil {
return nil, fmt.Errorf("after %d attempts: %w", i+1, ctx.Err())
}
}
return nil, fmt.Errorf("after %d attempts: %w", attempts, lastErr)
}
// New discovers the issuer and returns a configured DexAuth. ctx bounds discovery,
// including its retry back-off.
func New(ctx context.Context, cfg Config, opts ...Option) (*DexAuth, error) {
for name, val := range map[string]string{
"issuer": cfg.Issuer,
@@ -117,29 +165,31 @@ func New(ctx context.Context, cfg Config, opts ...Option) (*DexAuth, error) {
}
}
provider, err := oidc.NewProvider(ctx, cfg.Issuer)
if err != nil {
return nil, fmt.Errorf("oidc: discover issuer: %w", err)
}
d := &DexAuth{
cfg: cfg,
verifier: provider.Verifier(&oidc.Config{ClientID: cfg.ClientID}),
oauth: &oauth2.Config{
ClientID: cfg.ClientID,
ClientSecret: cfg.ClientSecret,
Endpoint: provider.Endpoint(),
RedirectURL: cfg.RedirectURL,
Scopes: []string{oidc.ScopeOpenID, "profile", "email"},
},
pending: newPendingStore(),
secret: []byte(cfg.SessionSecret),
sessionTTL: defaultSessionTTL,
now: time.Now,
cfg: cfg,
pending: newPendingStore(),
secret: []byte(cfg.SessionSecret),
sessionTTL: defaultSessionTTL,
now: time.Now,
discoveryAttempts: defaultDiscoveryAttempts,
discoveryDelay: defaultDiscoveryDelay,
}
for _, o := range opts {
o(d)
}
provider, err := discover(ctx, cfg.Issuer, d.discoveryAttempts, d.discoveryDelay)
if err != nil {
return nil, fmt.Errorf("oidc: discover issuer: %w", err)
}
d.verifier = provider.Verifier(&oidc.Config{ClientID: cfg.ClientID})
d.oauth = &oauth2.Config{
ClientID: cfg.ClientID,
ClientSecret: cfg.ClientSecret,
Endpoint: provider.Endpoint(),
RedirectURL: cfg.RedirectURL,
Scopes: []string{oidc.ScopeOpenID, "profile", "email"},
}
return d, nil
}
+10
View File
@@ -8,6 +8,7 @@ import (
"net/http"
"net/http/httptest"
"net/url"
"sync/atomic"
"testing"
"time"
@@ -34,6 +35,11 @@ type fakeIssuer struct {
sub string
email string
nonce string
// discoveryHits counts requests to the discovery document; the first
// unavailableFor of them answer 503 (issuer briefly down behind its proxy).
discoveryHits atomic.Int32
unavailableFor int32
}
const testKID = "test-key"
@@ -47,6 +53,10 @@ func newFakeIssuer(t *testing.T) *fakeIssuer {
mux := http.NewServeMux()
mux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, _ *http.Request) {
if f.discoveryHits.Add(1) <= f.unavailableFor {
http.Error(w, "no available server", http.StatusServiceUnavailable)
return
}
writeJSON(t, w, map[string]any{
"issuer": f.server.URL,
"authorization_endpoint": f.server.URL + "/authorize",
+1 -1
View File
@@ -66,7 +66,7 @@ func TestParseYouTubeVideoID(t *testing.T) {
func TestListPageShowsPasteFormOnlyWhenConnected(t *testing.T) {
render := func(connected bool) string {
var buf bytes.Buffer
if err := ListPage(listBuckets{}, Filter{}, PipelineStats{}, "", connected, nil, true).Render(context.Background(), &buf); err != nil {
if err := ListPage(listBuckets{}, Filter{}, PipelineStats{}, "", connected, nil, true, false).Render(context.Background(), &buf); err != nil {
t.Fatalf("render: %v", err)
}
return buf.String()
+136 -3
View File
@@ -18,6 +18,7 @@ import (
"git.d-ma.be/mathias/tapir/internal/adapters/chat"
"git.d-ma.be/mathias/tapir/internal/adapters/store"
"git.d-ma.be/mathias/tapir/internal/config"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/errkind"
"git.d-ma.be/mathias/tapir/internal/uploads"
@@ -27,7 +28,7 @@ import (
// *uploads.Intake satisfies it. It is the whole of what the handlers know about
// validation, spooling and queueing.
type Uploader interface {
Accept(ctx context.Context, userID, title string, src io.Reader) (id string, err error)
Accept(ctx context.Context, userID, title string, lang domain.Language, src io.Reader) (id string, err error)
}
const (
@@ -35,6 +36,8 @@ const (
// and the title field; the cap itself is enforced on the file part by Intake.
uploadOverhead = 1 << 20
maxTitleRunes = 200
// maxLanguageBytes bounds what is read of the language field; a valid value is at most four bytes, so a longer one fails the parse.
maxLanguageBytes = 16
// uploadPollEvery is how often an unfinished upload re-asks for its status.
uploadPollEvery = "2s"
// serviceSilentAfter is how long a job may go without any signal (heartbeat or
@@ -81,13 +84,14 @@ func (a *App) handleUploadPost(w http.ResponseWriter, r *http.Request) {
if !ok {
return
}
r.Body = http.MaxBytesReader(w, r.Body, a.UploadLimits.MaxBytes+uploadOverhead)
r.Body = http.MaxBytesReader(w, a.idleBoundedBody(w, r.Body), a.UploadLimits.MaxBytes+uploadOverhead)
mr, err := r.MultipartReader()
if err != nil {
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "Choose an audio file to upload.")
return
}
title := ""
lang := domain.LangSwedish
for {
part, err := mr.NextPart()
if errors.Is(err, io.EOF) {
@@ -120,11 +124,29 @@ func (a *App) handleUploadPost(w http.ResponseWriter, r *http.Request) {
return
}
title = cleanTitle(string(b))
case "language":
// A closed set read before the file: anything else is refused before the file
// is spooled, so a crafted value never reaches an engine (tapir#61).
b, err := io.ReadAll(io.LimitReader(part, maxLanguageBytes+1))
if err != nil {
if a.abandoned(w, r) {
return
}
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "That upload could not be read. Please try again.")
return
}
l, ok := domain.ParseLanguage(string(b))
if !ok {
a.logger().Info("upload rejected", "reason", "language")
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "Choose a language from the list.")
return
}
lang = l
case "file":
if title == "" {
title = titleFromFilename(part.FileName())
}
id, err := a.Uploads.Accept(r.Context(), userID, title, part)
id, err := a.Uploads.Accept(r.Context(), userID, title, lang, part)
if err != nil {
a.uploadFailure(w, r, userID, err)
return
@@ -136,6 +158,53 @@ func (a *App) handleUploadPost(w http.ResponseWriter, r *http.Request) {
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "Choose an audio file to upload.")
}
// defaultUploadIdleTimeout is how long an upload body may deliver nothing before its
// connection read is cut off (#56). A stall this long is not a slow link but a dead or
// hostile one; a legitimate upload, however slow, keeps delivering bytes.
const defaultUploadIdleTimeout = 60 * time.Second
// uploadIdleTimeout is the idle bound this App applies: UploadIdleTimeout when set, else
// defaultUploadIdleTimeout. cmd/tapir leaves it unset, so production runs the default.
func (a *App) uploadIdleTimeout() time.Duration {
if a.UploadIdleTimeout <= 0 {
return defaultUploadIdleTimeout
}
return a.UploadIdleTimeout
}
// idleBoundedBody bounds the SILENCE in an upload body, not its total duration (#56): every
// read first pushes the connection's read deadline idle into the future, so a body that
// keeps moving is never cut, and one that stops fails its read with a timeout. That read
// error reaches Accept as a client read error, so Accept returns and its receive slot is
// released. A whole-request ReadTimeout would instead cut a large upload on a slow link.
//
// If the writer cannot set a read deadline (a wrapper without Unwrap, or a test recorder)
// the body is returned unbounded and that is logged: a control that silently does not
// apply is worse than one that says so.
func (a *App) idleBoundedBody(w http.ResponseWriter, body io.ReadCloser) io.ReadCloser {
idle := a.uploadIdleTimeout()
rc := http.NewResponseController(w)
if err := rc.SetReadDeadline(time.Now().Add(idle)); err != nil {
a.logger().Warn("upload idle bound unavailable", "err", err)
return body
}
return &idleDeadlineBody{ReadCloser: body, rc: rc, idle: idle}
}
type idleDeadlineBody struct {
io.ReadCloser
rc *http.ResponseController
idle time.Duration
}
func (b *idleDeadlineBody) Read(p []byte) (int, error) {
_ = b.rc.SetReadDeadline(time.Now().Add(b.idle))
// Never cleared here: net/http clears the read deadline itself before its background
// read once the handler has the body, and an armed deadline after an error also bounds
// net/http's drain of an unread remainder before it replies.
return b.ReadCloser.Read(p)
}
// abandoned reports, logs at INFO and answers 499, because the client has already gone: its request
// context is cancelled, so there is nobody to answer and rendering a page would run its
// database read on a dead context and log a false ERROR. A hang-up mid-upload is routine.
@@ -209,6 +278,32 @@ func (a *App) handleUploadStatus(w http.ResponseWriter, r *http.Request) {
a.render(w, r, uploadStatus(*u, a.Chat != nil, a.now()))
}
// handleUploadDelete removes one of the caller's finished recordings and returns to the
// list. Another user's id is a 404 like every other upload route; a recording still being
// transcribed is refused (409) rather than deleted under the worker (#37).
func (a *App) handleUploadDelete(w http.ResponseWriter, r *http.Request) {
userID, ok := a.currentUserID(w, r)
if !ok {
return
}
id := r.PathValue("id")
if !uuidShape.MatchString(id) {
http.NotFound(w, r)
return
}
switch err := a.Store.DeleteUpload(r.Context(), userID, id); {
case err == nil:
http.Redirect(w, r, "/uploads", http.StatusSeeOther)
case errors.Is(err, store.ErrUploadNotFound):
http.NotFound(w, r)
case errors.Is(err, store.ErrUploadInProgress):
a.renderUploadsPage(w, r, userID, http.StatusConflict,
"This recording is still being transcribed. You can delete it once it has finished.")
default:
a.serverError(w, r, "delete upload", err)
}
}
func (a *App) loadOwnedUpload(w http.ResponseWriter, r *http.Request) (*domain.UploadedTranscript, bool) {
userID, ok := a.currentUserID(w, r)
if !ok {
@@ -243,6 +338,44 @@ func titleFromFilename(name string) string {
return t
}
// cloudProvider names the cloud service that produced a transcript, or false when it was
// served locally (or by an unknown endpoint). "Local" is config.IsLocalModel: the same
// definition the start-up validation uses, not a second string match here (#52).
//
// The endpoint is one alias, except for the two-engine path (ADR-034), which writes
// "<base> + <alt>, reconciled by <model>" (ensemble.go). That shape is taken apart exactly
// and each alias is judged whole, so a composite of local models is local, and anything
// else (a cloud alias, one with no provider prefix, one with a space or a connector word in
// it) is not. A string that is not that shape is one alias.
func cloudProvider(endpoint string) (string, bool) {
endpoint = strings.TrimSpace(endpoint)
if endpoint == "" {
return "", false
}
aliases := []string{endpoint}
if engines, reconciler, ok := strings.Cut(endpoint, ", reconciled by "); ok {
aliases = append(strings.Split(engines, " + "), reconciler)
}
for _, a := range aliases {
if a = strings.TrimSpace(a); !config.IsLocalModel(a) {
provider, _, _ := strings.Cut(a, "/")
if provider = strings.TrimSpace(provider); provider == "" {
provider = a
}
return provider, true
}
}
return "", false
}
// languageLabel is how the page names the language a recording was transcribed as.
func languageLabel(l domain.Language) string {
if l == domain.LangAuto {
return "Other (detected by the engine)"
}
return "Swedish"
}
// hasNote reports whether any phase carries a note. A note is the only warning for a
// recording that is shorter than its file claimed, so a record that has one is shown open.
func hasNote(steps []domain.ProgressStep) bool {
+31
View File
@@ -24,6 +24,14 @@ templ UploadsPage(v uploadsView) {
}
<form method="post" action="/uploads" enctype="multipart/form-data">
<p><label>Title (optional) <input type="text" name="title" maxlength="200"/></label></p>
<p>
<label>Language
<select name="language">
<option value="sv" selected>Swedish</option>
<option value="auto">Other (detected by the engine)</option>
</select>
</label>
</p>
<p><input type="file" name="file" accept="audio/*,.m4a,.mp3,.wav,.opus,.ogg,.mp4" required/></p>
<p><button type="submit">Upload and transcribe</button></p>
</form>
@@ -49,6 +57,7 @@ templ UploadPage(u domain.UploadedTranscript, chatEnabled bool, now time.Time) {
@Layout("Tapir — " + u.Title) {
<article class="account">
<h1>{ u.Title }</h1>
<p class="muted">Language: { languageLabel(u.Language) }</p>
@uploadStatus(u, chatEnabled, now)
<p><a href="/uploads">All recordings</a></p>
</article>
@@ -66,6 +75,12 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
Transcribed { uploads.FormatDuration(time.Duration(u.AudioSeconds * float64(time.Second))) }
of audio by <span class="chip">{ u.Endpoint }</span>.
</p>
if provider, cloud := cloudProvider(u.Endpoint); cloud {
<p class="cloud-note">
This recording's audio left the homelab: it was transcribed by a cloud service ({ provider })
because the local engine could not.
</p>
}
@reconciliationNote(u.Reconciliation)
if len(u.Progress) > 0 {
<details class="what-was-done" if hasNote(u.Progress) { open }>
@@ -80,6 +95,7 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
<summary>Transcript</summary>
<pre class="transcript">{ u.Content }</pre>
</details>
@deleteRecording(u.ID)
</div>
case domain.UploadFailed:
<div class="upload-status upload-failed flash flash-error" role="alert">
@@ -91,6 +107,7 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
@progressSteps(u.Progress, now, false)
</details>
}
@deleteRecording(u.ID)
</div>
default:
<div
@@ -116,6 +133,20 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
}
}
// deleteRecording is the confirm-then-delete control, shown only once a recording has
// finished (done or failed): deleting one that is still being processed is refused.
templ deleteRecording(id string) {
<details class="confirm-delete">
<summary class="btn-danger">Delete this recording…</summary>
<div class="confirm-body">
<p>This permanently deletes the recording and its transcript. Are you sure?</p>
<form method="post" action={ templ.SafeURL("/uploads/" + id + "/delete") }>
<button type="submit" class="btn-danger">Yes, permanently delete it</button>
</form>
</div>
</details>
}
// UploadChatPage is the no-JS full page for chatting with an uploaded transcript:
// the same chat panel as a video's (ADR-027), scoped to this recording.
templ UploadChatPage(u domain.UploadedTranscript, v chatView) {
+140
View File
@@ -0,0 +1,140 @@
package web_test
// A transcript produced by the cloud fallback means this recording's audio left the
// homelab, which the reader is told in plain words (#52). "Local" is config.IsLocalModel:
// one definition, not a second string match in the view.
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func setEndpoint(t *testing.T, id, endpoint string) {
t.Helper()
_, err := rawPool(t).Exec(context.Background(), `UPDATE uploaded_transcripts SET endpoint = $2 WHERE id = $1`, id, endpoint)
require.NoError(t, err)
}
func TestUploadStatus_ACloudServedTranscriptSaysTheAudioLeftTheHomelab(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
setEndpoint(t, id, "berget/kb-whisper")
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.Contains(t, b, "left the homelab")
assert.Contains(t, b, "cloud service (berget)")
}
func TestUploadStatus_ALocallyServedTranscriptCarriesNoCloudNote(t *testing.T) {
for _, endpoint := range []string{"iguana/kb-whisper", "koala/whisper", ""} {
t.Run(endpoint, func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
setEndpoint(t, id, endpoint)
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.NotContains(t, b, "left the homelab")
})
}
}
func TestUploadStatus_AnEndpointThatOnlyLooksLocalIsStillNamedAsCloud(t *testing.T) {
// The prefix alone must not decide it: IsLocalModel refuses path tricks.
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
setEndpoint(t, id, "iguana/../berget/kb-whisper")
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.Contains(t, b, "left the homelab")
}
// The two-engine path (ADR-034) writes one composite endpoint string. Every name in it is
// a local model, so the audio never left the homelab and the note must not say it did.
func TestUploadStatus_AReconciledAllLocalEndpointCarriesNoCloudNote(t *testing.T) {
for _, endpoint := range []string{
"iguana/kb-whisper + koala/whisper, reconciled by iguana/gemma4-26b",
"iguana/kb-whisper + koala/whisper, reconciled by iguana/gemma4-26b\n",
} {
t.Run(endpoint, func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
setEndpoint(t, id, endpoint)
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.NotContains(t, b, "left the homelab")
})
}
}
func TestUploadStatus_AReconciledEndpointNamingACloudEngineStillNamesTheCloudProvider(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
setEndpoint(t, id, "iguana/kb-whisper + berget/kb-whisper, reconciled by iguana/gemma4-26b")
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.Contains(t, b, "cloud service (berget)")
}
// An alias without a provider prefix is not a local model, wherever it sits in the
// composite: it must never be dropped from the judgement (PR #58 re-review).
func TestUploadStatus_AReconciledEndpointWithAnUnprefixedAliasIsNotCalledLocal(t *testing.T) {
for _, endpoint := range []string{
"iguana/kb-whisper + whisper-1, reconciled by iguana/gemma4-26b",
"iguana/kb-whisper + whisper 1, reconciled by iguana/gemma4-26b",
"iguana/kb-whisper + berget / x, reconciled by iguana/gemma4-26b",
} {
t.Run(endpoint, func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
setEndpoint(t, id, endpoint)
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.Contains(t, b, "left the homelab")
assert.NotContains(t, b, "cloud service ()", "the note names a provider")
})
}
}
// "by", "reconciled" and "+" are connectors only inside the composite format. As a whole
// endpoint, or as an alias within the composite, they are model names and are judged.
func TestUploadStatus_AConnectorWordUsedAsAModelNameIsStillJudged(t *testing.T) {
for _, endpoint := range []string{
"by",
"reconciled",
"+",
"iguana/kb-whisper + by, reconciled by iguana/gemma4-26b",
"iguana/kb-whisper + reconciled, reconciled by iguana/gemma4-26b",
"iguana/kb-whisper + +, reconciled by iguana/gemma4-26b",
"/x", // no provider at all: the note must still name something
} {
t.Run(endpoint, func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
setEndpoint(t, id, endpoint)
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.Contains(t, b, "left the homelab")
assert.NotContains(t, b, "cloud service ()")
})
}
}
func TestUploadStatus_AReconcilerThatIsNotLocalNamesTheCloudProvider(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
setEndpoint(t, id, "iguana/kb-whisper + koala/whisper, reconciled by berget/mistral-large")
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.Contains(t, b, "cloud service (berget)")
}
@@ -0,0 +1,85 @@
package web_test
// Deleting one recording (#37). Owner-scoped like every other upload route: another user's
// id is a 404 and the row survives; an upload still being processed is refused.
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func deleteReq(id string) *http.Request {
return httptest.NewRequest(http.MethodPost, "/uploads/"+id+"/delete", nil)
}
func uploadExists(t *testing.T, id string) bool {
t.Helper()
var n int
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT count(*) FROM uploaded_transcripts WHERE id = $1`, id).Scan(&n))
return n == 1
}
func TestUploadDelete_AnOwnerRemovesAFinishedOrFailedRecordingAndLandsOnTheList(t *testing.T) {
for _, status := range []string{"done", "failed"} {
t.Run(status, func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "Roadmap sync", status, "transcription_failed")
rec := do(t, app, deleteReq(id))
assert.Equal(t, http.StatusSeeOther, rec.Code)
assert.Equal(t, "/uploads", rec.Header().Get("Location"))
assert.False(t, uploadExists(t, id))
})
}
}
func TestUploadDelete_AnotherUsersRecordingIs404AndSurvives(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
theirs := seedUpload(t, otherUserID, "Their private meeting", "done", meetingText)
rec := do(t, app, deleteReq(theirs))
assert.Equal(t, http.StatusNotFound, rec.Code)
assert.True(t, uploadExists(t, theirs), "a non-owner's delete must not remove the row")
}
func TestUploadDelete_ARecordingStillBeingTranscribedIsRefusedWith409AndKept(t *testing.T) {
for _, status := range []string{"queued", "transcribing"} {
t.Run(status, func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", status, "")
rec := do(t, app, deleteReq(id))
assert.Equal(t, http.StatusConflict, rec.Code)
assert.Contains(t, body(t, rec), "still being transcribed")
assert.True(t, uploadExists(t, id))
})
}
}
func TestUploadDelete_ANonIDIsA404NotAServerError(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
assert.Equal(t, http.StatusNotFound, do(t, app, deleteReq("not-a-uuid")).Code)
}
func TestUploadPage_OffersDeleteOnlyForAFinishedOrFailedRecording(t *testing.T) {
for status, want := range map[string]bool{"done": true, "failed": true, "queued": false, "transcribing": false} {
t.Run(status, func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", status, "transcription_failed")
b := body(t, do(t, app, get("/uploads/"+id)))
assert.Equal(t, want, strings.Contains(b, `action="/uploads/`+id+`/delete"`))
})
}
}
@@ -14,6 +14,8 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
)
// A client that hangs up in the middle of an upload is routine, not a server fault. The
@@ -85,7 +87,7 @@ func TestUploadPost_AClientThatHangsUpAnywhereInTheBodyIsLoggedAtInfoAndNeverAsA
// context is cancelled, then Accept fails the way a read from a dead connection does.
type goneMidAccept struct{ cancel context.CancelFunc }
func (g goneMidAccept) Accept(context.Context, string, string, io.Reader) (string, error) {
func (g goneMidAccept) Accept(context.Context, string, string, domain.Language, io.Reader) (string, error) {
g.cancel()
return "", fmt.Errorf("receive: %w", io.ErrClosedPipe)
}
+2 -1
View File
@@ -16,6 +16,7 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/uploads"
"git.d-ma.be/mathias/tapir/internal/web"
)
@@ -132,7 +133,7 @@ func TestUploadPost_AcceptedAudioQueuesARowAndRedirectsToIt(t *testing.T) {
assert.Equal(t, "Sprint review", title)
assert.Equal(t, "/uploads/"+id, rec.Header().Get("Location"))
require.Len(t, sub.jobs, 1)
assert.Equal(t, uploads.Job{UserID: userID, UploadID: id, Path: sub.jobs[0].Path, Duration: audioOK.Duration}, sub.jobs[0])
assert.Equal(t, uploads.Job{UserID: userID, UploadID: id, Path: sub.jobs[0].Path, Language: domain.LangSwedish, Duration: audioOK.Duration}, sub.jobs[0])
spooled, err := os.ReadFile(sub.jobs[0].Path)
require.NoError(t, err)
assert.Equal(t, "synthetic audio bytes", string(spooled))
+161
View File
@@ -0,0 +1,161 @@
package web_test
// #56: a request body that stops arriving must not hold a receive slot forever. The bound is a
// read deadline on the connection, re-armed on every read, so it only exercises over a real
// socket and through the production handler stack (metrics.HTTPMiddleware wraps the writer).
import (
"bufio"
"bytes"
"errors"
"fmt"
"log/slog"
"mime/multipart"
"net"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/metrics"
"git.d-ma.be/mathias/tapir/internal/uploads"
"git.d-ma.be/mathias/tapir/internal/web"
)
func multipartBody(t *testing.T, content []byte) (body []byte, contentType string) {
t.Helper()
var buf bytes.Buffer
mw := multipart.NewWriter(&buf)
fw, err := mw.CreateFormFile("file", "a.m4a")
require.NoError(t, err)
_, err = fw.Write(content)
require.NoError(t, err)
require.NoError(t, mw.Close())
return buf.Bytes(), mw.FormDataContentType()
}
// rawUpload opens a connection and sends the request headers; the caller feeds the body.
func rawUpload(t *testing.T, addr, contentType string, length int) net.Conn {
t.Helper()
c, err := net.Dial("tcp", addr)
require.NoError(t, err)
t.Cleanup(func() { _ = c.Close() })
_, err = fmt.Fprintf(c, "POST /uploads HTTP/1.1\r\nHost: x\r\nContent-Type: %s\r\nContent-Length: %d\r\n\r\n", contentType, length)
require.NoError(t, err)
return c
}
// serverEnded reports whether the server ended the request (answered it or closed the
// connection) before wait elapsed, rather than leaving the client waiting.
func serverEnded(t *testing.T, c net.Conn, wait time.Duration) bool {
t.Helper()
require.NoError(t, c.SetReadDeadline(time.Now().Add(wait)))
_, err := http.ReadResponse(bufio.NewReader(c), nil)
var ne net.Error
return !errors.As(err, &ne) || !ne.Timeout()
}
// readStatus returns the response status code, or 0 if none arrived before wait elapsed.
func readStatus(t *testing.T, c net.Conn, wait time.Duration) int {
t.Helper()
require.NoError(t, c.SetReadDeadline(time.Now().Add(wait)))
resp, err := http.ReadResponse(bufio.NewReader(c), nil)
if err != nil {
return 0
}
_ = resp.Body.Close()
return resp.StatusCode
}
func newIdleBoundServer(t *testing.T, idle time.Duration) (addr string) {
t.Helper()
app, _ := newUploadApp(t, audioOK, 200<<20)
app.UploadIdleTimeout = idle
app.Uploads.(*uploads.Intake).MaxReceivesPerUser = 1
srv := httptest.NewServer(metrics.HTTPMiddleware(app.Router()))
t.Cleanup(srv.Close)
return strings.TrimPrefix(srv.URL, "http://")
}
func TestUploadPost_AConnectionReadDeadlineEndsABodyThatStallsAndReleasesItsReceiveSlot(t *testing.T) {
addr := newIdleBoundServer(t, 200*time.Millisecond)
body, ct := multipartBody(t, bytes.Repeat([]byte("a"), 8000))
stalled := rawUpload(t, addr, ct, len(body))
_, err := stalled.Write(body[:4000]) // half the body, then nothing more
require.NoError(t, err)
require.True(t, serverEnded(t, stalled, 3*time.Second),
"the server never ended the stalled request: it still holds its slot")
next := rawUpload(t, addr, ct, len(body))
_, err = next.Write(body)
require.NoError(t, err)
require.Equal(t, http.StatusSeeOther, readStatus(t, next, 3*time.Second),
"the same user's next upload gets the slot the stalled one held")
}
func TestUploadPost_TheReadDeadlineIsReArmedOnEveryReadSoASlowButMovingBodyIsAccepted(t *testing.T) {
addr := newIdleBoundServer(t, 300*time.Millisecond)
body, ct := multipartBody(t, bytes.Repeat([]byte("a"), 2000))
c := rawUpload(t, addr, ct, len(body))
// ~1.2 s in total, four times the idle bound, but never 300 ms without a byte.
const chunks = 20
step := len(body)/chunks + 1
for i := 0; i < len(body); i += step {
_, err := c.Write(body[i:min(i+step, len(body))])
require.NoError(t, err)
time.Sleep(60 * time.Millisecond)
}
require.Equal(t, http.StatusSeeOther, readStatus(t, c, 3*time.Second))
}
// cmd/tapir never sets UploadIdleTimeout, so the default is the value production runs.
// Shorter and legitimate uploads on a slow link are cut; much longer and the bound is moot.
func TestDefaultUploadIdleTimeoutIsSixtySeconds(t *testing.T) {
require.Equal(t, 60*time.Second, web.DefaultUploadIdleTimeout)
}
// The bound an App actually resolves to, not the constant: production's App leaves
// UploadIdleTimeout unset, so the zero value must resolve to exactly 60 s, and a set value
// must be used as given.
func TestAppUploadIdleTimeout_ResolvesUnsetToSixtySecondsAndHonoursASetValue(t *testing.T) {
require.Equal(t, 60*time.Second, (&web.App{}).UploadIdleTimeoutForTest())
require.Equal(t, 60*time.Second, (&web.App{UploadIdleTimeout: -time.Second}).UploadIdleTimeoutForTest())
require.Equal(t, 300*time.Millisecond, (&web.App{UploadIdleTimeout: 300 * time.Millisecond}).UploadIdleTimeoutForTest())
}
// With UploadIdleTimeout unset, a body with 100 ms gaps between reads is not cut (a resolved
// bound in milliseconds or nanoseconds would cut it). The exact value is pinned above.
func TestUploadPost_AnUnsetIdleTimeoutDoesNotCutABodyWithSubSecondGaps(t *testing.T) {
addr := newIdleBoundServer(t, 0)
body, ct := multipartBody(t, bytes.Repeat([]byte("a"), 1000))
c := rawUpload(t, addr, ct, len(body))
const chunks = 5
step := len(body)/chunks + 1
for i := 0; i < len(body); i += step {
time.Sleep(100 * time.Millisecond)
_, err := c.Write(body[i:min(i+step, len(body))])
require.NoError(t, err)
}
require.Equal(t, http.StatusSeeOther, readStatus(t, c, 3*time.Second))
}
// A writer that cannot set a read deadline leaves the body unbounded; that must be said in
// the log, not happen silently. The in-process recorder is such a writer.
func TestUploadPost_AWriterWithoutReadDeadlineSupportLogsThatTheIdleBoundIsUnavailable(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
buf := &lockedBuf{}
app.Log = slog.New(slog.NewTextHandler(buf, nil))
do(t, app, uploadReq(t, "t", "a.m4a", []byte("x")))
require.Contains(t, buf.String(), "upload idle bound unavailable")
}
@@ -0,0 +1,151 @@
package web_test
// The language of a recording is chosen on the upload form (tapir#61). These drive the real
// Router, Intake and Postgres store; the submitter is the process boundary and records the
// job the worker would receive.
import (
"bytes"
"context"
"mime/multipart"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
)
// langReq builds an upload whose parts come in the order given: "title", "language" and
// "file" (the file part always carries the same synthetic bytes).
func langReq(t *testing.T, order []string, language string) *http.Request {
t.Helper()
var buf bytes.Buffer
mw := multipart.NewWriter(&buf)
for _, part := range order {
switch part {
case "title":
require.NoError(t, mw.WriteField("title", "m"))
case "language":
require.NoError(t, mw.WriteField("language", language))
case "file":
fw, err := mw.CreateFormFile("file", "m.wav")
require.NoError(t, err)
_, err = fw.Write([]byte("RIFF....WAVE"))
require.NoError(t, err)
}
}
require.NoError(t, mw.Close())
req := httptest.NewRequest(http.MethodPost, "/uploads", &buf)
req.Header.Set("Content-Type", mw.FormDataContentType())
return req
}
func rowLanguage(t *testing.T, location string) domain.Language {
t.Helper()
id := strings.TrimPrefix(location, "/uploads/")
got, err := newStore(t).GetUpload(context.Background(), userID, id)
require.NoError(t, err)
return got.Language
}
func TestUploadPost_TheLanguageFieldReachesTheJobAndTheRow(t *testing.T) {
for _, tt := range []struct {
name string
language string
want domain.Language
}{
{"other", "auto", domain.LangAuto},
{"swedish", "sv", domain.LangSwedish},
} {
t.Run(tt.name, func(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
rec := do(t, app, langReq(t, []string{"title", "language", "file"}, tt.language))
require.Equal(t, http.StatusSeeOther, rec.Code)
require.Len(t, sub.jobs, 1)
assert.Equal(t, tt.want, sub.jobs[0].Language)
assert.Equal(t, tt.want, rowLanguage(t, rec.Header().Get("Location")))
})
}
}
func TestUploadPost_NoLanguageFieldIsSwedishAsBefore(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
rec := do(t, app, langReq(t, []string{"title", "file"}, ""))
require.Equal(t, http.StatusSeeOther, rec.Code)
require.Len(t, sub.jobs, 1)
assert.Equal(t, domain.LangSwedish, sub.jobs[0].Language)
assert.Equal(t, domain.LangSwedish, rowLanguage(t, rec.Header().Get("Location")))
}
// A part after the file is never read (the file part ends the stream), so a client that
// orders its fields wrongly gets Swedish, the old behaviour, not a guess.
func TestUploadPost_ALanguageFieldAfterTheFileIsNotReadAndTheUploadIsSwedish(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
rec := do(t, app, langReq(t, []string{"title", "file", "language"}, "auto"))
require.Equal(t, http.StatusSeeOther, rec.Code)
require.Len(t, sub.jobs, 1)
assert.Equal(t, domain.LangSwedish, sub.jobs[0].Language)
}
func TestUploadPost_ALanguageOutsideTheClosedSetIsRefusedBeforeAnythingIsSpooledOrStored(t *testing.T) {
for _, bad := range []string{"en", "SV", " sv", "sv,auto", "../auto", strings.Repeat("a", 5000)} {
t.Run(bad[:min(len(bad), 12)], func(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
rec := do(t, app, langReq(t, []string{"title", "language", "file"}, bad))
assert.Equal(t, http.StatusBadRequest, rec.Code)
assert.Empty(t, sub.jobs, "a refused post must not enqueue transcription")
rows, err := newStore(t).ListUploads(context.Background(), userID)
require.NoError(t, err)
assert.Empty(t, rows, "and must leave no row")
assert.Contains(t, body(t, rec), "Choose a language from the list.")
})
}
}
func TestUploadPage_TheFormOffersSwedishPreselectedAndOtherBeforeTheFilePart(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
b := body(t, do(t, app, get("/uploads")))
sel := strings.Index(b, `name="language"`)
file := strings.Index(b, `name="file"`)
require.True(t, sel >= 0, "the form has a language field")
assert.True(t, sel < file, "the field must come before the file part: the stream ends at the file")
assert.Regexp(t, `<option value="sv" selected`, b, "Swedish is preselected: today's behaviour")
assert.Contains(t, b, `<option value="auto"`)
}
func TestUploadPage_NamesTheLanguageInEveryState(t *testing.T) {
for _, status := range []string{"queued", "transcribing", "done", "failed"} {
for _, tt := range []struct {
language domain.Language
want string
}{
{domain.LangSwedish, "Swedish"},
{domain.LangAuto, "Other (detected by the engine)"},
} {
t.Run(status+"/"+string(tt.language), func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", status, "Synthetic transcript text.")
_, err := rawPool(t).Exec(context.Background(), `UPDATE uploaded_transcripts SET language = $2 WHERE id = $1`, id, string(tt.language))
require.NoError(t, err)
b := body(t, do(t, app, get("/uploads/"+id)))
assert.Contains(t, b, "Language: "+tt.want)
})
}
}
}
+3 -1
View File
@@ -13,6 +13,8 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
)
// Error paths of the upload routes log a kind, never the error's text, which can carry
@@ -37,7 +39,7 @@ func (l *lockedBuf) String() string {
type failingUploader struct{ err error }
func (f failingUploader) Accept(context.Context, string, string, io.Reader) (string, error) {
func (f failingUploader) Accept(context.Context, string, string, domain.Language, io.Reader) (string, error) {
return "", f.err
}
+19
View File
@@ -0,0 +1,19 @@
package web_test
// /uploads was mounted but nothing linked to it (#37). The shared Layout has no access to
// the App, so the entry point is the home list, shown only when uploads are mounted.
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestHomeList_LinksToRecordingsOnlyWhenUploadsAreMounted(t *testing.T) {
with, _ := newUploadApp(t, audioOK, 200<<20)
without := newApp(t)
assert.Contains(t, body(t, do(t, with, get("/"))), `href="/uploads"`)
assert.NotContains(t, body(t, do(t, without, get("/"))), `href="/uploads"`,
"a link to a route that is not mounted would be a dead end")
}
File diff suppressed because it is too large Load Diff
+4 -1
View File
@@ -107,9 +107,12 @@ templ flashBanner(code string) {
// #summary-list region; a non-HTMX request renders the whole page. flash carries
// a one-shot notification (e.g. "connected", "registered") surfaced on arrival
// after a POST→redirect.
templ ListPage(b listBuckets, f Filter, stats PipelineStats, flash string, hasConnected bool, channels []string, autoSummarize bool) {
templ ListPage(b listBuckets, f Filter, stats PipelineStats, flash string, hasConnected bool, channels []string, autoSummarize bool, uploadsEnabled bool) {
@Layout("Tapir — Summaries") {
@flashBanner(flash)
if uploadsEnabled {
<p class="muted"><a href="/uploads">Recordings</a>: upload audio and get a private transcript.</p>
}
if hasConnected {
@pasteForm()
}
File diff suppressed because it is too large Load Diff
+10
View File
@@ -122,6 +122,16 @@ var scenarioCoverage = map[string]string{
"I can ask about my recording and get a grounded answer": "TestUploadChatPost_SendsTheOwnersTranscriptWithTheUploadBudget",
"A long recording is not cut to its first third": "TestAnswerMaxCharsOverridesTheServiceCap",
"A transcript made by the cloud fallback says the audio left the homelab": "TestUploadStatus_ACloudServedTranscriptSaysTheAudioLeftTheHomelab",
"I choose the language of a recording": "TestUploadPost_TheLanguageFieldReachesTheJobAndTheRow",
"Swedish stays the default": "TestUploadPost_NoLanguageFieldIsSwedishAsBefore",
"A language outside the list is refused": "TestUploadPost_ALanguageOutsideTheClosedSetIsRefusedBeforeAnythingIsSpooledOrStored",
"A recording in another language never reaches the Swedish-only engine": "TestEnsemble_AnAutoLanguageUploadNeverReachesTheBaseEngineAndIsNotReconciled",
"I can delete a finished recording": "TestUploadDelete_AnOwnerRemovesAFinishedOrFailedRecordingAndLandsOnTheList",
"A recording still being transcribed cannot be deleted": "TestUploadDelete_ARecordingStillBeingTranscribedIsRefusedWith409AndKept",
"The home list links to my recordings when uploads are on": "TestHomeList_LinksToRecordingsOnlyWhenUploadsAreMounted",
// audio_upload.feature, ADR-034
"Two engines transcribe the audio and their disagreements are reconciled": "TestEnsemble_BothEnginesRunInOrderAndTheirDisagreementIsReconciled",
"Trivial disagreements never reach the model": "TestReconcile_TrivialDisputesAreResolvedWithoutAskingTheModel",
+143
View File
@@ -0,0 +1,143 @@
package lintgate
// Observes the exit status of `task lint` (the lint step of `task check`), not the lint
// rules themselves. It runs the repo's real Taskfile in a scratch module so a change to the
// lint command that stops it failing on findings (#41) turns this red.
import (
"bytes"
"errors"
"os"
"os/exec"
"path/filepath"
"regexp"
"strings"
"testing"
)
const scratchWithFinding = `package main
import "os"
func main() { os.Remove("x") }
`
func scratchModule(t *testing.T) string {
t.Helper()
taskfile, err := os.ReadFile(filepath.Join("..", "..", "Taskfile.yml"))
if err != nil {
t.Fatalf("read Taskfile.yml: %v", err)
}
dir := t.TempDir()
for name, body := range map[string]string{
"Taskfile.yml": string(taskfile),
"go.mod": "module scratch\n\ngo 1.21\n",
"main.go": scratchWithFinding,
} {
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil {
t.Fatalf("write %s: %v", name, err)
}
}
return dir
}
// ansi matches the colour escape sequences task prints when it thinks it has a terminal.
var ansi = regexp.MustCompile("\x1b\\[[0-9;]*m")
// hasSkipNotice reports whether the skip notice was printed as output. Task echoes the command
// line first, and that line contains the same words, so a substring match would be vacuous.
func hasSkipNotice(out string) bool {
for _, line := range strings.Split(ansi.ReplaceAllString(out, ""), "\n") {
if strings.HasPrefix(line, "golangci-lint not installed") {
return true
}
}
return false
}
// runLint runs `task lint` in dir with the given PATH and returns its combined output and exit code.
func runLint(t *testing.T, dir, path string) (string, int) {
t.Helper()
return runTask(t, dir, path, "lint")
}
func runTask(t *testing.T, dir, path string, args ...string) (string, int) {
t.Helper()
taskBin, err := exec.LookPath("task")
if err != nil {
t.Skip("task not installed")
}
cmd := exec.Command(taskBin, args...)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "PATH="+path)
var out bytes.Buffer
cmd.Stdout, cmd.Stderr = &out, &out
err = cmd.Run()
var exitErr *exec.ExitError
switch {
case err == nil:
return out.String(), 0
case errors.As(err, &exitErr):
return out.String(), exitErr.ExitCode()
default:
t.Fatalf("run task lint: %v", err)
return "", -1
}
}
func TestTaskLint_ExitsNonZeroWhenGolangciLintReportsAFinding(t *testing.T) {
if _, err := exec.LookPath("golangci-lint"); err != nil {
t.Skip("golangci-lint not installed")
}
out, code := runLint(t, scratchModule(t), os.Getenv("PATH"))
if code == 0 {
t.Fatalf("task lint exited 0 on a module with an unchecked error; output:\n%s", out)
}
if hasSkipNotice(out) {
t.Fatalf("task lint blamed a missing binary for a lint failure; output:\n%s", out)
}
if !strings.Contains(out, "errcheck") {
t.Fatalf("a non-zero exit without the finding means the linter crashed, not that it found something; output:\n%s", out)
}
}
// The lint step only gates anything if `task check` runs it.
func TestTaskCheck_RunsTheLintStep(t *testing.T) {
out, code := runTask(t, scratchModule(t), os.Getenv("PATH"), "--dry", "check")
if code != 0 {
t.Fatalf("task --dry check exited %d; output:\n%s", code, out)
}
if !strings.Contains(out, "task: [lint]") {
t.Fatalf("task check no longer runs the lint step; output:\n%s", out)
}
}
func TestTaskLint_SkipsWithANoticeWhenGolangciLintIsAbsent(t *testing.T) {
taskBin, err := exec.LookPath("task")
if err != nil {
t.Skip("task not installed")
}
bin := t.TempDir() // a PATH holding task and nothing else
if err := os.Symlink(taskBin, filepath.Join(bin, "task")); err != nil {
t.Fatalf("symlink task: %v", err)
}
out, code := runLint(t, scratchModule(t), bin)
if code != 0 {
t.Fatalf("task lint exited %d without golangci-lint on PATH; output:\n%s", code, out)
}
if !hasSkipNotice(out) {
t.Fatalf("expected the skip notice; output:\n%s", out)
}
}
// CI runs task with colour on, so the notice line starts with an escape sequence; a bare
// prefix match missed it there while passing on a terminal-less local run.
func TestHasSkipNotice_SeesTheNoticeBehindColourCodes(t *testing.T) {
out := "\x1b[32mtask: [lint] ...\n\x1b[0mgolangci-lint not installed — skipping (CI installs it)\n"
if !hasSkipNotice(out) {
t.Fatalf("notice not found in coloured output: %q", out)
}
if hasSkipNotice("\x1b[32mtask: [lint] if command -v golangci-lint ...; echo \"golangci-lint not installed\"\n") {
t.Fatal("the echoed command line must not count as the notice")
}
}