Compare commits

..
Author SHA1 Message Date
mathiasandClaude Sonnet 5.5 12d1365607 test(web): the 499 test cancels the client mid-Accept, as a real hang-up does
CI / Lint / Test / Vet (pull_request) Successful in 27s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:47:13 +02:00
mathiasandClaude Sonnet 5.5 9d7afb2bcb fix(web): the title-part hang-up is handled like the rest, and an abandoned request is answered 499, not an implicit 200 (fourth review)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:47:13 +02:00
mathiasandClaude Sonnet 5.5 1266e18038 fix(uploads): every worker log carries a kind, including store and temp-file errors (fourth review)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:47:13 +02:00
mathiasandClaude Sonnet 5.5 51ee5d48e5 feat(errkind): a database error is reduced to its SQLSTATE, never its message, which can quote a transcript value (fourth review)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:47:13 +02:00
mathiasandClaude Sonnet 5.5 cff5cd2f0a test: fourth review -- a hang-up anywhere in the body, an abandoned request is not an implicit 200, a database error is logged as its SQLSTATE (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:46:07 +02:00
mathiasandClaude Sonnet 5.5 fc8ac376fa test: fourth review -- a hang-up anywhere in the body, an abandoned request is not an implicit 200, a database error is logged as its SQLSTATE (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:46:07 +02:00
mathiasandClaude Sonnet 5.5 73fe203d62 test: fourth review -- a hang-up anywhere in the body, an abandoned request is not an implicit 200, a database error is logged as its SQLSTATE (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:46:07 +02:00
mathiasandClaude Sonnet 5.5 fbc3ee61c1 test: fourth review -- a hang-up anywhere in the body, an abandoned request is not an implicit 200, a database error is logged as its SQLSTATE (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:46:07 +02:00
mathiasandClaude Sonnet 5.5 e97d4400b0 test: fourth review -- a hang-up anywhere in the body, an abandoned request is not an implicit 200, a database error is logged as its SQLSTATE (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:46:07 +02:00
mathiasandClaude Sonnet 5.5 6e50a97b6d docs(adr): third review -- the hang-up path, the client-read marker, the note kept visible
CI / Lint / Test / Vet (pull_request) Successful in 26s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:36:42 +02:00
mathiasandClaude Sonnet 5.5 815ee98d4c test(web): the title input that proves a bidi character is dropped, not replaced
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:36:42 +02:00
mathiasandClaude Sonnet 5.5 067620d06a chore(web): regenerate uploads_templ.go
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:36:42 +02:00
mathiasandClaude Sonnet 5.5 91e475807c fix(web): a finished upload's record opens when a step carries a note (third review)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:36:42 +02:00
mathiasandClaude Sonnet 5.5 6d044ba403 fix(web): a client that hangs up is logged at INFO and not answered; the 400 follows the marker, not io.ErrUnexpectedEOF; hasNote (third review)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:36:42 +02:00
mathiasandClaude Sonnet 5.5 632dbeba0e feat(uploads): ErrClientRead marks an error from reading the client's body (third review)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:36:42 +02:00
mathiasandClaude Sonnet 5.5 fb5dc9d461 test: third review -- a real disconnect is not an error, a DB drop is not the client's, a note stays visible, every bidi character and the 80% boundary pinned (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:35:29 +02:00
mathiasandClaude Sonnet 5.5 272a5db75f test: third review -- a real disconnect is not an error, a DB drop is not the client's, a note stays visible, every bidi character and the 80% boundary pinned (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:35:29 +02:00
mathiasandClaude Sonnet 5.5 1b0fc668bc test: third review -- a real disconnect is not an error, a DB drop is not the client's, a note stays visible, every bidi character and the 80% boundary pinned (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:35:29 +02:00
mathiasandClaude Sonnet 5.5 375da83c5e test: third review -- a real disconnect is not an error, a DB drop is not the client's, a note stays visible, every bidi character and the 80% boundary pinned (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:35:29 +02:00
mathiasandClaude Sonnet 5.5 af4e9c8632 test: third review -- a real disconnect is not an error, a DB drop is not the client's, a note stays visible, every bidi character and the 80% boundary pinned (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:35:29 +02:00
mathiasandClaude Sonnet 5.5 04a3bc261f test: third review -- a real disconnect is not an error, a DB drop is not the client's, a note stays visible, every bidi character and the 80% boundary pinned (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:35:29 +02:00
mathiasandClaude Sonnet 5.5 08ffc2ff62 test(web): Go escapes for the bidirectional characters, which staticcheck ST1018 requires -- task check does not gate lint (#41)
CI / Lint / Test / Vet (pull_request) Successful in 25s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:23:50 +02:00
mathiasandClaude Sonnet 5.5 e70e56a531 test(web): the render tool shows a step note
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:23:01 +02:00
mathiasandClaude Sonnet 5.5 b46a0dd0c9 docs(adr): second review -- the note instead of a refusal, room before spooling, the callable chat model, and the image that was actually built (ADR-035 decision 8)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:22:41 +02:00
mathiasandClaude Sonnet 5.5 fe6c0fae3a test(acceptance): follow the renamed scenario
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:22:41 +02:00
mathiasandClaude Sonnet 5.5 2a616c6ae1 docs(use-cases): a short recording is noted, not refused
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:22:41 +02:00
mathiasandClaude Sonnet 5.5 bcec4f6e86 docs(homelab): engine timeout 20m
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:22:41 +02:00
mathiasandClaude Sonnet 5.5 be20adc76b chore(fixtures): an obviously invented context phrase (second review F10 residual)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:22:41 +02:00
mathiasandClaude Sonnet 5.5 1b97851a39 chore(fixtures): an obviously invented context phrase (second review F10 residual)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:22:41 +02:00
mathiasandClaude Sonnet 5.5 b59b1a22dd test(web): joiners and variation selectors survive the title cleaning
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:22:41 +02:00
mathiasandClaude Sonnet 5.5 22c0317c71 fix(web): a body cut off mid-file is a 400; bidirectional controls are stripped from a title (second review)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:22:41 +02:00
mathiasandClaude Sonnet 5.5 9a2f24b3c9 fix(uploads): refuse a full queue before spooling, probing or creating a row (second review)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:22:41 +02:00
mathiasandClaude Sonnet 5.5 3f5b9345a0 feat(uploads): HasRoom -- say before spooling whether a submit would be accepted (second review)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:22:41 +02:00
mathiasandClaude Sonnet 5.5 16db156406 fix(config): the engine timeout defaults to 20 minutes so two wedged engines leave the job time for a fallback (second review)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:22:41 +02:00
mathiasandClaude Sonnet 5.5 471425cd4c fix(cmd): the configured upload chat model is in the shared chat service's list, so it is no longer silently swapped (second review finding 2)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:22:41 +02:00
mathiasandClaude Sonnet 5.5 8f79132fcb test: second review -- the upload chat model outside the chain, room checked before spooling, a cut-off body is a 400, bidi controls in titles, the ensemble's deadline wiring, an honest test name (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:20:38 +02:00
mathiasandClaude Sonnet 5.5 b1f9d5f4b7 test: second review -- the upload chat model outside the chain, room checked before spooling, a cut-off body is a 400, bidi controls in titles, the ensemble's deadline wiring, an honest test name (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:20:38 +02:00
mathiasandClaude Sonnet 5.5 5bd2e3d3d0 test: second review -- the upload chat model outside the chain, room checked before spooling, a cut-off body is a 400, bidi controls in titles, the ensemble's deadline wiring, an honest test name (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:20:38 +02:00
mathiasandClaude Sonnet 5.5 b8a6e2f6f5 test: second review -- the upload chat model outside the chain, room checked before spooling, a cut-off body is a 400, bidi controls in titles, the ensemble's deadline wiring, an honest test name (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:20:38 +02:00
mathiasandClaude Sonnet 5.5 335ad0cd85 test: second review -- the upload chat model outside the chain, room checked before spooling, a cut-off body is a 400, bidi controls in titles, the ensemble's deadline wiring, an honest test name (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:20:38 +02:00
mathiasandClaude Sonnet 5.5 59bc373068 test: second review -- the upload chat model outside the chain, room checked before spooling, a cut-off body is a 400, bidi controls in titles, the ensemble's deadline wiring, an honest test name (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:20:38 +02:00
mathiasandClaude Sonnet 5.5 d505f64c99 test: second review -- the upload chat model outside the chain, room checked before spooling, a cut-off body is a 400, bidi controls in titles, the ensemble's deadline wiring, an honest test name (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:20:38 +02:00
mathiasandClaude Sonnet 5.5 dbe37e5756 feat(web): style for a step note
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:20:02 +02:00
mathiasandClaude Sonnet 5.5 0962b6f639 chore(web): regenerate uploads_templ.go
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:20:02 +02:00
mathiasandClaude Sonnet 5.5 9314454ab1 feat(web): show a step's note under it, escaped (second review finding 1)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:20:02 +02:00
mathiasandClaude Sonnet 5.5 9c8ac0f9fb fix(web): no message for the removed incomplete kind (second review finding 1)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:20:02 +02:00
mathiasandClaude Sonnet 5.5 f7284ccdc9 fix(uploads): a file much shorter than it claimed is transcribed and noted, not refused; the tolerances are named (second review finding 1)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:20:02 +02:00
mathiasandClaude Sonnet 5.5 4943c80dbd feat(progress): Annotate -- a note on a step, saved at once (second review finding 1)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:20:02 +02:00
mathiasandClaude Sonnet 5.5 8eca8f721b feat(domain): ProgressStep.Note; UploadErrIncomplete removed -- a short file is noted, not refused (second review finding 1)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:20:02 +02:00
mathiasandClaude Sonnet 5.5 406c0232c1 test: a shortfall is noted on the audio step, not refused; the 80%, two-minute and 30-second tolerances are pinned (red; second review finding 1)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:19:32 +02:00
mathiasandClaude Sonnet 5.5 ac530de91a test: a shortfall is noted on the audio step, not refused; the 80%, two-minute and 30-second tolerances are pinned (red; second review finding 1)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:19:32 +02:00
mathiasandClaude Sonnet 5.5 a90107b73b test: a shortfall is noted on the audio step, not refused; the 80%, two-minute and 30-second tolerances are pinned (red; second review finding 1)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:19:32 +02:00
mathiasandClaude Sonnet 5.5 0ef83f2ccb test: a shortfall is noted on the audio step, not refused; the 80%, two-minute and 30-second tolerances are pinned (red; second review finding 1)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:19:32 +02:00
mathiasandClaude Sonnet 5.5 71cd50aa88 fix(docker): install xz in the ffmpeg stage -- the golang image has none; found by building the image, which CI does not do on a pull request (second review)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 17:19:06 +02:00
mathiasandClaude Sonnet 5.5 aa54089230 docs(claude): errors are logged and shown as kinds, and what carries transcript text must be local (ADR-035)
CI / Lint / Test / Vet (pull_request) Successful in 25s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:50:56 +02:00
mathiasandClaude Sonnet 5.5 8749ffe6a9 docs(homelab): the engine timeout, a local chat model, ffmpeg in the image, and 31B with Low Power Mode off
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:50:56 +02:00
mathiasandClaude Sonnet 5.5 67879f85eb test(acceptance): map the hardening scenarios, and follow the two tests renamed in the review
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:50:43 +02:00
mathiasandClaude Sonnet 5.5 08b7654d06 docs(use-cases): scenarios for the review's hardening (ADR-035 decision 8) -- the coverage gate is red until mapped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:50:35 +02:00
mathiasandClaude Sonnet 5.5 f03b0213d0 docs(adr): record the owner's decisions on #34, #35 and ADR-034, and the adversarial review's hardening (ADR-035 decision 8)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:50:22 +02:00
mathiasandClaude Sonnet 5.5 cb22832413 feat(docker): static ffmpeg and ffprobe in the distroless image, pinned by version and sha256 (#34 option A)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:50:02 +02:00
mathiasandClaude Sonnet 5.5 5d4324c2a2 chore(fixtures): invented text in place of words that may come from the private recording; the research note no longer names the private file (adversarial review F10)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:49:54 +02:00
mathiasandClaude Sonnet 5.5 6596e83cd5 chore(fixtures): invented text in place of words that may come from the private recording; the research note no longer names the private file (adversarial review F10)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:49:54 +02:00
mathiasandClaude Sonnet 5.5 bc90e3aec7 chore(fixtures): invented text in place of words that may come from the private recording; the research note no longer names the private file (adversarial review F10)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:49:54 +02:00
mathiasandClaude Sonnet 5.5 1a57489d04 chore(fixtures): invented text in place of words that may come from the private recording; the research note no longer names the private file (adversarial review F10)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:49:54 +02:00
mathiasandClaude Sonnet 5.5 ede6c0e5ea chore(fixtures): invented text in place of words that may come from the private recording; the research note no longer names the private file (adversarial review F10)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:49:54 +02:00
mathiasandClaude Sonnet 5.5 5928925589 chore(fixtures): invented text in place of words that may come from the private recording; the research note no longer names the private file (adversarial review F10)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:49:54 +02:00
mathiasandClaude Sonnet 5.5 e5bba3cd23 chore(fixtures): invented text in place of words that may come from the private recording; the research note no longer names the private file (adversarial review F10)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:49:54 +02:00
mathiasandClaude Sonnet 5.5 2df8a7eb20 chore(fixtures): invented text in place of words that may come from the private recording; the research note no longer names the private file (adversarial review F10)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:49:54 +02:00
mathiasandClaude Sonnet 5.5 6e7c772b31 chore(fixtures): invented text in place of words that may come from the private recording; the research note no longer names the private file (adversarial review F10)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:49:54 +02:00
mathiasandClaude Sonnet 5.5 a2027c9f9b chore(fixtures): invented text in place of words that may come from the private recording; the research note no longer names the private file (adversarial review F10)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:49:54 +02:00
mathiasandClaude Sonnet 5.5 e38ea3175d test: assertions the adversarial review found missing -- log lines, immediate saves, snapshot isolation, store and temp-file failures, bounded snippet, identical list (review F9)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:48:58 +02:00
mathiasandClaude Sonnet 5.5 cdb709b9e2 test: assertions the adversarial review found missing -- log lines, immediate saves, snapshot isolation, store and temp-file failures, bounded snippet, identical list (review F9)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:48:58 +02:00
mathiasandClaude Sonnet 5.5 0d8bf0f94c test: assertions the adversarial review found missing -- log lines, immediate saves, snapshot isolation, store and temp-file failures, bounded snippet, identical list (review F9)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:48:58 +02:00
mathiasandClaude Sonnet 5.5 5b3e550d9b test: assertions the adversarial review found missing -- log lines, immediate saves, snapshot isolation, store and temp-file failures, bounded snippet, identical list (review F9)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:48:58 +02:00
mathiasandClaude Sonnet 5.5 cb0ed8642a test: assertions the adversarial review found missing -- log lines, immediate saves, snapshot isolation, store and temp-file failures, bounded snippet, identical list (review F9)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:48:58 +02:00
mathiasandClaude Sonnet 5.5 b8a4778513 test(web): a malformed multipart header is a 400 whose text is not logged; the file-name case no longer demands a 303
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:47:49 +02:00
mathiasandClaude Sonnet 5.5 9b82c06d92 fix(web): 429 for a busy queue; a non-UUID id and a malformed or odd-byte title are 4xx or cleaned, never 500 (review F6, F7)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:47:49 +02:00
mathiasandClaude Sonnet 5.5 93b4dbec1e feat(uploads): a full queue is a busy rejection, the row failed honestly (review F6)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:47:49 +02:00
mathiasandClaude Sonnet 5.5 f9ad85cac4 feat(uploads): a bounded queue, in total and per user (review F6)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:47:49 +02:00
mathiasandClaude Sonnet 5.5 a1d6754ae9 feat(uploads): ReasonBusy, a rejection that is the user's to retry (review F6)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:47:49 +02:00
mathiasandClaude Sonnet 5.5 e4584fb2ac feat(domain): UploadErrBusy -- the queue was full (review F6)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:47:49 +02:00
mathiasandClaude Sonnet 5.5 019ee636c8 test: a bounded queue answered as 'busy', and bad input answered with a 4xx (red; adversarial review F6, F7)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:46:13 +02:00
mathiasandClaude Sonnet 5.5 0da24d42f4 test: a bounded queue answered as 'busy', and bad input answered with a 4xx (red; adversarial review F6, F7)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:46:13 +02:00
mathiasandClaude Sonnet 5.5 2891ffabae test: a bounded queue answered as 'busy', and bad input answered with a 4xx (red; adversarial review F6, F7)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:46:13 +02:00
mathiasandClaude Sonnet 5.5 aa0322095b test: a bounded queue answered as 'busy', and bad input answered with a 4xx (red; adversarial review F6, F7)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:46:13 +02:00
mathiasandClaude Sonnet 5.5 a7e2cc43c0 test(cmd): the built worker refuses audio longer than the configured maximum (review F2)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:45:25 +02:00
mathiasandClaude Sonnet 5.5 fdcdee7413 test(cmd): the uploads store double records the kind a job failed with
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:45:25 +02:00
mathiasandClaude Sonnet 5.5 dc100d1cd4 test(web): the job carries the claimed duration
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:45:25 +02:00
mathiasandClaude Sonnet 5.5 6b83d72655 fix(cmd): the converter is capped and the worker measures, at the configured maximum (review F2)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:45:25 +02:00
mathiasandClaude Sonnet 5.5 07050cf9b6 feat(web): say in words why a recording was refused for its real length (review F2)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:45:25 +02:00
mathiasandClaude Sonnet 5.5 16cd246241 feat(uploads): the worker measures the converted audio against the limit and the claim (review F2)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:45:25 +02:00
mathiasandClaude Sonnet 5.5 132f46b047 feat(uploads): the job carries the duration the file claimed (review F2)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:45:25 +02:00
mathiasandClaude Sonnet 5.5 12576aa129 feat(audio): MaxSeconds caps the converted length, so a header that lies cannot become hours of WAV (review F2)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:45:25 +02:00
mathiasandClaude Sonnet 5.5 ba39864b2b feat(domain): UploadErrTooLong and UploadErrIncomplete -- the two ways a file's claimed length can be wrong (review F2)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:45:25 +02:00
mathiasandClaude Sonnet 5.5 741eccb575 test: the duration limit is checked against what is converted, not what the header claims (red; adversarial review F2)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:43:46 +02:00
mathiasandClaude Sonnet 5.5 5640d95f40 test: the duration limit is checked against what is converted, not what the header claims (red; adversarial review F2)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:43:46 +02:00
mathiasandClaude Sonnet 5.5 8f62d053ee test: the duration limit is checked against what is converted, not what the header claims (red; adversarial review F2)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:43:46 +02:00
mathiasandClaude Sonnet 5.5 52351c2cc1 test: the duration limit is checked against what is converted, not what the header claims (red; adversarial review F2)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:43:46 +02:00
mathiasandClaude Sonnet 5.5 29b3b725d1 test: the duration limit is checked against what is converted, not what the header claims (red; adversarial review F2)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:43:46 +02:00
mathiasandClaude Sonnet 5.5 7a468fdc09 fix(cmd): every local engine call carries its own deadline (review F4)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:42:46 +02:00
mathiasandClaude Sonnet 5.5 c9b7df9f95 feat(config): TAPIR_TRANSCRIBE_ENGINE_TIMEOUT, default 25 minutes (review F4)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:42:46 +02:00
mathiasandClaude Sonnet 5.5 c8ec468e47 feat(transcribe): WithTimeout -- each engine call gets its own deadline so a wedged engine fails while the job lives (review F4)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:42:46 +02:00
mathiasandClaude Sonnet 5.5 81a60d5fd4 test: a per-engine deadline so a wedged primary reaches the fallback (red; adversarial review F4)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:41:38 +02:00
mathiasandClaude Sonnet 5.5 c8c7356f19 test: a per-engine deadline so a wedged primary reaches the fallback (red; adversarial review F4)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:41:38 +02:00
mathiasandClaude Sonnet 5.5 8d1b7d887f test: a per-engine deadline so a wedged primary reaches the fallback (red; adversarial review F4)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:41:38 +02:00
mathiasandClaude Sonnet 5.5 21b337702a test: a per-engine deadline so a wedged primary reaches the fallback (red; adversarial review F4)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:41:38 +02:00
mathiasandClaude Sonnet 5.5 d16c2998d0 test(reconcile): the stored reason is a kind -- the old assertion encoded the leak the review found
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:40:55 +02:00
mathiasandClaude Sonnet 5.5 fb79dbb045 fix(uploads.go): an error reaches the log, the row and the page as a kind, never its text (adversarial review F3, F5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:40:55 +02:00
mathiasandClaude Sonnet 5.5 a2587eae9b fix(worker.go): an error reaches the log, the row and the page as a kind, never its text (adversarial review F3, F5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:40:55 +02:00
mathiasandClaude Sonnet 5.5 0b35278cdf fix(transcribe): an error reaches the log, the row and the page as a kind, never its text (adversarial review F3, F5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:40:55 +02:00
mathiasandClaude Sonnet 5.5 16a8833f79 fix(reconcile.go): an error reaches the log, the row and the page as a kind, never its text (adversarial review F3, F5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:40:55 +02:00
mathiasandClaude Sonnet 5.5 03a122c9f5 fix(tracker.go): an error reaches the log, the row and the page as a kind, never its text (adversarial review F3, F5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:40:55 +02:00
mathiasandClaude Sonnet 5.5 31f5a74d2a test(web): the upload routes' error paths log a kind, not the text (red; review F5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:40:10 +02:00
mathiasandClaude Sonnet 5.5 166ff849c0 test(uploads): a failure logs a kind, never the error text or the spool path (red; review F5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:40:10 +02:00
mathiasandClaude Sonnet 5.5 e9863c2e12 test(progress): a keyed failure logs its kind, not the error's text (red; review F3)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:40:10 +02:00
mathiasandClaude Sonnet 5.5 8f3300b377 test(transcribe): a failed engine's reason is a kind, not its error text (red; review F3)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:40:10 +02:00
mathiasandClaude Sonnet 5.5 f6c3f2af23 test(reconcile): the skipped reason carries a kind, not the model's text (red; review F3)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:40:10 +02:00
mathiasandClaude Sonnet 5.5 2d1b08458b feat(transcribe): StatusError.HTTPStatus -- its code, never its snippet, becomes the kind (review F3)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:39:38 +02:00
mathiasandClaude Sonnet 5.5 2608ad2f09 feat(llm): HTTPError -- a status code to branch on, a bounded body in the text (review F3)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:39:38 +02:00
mathiasandClaude Sonnet 5.5 81f3d7ffc6 test(transcribe): a StatusError exposes its code as a kind, not its snippet (red; adversarial review F3)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:39:25 +02:00
mathiasandClaude Sonnet 5.5 8c82d67b12 test(llm): a non-200 is an HTTP status error whose kind carries no body, and the body is bounded (red; adversarial review F3)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:39:25 +02:00
mathiasandClaude Sonnet 5.5 d8eea66c4c feat(errkind): Of -- a kind from a fixed vocabulary, never the error's text (review F3, F5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:39:10 +02:00
mathiasandClaude Sonnet 5.5 dc0c64ddeb test(errkind): an error reduced to a safe kind, never its text (red; adversarial review F3, F5)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:39:04 +02:00
mathiasandClaude Sonnet 5.5 99ac5580e0 test(web): the upload chat helper says which models are offered
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:38:30 +02:00
mathiasandClaude Sonnet 5.5 ef815edded fix(web): upload chat offers and accepts local models only and fails closed; a forged cloud model is replaced before anything is sent (review F1)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:38:30 +02:00
mathiasandClaude Sonnet 5.5 4933504982 feat(web): App.UploadChatModels -- the models upload chat offers and accepts (review F1)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:38:30 +02:00
mathiasandClaude Sonnet 5.5 e121ef241d fix(cmd): wire the local-only chat models into the upload routes (review F1)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:38:30 +02:00
mathiasandClaude Sonnet 5.5 5ae9212869 feat(cmd): uploadChatModels -- the chat switcher for a private recording is local models only (review F1)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:38:30 +02:00
mathiasandClaude Sonnet 5.5 a9a2a9270f fix(config): a strict shape for a local model alias; the upload chat model must be local (adversarial review F1, F8)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:38:30 +02:00
mathiasandClaude Sonnet 5.5 7fe746173d test(web): the upload chat offers and accepts local models only, and fails closed (red; adversarial review F1)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:37:49 +02:00
mathiasandClaude Sonnet 5.5 4b16e33715 test(cmd): the upload chat switcher never offers a cloud model (red; adversarial review F1)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:37:49 +02:00
mathiasandClaude Sonnet 5.5 896ce6a952 test(config): a strict shape for a local model alias, and the upload chat model must be local (red; adversarial review F1, F8)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 16:37:49 +02:00
mathiasandClaude Sonnet 5.5 7029dd1d7f test(e2e): a finished job's phases are all closed and it was reporting while it ran (ADR-035)
CI / Lint / Test / Vet (pull_request) Successful in 23s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:05:36 +02:00
mathiasandClaude Sonnet 5.5 c977618979 docs(claude): the progress tracker, the two timestamps, and the log's privacy rule (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:04:32 +02:00
mathiasandClaude Sonnet 5.5 0763e6238b docs(data-model): progress, progress_at, alive_at, and every judged spot stored (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:04:32 +02:00
mathiasandClaude Sonnet 5.5 6722433eb0 test(acceptance): map the progress, log and judged-spot scenarios to their covering tests
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:04:23 +02:00
mathiasandClaude Sonnet 5.5 ac75971030 docs(use-cases): scenarios for progress, the log and the judged spots (ADR-035) -- the coverage gate is red until they are mapped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:04:19 +02:00
mathiasandClaude Sonnet 5.5 02a065b826 test(web): opt-in renderer of every progress state for a screenshot review (TAPIR_SHOT_DIR)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:03:55 +02:00
mathiasandClaude Sonnet 5.5 105777f1aa fix(web): the spinner sits in the same mark column as the other glyphs; a lost job's running phase gets a warning mark (screenshot review)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:03:55 +02:00
mathiasandClaude Sonnet 5.5 18d8050a2b fix(web): the progress bar ends at the time column and the lost mark is error-coloured (screenshot review)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:03:55 +02:00
mathiasandClaude Sonnet 5.5 c06d285a3a test(web): the lost job's running phase carries a warning mark (red; found in the screenshot review)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:03:32 +02:00
mathiasandClaude Sonnet 5.5 f74fbbc40e test(cmd): the uploads store double has the progress methods the worker now needs
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:01:37 +02:00
mathiasandClaude Sonnet 5.5 cd7595a048 feat(web): a spot shows the words before it, and an empty list shows no section (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:01:21 +02:00
mathiasandClaude Sonnet 5.5 24d1a99ed0 feat(web): list only the spots to check, in order and capped (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:01:21 +02:00
mathiasandClaude Sonnet 5.5 8cb0b0a12d test(web): the kept-spot assertion looks for what the page actually renders (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:00:56 +02:00
mathiasandClaude Sonnet 5.5 6dfd53ac71 test(web): the page lists only changed and uncertain spots, with the words before each, though every judged spot is stored (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:00:49 +02:00
mathiasandClaude Sonnet 5.5 d62331b5f9 test(web): the quiet note is capitalised as the page writes it
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:00:23 +02:00
mathiasandClaude Sonnet 5.5 a7668b670e feat(web): the upload page shows the job's phases, a bar only where the count is real, and says when the service is lost (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:00:23 +02:00
mathiasandClaude Sonnet 5.5 89c9dd6705 feat(web): tell running from quiet from lost; poll every two seconds (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:00:23 +02:00
mathiasandClaude Sonnet 5.5 d1dcb131f2 feat(web): progress styles in tapir's own tokens -- a mono step log, a CSS spinner that respects reduced motion (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 09:00:23 +02:00
mathiasandClaude Sonnet 5.5 d0f2dcf838 test(web): the page shows real phases, tells running from quiet from lost, and never invents a percentage (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:57:58 +02:00
mathiasandClaude Sonnet 5.5 a97598586c test(reconcile): a span across a segment edge leaves neither segment holding a replaced word -- a mutation that skipped this survived
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:56:53 +02:00
mathiasandClaude Sonnet 5.5 976aeb71c6 feat(reconcile): report the comparison and deciding phases, store every judged spot, merge the words into the segments (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:56:53 +02:00
mathiasandClaude Sonnet 5.5 f74ad726e2 feat(domain): a judged spot carries the few words before it (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:56:53 +02:00
mathiasandClaude Sonnet 5.5 9297f728c2 test(reconcile): every judged spot with its left context; segments rewritten per segment, or left alone when they do not line up (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:55:43 +02:00
mathiasandClaude Sonnet 5.5 520ab666f8 test(reconcile): the comparison and the deciding phase report real counts, and a shutdown is neither done nor failed (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:55:43 +02:00
mathiasandClaude Sonnet 5.5 0b7bf08c95 test(reconcile): kept spots are stored, segments carry the merged words, the cap is an option -- three old assertions changed on purpose (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:55:43 +02:00
mathiasandClaude Sonnet 5.5 aefc8103f7 feat(transcribe): the ensemble declares and reports its phases; a failed engine shows failed, a shutdown does not (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:54:31 +02:00
mathiasandClaude Sonnet 5.5 f064dec185 test(transcribe): the ensemble declares and reports its phases; a failed engine is shown failed, a shutdown is not (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:54:07 +02:00
mathiasandClaude Sonnet 5.5 c6319b9bc6 refactor(uploads): the worker's own failure is FailRunning
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:53:43 +02:00
mathiasandClaude Sonnet 5.5 5f279d7e71 feat(progress): keyed Fail for a soft failure, Note for the operator, FailRunning for the job's own (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:53:43 +02:00
mathiasandClaude Sonnet 5.5 a6b8a2b744 test(progress): a keyed soft failure that logs why, a note that costs no write, and FailRunning for the job's own failure (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:53:33 +02:00
mathiasandClaude Sonnet 5.5 9393c27cdb test(uploads): the worker waits for its heartbeat to finish -- a mutation that skipped the wait survived until now
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:53:10 +02:00
mathiasandClaude Sonnet 5.5 538ed93ccd test(uploads): a slow heartbeat write, so one is in flight when the job ends
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:53:10 +02:00
mathiasandClaude Sonnet 5.5 27a88a0a1f feat(uploads): the worker reports real phases, beats on its own, and logs each one (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:53:10 +02:00
mathiasandClaude Sonnet 5.5 eb4d4b67e6 test(uploads): the worker reports real phases, beats on its own, and logs them in order without the transcript (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:52:20 +02:00
mathiasandClaude Sonnet 5.5 3f66c35497 test(uploads): the worker's store double records progress saves and heartbeats
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:52:20 +02:00
mathiasandClaude Sonnet 5.5 f8526d4549 feat(store): SaveProgress and Heartbeat; Get returns the phases and both timestamps (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:51:39 +02:00
mathiasandClaude Sonnet 5.5 41ebba3963 feat(store): migration 020 -- uploaded_transcripts.progress, progress_at, alive_at (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:51:11 +02:00
mathiasandClaude Sonnet 5.5 256e5b1a66 test(store): progress and heartbeat on an upload row -- round trip, alive is not progress, another user's write changes nothing (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:51:04 +02:00
mathiasandClaude Sonnet 5.5 8ec43ad393 test(progress): read the log buffer safely while the heartbeat writes to it
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:50:39 +02:00
mathiasandClaude Sonnet 5.5 4bfe578a05 feat(progress): Tracker -- one writer for the page and the log, throttled counts, a heartbeat of its own (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:50:39 +02:00
mathiasandClaude Sonnet 5.5 c9ebe526cc feat(domain): progress steps and the two timestamps that tell alive from progressing (ADR-035)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:50:39 +02:00
mathiasandClaude Sonnet 5.5 7bf5f6b8b7 test(progress): a tracker that shows the plan, times each step, throttles counts and heartbeats on its own (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:49:44 +02:00
mathiasandClaude Sonnet 5.5 7e9f5d7ca5 docs(adr): ADR-035 -- progress as real phases, the same transitions in the log, every judged spot kept (#42 #43 #44)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:49:06 +02:00
mathiasandClaude Sonnet 5.5 1a89eb3868 docs(adr-034): record the first blind check; bar ambiguity and the unanswered second condition
CI / Lint / Test / Vet (pull_request) Successful in 23s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:01:31 +02:00
mathiasandClaude Sonnet 5.5 2e9d76dad0 docs(research): first blind check of two-engine reconcile on a real recording (aggregates only)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 08:01:31 +02:00
mathiasandClaude Sonnet 5.5 7f8e3677b3 docs(adr-034): gemma4-31b slowness explained; reopen the 26B-vs-31B reconciler choice
CI / Lint / Test / Vet (pull_request) Successful in 22s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 07:13:10 +02:00
mathiasandClaude Sonnet 5.5 e3aa661cf2 docs(research): Low Power Mode confirmed as the cause of slow gemma4-31b (22.3 vs 7.9 tok/s decode)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-02 07:13:10 +02:00
mathiasandClaude Sonnet 5.5 7ae888a468 docs(claude): two-engine reconciliation, and that task check does not gate lint (#41)
CI / Lint / Test / Vet (pull_request) Successful in 21s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:47 +02:00
mathiasandClaude Sonnet 5.5 b9e0dfa1c9 docs(homelab): two-engine mode, its settings and why every model must be local
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:47 +02:00
mathiasandClaude Sonnet 5.5 8dbcb9a9eb docs(data-model): reconciliation column
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:47 +02:00
mathiasandClaude Sonnet 5.5 26cc00c1cc test(acceptance): map the two-engine scenarios to their covering tests
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:37 +02:00
mathiasandClaude Sonnet 5.5 36a80ac4f5 docs(use-cases): scenarios for two-engine reconciliation (ADR-034)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:22 +02:00
mathiasandClaude Sonnet 5.5 2497554683 docs(adr): ADR-034 records the first live smoke, including the wrong call
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:22 +02:00
mathiasandClaude Sonnet 5.5 c3c9b1e48a test(e2e): two engines + local reconciler, live
TAPIR_E2E_RECONCILE=1 runs pianissimo + kb-whisper + gemma4-26b on the audio and
asserts the reconciliation is recorded, every contributor is named in the endpoint, and
the page says what was done. Run 2026-10-01 on 82 s of synthetic Swedish: 3 disputes, all
3 changed; against the known script 2 were right (virket, a missing 'procent') and 1 was
WRONG (Maria -> Vi replaced a correct base reading). A smoke test, not a measurement.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:27:22 +02:00
mathiasandClaude Sonnet 5.5 ed8ca54674 feat(cmd): buildTranscriber -- the base-engine setting turns on the two-engine ensemble
Default is the single-engine chain exactly as before. With TAPIR_RECONCILE_BASE_MODEL set,
the base engine and the primary run on the same WAV and a LOCAL model (config has already
refused any non-local alias) reconciles them, at temperature 0 with a short reply -- the
conditions its accuracy was measured under. The cloud fallback is unchanged and runs
only when both local engines fail.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:26:02 +02:00
mathiasandClaude Sonnet 5.5 7338d9d603 test(e2e): discard the deferred Close error golangci-lint flags
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:26:02 +02:00
mathiasandClaude Sonnet 5.5 3d755fd57a fix(transcribe): check (discard) the deferred Close errors golangci-lint flags
Found because the project's lint step cannot fail: the Taskfile runs
'golangci-lint run ./... || echo "not installed -- skipping"', so a failing lint
prints that message and exits 0. CI's log showed these findings and still passed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:26:02 +02:00
mathiasandClaude Sonnet 5.5 b492a3144e test(cmd): the base-engine setting turns on the ensemble, and nothing else does (red)
Red: buildTranscriber is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:25:10 +02:00
mathiasandClaude Sonnet 5.5 505670c0e1 feat(llm): WithTemperature, default unchanged
Zero is sent as zero (the field has no omitempty). The reconciler's measured accuracy
was at temperature 0, so it runs there; every other caller keeps 0.2.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:25:10 +02:00
mathiasandClaude Sonnet 5.5 7891675393 test(llm): a temperature option; zero is sent as zero, not 'unset' (red)
The reconciler was measured at temperature 0 but the client hardcodes 0.2. Zero must
be sent explicitly, and the default must not change for existing callers.

Red: WithTemperature is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:54 +02:00
mathiasandClaude Sonnet 5.5 8110fe56ab style(config): gofumpt the new test rows
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:38 +02:00
mathiasandClaude Sonnet 5.5 61fb50ce99 feat(config): two-engine reconciliation settings with a local-only guard (ADR-034)
TAPIR_RECONCILE_BASE_MODEL turns it on and needs TAPIR_RECONCILE_MODEL; every model must
be a local alias (koala/ or iguana/, matched as a PREFIX): a cloud adjudicator is a
start-up error because private meeting content must not reach it; the two engines must
differ; default job timeout 60m. Off by default. Mutation-checked.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:34 +02:00
mathiasandClaude Sonnet 5.5 f25fdaf398 test(config): a cloud alias that merely contains a local host name is still cloud
Mutation-testing showed a substring match instead of a prefix match went unnoticed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:34 +02:00
mathiasandClaude Sonnet 5.5 ad071d44cb test(config): reconciliation settings -- off by default, local only, complete (red)
Off by default; a base engine needs an adjudicator; every model must be a local alias
(koala/ or iguana/): a cloud adjudicator is a start-up error because private meeting
content must not reach it; the base engine must differ from the other engine; default
job timeout 60m (two engines plus adjudication).

Red: the Config fields do not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:20 +02:00
mathiasandClaude Sonnet 5.5 59f2893434 docs(adr): ADR-034 names the settings for what they are
TAPIR_RECONCILE_BASE_MODEL, not 'second model': the engine added to turn the
feature on is the one whose TEXT is the base, and the old name hid that. All three
models must be local aliases; the base must differ from the other engine. ADR is
still Proposed, so this is an in-place correction.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:24:09 +02:00
mathiasandClaude Sonnet 5.5 6fb7a52980 feat(web): say what reconciliation did, and never claim a second opinion that was not there
CI / Lint / Test / Vet (pull_request) Successful in 21s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Reconciled: the two engines, the deciding model and the exact counts; the contested
spots (changed or uncertain) listed with a minute mark and both readings, the first
50 shown and the rest counted from the exact totals. Single-engine and
model-unavailable results say why. A nil reconciliation renders nothing. Readings are
escaped. Mutation-checked: the cap, the minute rounding and the 'more' count.

uploads.go and the templ pair are one commit: the helpers and the template need each other.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:23:39 +02:00
mathiasandClaude Sonnet 5.5 d6ba11d288 test(web): the page says what reconciliation did, plainly, and never claims more (red)
Reconciled: who the two engines and the deciding model were, and the counts
(differences, settled automatically, decided by the model, uncertain). The contested
spots are listed with a minute mark, both readings and which was used, uncertain ones
marked, readings escaped, and a very long list shortened with its true count. A
single-engine transcript makes no claim of a second opinion; one-engine and
model-unavailable cases say why.

Red: the page renders none of this yet.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:23:00 +02:00
mathiasandClaude Sonnet 5.5 ae4f9fd775 feat(store): persist and read back the reconciliation
CompleteUpload writes it (SQL NULL when there was none, never the JSON text 'null':
the page claims a second opinion only if the column is non-NULL); GetUpload reads it,
List does not. Mutation-checked.

domain.go and the store are one commit: the store tests need the new field.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:22:09 +02:00
mathiasandClaude Sonnet 5.5 e838f6bb6d feat(store): migration 019 -- uploaded_transcripts.reconciliation
JSONB, NULL for a single engine so the page cannot claim a second opinion. On the
already FORCE'd, RLS-scoped table, so the exact-set isolation test is unaffected.
Up and down are one commit: golang-migrate pairs them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:21:52 +02:00
mathiasandClaude Sonnet 5.5 5bd41077af test(store): the reconciliation survives the round trip, and its absence stays absent (red)
Counts and contested spots round-trip; a single-engine transcript has NO
reconciliation (nil, not empty -- the page must not claim a second opinion); a
skipped reconciliation keeps its reason.

Red: Transcription.Reconciliation is stored nowhere and UploadedTranscript has none.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:21:36 +02:00
mathiasandClaude Sonnet 5.5 34bc8dbdee feat(transcribe): Ensemble -- two local engines, a local reconciler, fail-soft at every step
Runs the fast engine then the other from one WAV and reconciles; one engine down
uses the other and says so; the model down gives the base text unreconciled; the
cloud fallback runs only when BOTH local engines fail and is recorded as such; a
cancelled job stops everything and never reaches the cloud. The endpoint lists every
contributor. Mutation-checked, including both cancellation points.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:21:17 +02:00
mathiasandClaude Sonnet 5.5 8778376aa6 test(transcribe): a shutdown in the middle of either engine stops the job
Mutation-testing showed each of the two cancellation checks was redundant with the
other in my first test. Now a cancel during the first engine must not start the second
or the cloud, and a cancel during the second must not be recorded as a single-engine
success ('kb-whisper failed').

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:21:17 +02:00
mathiasandClaude Sonnet 5.5 ad39b6154c test(transcribe): two engines, then a reconciler -- and every way it can fail softly (red)
Both run in order (the fast one first) and are reconciled; identical texts cost no
model call; one engine down uses the other and says so, with the cloud fallback
untouched; only when BOTH local engines fail does the fallback run, and the row says
berget served it; both failing with no fallback reports both errors; a cancelled
job never reaches the cloud; a model that is down yields the base text unreconciled,
not a failed job.

Red: transcribe.Ensemble is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:20:40 +02:00
mathiasandClaude Sonnet 5.5 f25acebe4c feat(reconcile): merge two transcripts with a model choosing between disagreements
CI / Lint / Test / Vet (pull_request) Successful in 20s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Base text stands except where the model picks the alternative's reading; trivial
disputes skip the model; the reading order alternates; the base's context goes to
the model. Fails soft: three consecutive model errors return the base text
unreconciled (none of the partial changes) with the reason; a verdict returned with
an error is ignored; a cancelled job is an error, not 'skipped'. Counts are exact,
the stored list of contested spots is capped at 200 and carries the second each
starts at. Mutation-checked on nine behaviours.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:20:12 +02:00
mathiasandClaude Sonnet 5.5 7f9e967ad0 test(reconcile): a verdict returned with an error is not used
Mutation-testing showed the forced-unsure line was untested because my fake
always returned unsure with its errors.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:20:12 +02:00
mathiasandClaude Sonnet 5.5 0fead2e102 feat(domain): Reconciliation -- how a merged transcript came about (ADR-034)
Status (reconciled / single_engine / skipped), exact counts, and a bounded list of
contested spots (changed or unsure) with a position and both readings.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:19:54 +02:00
mathiasandClaude Sonnet 5.5 a1e16fa658 test(reconcile): fixtures need two shared words between disputes
Disputes separated by a single shared word are merged into one decision by design,
so several of my fixtures were secretly one dispute. Corrected before any
implementation existed; the rule is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:19:22 +02:00
mathiasandClaude Sonnet 5.5 b20558ffa6 test(reconcile): merging two transcripts, and failing soft (red)
Identical texts and trivial disputes cost no model call; an alt verdict takes the
alternative's own words, base/unsure keep the base and only unsure is flagged;
insertions and deletions land in the right place; the reading order alternates; the
model gets the base's context. Three consecutive model errors stop the whole
reconciliation (base text, none of the partial changes, reason recorded) while a
success in between resets the breaker; a cancelled job is an error, not 'skipped'.
Spots carry the second they start at; the stored list is capped, the counts are not.

Red: Reconcile, Options, MaxContested and domain.Reconcile* are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:19:00 +02:00
mathiasandClaude Sonnet 5.5 5e54c9beff feat(reconcile): LLM adjudicator with the measured prompt, plus an order-checked wrapper
Sends the exact wording that scored 92% (gemma4-26b) and 97% (gemma4-31b); maps
1/2 back to base/alt under both orders; an unreadable reply is unsure, a transport
error is an error. OrderChecked asks twice with the order swapped and keeps only
agreement. Mutation-checked: answer mapping, unparseable-as-guess, the word
boundary and the order check each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:18:11 +02:00
mathiasandClaude Sonnet 5.5 2bfe84ee64 test(reconcile): a number or word that merely starts with 1 or 2 is not an answer
Mutation-testing the adjudicator showed dropping the word boundary went unnoticed:
'ANSWER: 12' would be read as 1.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:18:11 +02:00
mathiasandClaude Sonnet 5.5 481321a75c test(reconcile): the adjudicator sends the measured prompt and maps answers back (red)
Readings in the requested order; 1/2 mapped to base/alt under both orders;
UNSURE, an unparseable or out-of-range reply is unsure rather than a guess; an
empty reading is described, not blank; a model error is an error, distinguishable
from 'unsure'. The optional order-checked wrapper asks twice with the order swapped
and keeps only agreement.

Red: Request, Verdict, NewLLMAdjudicator and NewOrderChecked are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:17:42 +02:00
mathiasandClaude Sonnet 5.5 2600e6c45a feat(reconcile): find disputes, resolve trivial ones without a model, give context
Disputes carry both readings with their own spelling; differences separated by
one shared word merge into one decision. Compound splits/joins and one or two
filler particles are resolved without a model; 'nej', numbers and content words
never are. Mutation-checked (merge distance, filler limit, 'nej', compound rule).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:17:21 +02:00
mathiasandClaude Sonnet 5.5 4e43327792 test(reconcile): finding disputes, resolving the trivial ones, and their context (red)
A dispute carries both readings with their own spelling; differences separated by
a single shared word are one decision, two shared words keep them apart; compound
splits/joins and a short run of filler words are resolved without a model, while
numbers, content words, and 'nej' (an answer) are not.

Red: FindDisputes, AutoResolve and ContextAround are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:16:55 +02:00
mathiasandClaude Sonnet 5.5 92a39da45c feat(reconcile): minimal word-level diff (Myers) with a bounded edit distance
Prefix/suffix trimmed first; adjacent deletes and inserts read as one Replace;
beyond 3,000 edits (two texts sharing almost nothing) the middle is one Replace
instead of gigabytes of trace. Mutation-checked: a wrong tie-break, missing
replace coalescing, wrong suffix accounting and a punctuation-bearing key each
turn tests red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:16:31 +02:00
mathiasandClaude Sonnet 5.5 aa5607b002 feat(reconcile): tokenise for word-level comparison
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:16:31 +02:00
mathiasandClaude Sonnet 5.5 984d92e3a9 test(reconcile): word-level diff is minimal, tiles both inputs and is fast (red)
A random-input property test compares the equal runs against a dynamic-programming
LCS: a diff that is valid but not minimal would turn real agreement into disputes
the model is then asked about. A 6,500-word, 380-edit case mirrors a real hour.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:15:44 +02:00
mathiasandClaude Sonnet 5.5 00fd884c6c test(reconcile): tokenising for word-level comparison (red)
Keeps the word as written, keys it without case/punctuation/hyphens/plus, joins a
lone dash or full stop to its neighbour so it is not a word the engines can
disagree about. Red: package reconcile does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:15:44 +02:00
mathiasandClaude Sonnet 5.5 de9871c9c3 docs(adr): ADR-034 transcribe with two engines, reconcile with a local model
CI / Lint / Test / Vet (pull_request) Successful in 29s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Proposed, decider pending. Two local engines on one WAV; word-level diff with
pianissimo as the base (readability and timing, NOT accuracy -- that was a tie);
trivial disputes resolved without a model; the rest adjudicated by a LOCAL model
(alias must start koala/ or iguana/) whose only freedom is choosing between the
two readings. Reconciliation never fails a job; the page says what was done and
lists the uncertain spots. Off by default. Validation: re-upload the real
recording and keep the feature only if the reconciled reading is right >= 85% of
the time and no worse than the better single engine.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:15:12 +02:00
mathiasandClaude Sonnet 5.5 e182a6b07d docs(research): can a local model adjudicate two ASR transcripts?
Eleven local models, 120 calls each, scored against the speaker's blind answers
on 30 disagreements, with pass criteria fixed before the run. gemma4-31b (97%)
and gemma4-26b (92%) pass; nothing else clears accuracy and order-consistency.
Records why 31b is slow (MLX gives the same speed; Low Power Mode is the
unproven suspect) and the caveats: 30 items, one recording, thinking models
under-budgeted. Aggregate numbers only; this repo is public.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 23:14:27 +02:00
mathiasandClaude Sonnet 5.5 ebc6cc3ff8 docs(research): kb-whisper vs pianissimo on one real hour of Swedish speech
CI / Lint / Test / Vet (pull_request) Successful in 22s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
A near tie on accuracy (blind A/B by the speaker: 17 vs 13 on the 30 spans that
change meaning) and a 6x difference in speed (40 s vs 4m08s for 63.7 min).
Different error profiles: pianissimo better on terms and numbers, kb-whisper on
everyday words. Aggregate numbers only -- no transcript text or domain terms,
this repo is public. Also records the mid-word wrapping bug the comparison found
and its fix.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 19:45:56 +02:00
mathiasandClaude Sonnet 5.5 d11afb5ff1 fix(transcribe): rebuild the transcript from segments so split words are whole
CI / Lint / Test / Vet (pull_request) Successful in 22s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
whisper.cpp wraps at ~60 chars and can wrap inside a word; the response's text
field joins segments with newlines, so chat saw 'nyt' / 't godkännande'. Text
is now rebuilt from the raw segments: concatenated when any segment carries a
leading space (whisper.cpp), joined by a space when none do (berget's sentence
level), whitespace collapsed; the text field is only a fallback.

Checked against the first real one-hour transcript with pianissimo's vocabulary
as an independent yardstick: 36 spans changed, 31 now consist entirely of words
pianissimo also produced, words unknown to pianissimo fell 224 -> 177 and word
agreement rose 92.4% -> 93.1%. Five rejoined spans still contain an unknown
word (names/loanwords or an unlucky glue) -- a residual risk, small.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 19:35:14 +02:00
mathiasandClaude Sonnet 5.5 500e11962b test(transcribe): rebuild Text from segments so a split word is not broken (red)
Found by comparing the first real one-hour transcript against pianissimo:
whisper.cpp wraps at ~60 chars and can wrap INSIDE a word. The continuation
segment has no leading space, the text field joins segments with newlines, and
a segment can be a lone '.'. Measured 2026-10-01: 7% of segments (96/1332) in a
synthetic hour, 42/1302 lines in the real one. My earlier tests used short
sentences and could not see it. berget's segments are whole sentences with no
leading space, so the two backends need different joins.

Red: Text still joins by newline.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 19:34:54 +02:00
mathiasandClaude Sonnet 5.5 206fc2f5ca feat(web): log upload rejections by reason only
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 19:28:10 +02:00
mathiasandClaude Sonnet 5.5 5368d5f8c2 test(web): a rejected upload is logged with its reason, never the filename or title (red)
Rejections are synchronous and leave no row, so an operator could not tell 'the
user never retried' from 'the page refused it' (it happened during the first
real-recording trial). The reason is operational; the filename and title are
the user's private content.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 19:28:06 +02:00
mathiasandClaude Sonnet 5.5 365dff5c85 docs(data-model): had_video
CI / Lint / Test / Vet (pull_request) Successful in 26s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:55 +02:00
mathiasandClaude Sonnet 5.5 adaaa9918f test(acceptance): map the video-track scenarios to their covering tests
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:55 +02:00
mathiasandClaude Sonnet 5.5 364a28eecd docs(use-cases): a video track is accepted and the user is told; no-audio is rejected
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:55 +02:00
mathiasandClaude Sonnet 5.5 1fa7ca3a8c test(e2e): a video file (camera-off Teams meeting) is transcribed and the user is told
Run 2026-10-01 with 82 s of synthetic Swedish muxed with black video (H.264 +
AAC .mp4, built with the real ffmpeg): accepted, had_video=true, transcribed by
iguana/kb-whisper, the page said only its audio was used, and chat answered.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:48 +02:00
mathiasandClaude Sonnet 5.5 cca0a7461b feat(web): tell the user when a video track was ignored, in every state
A Teams meeting with the camera off is still a video file. It is accepted, only
its audio is transcribed, and the page says so in queued, transcribing, done and
failed -- not just at the end.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:14 +02:00
mathiasandClaude Sonnet 5.5 5175496809 test(web): match the notice's real casing
My assertion expected a lowercase 'only'; the sentence begins 'Only'. The user
facing wording is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:14 +02:00
mathiasandClaude Sonnet 5.5 f102199097 feat(uploads): Intake passes whether the file had a video track to the store
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:00 +02:00
mathiasandClaude Sonnet 5.5 01eab1ab6f feat(uploads): a video track is no longer a reason to reject (ADR-033)
Only no-audio, unreadable length and too-long are refused. ReasonVideo is gone.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:43:00 +02:00
mathiasandClaude Sonnet 5.5 85cf89c86c feat(store): CreateUpload records had_video; Get/List return it
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:50 +02:00
mathiasandClaude Sonnet 5.5 603d02e504 feat(domain): UploadedTranscript.HadVideo
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:40 +02:00
mathiasandClaude Sonnet 5.5 3c4d2880f3 feat(store): migration 018 -- uploaded_transcripts.had_video
Up and down are one commit: golang-migrate pairs them. RLS is unchanged: the
column is on the already-FORCE'd table, so the exact-set isolation test still
holds.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:33 +02:00
mathiasandClaude Sonnet 5.5 1157854441 test(cmd): fake store follows the CreateUpload signature (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:33 +02:00
mathiasandClaude Sonnet 5.5 7aec2538be test(web): a video file is accepted and the user is told the video was ignored (red)
Video with no audio is rejected for having none. A file with a video track is
accepted and marked had_video; every state of its page (queued, transcribing,
done, failed) says only its audio was used and the video was discarded, and a
plain audio file shows no such notice.

Red: the had_video column does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:33 +02:00
mathiasandClaude Sonnet 5.5 eb55c7d9d1 test(store): had_video is recorded on creation and survives done/failed (red)
Red: CreateUpload takes no hadVideo and the row has no HadVideo.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:13 +02:00
mathiasandClaude Sonnet 5.5 ab62cdaf1c test(uploads): Accept records whether the file had a video track (red)
Red: Store.CreateUpload does not take hadVideo yet.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:13 +02:00
mathiasandClaude Sonnet 5.5 6cbb7b2df0 test(uploads): a recording with a video track is accepted (red)
A Teams meeting with the camera off is still an .mp4 with a video track.
Video with no audio is now rejected as having no audio.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:42:04 +02:00
mathiasandClaude Sonnet 5.5 1e192e53ef docs(adr): ADR-033 accepts files with a video track, uses the audio, says so
The first draft rejected video and named #28. The maintainer's first real
recording -- a Teams meeting with the camera off -- is still an .mp4 with a
video track, so that rejected the primary use case. Decision 1 and 4 now:
accept, extract audio only, discard the video, record had_video, and tell the
user. A file with no audio stream is still rejected. Video features stay #28.
In-place amendment: the ADR is still Proposed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 18:41:49 +02:00
mathiasandClaude Sonnet 5.5 6ebb4e49e9 test(audio): opt-in live ffprobe check on hostile playlists and real audio
CI / Lint / Test / Vet (pull_request) Successful in 40s
CI / Build & Import (pull_request) Skipped
CI / Deploy via GitOps (pull_request) Skipped
Skipped unless TAPIR_LIVE_FFPROBE is set. Synthetic/crafted files only.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:14:29 +02:00
mathiasandClaude Sonnet 5.5 291a360a83 feat(audio): restrict ffmpeg/ffprobe input to local files and audio demuxers
The upload is attacker-supplied and its CONTENT can steer ffmpeg to open other
files or URLs (HLS/concat playlists). Pins -protocol_whitelist file,pipe and
-format_whitelist to the containers real recordings use, before the input
(after -i it would apply to the output).

Verified with the real static ffmpeg 7.0.2 on koala: real AAC .m4a and WAV
still probe/convert; a crafted concat file is rejected with 'Format not on
whitelist'. Note ffmpeg 7.0.2 ALREADY refused the playlists tried without the
whitelist, so the live check cannot prove the whitelist -- the argv unit tests
do; this protects against a different ffmpeg version being put in the image.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:14:29 +02:00
mathiasandClaude Sonnet 5.5 4a0c51d877 test(audio): attacker-supplied input must not steer ffmpeg to other files or URLs (red)
A file's CONTENT can tell ffmpeg to open other paths or URLs (HLS, concat
playlists). ffmpeg 7.0.2 refuses the ones tried, but that is version luck and
the image's ffmpeg is still to be chosen (#34). Pin protocol_whitelist to
file,pipe and format_whitelist to the demuxers real audio uses, before the input.

Red: audio.AllowedDemuxers is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:13:54 +02:00
mathiasandClaude Sonnet 5.5 1c726a810e docs(homelab): upload chat budget is 90000 and the chat model is gemma4-26b
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:08:33 +02:00
mathiasandClaude Sonnet 5.5 77fe6954e5 docs(adr): ADR-033 decision 8 -- chat budget and model from measurement
The proposed 120,000-char budget was a guess; measured, gemma4-26b silently
loses the head at that size and phi4-mini (first in the offered list) 400s on
an hour of speech. Budget is now 90,000, uploads open on gemma4-26b, the
retrieval trigger is concrete, and the iguana contention hazard is recorded.
ADR-033 is still Proposed, so this is an in-place correction, not a supersession.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:08:28 +02:00
mathiasandClaude Sonnet 5.5 4d3e2a471a docs(research): chat over a long transcript -- what the offered models really hold
phi4-mini 400s at ~13k tokens; gemma4-26b is correct to 90,000 chars and
silently loses the HEAD at 120,000 (no error, no truncation flag). The
end-needle test alone passed at 120k -- only a head-needle exposed it.
Records the default-model trap and the iguana contention hazard.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:08:13 +02:00
mathiasandClaude Sonnet 5.5 548bfaee85 test(e2e): use the same wireUploads as serve
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:41 +02:00
mathiasandClaude Sonnet 5.5 ad2e3e0e8e fix(cmd): serve now actually sets the upload chat budget and model
cmdServe assigned Uploads and UploadLimits but not UploadChatMaxChars, so chat
over a long recording would have used the 18,000-char caption default. It now
calls wireUploads, which is tested.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:41 +02:00
mathiasandClaude Sonnet 5.5 e6738bcd3e feat(cmd): wireUploads -- limits, chat budget and chat model into the web App
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:40 +02:00
mathiasandClaude Sonnet 5.5 3bb4e30f4c test(cmd): wireUploads sets the budget and model, and never a typed-nil Intake (red)
I wired Uploads and UploadLimits in cmdServe but not UploadChatMaxChars: chat
over a long recording would have silently used the 18,000-char caption
default. A helper that main and the e2e test both call makes that a tested fact.

Red: wireUploads is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:33 +02:00
mathiasandClaude Sonnet 5.5 6c39ab86cf feat(web): chat over an upload opens with App.UploadChatModel
The Chatter's own default is the summarizer's (koala/phi4-mini, 8,192-token
context), which answers HTTP 400 on an hour of speech. Empty keeps the old
behaviour.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:33 +02:00
mathiasandClaude Sonnet 5.5 4b6ac2a213 test(web): chat over an upload opens with the upload chat model (red)
Red: App.UploadChatModel is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:20 +02:00
mathiasandClaude Sonnet 5.5 af66419a44 feat(config): measured chat budget (90,000) and upload chat model (gemma4-26b)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:20 +02:00
mathiasandClaude Sonnet 5.5 4b09dc8f36 test(config): chat defaults for uploads come from measurement, not a guess (red)
iguana/gemma4-26b answered correctly at 90,000 chars but SILENTLY dropped the
head of a 120,000-char transcript (prompt tokens fell to 16k; 'not in the
text'); koala/phi4-mini has an 8,192-token context and answers HTTP 400 at
~13k tokens. So the budget is 90,000 and the upload chat model defaults to
gemma4-26b. Red: UploadChatModel is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:09 +02:00
mathiasandClaude Sonnet 5.5 8f16ab671a docs(claude): audio upload supersedes the STT deferral for user-supplied audio only
The 'no audio-download + STT' bullet is narrowed to YouTube (ADR-007 stands);
adds the upload packages, the exact-set isolation test and the pending ffmpeg
decision to the build state.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:02:11 +02:00
mathiasandClaude Sonnet 5.5 39aeda7c69 docs(homelab): speech-to-text endpoints, the ingress 413, and the upload knobs
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:02:05 +02:00
mathiasandClaude Sonnet 5.5 0fc25a24ed docs(data-model): uploaded_transcripts, and fix the stale claim that transcripts is RLS'd
Adds the private uploaded_transcripts entity and the exact-set isolation rule
(ADR-033). Also corrects two lists that still named `transcripts` as an RLS'd
table: it has been shared non-RLS since ADR-021 / migration 015, and these
lists sit next to the very boundary that matters.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:02:05 +02:00
mathiasandClaude Sonnet 5.5 07375f75de test(acceptance): map audio_upload.feature scenarios to their covering tests
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:01:38 +02:00
mathiasandClaude Sonnet 5.5 f9d6df9f84 docs(use-cases): audio_upload.feature -- scenarios for #33 (ADR-033)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:01:20 +02:00
mathiasandClaude Sonnet 5.5 3464cdf876 test(e2e): opt-in live upload -> transcript -> grounded chat check
Real ffprobe/ffmpeg, the live gateway and a real chat model, a private
embedded Postgres. Skipped unless TAPIR_E2E_AUDIO is set. Run 2026-10-01 with
82 s of synthetic Swedish as AAC .m4a:
- primary: iguana/kb-whisper, 82.6 s audio, 1,186 chars; asked who owns the
  timber order, the answer named Anders and the 15 November deadline from the
  transcript; the spool directory was empty afterwards.
- primary forced to fail (TAPIR_TRANSCRIBE_MODEL=iguana/does-not-exist): the
  fallback served it and the row recorded endpoint=berget/kb-whisper.
Synthetic audio only; this repo is public.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:01:06 +02:00
mathiasandClaude Sonnet 5.5 4fdb212cef feat(web): chat with an uploaded transcript, reusing the ADR-027 path (ADR-033)
GET/POST /uploads/{id}/chat, mounted only when both an Uploader and a Chatter
are wired. The row comes from the user-scoped private store, so another user's
upload is a 404 before any model is involved. The owner's transcript goes to
the SAME Chatter with App.UploadChatMaxChars instead of the 18,000-char caption
default. The chat panel is shared, not forked: chatView gains PostURL/Subject
(empty = a YouTube video, so existing callers are unchanged). The done card
links to chat only when chat is wired.

Mutation-checked: dropping the upload budget, ignoring the done-status guard,
or scoping the lookup to the wrong user each turn tests red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:58:25 +02:00
mathiasandClaude Sonnet 5.5 a39f58b730 test(web): match the HTML-escaped tail of the inline chat error
templ escapes the apostrophe in "couldn't", so the assertion could never match
the rendered page. A test defect, not a loosening: the failure must still be
shown inline with the conversation kept.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:58:25 +02:00
mathiasandClaude Sonnet 5.5 10ca45fc0a test(web): chat over an uploaded transcript reuses the ADR-027 path (red)
Owner sees the panel posting to /uploads/{id}/chat; an unfinished upload
redirects to its status; ANOTHER USER'S upload is 404 on GET and POST and no
model is ever called with it; the owner's transcript is sent with the upload
budget (not the 18,000-char caption default); an unoffered model falls back; a
model failure is inline and keeps the conversation; routes are absent without a
chat backend; the done card links to chat only when chat is wired.

Red: App.UploadChatMaxChars is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:57:47 +02:00
mathiasandClaude Sonnet 5.5 c72d366935 feat(chat): per-request transcript budget so a long recording is not cut to its head (ADR-033)
chat.Request.MaxChars overrides the Service cap when positive. The 18,000-char
default (~22 min of speech) suits captions; an uploaded hour is ~50,000 chars
and truncation keeps only the head. This is reuse of the existing chat path,
not a fork.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:57:01 +02:00
mathiasandClaude Sonnet 5.5 ec29c5e761 style(chat): gofumpt the new test
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:57:01 +02:00
mathiasandClaude Sonnet 5.5 867afefb08 test(chat): a request may carry its own transcript budget (red)
The 18,000-char default is ~22 min of speech and keeps only the head; an
uploaded hour is ~50,000 chars. Red: Request.MaxChars is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:56:51 +02:00
mathiasandClaude Sonnet 5.5 1c5d85f5c6 feat(cmd): serve mounts audio upload when enabled (ADR-033)
Wires buildUploads into cmdServe: Intake and limits into the web App, the
worker on the server's ctx. Logs the two facts an operator must not miss: the
queue is process-local (single replica), and a configured fallback is a cloud
endpoint that private audio can leave the homelab through.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:56:22 +02:00
mathiasandClaude Sonnet 5.5 9b3175db09 feat(cmd): buildUploads -- spool, sweep, fail interrupted rows, wire the chain (ADR-033)
Returns nils when disabled (never a typed-nil Intake). When enabled: private
0700 spool dir, delete audio left by a crash, fail rows a restart left queued
or transcribing (the audio is not retained, so they cannot resume), then build
the iguana->berget chain, worker and intake. An empty
TAPIR_TRANSCRIBE_FALLBACK_MODEL keeps private audio on the homelab. Enabling
with no gateway fails at start-up, not at the first upload.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:56:21 +02:00
mathiasandClaude Sonnet 5.5 eb9357a5de test(cmd): start-up removes spooled audio left by a crash (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:56:11 +02:00
mathiasandClaude Sonnet 5.5 3e802adade test(cmd): buildUploads wiring -- off by default, sweep first, private spool (red)
Red: buildUploads is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:55:55 +02:00
mathiasandClaude Sonnet 5.5 b8798dbd5d feat(config): upload and transcription settings (ADR-033)
Embedded UploadConfig. Off by default until the image carries ffmpeg; 200 MB /
90 min; iguana primary with a berget fallback that an explicit empty
TAPIR_TRANSCRIBE_FALLBACK_MODEL disables (private audio then never leaves the
homelab); zero or negative limits fail at load instead of rejecting every
upload at runtime.

uploads.go and the two-line hook in config.go are one commit: the tests need both.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:55:32 +02:00
mathiasandClaude Sonnet 5.5 fb521f2b89 test(config): upload and transcription settings (red)
Off by default until the image has ffmpeg; 200 MB / 90 min (the longest length
measured); iguana primary, berget fallback that an explicit empty value
disables; zero or negative limits are refused rather than silently rejecting
every upload.

Red: the Config fields do not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:55:11 +02:00
mathiasandClaude Sonnet 5.5 a8dadb8ca4 feat(uploads): in-process transcription worker, one job at a time (ADR-033)
Queue -> StartTranscribing -> ffmpeg to 16 kHz WAV -> gateway chain ->
CompleteUpload, with a per-job timeout so a hung endpoint ends in failed, not
a spinner. The upload and the WAV are deleted on every path (audio is never
retained). A shutdown mid-job is not a failure: the row is left for the
start-up sweep. Cancellation wins over a non-empty queue, so queued jobs are
never started on shutdown; stop() deletes their audio and refuses new jobs.

Concurrency is 1 on purpose: two simultaneous long jobs on iguana were never
measured, and a Metal OOM there latches whisper (infra#273). The queue is
process-local, so ADR-018's single-replica assumption now has a second
dependent.

Mutation-checked, -race clean: cleanup, shutdown-vs-failure, timeout,
concurrency and stop-cleanup each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:54:52 +02:00
mathiasandClaude Sonnet 5.5 3cfc95640e test(uploads): a queued job at shutdown must never be started; timeout test fails fast
Mutation-testing the worker showed deleting stop()'s cleanup went unnoticed:
Process was incidentally deleting the audio of jobs it should never have
started. The stop test now asserts only the running job was converted (red on
the old code). The timeout test no longer hangs for the 10-minute go test
default when the timeout is removed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:54:52 +02:00
mathiasandClaude Sonnet 5.5 52417e6b47 test(uploads): in-process transcription worker (red)
Success path deletes both the upload and the WAV; failures are typed and still
delete the audio; a shutdown mid-job leaves the row for the start-up sweep
(it is not a transcription failure); a hung endpoint is failed by the job
timeout, never an eternal spinner; strictly one job at a time (two long jobs on
iguana were never measured); stopping deletes queued audio and refuses new jobs.

Red: uploads.Worker is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:47:22 +02:00
mathiasandClaude Sonnet 5.5 4352446f36 feat(audio): ffmpeg adapter -- always transcode to 16 kHz mono PCM (ADR-033)
Replaces whisper-server --convert. A zero exit that wrote nothing is an error.
Also fixes the red test's stub, which wrote an empty file and so (correctly)
tripped the no-output check.

Checked against real ffmpeg on iguana with the same argv: AAC .m4a in,
pcm_s16le 16000 Hz mono WAV out, duration preserved.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:46:34 +02:00
mathiasandClaude Sonnet 5.5 3bc33fc34c test(audio): ffmpeg transcode to 16 kHz mono PCM, always (red)
Red: audio.FFmpeg is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:46:14 +02:00
mathiasandClaude Sonnet 5.5 2227cfff7e test(transcribe): opt-in live check against the real gateway
The httptest fake proves only what we THINK the gateway accepts. Skipped
unless TAPIR_LIVE_WAV and TAPIR_LIVE_GATEWAY_URL are set. Run once on
2026-10-01 with 11 min of synthetic Swedish (21 MB, above the 8-10 MB ingress
cap, via the NodePort): both iguana/kb-whisper (52.8 s) and berget/kb-whisper
(35.4 s) accepted the streamed, computed-Content-Length body and returned
usable verbose_json. Synthetic audio only; this repo is public.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:46:14 +02:00
mathiasandClaude Sonnet 5.5 cae423750c feat(transcribe): gateway client with an iguana->berget fallback chain (ADR-033)
Streams the WAV from disk with a computed Content-Length (no buffering, no
chunking), posts verbose_json with field `file` and language=sv, clamps the
final segment to the audio duration, and treats an empty transcript as a
failure. The chain falls back on ANY primary error, records the serving
endpoint, and never falls back once the context is cancelled.

gateway.go and chain.go are one commit: the chain's tests use the gateway's
StatusError and ErrEmptyTranscript, so neither compiles alone.

Mutation-checked: field name, clamp, cancel-fallback, content length and the
recorded endpoint each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:44:34 +02:00
mathiasandClaude Sonnet 5.5 bb285d8757 test(transcribe): gateway client and iguana->berget chain (red)
Gateway: field is `file`, no Expect header, a computed Content-Length (not
chunked, not buffered -- a 90 min WAV is ~170 MB), verbose_json normalised
with the last segment clamped to the audio duration, errors independent of
the body, an empty transcript is a failure. Chain: ANY primary failure falls
back and the row records the fallback endpoint; a cancelled job never falls
back to the cloud.

Red: package transcribe does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:43:51 +02:00
mathiasandClaude Sonnet 5.5 e12a5e7e89 feat(web): upload routes -- form, multipart intake, honest status (ADR-033)
GET/POST /uploads, GET /uploads/{id}, GET /uploads/{id}/status, mounted only
when an Uploader is wired. The multipart body is streamed to Intake, never
buffered. Rejections are synchronous: 422 (video, too long, unreadable) or 413
with the user-facing message and no row. Another user's id is a 404 that does
not leak the title. Unfinished states poll; done/failed render without
hx-trigger so nothing polls forever (ADR-025). A done transcript names the
endpoint that served it.

handlers.go, uploads.go and the templ pair are one commit: the App fields and
Store methods are what make the handlers compile.

Mutation-checked: dropping the 413 mapping, the not-found mapping, or letting
a done card poll each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:42:58 +02:00
mathiasandClaude Sonnet 5.5 1bc98813fc test(web): read the response body once; match the message's real case
Two defects in my red tests, found when going green: body() drains the
recorder so a second call read empty, and two assertions expected a different
case than the user-facing message. The assertions were not weakened.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:42:40 +02:00
mathiasandClaude Sonnet 5.5 e1ed6ea6f4 test(web): upload routes -- form, multipart POST, rejections, owner-only, honest status (red)
Real Router + real Intake + real Postgres; only ffprobe and the worker are
faked. Covers: the form states the limits up front; accepted audio queues a
row and redirects; video / too-long / too-large are rejected synchronously
(422/422/413) with a message and NO row; a non-owner gets 404 without the
title leaking; unfinished states keep polling and terminal states stop (no
stuck spinner); a done transcript names the endpoint that served it.

Red: App.Uploads / App.UploadLimits are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:41:42 +02:00
mathiasandClaude Sonnet 5.5 f84389028d feat(audio): ffprobe adapter implementing uploads.Prober (ADR-033)
Cover art is not video; non-zero exit is the file's fault (ErrUnreadable),
a missing binary / cancelled ctx / bad output is ours. WaitDelay bounds a
cancelled probe whose child holds the pipe open.

Checked against real ffprobe on iguana: JSON shape matches the fixtures and
/etc/hosts exits 1 with 'Invalid data found when processing input'. The unit
tests drive a stub process; ffprobe is not installed on koala.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:40:51 +02:00
mathiasandClaude Sonnet 5.5 0ab067c510 test(audio): ffprobe adapter -- cover art is not video, exit code vs missing binary (red)
Red: package audio does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:40:29 +02:00
mathiasandClaude Sonnet 5.5 1910d3dfb5 feat(uploads): Intake -- spool, probe, validate, queue; clean up on every failure (ADR-033)
Bounded read (cap+1 byte tells 'at the limit' from 'over'); a file that fails
validation creates no row; a prober fault is our error, not the user's bad
file; a row whose job cannot be submitted is failed, never left queued.

Mutation-checked: cap off-by-one, missing cleanup and deleting the handed-over
file each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:40:13 +02:00
mathiasandClaude Sonnet 5.5 b98e9df462 test(uploads): Accept -- size cap, probe verdicts, cleanup on every failure path (red)
Red: uploads.Job, Intake and ErrUnreadable are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:40:03 +02:00
mathiasandClaude Sonnet 5.5 e3e1f17c40 feat(uploads): validation policy for probed audio uploads (ADR-033)
Video is checked first so it is always refused as video; limits and reasons
are typed so callers and tests branch on Reason, not message text.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:39:51 +02:00
mathiasandClaude Sonnet 5.5 f42b8e23fc test(uploads): validation policy -- video, no-audio, too-long, unreadable (red)
Probe-based, not extension-based: a Teams voice-only export is typically an
.mp4 with no video stream, indistinguishable from video by name (ADR-033).
Red: package uploads does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:39:20 +02:00
mathiasandClaude Sonnet 5.5 836a501576 feat(store): Store API for private uploaded transcripts (ADR-033)
Create/Start/Complete/Fail/Get/List plus the start-up sweep that fails
unfinished rows as interrupted (audio is not retained, so no resume). Every
call goes through withUser AND names user_id in its predicate, so isolation
does not rest on RLS alone and stays testable under the superuser test
connection; a non-owner gets ErrUploadNotFound, never a distinct forbidden.
List drops transcript text.

Mutation-checked: removing the user predicate from Get or from the UPDATEs
turns the cross-user tests red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:38:25 +02:00
mathiasandClaude Sonnet 5.5 c29c0a7127 feat(domain): types for private uploaded transcripts (ADR-033)
Segment, Transcription, UploadStatus, UploadErrorKind and UploadedTranscript.
Error kinds are typed, never a provider's string. UploadedTranscript is a
distinct type from the shared public-caption Transcript on purpose.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:37:39 +02:00
mathiasandClaude Sonnet 5.5 4b557209c8 test(store): Store API for uploaded transcripts (red)
Lifecycle queued -> transcribing -> done|failed, per-row serving endpoint,
segments JSONB round trip, owner-only get/write (not-found, not forbidden),
list without transcript text, and the start-up sweep that fails unfinished
rows as `interrupted` for every user (audio is not retained, so no resume).

Red: CreateUpload and friends are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:37:30 +02:00
mathiasandClaude Sonnet 5.5 e36c4c1e0c feat(store): migration 017 -- private RLS-scoped uploaded_transcripts (ADR-033)
Separate table, not a user_id on the shared transcripts (ADR-021). ENABLE +
FORCE ROW LEVEL SECURITY with the migration-003 policy, user_id NOT NULL
ON DELETE CASCADE, typed error_kind, per-row serving endpoint, JSONB segments.
CHECK constraints: done => content, failed => error_kind.

Up and down are one commit: golang-migrate pairs them.

Mutation-checked: dropping FORCE, the policy, or the cascade each turn a test
red. FORCE is only observable structurally -- the app role is not the table
owner, so no behavioural test can see it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:36:50 +02:00
mathiasandClaude Sonnet 5.5 066ae6966a test(store): isolation tests for the private uploaded_transcripts table (red)
Drives RLS directly as the non-superuser app role: deny-all with no scope,
own-rows-only scoped reads, zero-row cross-user UPDATE/DELETE, rejected
cross-user INSERT, FORCE + NOT NULL user_id, cascade on DeleteUser. Also pins
the EXACT set of public tables with row security off, so a new table added
without RLS fails CI rather than passing it (ADR-033). The existing test only
named transcripts; #33's "the only non-RLS table" is false -- invitations,
user_identities and schema_migrations are off too.

Red: relation "uploaded_transcripts" does not exist. Set test passes today
and guards the migration that follows.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:36:22 +02:00
mathiasandClaude Sonnet 5.5 38b3f1cf33 docs(adr): ADR-033 upload audio -> private Swedish transcript -> chat (#33)
Proposed, decider pending. Supersedes ADR-007's STT deferral for
user-supplied audio only; the YouTube audio-download rejection stands.
Settles #28 decision 1 as a separate RLS-scoped uploaded_transcripts table
and records the measured length ceiling, the ingress 413, and the 18k-char
chat truncation that #33 assumed did not exist. Two DECISION NEEDED items
(ffmpeg in the runtime image; berget fallback for private audio).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:35:04 +02:00
mathiasandClaude Sonnet 5.5 feb108def3 docs(research): measure the audio length ceiling before designing chunking (#33)
No transcription ceiling found up to 90 min: one 174 MB request through the
LiteLLM NodePort transcribed in 440 s. The first real failure is the public
gateway ingress rejecting bodies over ~8-10 MB with 413, not whisper or a
Metal OOM. Also records that srt arrives wrapped in a JSON envelope, that the
two backends differ in timestamp granularity, and that the existing chat path
truncates at 18k chars (~22 min of speech).

Synthetic Swedish TTS fixture only; no real transcript.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:33:34 +02:00
61 changed files with 870 additions and 3679 deletions
+4 -38
View File
@@ -1434,12 +1434,7 @@ an hour of Swedish *was* an unknown here; it is now measured, see decision 8.)
empty disables), recorded per row, and called out in Open items rather than defaulted empty disables), recorded per row, and called out in Open items rather than defaulted
silently. silently.
- Swedish only (`language=sv`). Language detection and routing (#28 decision 2) are not - Swedish only (`language=sv`). Language detection and routing (#28 decision 2) are not
touched. **Amended 2026-10-04 (#61, interim):** the language is now a per-upload choice on touched.
the form, `sv` (preselected, and what an upload that names none gets) or `auto` ("Other":
`language` is omitted so the engine detects it), a closed set held by the form, the request
code and a CHECK constraint on `uploaded_transcripts.language` (migration 021). Automatic
detection and a general-purpose engine for other languages are still not built (#61); an
`auto` upload goes to the same configured engine, which is not tuned for it.
- `-H 'Expect:'`, multipart field `file` — encoded in the adapter and its tests, not rediscovered. - `-H 'Expect:'`, multipart field `file` — encoded in the adapter and its tests, not rediscovered.
7. **Jobs run in-process, one at a time (ADR-018 stands).** Transcription is minutes (~7.5 min 7. **Jobs run in-process, one at a time (ADR-018 stands).** Transcription is minutes (~7.5 min
@@ -1452,25 +1447,6 @@ an hour of Swedish *was* an unknown here; it is now measured, see decision 8.)
re-uploads. The single-replica assumption gains a second dependent (the semaphore is re-uploads. The single-replica assumption gains a second dependent (the semaphore is
process-local) and is recorded here. process-local) and is recorded here.
*Amended 2026-10-03 (#50, #47).* The process now handles SIGTERM (what Kubernetes sends on
every deploy; only SIGINT was handled, so shutdown code never ran). On shutdown the running job
is recorded `interrupted` at once and the process waits, bounded to 10 s each, for in-flight
requests and for that write; the start-up sweep stays the backstop for a kill. **Known
limitation, deliberately not built: a deploy still loses a job in flight** (finishing it needs
a drain of up to the job time, a ~330 s termination grace and `strategy: Recreate`, i.e.
downtime on every deploy; a 90-minute file takes ~4.5 min and a deploy lands on a running job
perhaps one time in five at current volume). Jobs still queued at shutdown stay `queued` until
the next start's sweep. The user re-uploads, with an honest message.
*Receive limits (#47).* Request bodies are received at most 4 at a time, 2 per user, answered
as busy beyond that. Together with the 20-job queue and the running job that bounds the spool
at about (4 + 20 + 1) x 200 MiB = 5 GiB of the 6 Gi `emptyDir` (infra): raising any of those or
`TAPIR_UPLOAD_MAX_BYTES` must revisit that sum (pinned by `TestNewIntake_Defaults...`, not
enforced against infra). **A slow-drip body holds its slot without a time bound**: the server
sets `ReadHeaderTimeout` only, because a `ReadTimeout` would cut legitimate 200 MiB uploads on
slow links; two slow bodies take a user's two slots and two such users make everyone's upload
busy. It needs a valid session; recorded as a limitation with a follow-up issue.
8. **Chat reuses `chat.Service`; it is not forked.** The uploaded-transcript chat handlers 8. **Chat reuses `chat.Service`; it is not forked.** The uploaded-transcript chat handlers
look the row up through the RLS-scoped store, so a non-owner gets 404 by the same mechanism look the row up through the RLS-scoped store, so a non-owner gets 404 by the same mechanism
as ADR-027's summary gate, **before any model is called**. ADR-027's decisions (ephemeral, as ADR-027's summary gate, **before any model is called**. ADR-027's decisions (ephemeral,
@@ -1555,15 +1531,9 @@ deferred. The scope cut — chat first, no rich analysis, no video — is what k
#33 specifies, with the serving endpoint shown **(recommended, matches the issue)**; (B) #33 specifies, with the serving endpoint shown **(recommended, matches the issue)**; (B)
disabled by default, a failed iguana job fails honestly and the user retries. *Blocks:* disabled by default, a failed iguana job fails honestly and the user retries. *Blocks:*
the production default only; the adapter ships either way. the production default only; the adapter ships either way.
- ~~Verify Tapir's ingress `proxy-body-size` against the 200 MB cap (`infra`).~~ **Resolved 2026-10-03:** real unauthenticated bodies of 2, 30, 120 and 210 MiB all pass NPM (TLS, piguard) and Traefik to tapir (a 302 to login); there is no body limit in that range. Uploads were enabled in production the same day, local engines only (infra#520). - Verify Tapir's ingress `proxy-body-size` against the 200 MB cap (`infra`).
- ~~Per-transcript deletion is not built; account deletion cascades.~~ **Built 2026-10-03 - Per-transcript deletion is not built; account deletion cascades. A private-content store
(#37):** `POST /uploads/{id}/delete`, owner-scoped (a non-owner gets 404 by the same RLS without a delete button is a gap worth a ticket, not a silent omission.
mechanism), confirm-then-delete on the recording's page. A recording that is **queued or
transcribing is refused (409) until it ends**, because the worker would otherwise write to a
row that is gone; a job whose service was lost stays undeletable until the next start's sweep
marks it failed. The refusal reflects the row's state when the delete was attempted: a
recording that finishes in the same instant still gets a 409 once, and a retry deletes it.
The home list now links to `/uploads` when uploads are mounted.
**Reversibility.** Additive: one table (down-migration drops it), one adapter, one upload **Reversibility.** Additive: one table (down-migration drops it), one adapter, one upload
surface, one image change. Removing the upload route removes the feature; `transcripts`, `videos` surface, one image change. Removing the upload route removes the feature; `transcripts`, `videos`
@@ -1650,10 +1620,6 @@ on the slow numbers and is reopened: 31B scored 97% against 92%.
job timeout rises from 30 to 60 minutes: ~4 min of kb-whisper and ~40 s of pianissimo per hour of job timeout rises from 30 to 60 minutes: ~4 min of kb-whisper and ~40 s of pianissimo per hour of
audio, plus the adjudication. audio, plus the adjudication.
**Amended 2026-10-04 (#61, interim):** reconcile is for Swedish. A recording whose language is
`auto` is never sent to the base engine (it is Swedish-only) and is not reconciled: the other
engine transcribes alone, then the cloud fallback if that fails and the job is live, and the
page does not claim a second opinion.
8. **Not decided here.** A cloud adjudicator (rejected: private content); tuning the prompt; making 8. **Not decided here.** A cloud adjudicator (rejected: private content); tuning the prompt; making
`gemma4-31b` the default; a "reconciling" progress state (the page says "transcribing" for the whole `gemma4-31b` the default; a "reconciling" progress state (the page says "transcribing" for the whole
job); feeding the uncertain spots to chat; whether pianissimo should also be a middle tier in the job); feeding the uncertain spots to chat; whether pianissimo should also be a middle tier in the
+1 -1
View File
@@ -34,7 +34,7 @@ tasks:
lint: lint:
desc: golangci-lint (installed by CI; skipped locally if absent). desc: golangci-lint (installed by CI; skipped locally if absent).
cmds: cmds:
- 'if command -v golangci-lint >/dev/null 2>&1; then golangci-lint run ./...; else echo "golangci-lint not installed — skipping (CI installs it)"; fi' - 'command -v golangci-lint >/dev/null 2>&1 && golangci-lint run ./... || echo "golangci-lint not installed — skipping (CI installs it)"'
test: test:
desc: Run all tests. -p 1 serializes packages so the embedded-postgres tests desc: Run all tests. -p 1 serializes packages so the embedded-postgres tests
+16 -17
View File
@@ -17,9 +17,9 @@ import (
"errors" "errors"
"fmt" "fmt"
"log/slog" "log/slog"
"net"
"net/http" "net/http"
"os" "os"
"os/signal"
"sync" "sync"
"time" "time"
@@ -41,7 +41,7 @@ func main() {
} }
log := slog.New(slog.NewTextHandler(os.Stderr, nil)) log := slog.New(slog.NewTextHandler(os.Stderr, nil))
ctx, stop := shutdownContext(context.Background()) ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop() defer stop()
var err error var err error
@@ -215,11 +215,9 @@ func cmdServe(ctx context.Context, log *slog.Logger) error {
if err != nil { if err != nil {
return err return err
} }
var workerDone chan struct{} // nil without uploads: nothing to wait for at shutdown
wireUploads(app, cfg, intake) wireUploads(app, cfg, intake)
if intake != nil { if intake != nil {
workerDone = make(chan struct{}) go worker.Run(ctx)
go func() { worker.Run(ctx); close(workerDone) }()
log.Info("audio upload enabled", "model", cfg.TranscribeModel, "fallback", cfg.TranscribeFallbackModel, log.Info("audio upload enabled", "model", cfg.TranscribeModel, "fallback", cfg.TranscribeFallbackModel,
"max_mb", cfg.UploadMaxBytes>>20, "max_minutes", int(cfg.UploadMaxDuration.Minutes())) "max_mb", cfg.UploadMaxBytes>>20, "max_minutes", int(cfg.UploadMaxDuration.Minutes()))
log.Warn("audio upload assumes a SINGLE replica: the transcription queue is process-local (ADR-033, ADR-018)") log.Warn("audio upload assumes a SINGLE replica: the transcription queue is process-local (ADR-033, ADR-018)")
@@ -352,19 +350,20 @@ func cmdServe(ctx context.Context, log *slog.Logger) error {
}() }()
} }
// Graceful shutdown on signal: stop accepting, drain in-flight requests, and let the // Graceful shutdown on signal: stop accepting, drain in-flight requests.
// worker record the job it was running as interrupted (#50). go func() {
var extra []*http.Server <-ctx.Done()
if metricsSrv != nil { shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
extra = append(extra, metricsSrv) defer cancel()
} _ = srv.Shutdown(shutdownCtx)
ln, err := net.Listen("tcp", srv.Addr) if metricsSrv != nil {
if err != nil { _ = metricsSrv.Shutdown(shutdownCtx)
return fmt.Errorf("serve: listen: %w", err) }
} }()
log.Info("serving web ui", "addr", cfg.HTTPAddr, "user", cfg.UserID) log.Info("serving web ui", "addr", cfg.HTTPAddr, "user", cfg.UserID)
if err := serveHTTP(ctx, srv, ln, extra, workerDone, shutdownGrace); err != nil { if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
return err return fmt.Errorf("serve: %w", err)
} }
return nil return nil
} }
-55
View File
@@ -1,55 +0,0 @@
package main
import (
"context"
"errors"
"fmt"
"net"
"net/http"
"os"
"os/signal"
"syscall"
"time"
)
// shutdownGrace bounds each wait at shutdown (in-flight requests, then the worker). Two of
// them stay inside Kubernetes' default 30 s termination grace.
const shutdownGrace = 10 * time.Second
// shutdownContext is cancelled by SIGINT (a terminal) or SIGTERM (what Kubernetes sends
// on every deploy). Handling only SIGINT meant a deploy killed the process before any
// shutdown code ran (#50).
func shutdownContext(parent context.Context) (context.Context, context.CancelFunc) {
return signal.NotifyContext(parent, os.Interrupt, syscall.SIGTERM)
}
// serveHTTP serves srv on ln until it fails or ctx ends, then shuts every server down and
// waits for in-flight requests and for workerDone (nil: no worker) to finish, each for at
// most grace. It must not return earlier: Serve returns http.ErrServerClosed as soon as
// Shutdown STARTS, and returning from main kills the drain and the worker's last write.
func serveHTTP(ctx context.Context, srv *http.Server, ln net.Listener, extra []*http.Server, workerDone <-chan struct{}, grace time.Duration) error {
errc := make(chan error, 1)
go func() { errc <- srv.Serve(ln) }()
select {
case err := <-errc:
if !errors.Is(err, http.ErrServerClosed) {
return fmt.Errorf("serve: %w", err)
}
return nil
case <-ctx.Done():
}
shutdownCtx, cancel := context.WithTimeout(context.Background(), grace)
defer cancel()
for _, s := range append([]*http.Server{srv}, extra...) {
_ = s.Shutdown(shutdownCtx)
}
if workerDone != nil {
select {
case <-workerDone:
case <-time.After(grace):
}
}
return nil
}
-124
View File
@@ -1,124 +0,0 @@
package main
import (
"context"
"net/http"
"net/http/httptest"
"os"
"sync"
"syscall"
"testing"
"time"
)
// Kubernetes stops a pod with SIGTERM; only SIGINT was handled, so a deploy killed the
// process without running any shutdown code (#50).
func TestShutdownContext_IsCancelledBySIGTERMAndSIGINT(t *testing.T) {
for _, sig := range []syscall.Signal{syscall.SIGTERM, syscall.SIGINT} {
t.Run(sig.String(), func(t *testing.T) {
ctx, stop := shutdownContext(context.Background())
defer stop()
if err := syscall.Kill(os.Getpid(), sig); err != nil {
t.Fatalf("send %v: %v", sig, err)
}
select {
case <-ctx.Done():
case <-time.After(2 * time.Second):
t.Fatalf("context not cancelled by %v", sig)
}
})
}
}
// serveHTTP must not return while a request is in flight or the worker is still recording
// its job: Serve returns http.ErrServerClosed as soon as Shutdown STARTS, and returning
// from it let the process exit before either finished (#50). Each half is tested with the
// other already satisfied, so neither can hold the call open on the other's behalf.
func pendingFor(t *testing.T, returned <-chan error, why string) {
t.Helper()
select {
case err := <-returned:
t.Fatalf("serveHTTP returned (%v) %s", err, why)
case <-time.After(150 * time.Millisecond):
}
}
func TestServeHTTP_DrainsAnInFlightRequestBeforeReturning(t *testing.T) {
release := make(chan struct{})
entered := make(chan struct{})
srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
close(entered)
<-release
_, _ = w.Write([]byte("ok"))
}))
srv.Start()
defer srv.Close()
var once sync.Once
releaseHandler := func() { once.Do(func() { close(release) }) }
defer releaseHandler() // runs before srv.Close: a failing test must not hang on the blocked handler
ctx, cancel := context.WithCancel(context.Background())
workerDone := make(chan struct{})
close(workerDone) // the worker is already done: only the request can hold serveHTTP open
returned := make(chan error, 1)
go func() { returned <- serveHTTP(ctx, srv.Config, srv.Listener, nil, workerDone, 5*time.Second) }()
go func() { _, _ = http.Get(srv.URL) }()
<-entered
cancel()
pendingFor(t, returned, "while a request was still in flight")
releaseHandler()
select {
case err := <-returned:
if err != nil {
t.Fatalf("serveHTTP: %v", err)
}
case <-time.After(2 * time.Second):
t.Fatal("serveHTTP did not return once the request finished")
}
}
func TestServeHTTP_WaitsForTheWorkerBeforeReturning(t *testing.T) {
srv := httptest.NewUnstartedServer(http.NotFoundHandler())
srv.Start()
defer srv.Close()
ctx, cancel := context.WithCancel(context.Background())
workerDone := make(chan struct{})
returned := make(chan error, 1)
go func() { returned <- serveHTTP(ctx, srv.Config, srv.Listener, nil, workerDone, 5*time.Second) }() // no request in flight
cancel()
pendingFor(t, returned, "before the worker finished")
close(workerDone)
select {
case err := <-returned:
if err != nil {
t.Fatalf("serveHTTP: %v", err)
}
case <-time.After(2 * time.Second):
t.Fatal("serveHTTP did not return once the worker finished")
}
}
func TestServeHTTP_GivesUpOnAWorkerThatNeverFinishesAfterTheGrace(t *testing.T) {
srv := httptest.NewUnstartedServer(http.NotFoundHandler())
srv.Start()
defer srv.Close()
ctx, cancel := context.WithCancel(context.Background())
returned := make(chan error, 1)
go func() {
returned <- serveHTTP(ctx, srv.Config, srv.Listener, nil, make(chan struct{}), 200*time.Millisecond)
}()
cancel()
select {
case <-returned:
case <-time.After(3 * time.Second):
t.Fatal("a stuck worker must not hold the process past the grace period")
}
}
+2 -2
View File
@@ -25,7 +25,7 @@ type fakeUploadsStore struct {
failedKinds []domain.UploadErrorKind failedKinds []domain.UploadErrorKind
} }
func (f *fakeUploadsStore) CreateUpload(context.Context, string, string, domain.Language, bool) (string, error) { func (f *fakeUploadsStore) CreateUpload(context.Context, string, string, bool) (string, error) {
return "id", nil return "id", nil
} }
@@ -44,7 +44,7 @@ func (f *fakeUploadsStore) SaveProgress(context.Context, string, string, []domai
return nil return nil
} }
func (f *fakeUploadsStore) Heartbeat(context.Context, string, string, time.Time) error { return nil } func (f *fakeUploadsStore) Heartbeat(context.Context, string, string) error { return nil }
func (f *fakeUploadsStore) FailInterruptedUploads(context.Context) (int, error) { func (f *fakeUploadsStore) FailInterruptedUploads(context.Context) (int, error) {
f.sweeps++ f.sweeps++
+1 -2
View File
@@ -14,8 +14,7 @@ it** — endpoints and aliases drift, and this file is a snapshot (2026-06-06),
not LiteLLM. Use `30401` for off-cluster LiteLLM. `confirm` resolved. not LiteLLM. Use `30401` for off-cluster LiteLLM. `confirm` resolved.
- **Speech-to-text (ADR-033):** the same gateway serves `POST /v1/audio/transcriptions` for - **Speech-to-text (ADR-033):** the same gateway serves `POST /v1/audio/transcriptions` for
`iguana/kb-whisper` (primary, local) and `berget/kb-whisper` (fallback, EU cloud — private audio `iguana/kb-whisper` (primary, local) and `berget/kb-whisper` (fallback, EU cloud — private audio
leaves the homelab if it is used: when the primary fails **or returns an empty transcript**, leaves the homelab if it is used). **Call it via the NodePort / in-cluster address above, NOT
e.g. a silent or music-only file; the transcript page then says so, #52). **Call it via the NodePort / in-cluster address above, NOT
`https://llm-api.d-ma.be`:** that public ingress (openresty) returns `413` for bodies over `https://llm-api.d-ma.be`:** that public ingress (openresty) returns `413` for bodies over
roughly 8–10 MB (~5 minutes of 16 kHz WAV), while the NodePort took 174 MB (90 minutes) in one roughly 8–10 MB (~5 minutes of 16 kHz WAV), while the NodePort took 174 MB (90 minutes) in one
request (measured 2026-10-01, `docs/research/audio-length-ceiling-2026-10-01.md`). Send multipart request (measured 2026-10-01, `docs/research/audio-length-ceiling-2026-10-01.md`). Send multipart
@@ -1,69 +0,0 @@
# ADR-034 second blind check: protocol (ready to run; tapir#33, D9, D10)
This repo is public and the recordings are private, so this file holds the method only. The
sheet, the key and the filled-in verdicts stay outside the repo, and the trial data is deleted
after the check. Results go back here as aggregate numbers, as in
`reconcile-blind-check-2026-10-02.md`.
## What the check decides
ADR-034 (two engines, a local model reconciles their disagreements) is `Proposed` and off in
production. Two conditions were set for keeping it. The first run answered the first one on one
reading only and did not answer the second:
1. **The reconciled reading is right at least 85% of the time.** Open flaw in the bar: it did not
say how ties count. The first run gave 95% (decided items only) or 70% (all 30), so it was met on
one reading and missed on the other.
2. **It is no worse than the better single engine.** Not answered: before ADR-035 the spots the
model *kept* as the base were not stored, so kb-whisper alone could not be compared.
ADR-035 now stores every judged spot (`reconciliation.contested`: both readings, which one was
used, unsure or not, a few words of left context, a time). That makes condition 2 measurable.
## Rules to fix BEFORE looking at any result (the owner confirms or changes them)
| Rule | Proposed |
|---|---|
| Who judges | The speaker, hearing the audio where the text cannot settle it (D9). Not Opus: it cannot hear, and its agreement with the speaker was only shown on items picked as decidable. |
| Sample | 30 spots drawn at random with a fixed seed, from ALL judged spots (changed and kept-base alike), not only the changed ones. Insertions and deletions that cannot be placed are listed separately and not sampled. |
| Verdict per spot | `x`, `y` (the two readings, shown in random order, key kept apart), `neither`, or `cannot tell`. |
| Condition 1 | Reconciled is right on at least 85% of the spots where the judge could tell (`cannot tell` excluded, reported, and `neither` counts as not right). Also report all-30. Pass needs the decided-only figure at 85% or more AND the all-30 figure above 70%. |
| Condition 2 | Over the same spots, the reconciled text is right at least as often as the better single engine (base engine alone is right where the base reading is right; the other engine alone where the alt reading is right). Ties favour the single engine. |
| If it fails | Switch reconcile off by configuration; the single-engine path is unchanged (ADR-034 reversibility). |
Why a fixed all-30 floor: the literal 70% last time sat well under the bar, and "decided only" can
hide a model that mostly lands on spelling differences.
## What a run needs (D10: the owner is present)
- The same real recording as before (about 60 minutes). Only the owner has it.
- A **throwaway** tapir instance with its own Postgres, not the production pod, so production's
key pin and deployment are untouched. Settings: `TAPIR_RECONCILE_BASE_MODEL=iguana/pianissimo`,
`TAPIR_TRANSCRIBE_MODEL=iguana/kb-whisper`, `TAPIR_RECONCILE_MODEL=iguana/gemma4-31b`, cloud
fallbacks empty. Job time was 21 min 49 s per 63.6 min last run.
- iguana with Low Power Mode off and gemma4-31b able to load (the production-iguana model load
that needs a quiet window; D6).
## Steps
1. Start the throwaway instance, upload the recording, wait for `done` (about 22 minutes).
2. Read `reconciliation.contested` from the throwaway database. From it build, **outside the repo**:
`sheet` (spot number, time, left context, reading x, reading y, random order) and `key` (which
of x/y is the base, which is the other engine, which one the reconciler used).
3. The speaker fills in the sheet (about 20 to 30 minutes for 30 spots, audio open at the times).
The agent does not see the key until the sheet is returned.
4. Score with the rules above. Write the aggregate numbers into a dated note next to this one,
and amend ADR-034's validation section (kept, or switched off, and the tie rule used).
5. Delete the throwaway instance, its database and the sheet, as last time.
6. Only if the check passes and the owner wants it in production: enabling reconcile there is
the separate D10 step (it needs `iguana/pianissimo` and the reconciler added to the tapir key
pin in the infra repo plus the env vars on the deployment, `risk/medium`, owner present).
## Open and owner-only
- Confirm or change the rules table above before the run.
- The **tool that builds the sheet and key** is the one piece not in the repo (last time it was a
one-off script). It can be built test-first on request; until then the agent builds it ad hoc
outside the repo.
- A Berget adjudicator (D8, tapir#45) is a different question with its own measurement and is
not part of this check: this check judges the local setup that ADR-034 describes.
@@ -1,115 +0,0 @@
# Sprint: audio upload follow-ups (Sweep Sprint, 2026-10)
Method: Sweep Sprint (phase 1 parallel read-only research per issue: **done**; phase 2 sequential
execution, one issue per commit: **v0.31.1 and v0.32.0 shipped 2026-10-03/04, the owner-held items
below remain open**). Closing oath below.
Milestones: [v0.31.1 - security and robustness](https://git.d-ma.be/mathias/tapir/milestone/10),
[v0.32.0 - audio upload follow-ups](https://git.d-ma.be/mathias/tapir/milestone/11).
Context: v0.31.0 shipped audio upload (ADR-033/034/035) and is live in production behind auth at
`/uploads`, with the Berget cloud fallback on for audio (infra#522). A real 63.7 min upload
transcribed in 2m53s with no errors. The phase-1 investigation found two pre-existing or new
weaknesses an authenticated user can reach, so the first release is a hotfix.
## Oath (closing gates, every clause must hold; blocked = descoped and tracked, never "done")
| # | Clause | Evidence |
|---|--------|----------|
| SG1 | Every issue in the sprint is closed with a linked commit, or moved out with a stated reason | issue list of both milestones |
| SG2 | Each fix has a test that was seen red on main first, named after the mechanism it observes | red/green output in the issue |
| SG3 | `task check` AND `golangci-lint run ./...` exit 0 (checked by `$?`, not by grep); `task check` does not gate lint until #41 | gate output |
| SG4 | Control code (#48, #49, the upload cap in #47) has a recorded adversarial review verified by execution, verdict in the issue | gatekeeper verdict comment |
| SG5 | Docs current in the same commit: ADR-033/035 text, feature file, use-case coverage test | diff |
| SG6 | Each release tagged (v0.31.1, v0.32.0), pushed, deployed, and observed live (pod running the new image, no restart loop, a real upload still works) | `kubectl`/log + one upload |
| SG7 | Learnings written to the brain; deferred work filed as issues; decisions that are the owner's recorded as DECISION NEEDED | brain ids + issue numbers |
## Work order
### v0.31.1 (security and robustness; hotfix, do first)
1. **#41 lint gate.** `task check` swallows lint findings (`|| echo`). Make the gate fail on findings,
fix what it surfaces. First, so every later gate means something. (Decision D1.)
2. **#48 cross-origin POST protection.** `http.NewCrossOriginProtection()` around `Router()`.
Red: foreign-origin POST to `/account/delete`, `/uploads`, `/uploads/{id}/chat` is refused.
3. **#49 ffprobe/ffmpeg limits.** `-max_streams 32` + narrower probe output; probe stdout cap;
probe deadline. Three commits.
4. **#47 concurrent-upload cap (remaining item).** Bound the spool: reservation per accepted upload
against the 6Gi emptyDir so concurrent uploads cannot evict the pod.
5. **#50 SIGTERM interim.** Handle SIGTERM; record the interrupted job at once; fix the false
comment. Full drain recorded as a limitation. (Decision D2.)
Release v0.31.1: one adversarial review round over 2-5, then tag, push, observe.
### v0.32.0 (follow-ups)
6. **#52** label a cloud-served transcript in plain words; document the empty-transcript trigger.
7. **#53** bound the LLM client's response body read.
8. **#51** tracker: save timeout, then app-clock heartbeat.
9. **#37** nav link to uploads and per-recording delete (queued/transcribing: refuse until it
ends). (Decision D3.)
10. **#36** chat default model after measurement. (Decision D4.)
11. **#38** measurements (Berget length limit, large models on production iguana). (D5, D6.)
12. **#46** remainder of the hardening list after the items above are split out.
13. **#45** evaluate Berget as the reconciler: a question, not work. (D8.)
## DECISION NEEDED (owner's call; recommendation first)
| # | Question | Recommendation | Blocks |
|---|----------|----------------|--------|
| D1 | #41: should any linter be allowed to opt out of the gate? | No opt-out; fix findings | #41 |
| D2 | #50: full SIGTERM drain, or record as a limitation and ship the interim fix? | Limitation + interim; a drain costs 4-6 h and downtime on every deploy at current volume | #50 |
| D3 | #37: delete of a queued/transcribing recording | Refuse until it ends | #37 |
| D4 | #36: make a 31B model the chat default? | Only after measurement, conditional | #36 |
| D5 | #38: Berget length measurement needs a throwaway key | Yes, throwaway key | #38 |
| D6 | #38: load large models on production iguana | Yes, in a quiet window with the owner present | #38 |
| D7 | #46: where the ffmpeg tarball lives | Do nothing now; file an issue | none |
| D8 | #45: evaluate Berget as the reconciler | **Decided 2026-10-04 by the owner: yes, Berget may be used, and each user must be able to keep a recording on local models only** (reverses the recommendation). Opt-out first: #60; #45 stays as the model measurement | #60, #45 |
| D9 | Who judges ADR-034's second blind check | The owner; do not reuse the one-off Opus authorisation. Protocol: `docs/research/reconcile-second-check-protocol.md` | ADR-034 |
| D10 | When to enable reconcile in production | Needs `iguana/pianissimo` in the key pin and env vars; owner present | ADR-034 |
## Dropped or reduced, with reasons
- **#46 item 1 (`alis`/`dref` external reference):** checked, not a problem. Under the application's
flags ffmpeg skips the external track and writes a 78-byte WAV.
- **#46 item 2 / #47 item 2:** merged into #49.
- **#47 item 3 (full drain):** reduced to #50 plus a recorded limitation.
- **#46 `Beat` taking the tracker mutex:** incorrect in #46; only `logStill` does. Corrected in #51.
## Human-present items (do not run unattended)
- Production iguana model loads (#38, D6).
- Enabling reconcile in production (D10).
- Any key change or `apply` in infra (infra#523, infra#524 are open and not part of this sprint).
## Release plan
SemVer: v0.31.1 patch (fixes, no new behaviour); v0.32.0 minor (nav link, delete, cloud labelling).
Every push to tapir main redeploys the pod and fails any job in flight: push when no upload is
running (check the pod log first).
## Status (2026-10-04)
**Shipped and deployed:** v0.31.1 (PR #55, merge e037c6b) and v0.32.0 (PR #58, merge 85f4cc3; pod
tapir-597dbfccc-ng78q 1/1, 0 restarts, image 85f4cc3). Each went through an adversarial review
round by agent with the verdicts recorded on the PR (v0.31.1: BLOCK then APPROVE; v0.32.0: BLOCK,
BLOCK, APPROVE, APPROVE; every block was a real defect, fixed test-first).
**Live evidence:** a foreign-origin `POST` to `/account/delete` and to `/uploads/{id}/delete` is
refused (403) on production, a same-origin one reaches the login redirect (302).
**Not observed live (needs an authenticated session, so it is the owner's look; closed on word or screenshot):**
- #52, cloud side: a transcript served by the Berget fallback names `berget`; a reconciled all-local
one shows no note. The local side was observed on 2026-10-04 (a new clip served by iguana: chip
`iguana/…`, no cloud note). The cloud side is carried as an acceptance item in #60.
**Observed live since:** #37 (owner, 2026-10-04: link, confirm-then-delete, no control while transcribing; closed). #50: at the 2026-10-04 rollout the old pod went `Terminating` then `Completed`
(not `Error`), 0 restarts; #50 closed. No job was in flight, so a drained job is still unobserved
(a recorded limitation, D2).
**Still open (owner's call, milestone 11):** #36, #38, #45 (now the Berget adjudicator measurement), #60 (per-recording local-only choice, 4 sub-decisions), #46 remainder, #56 (slow-drip bodies hold
a receive slot), #57 (shutdown logging, second signal). Decisions D4-D6, D9, D10 above are unchanged; D8 is decided (above).
**Deferred and filed from the reviews:** #56, #57, #59 (start-up exits when the identity provider's OIDC
discovery returns 503 (seen once at 19:40 on 2026-10-03; the pod restarted itself and recovered)).
-35
View File
@@ -240,24 +240,6 @@ Feature: Upload an audio recording, get a private Swedish transcript, discuss it
Then it is transcribed anyway and its audio step says how long the audio really is Then it is transcribed anyway and its audio step says how long the audio really is
And a complete recording whose header over-estimated its length is not refused And a complete recording whose header over-estimated its length is not refused
Scenario: A transcript made by the cloud fallback says the audio left the homelab
When the transcript was produced by an endpoint that is not local
Then its page says in plain words that the audio left the homelab and names the provider
And a locally transcribed recording carries no such note
Scenario: I can delete a finished recording
When I confirm deleting a done or failed recording
Then the recording and its transcript are gone and I am back on my list
And another user's recording is a 404 and is untouched
Scenario: A recording still being transcribed cannot be deleted
When I try to delete a recording that is queued or transcribing
Then I am told it is still being transcribed and nothing is removed
Scenario: The home list links to my recordings when uploads are on
Given audio upload is enabled
Then the home list links to the recordings page
Scenario: A full queue is a retry, not an error Scenario: A full queue is a retry, not an error
When too many recordings are already waiting When too many recordings are already waiting
Then the upload is answered as busy, its row is failed honestly and its audio is deleted Then the upload is answered as busy, its row is failed honestly and its audio is deleted
@@ -274,20 +256,3 @@ Feature: Upload an audio recording, get a private Swedish transcript, discuss it
Scenario: Input that cannot be right is a client error Scenario: Input that cannot be right is a client error
When the id is not an id, or a header or title holds control bytes When the id is not an id, or a header or title holds control bytes
Then the answer is a 4xx or the title is cleaned, never a server error Then the answer is a 4xx or the title is cleaned, never a server error
Scenario: I choose the language of a recording
When I upload a recording and choose Swedish or Other on the form
Then the recording is transcribed as that language and its page says which
Scenario: Swedish stays the default
When an upload names no language
Then it is transcribed as Swedish, as before the choice existed
Scenario: A language outside the list is refused
When an upload names a language the form does not offer
Then it is answered 400 before anything is stored or sent to an engine
Scenario: A recording in another language never reaches the Swedish-only engine
When I choose Other for a recording
Then the language is left for the engine to detect and the Swedish-only base engine and the reconcile are not used
And the engine that does run is the configured one, which is not tuned for other languages
+2 -6
View File
@@ -47,15 +47,11 @@ func (f *FFmpeg) ToWAV(ctx context.Context, src, dst string) error {
args = append(args, "-f", "wav", dst) args = append(args, "-f", "wav", dst)
cmd := exec.CommandContext(ctx, bin, args...) cmd := exec.CommandContext(ctx, bin, args...)
cmd.WaitDelay = killGrace cmd.WaitDelay = killGrace
// Only a bounded head of ffmpeg's output is kept: a hostile file made it print ~0.4 if out, err := cmd.CombinedOutput(); err != nil {
// bytes per input byte, all of which CombinedOutput would buffer and copy (#49).
out := &headWriter{max: maxStderr}
cmd.Stdout, cmd.Stderr = out, out
if err := cmd.Run(); err != nil {
if ctxErr := ctx.Err(); ctxErr != nil { if ctxErr := ctx.Err(); ctxErr != nil {
return fmt.Errorf("audio: ffmpeg: %w", ctxErr) return fmt.Errorf("audio: ffmpeg: %w", ctxErr)
} }
return fmt.Errorf("audio: ffmpeg: %w: %s", err, strings.TrimSpace(out.buf.String())) return fmt.Errorf("audio: ffmpeg: %w: %s", err, strings.TrimSpace(string(out)))
} }
// A zero exit that wrote nothing is still a failure: never hand whisper a // A zero exit that wrote nothing is still a failure: never hand whisper a
// missing file and let the error surface three layers later. // missing file and let the error surface three layers later.
-29
View File
@@ -109,32 +109,3 @@ printf data > "$last"`)
require.NoError(t, err) require.NoError(t, err)
assert.NotContains(t, " "+strings.ReplaceAll(string(raw), "\n", " ")+" ", " -t ") assert.NotContains(t, " "+strings.ReplaceAll(string(raw), "\n", " ")+" ", " -t ")
} }
func TestFFmpegToWAV_CapsTheStreamCountBeforeTheInput(t *testing.T) {
// ffmpeg also allocated ~570 MB opening the 900-stream file (#49).
rec := filepath.Join(t.TempDir(), "argv")
bin := stub(t, `printf '%s\n' "$@" > `+rec+`
for a; do last="$a"; done
printf data > "$last"`)
err := (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/spool/in", filepath.Join(t.TempDir(), "o.wav"))
require.NoError(t, err)
args := argvOf(t, rec)
ms, in := indexOf(args, "-max_streams"), indexOf(args, "-i")
require.GreaterOrEqual(t, ms, 0, "argv: %v", args)
assert.Equal(t, "32", args[ms+1])
assert.Less(t, ms, in, "an input option placed after -i is applied to the output")
}
func TestFFmpegToWAV_ABoundedPartOfAFloodingStderrIsKept(t *testing.T) {
// A hostile mp3 made ffmpeg print ~0.4 bytes of stderr per input byte (83 MB for a
// 209 MB file), all of it buffered, then copied twice into the error (#49 review).
bin := stub(t, `head -c 5000000 /dev/zero | tr '\0' 'e' >&2; exit 1`)
err := (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/spool/in", filepath.Join(t.TempDir(), "o.wav"))
require.Error(t, err)
assert.Less(t, len(err.Error()), 16*1024, "the error carries a bounded head of stderr, not all of it")
assert.Contains(t, err.Error(), "eeee", "and still carries some of it")
}
+9 -85
View File
@@ -4,7 +4,6 @@
package audio package audio
import ( import (
"bytes"
"context" "context"
"encoding/json" "encoding/json"
"errors" "errors"
@@ -20,51 +19,6 @@ import (
type FFprobe struct { type FFprobe struct {
// Bin is the ffprobe executable ("ffprobe" resolves via PATH). // Bin is the ffprobe executable ("ffprobe" resolves via PATH).
Bin string Bin string
// MaxOutput bounds how much of ffprobe's stdout is read; zero means DefaultMaxOutput.
MaxOutput int
// Timeout bounds one probe; zero means DefaultProbeTimeout. A probe is milliseconds of
// work on a real file, so one that runs this long is the FILE's doing (#49).
Timeout time.Duration
}
// DefaultProbeTimeout is the probe deadline when FFprobe.Timeout is unset.
const DefaultProbeTimeout = 45 * time.Second
// DefaultMaxOutput is far above the narrowed probe output of a legitimate file (a few
// hundred bytes) and far below what a hostile one can make ffprobe print.
const DefaultMaxOutput = 1 << 20
// maxStderr is how much of ffprobe's stderr is kept for the error message.
const maxStderr = 4096
// errOutputCap is the write error that stops ffprobe once it prints past the cap.
var errOutputCap = errors.New("audio: ffprobe output exceeds the cap")
// capWriter collects up to max bytes and then refuses further writes, which closes the
// pipe and ends the process instead of buffering whatever it prints.
type capWriter struct {
buf bytes.Buffer
max int
}
func (w *capWriter) Write(p []byte) (int, error) {
if w.buf.Len()+len(p) > w.max {
return 0, errOutputCap
}
return w.buf.Write(p)
}
// headWriter keeps the first max bytes and silently drops the rest.
type headWriter struct {
buf bytes.Buffer
max int
}
func (w *headWriter) Write(p []byte) (int, error) {
if room := w.max - w.buf.Len(); room > 0 {
w.buf.Write(p[:min(room, len(p))])
}
return len(p), nil
} }
// AllowedDemuxers are the input formats an upload may be. The input is // AllowedDemuxers are the input formats an upload may be. The input is
@@ -76,22 +30,9 @@ func (w *headWriter) Write(p []byte) (int, error) {
// formats. "mov" covers .mp4/.m4a/.3gp; "matroska" covers .webm. // formats. "mov" covers .mp4/.m4a/.3gp; "matroska" covers .webm.
const AllowedDemuxers = "wav,mov,mp4,m4a,3gp,mp3,ogg,matroska,webm,flac,aac,opus,aiff,amr,asf" const AllowedDemuxers = "wav,mov,mp4,m4a,3gp,mp3,ogg,matroska,webm,flac,aac,opus,aiff,amr,asf"
// MaxStreams bounds how many streams a file may declare. A 1.3 MB file declaring 900 of
// them cost ~570 MB of RSS in each of ffprobe and ffmpeg (#49), against a 512 Mi pod;
// real recordings have a handful.
const MaxStreams = "32"
// inputRestrictions are ffmpeg/ffprobe INPUT options and must precede the input. // inputRestrictions are ffmpeg/ffprobe INPUT options and must precede the input.
// Local files and pipes only: no network, no other protocols, and a bounded stream count. // Local files and pipes only: no network, no other protocols.
var inputRestrictions = []string{ var inputRestrictions = []string{"-protocol_whitelist", "file,pipe", "-format_whitelist", AllowedDemuxers}
"-protocol_whitelist", "file,pipe",
"-format_whitelist", AllowedDemuxers,
"-max_streams", MaxStreams,
}
// probeEntries is exactly what ParseProbe reads. -show_format/-show_streams print every
// tag too, and a 120 MB WAV made 491 MB of JSON that Output() buffers.
const probeEntries = "format=duration:stream=codec_type:stream_disposition=attached_pic"
// killGrace bounds how long Probe waits for a cancelled ffprobe's pipes to close. // killGrace bounds how long Probe waits for a cancelled ffprobe's pipes to close.
const killGrace = 2 * time.Second const killGrace = 2 * time.Second
@@ -107,38 +48,21 @@ func (f *FFprobe) Probe(ctx context.Context, path string) (uploads.Probe, error)
bin = "ffprobe" bin = "ffprobe"
} }
args := append([]string{"-v", "error"}, inputRestrictions...) args := append([]string{"-v", "error"}, inputRestrictions...)
args = append(args, "-print_format", "json", "-show_entries", probeEntries, path) args = append(args, "-print_format", "json", "-show_format", "-show_streams", path)
timeout := f.Timeout cmd := exec.CommandContext(ctx, bin, args...)
if timeout <= 0 {
timeout = DefaultProbeTimeout
}
probeCtx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
cmd := exec.CommandContext(probeCtx, bin, args...)
cmd.WaitDelay = killGrace cmd.WaitDelay = killGrace
maxOut := f.MaxOutput out, err := cmd.Output()
if maxOut <= 0 { if err != nil {
maxOut = DefaultMaxOutput if ctxErr := ctx.Err(); ctxErr != nil {
}
stdout, stderr := &capWriter{max: maxOut}, &headWriter{max: maxStderr}
cmd.Stdout, cmd.Stderr = stdout, stderr
if err := cmd.Run(); err != nil {
if ctxErr := ctx.Err(); ctxErr != nil { // the caller gave up: not a verdict on the file
return uploads.Probe{}, fmt.Errorf("audio: ffprobe: %w", ctxErr) return uploads.Probe{}, fmt.Errorf("audio: ffprobe: %w", ctxErr)
} }
if probeCtx.Err() != nil { // our own deadline passed while the caller was still waiting
return uploads.Probe{}, fmt.Errorf("%w: ffprobe did not finish within %s", uploads.ErrUnreadable, timeout)
}
if errors.Is(err, errOutputCap) {
return uploads.Probe{}, fmt.Errorf("%w: ffprobe printed more than %d bytes", uploads.ErrUnreadable, maxOut)
}
var exit *exec.ExitError var exit *exec.ExitError
if errors.As(err, &exit) { if errors.As(err, &exit) {
return uploads.Probe{}, fmt.Errorf("%w: ffprobe: %s", uploads.ErrUnreadable, stderr.buf.String()) return uploads.Probe{}, fmt.Errorf("%w: ffprobe: %s", uploads.ErrUnreadable, exit.Stderr)
} }
return uploads.Probe{}, fmt.Errorf("audio: run ffprobe: %w", err) return uploads.Probe{}, fmt.Errorf("audio: run ffprobe: %w", err)
} }
p, err := ParseProbe(stdout.buf.Bytes()) p, err := ParseProbe(out)
if err != nil { if err != nil {
return uploads.Probe{}, fmt.Errorf("audio: ffprobe output: %w", err) return uploads.Probe{}, fmt.Errorf("audio: ffprobe output: %w", err)
} }
-75
View File
@@ -184,78 +184,3 @@ func TestAllowedDemuxers_ExcludeThePlaylistFormats(t *testing.T) {
assert.Contains(t, strings.Split(audio.AllowedDemuxers, ","), good) assert.Contains(t, strings.Split(audio.AllowedDemuxers, ","), good)
} }
} }
func TestFFprobeProbe_CapsTheStreamCountBeforeTheInput(t *testing.T) {
// A 1.3 MB file declaring 900 streams cost ~570 MB of RSS in ffprobe (#49) against
// a 512 Mi pod. -max_streams is an input option: it must precede the path.
bin, argvFile := argvStub(t)
_, err := (&audio.FFprobe{Bin: bin}).Probe(context.Background(), "/spool/upload-1")
require.NoError(t, err)
args := argvOf(t, argvFile)
ms := indexOf(args, "-max_streams")
require.GreaterOrEqual(t, ms, 0, "argv: %v", args)
assert.Equal(t, "32", args[ms+1])
assert.Less(t, ms, len(args)-1, "must come before the path, which stays last")
}
func TestFFprobeProbe_AsksOnlyForTheFieldsParseProbeReads(t *testing.T) {
// -show_format/-show_streams print every tag: a 120 MB WAV produced 491 MB of JSON
// that cmd.Output() buffers in Go (#49).
bin, argvFile := argvStub(t)
_, err := (&audio.FFprobe{Bin: bin}).Probe(context.Background(), "/spool/upload-1")
require.NoError(t, err)
args := argvOf(t, argvFile)
assert.Equal(t, -1, indexOf(args, "-show_format"), "argv: %v", args)
assert.Equal(t, -1, indexOf(args, "-show_streams"), "argv: %v", args)
se := indexOf(args, "-show_entries")
require.GreaterOrEqual(t, se, 0, "argv: %v", args)
assert.Equal(t, "format=duration:stream=codec_type:stream_disposition=attached_pic", args[se+1])
}
func TestFFprobeProbe_OutputPastTheCapIsUnreadableNotBuffered(t *testing.T) {
// ffprobe's stdout is attacker-influenced (tags, 491 MB observed). Past the cap the
// probe is stopped and the file is reported as unreadable (#49).
bin := stub(t, `head -c 200000 /dev/zero | tr '\0' 'a'`)
_, err := (&audio.FFprobe{Bin: bin, MaxOutput: 4096}).Probe(context.Background(), "/x")
assert.ErrorIs(t, err, uploads.ErrUnreadable)
}
func TestFFprobeProbe_OutputWithinTheCapIsParsed(t *testing.T) {
bin := stub(t, `echo '{"streams":[{"codec_type":"audio"}],"format":{"duration":"90"}}'`)
got, err := (&audio.FFprobe{Bin: bin, MaxOutput: 4096}).Probe(context.Background(), "/x")
require.NoError(t, err)
assert.True(t, got.HasAudio)
}
func TestFFprobeProbe_AHungProbeTimesOutOnItsOwnDeadlineAsUnreadableNotAsOurFault(t *testing.T) {
// Accept runs the probe on the request context, which has no deadline of its own;
// a probe that never returns would pin a request (#49). `exec` so the sleep is the
// process itself and not a child holding the pipe open.
bin := stub(t, `exec sleep 30`)
start := time.Now()
_, err := (&audio.FFprobe{Bin: bin, Timeout: 200 * time.Millisecond}).Probe(context.Background(), "/x")
assert.ErrorIs(t, err, uploads.ErrUnreadable)
assert.Less(t, time.Since(start), 3*time.Second)
}
func TestFFprobeProbe_ParentCancellationIsStillNotUnreadableWhenATimeoutIsSet(t *testing.T) {
bin := stub(t, `exec sleep 30`)
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
_, err := (&audio.FFprobe{Bin: bin, Timeout: time.Minute}).Probe(ctx, "/x")
require.Error(t, err)
assert.False(t, errors.Is(err, uploads.ErrUnreadable), "the caller giving up is not a verdict on the file")
assert.ErrorIs(t, err, context.DeadlineExceeded)
}
-156
View File
@@ -1,156 +0,0 @@
package llm
// Observes how much of a gateway response the client will read. The body is read before
// the status is looked at, so a misbehaving gateway or proxy that streams forever was
// bounded only by the client's overall timeout (#53).
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"time"
)
// endlessGateway answers with the given status and streams filler until the client hangs
// up; wrote reports how many bytes it managed to send.
func endlessGateway(t *testing.T, status int, wrote *atomic.Int64) *httptest.Server {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(status)
chunk := []byte(strings.Repeat("a", 4096))
for {
n, err := w.Write(chunk)
wrote.Add(int64(n))
if err != nil {
return
}
if f, ok := w.(http.Flusher); ok {
f.Flush()
}
}
}))
t.Cleanup(srv.Close)
return srv
}
func TestClient_ASuccessBodyLargerThanTheCapIsAnErrorAndTheClientStopsReading(t *testing.T) {
var wrote atomic.Int64
srv := endlessGateway(t, http.StatusOK, &wrote)
c := New(srv.URL, "", "m", 30*time.Second, WithMaxResponseBytes(8192))
start := time.Now()
_, err := c.Complete(context.Background(), "s", "u")
if err == nil || !strings.Contains(err.Error(), "exceeds") {
t.Fatalf("want a response-too-large error, got %v", err)
}
if time.Since(start) > 5*time.Second {
t.Fatalf("the client kept reading: took %s", time.Since(start))
}
// Hanging up lets the server's writes fail; allow what the transport had already buffered.
time.Sleep(100 * time.Millisecond)
if got := wrote.Load(); got > 8192+1<<20 {
t.Fatalf("server managed to send %d bytes after a cap of 8192", got)
}
}
func TestClient_AFailingBodyIsReadOnlyUpToTheErrorBound(t *testing.T) {
var wrote atomic.Int64
srv := endlessGateway(t, http.StatusBadGateway, &wrote)
c := New(srv.URL, "", "m", 30*time.Second)
start := time.Now()
_, err := c.Complete(context.Background(), "s", "u")
he, ok := err.(*HTTPError)
if !ok {
t.Fatalf("want *HTTPError, got %T: %v", err, err)
}
if he.Code != http.StatusBadGateway {
t.Fatalf("code %d", he.Code)
}
if len(he.Body) > maxErrorBody+1 {
t.Fatalf("kept %d bytes of an error body, bound is %d", len(he.Body), maxErrorBody)
}
if time.Since(start) > 5*time.Second {
t.Fatalf("the client kept reading an endless error body: took %s", time.Since(start))
}
}
// sizedGateway answers 200 with a valid completion padded with trailing whitespace to
// exactly size bytes, so the boundary of the cap can be probed to the byte.
func sizedGateway(t *testing.T, size int) *httptest.Server {
t.Helper()
const base = `{"choices":[{"message":{"content":"ok"}}]}`
payload := base + strings.Repeat(" ", size-len(base))
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(payload))
}))
t.Cleanup(srv.Close)
return srv
}
func TestClient_ABodyOfExactlyTheCapIsReadAndOneByteMoreIsRefused(t *testing.T) {
const capBytes = 4096
for _, tt := range []struct {
name string
size int
wantErr bool
}{
{"one under", capBytes - 1, false},
{"exactly the cap", capBytes, false},
{"one over", capBytes + 1, true},
} {
t.Run(tt.name, func(t *testing.T) {
c := New(sizedGateway(t, tt.size).URL, "", "m", 5*time.Second, WithMaxResponseBytes(capBytes))
got, err := c.Complete(context.Background(), "s", "u")
if tt.wantErr {
if err == nil || !strings.Contains(err.Error(), "exceeds") {
t.Fatalf("want a response-too-large error, got %q, %v", got, err)
}
return
}
if err != nil || got != "ok" {
t.Fatalf("want ok, got %q, %v", got, err)
}
})
}
}
// New is what production calls: no option. The default is therefore the live bound.
func TestClient_TheDefaultCapIsEightMiBWhenNoOptionIsGiven(t *testing.T) {
for _, tt := range []struct {
name string
size int
wantErr bool
}{
{"exactly 8 MiB", 8 << 20, false},
{"8 MiB plus one", 8<<20 + 1, true},
} {
t.Run(tt.name, func(t *testing.T) {
c := New(sizedGateway(t, tt.size).URL, "", "m", 10*time.Second)
_, err := c.Complete(context.Background(), "s", "u")
if (err != nil) != tt.wantErr {
t.Fatalf("size %d: err = %v, wantErr %v", tt.size, err, tt.wantErr)
}
})
}
}
func TestClient_ANonPositiveCapOptionKeepsTheDefault(t *testing.T) {
for _, n := range []int64{0, -1} {
c := New(sizedGateway(t, 1024).URL, "", "m", 5*time.Second, WithMaxResponseBytes(n))
if got, err := c.Complete(context.Background(), "s", "u"); err != nil || got != "ok" {
t.Fatalf("WithMaxResponseBytes(%d): want ok from the default cap, got %q, %v", n, got, err)
}
}
}
+3 -25
View File
@@ -37,21 +37,6 @@ type Client struct {
temp float64 temp float64
httpClient *http.Client httpClient *http.Client
usageHook func(model string, prompt, completion int) usageHook func(model string, prompt, completion int)
maxBody int64
}
// defaultMaxResponseBytes bounds a success body. A completion is a few KB of text; this
// leaves room for a long one and none for a gateway that streams forever (#53).
const defaultMaxResponseBytes = 8 << 20
// WithMaxResponseBytes overrides the bound on a success response body. A non-positive n is
// ignored (keeps the default).
func WithMaxResponseBytes(n int64) Option {
return func(c *Client) {
if n > 0 {
c.maxBody = n
}
}
} }
// Option configures a Client at construction. Variadic so the existing 4-arg // Option configures a Client at construction. Variadic so the existing 4-arg
@@ -116,7 +101,6 @@ func New(baseURL, apiKey, model string, timeout time.Duration, opts ...Option) *
maxTokens: defaultMaxTokens, maxTokens: defaultMaxTokens,
temp: defaultTemperature, temp: defaultTemperature,
httpClient: &http.Client{Timeout: timeout}, httpClient: &http.Client{Timeout: timeout},
maxBody: defaultMaxResponseBytes,
} }
for _, opt := range opts { for _, opt := range opts {
opt(c) opt(c)
@@ -200,18 +184,12 @@ func (c *Client) Complete(ctx context.Context, system, user string) (string, err
} }
defer resp.Body.Close() //nolint:errcheck defer resp.Body.Close() //nolint:errcheck
if resp.StatusCode != http.StatusOK { out, err := io.ReadAll(resp.Body)
// Only the bounded head is ever used (HTTPError.Error truncates to it), so read no more.
// A read error here only shortens a diagnostic; the status code below is what callers act on.
head, _ := io.ReadAll(io.LimitReader(resp.Body, maxErrorBody+1))
return "", &HTTPError{Code: resp.StatusCode, Body: string(head)}
}
out, err := io.ReadAll(io.LimitReader(resp.Body, c.maxBody+1))
if err != nil { if err != nil {
return "", fmt.Errorf("read response: %w", err) return "", fmt.Errorf("read response: %w", err)
} }
if int64(len(out)) > c.maxBody { if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("LLM response exceeds %d bytes", c.maxBody) return "", &HTTPError{Code: resp.StatusCode, Body: string(out)}
} }
var cr chatResponse var cr chatResponse
@@ -1,2 +0,0 @@
ALTER TABLE uploaded_transcripts
DROP COLUMN language;
@@ -1,13 +0,0 @@
-- Migration 021: the language an upload is transcribed as (tapir#61).
--
-- 'sv' Swedish: the engines are sent language=sv. Every row before this migration was
-- transcribed that way (the request hard-coded it), so the default is true history.
-- 'auto' Other: the engine detects the language and the Swedish-only engine is not used.
--
-- A closed set held by the table as well as by the form and the request code, so a second
-- writer or a bug cannot store a value the engines are never meant to receive. The column
-- sits on the already-FORCE'd, RLS-scoped uploaded_transcripts, so the exact-set isolation
-- test is unaffected.
ALTER TABLE uploaded_transcripts
ADD COLUMN language TEXT NOT NULL DEFAULT 'sv'
CONSTRAINT uploaded_transcripts_language_check CHECK (language IN ('sv', 'auto'));
+12 -51
View File
@@ -17,10 +17,6 @@ import (
// nothing about whether someone else's upload exists (ADR-033). // nothing about whether someone else's upload exists (ADR-033).
var ErrUploadNotFound = errors.New("store: uploaded transcript not found") var ErrUploadNotFound = errors.New("store: uploaded transcript not found")
// ErrUploadInProgress is returned by DeleteUpload for an upload that is still queued or
// transcribing: the worker would write its result to a row that no longer exists.
var ErrUploadInProgress = errors.New("store: uploaded transcript is still being processed")
// Uploaded transcripts are PRIVATE per-user content, stored in their own // Uploaded transcripts are PRIVATE per-user content, stored in their own
// RLS-scoped table (migration 017) — never in the shared `transcripts` (ADR-021). // RLS-scoped table (migration 017) — never in the shared `transcripts` (ADR-021).
// Every method goes through withUser, so row-level security confines it to the // Every method goes through withUser, so row-level security confines it to the
@@ -31,12 +27,12 @@ var ErrUploadInProgress = errors.New("store: uploaded transcript is still being
// CreateUpload records a new upload in the queued state and returns its id. // CreateUpload records a new upload in the queued state and returns its id.
// hadVideo records that the file carried a video track (only its audio is used); // hadVideo records that the file carried a video track (only its audio is used);
// it is kept for the life of the row so the page can say so. // it is kept for the life of the row so the page can say so.
func (s *Store) CreateUpload(ctx context.Context, userID, title string, lang domain.Language, hadVideo bool) (string, error) { func (s *Store) CreateUpload(ctx context.Context, userID, title string, hadVideo bool) (string, error) {
var id string var id string
if err := s.withUser(ctx, userID, func(tx pgx.Tx) error { if err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
return tx.QueryRow(ctx, return tx.QueryRow(ctx,
`INSERT INTO uploaded_transcripts (user_id, title, language, had_video) VALUES ($1, $2, $3, $4) RETURNING id`, `INSERT INTO uploaded_transcripts (user_id, title, had_video) VALUES ($1, $2, $3) RETURNING id`,
userID, title, string(lang), hadVideo).Scan(&id) userID, title, hadVideo).Scan(&id)
}); err != nil { }); err != nil {
return "", fmt.Errorf("store: create upload: %w", err) return "", fmt.Errorf("store: create upload: %w", err)
} }
@@ -84,56 +80,23 @@ func (s *Store) SaveProgress(ctx context.Context, userID, id string, steps []dom
WHERE id = $1 AND user_id = $2`, b, at) WHERE id = $1 AND user_id = $2`, b, at)
} }
// Heartbeat stamps the upload as alive at the given time. It does not touch progress_at: // Heartbeat stamps the upload as alive now. It does not touch progress_at: alive says
// alive says the service is still reporting, not that work is moving (ADR-035). The time // the service is still reporting, not that work is moving (ADR-035).
// comes from the application clock, the one progress_at and the page's health check use, func (s *Store) Heartbeat(ctx context.Context, userID, id string) error {
// so a database clock that disagrees cannot make a healthy job look lost (#51).
func (s *Store) Heartbeat(ctx context.Context, userID, id string, at time.Time) error {
return s.updateUpload(ctx, userID, id, "heartbeat", return s.updateUpload(ctx, userID, id, "heartbeat",
`UPDATE uploaded_transcripts SET alive_at = $3 `UPDATE uploaded_transcripts SET alive_at = NOW()
WHERE id = $1 AND user_id = $2`, at) WHERE id = $1 AND user_id = $2`)
} }
// FailUpload moves a queued or transcribing upload to failed with a typed reason. A // FailUpload moves an upload to failed with a typed reason.
// finished upload is left as it was: a shutdown can record a job as failed after its
// transcript was already committed (the commit lands, the caller sees a cancelled
// context), and that must not hide the text. Such a call touches no row and returns
// ErrUploadNotFound.
func (s *Store) FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error { func (s *Store) FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error {
return s.updateUpload(ctx, userID, id, "fail upload", return s.updateUpload(ctx, userID, id, "fail upload",
`UPDATE uploaded_transcripts `UPDATE uploaded_transcripts
SET status = 'failed', error_kind = $3, finished_at = NOW() SET status = 'failed', error_kind = $3, finished_at = NOW()
WHERE id = $1 AND user_id = $2 AND status IN ('queued', 'transcribing')`, WHERE id = $1 AND user_id = $2`,
string(kind)) string(kind))
} }
// DeleteUpload removes one of the caller's finished (done or failed) uploads. An upload
// still queued or transcribing is refused with ErrUploadInProgress and left untouched; one
// that is not the caller's is ErrUploadNotFound, as for every other write (#37).
func (s *Store) DeleteUpload(ctx context.Context, userID, id string) error {
return s.withUser(ctx, userID, func(tx pgx.Tx) error {
ct, err := tx.Exec(ctx,
`DELETE FROM uploaded_transcripts
WHERE id = $1 AND user_id = $2 AND status IN ('done', 'failed')`, id, userID)
if err != nil {
return fmt.Errorf("store: delete upload: %w", err)
}
if ct.RowsAffected() > 0 {
return nil
}
var exists bool
if err := tx.QueryRow(ctx,
`SELECT EXISTS (SELECT 1 FROM uploaded_transcripts WHERE id = $1 AND user_id = $2)`,
id, userID).Scan(&exists); err != nil {
return fmt.Errorf("store: delete upload: %w", err)
}
if exists {
return ErrUploadInProgress
}
return ErrUploadNotFound
})
}
// updateUpload runs one UPDATE keyed on ($1 = id, $2 = userID) plus extra args, // updateUpload runs one UPDATE keyed on ($1 = id, $2 = userID) plus extra args,
// and maps "no row touched" to ErrUploadNotFound — which is what a write to another // and maps "no row touched" to ErrUploadNotFound — which is what a write to another
// user's upload looks like under RLS (the row is invisible, not forbidden). // user's upload looks like under RLS (the row is invisible, not forbidden).
@@ -221,7 +184,7 @@ func (s *Store) FailInterruptedUploads(ctx context.Context) (int, error) {
return total, nil return total, nil
} }
const uploadSelect = `SELECT id, user_id, title, language, status, content, segments, endpoint, const uploadSelect = `SELECT id, user_id, title, status, content, segments, endpoint,
audio_seconds, error_kind, had_video, reconciliation, progress, progress_at, alive_at, audio_seconds, error_kind, had_video, reconciliation, progress, progress_at, alive_at,
created_at, finished_at FROM uploaded_transcripts` created_at, finished_at FROM uploaded_transcripts`
@@ -234,13 +197,11 @@ func scanUpload(row pgx.Row, u *domain.UploadedTranscript, withText bool) error
seconds *float64 seconds *float64
finished, progressAt, aliveAt *time.Time finished, progressAt, aliveAt *time.Time
) )
var language string if err := row.Scan(&u.ID, &u.UserID, &u.Title, &status, &content, &segments, &endpoint,
if err := row.Scan(&u.ID, &u.UserID, &u.Title, &language, &status, &content, &segments, &endpoint,
&seconds, &errKind, &u.HadVideo, &reconciliation, &progress, &progressAt, &aliveAt, &seconds, &errKind, &u.HadVideo, &reconciliation, &progress, &progressAt, &aliveAt,
&u.CreatedAt, &finished); err != nil { &u.CreatedAt, &finished); err != nil {
return err return err
} }
u.Language = domain.Language(language)
u.Status = domain.UploadStatus(deref(status)) u.Status = domain.UploadStatus(deref(status))
u.Endpoint = deref(endpoint) u.Endpoint = deref(endpoint)
u.ErrorKind = domain.UploadErrorKind(deref(errKind)) u.ErrorKind = domain.UploadErrorKind(deref(errKind))
@@ -1,68 +0,0 @@
package store_test
// Deleting one recording (#37). A finished or failed upload can be removed by its owner;
// one that is queued or transcribing is refused until it ends, because the worker would
// otherwise write its result to a row that is gone (D3 in the sprint plan).
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/store"
"git.d-ma.be/mathias/tapir/internal/domain"
)
func TestDeleteUpload_RemovesAFinishedOrFailedUploadOfTheOwner(t *testing.T) {
for name, finish := range map[string]func(*store.Store, string) error{
"done": func(s *store.Store, id string) error {
return s.CompleteUpload(context.Background(), userA, id, domain.Transcription{Text: "synthetic", Endpoint: "iguana/kb-whisper"})
},
"failed": func(s *store.Store, id string) error {
return s.FailUpload(context.Background(), userA, id, domain.UploadErrTranscriptionFailed)
},
} {
t.Run(name, func(t *testing.T) {
s := newStore(t)
id := startedUpload(t, s)
require.NoError(t, s.StartTranscribing(context.Background(), userA, id))
require.NoError(t, finish(s, id))
require.NoError(t, s.DeleteUpload(context.Background(), userA, id))
_, err := s.GetUpload(context.Background(), userA, id)
assert.ErrorIs(t, err, store.ErrUploadNotFound)
})
}
}
func TestDeleteUpload_RefusesAnUploadThatIsStillQueuedOrTranscribingAndKeepsTheRow(t *testing.T) {
s := newStore(t)
ctx := context.Background()
queued := startedUpload(t, s)
running, err := s.CreateUpload(ctx, userA, "running", domain.LangSwedish, false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, running))
for _, id := range []string{queued, running} {
assert.ErrorIs(t, s.DeleteUpload(ctx, userA, id), store.ErrUploadInProgress)
_, err := s.GetUpload(ctx, userA, id)
assert.NoError(t, err, "a refused delete must leave the row")
}
}
func TestDeleteUpload_AnotherUsersUploadIsNotFoundAndSurvives(t *testing.T) {
s := newStore(t)
ctx := context.Background()
id := startedUpload(t, s)
require.NoError(t, s.StartTranscribing(ctx, userA, id))
require.NoError(t, s.FailUpload(ctx, userA, id, domain.UploadErrTranscriptionFailed))
assert.ErrorIs(t, s.DeleteUpload(ctx, userB, id), store.ErrUploadNotFound,
"existence is not disclosed to a non-owner")
_, err := s.GetUpload(ctx, userA, id)
assert.NoError(t, err, "B's delete must not have removed A's row")
}
@@ -1,47 +0,0 @@
package store_test
// A shutdown can record a job as failed after its transcript was already committed (the
// commit lands, the caller sees a cancelled context). FailUpload must not be able to turn a
// finished row into a failed one and hide the text (#50 review).
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
)
func TestFailUpload_LeavesAFinishedUploadAsItWas(t *testing.T) {
s := newStore(t)
ctx := context.Background()
id := startedUpload(t, s)
require.NoError(t, s.StartTranscribing(ctx, userA, id))
require.NoError(t, s.CompleteUpload(ctx, userA, id, domain.Transcription{Text: "synthetic text", Endpoint: "iguana/kb-whisper"}))
_ = s.FailUpload(ctx, userA, id, domain.UploadErrInterrupted)
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, domain.UploadDone, got.Status)
assert.Equal(t, "synthetic text", got.Content)
assert.Empty(t, got.ErrorKind)
}
func TestFailUpload_StillFailsAQueuedOrTranscribingUpload(t *testing.T) {
s := newStore(t)
ctx := context.Background()
queued := startedUpload(t, s)
running, err := s.CreateUpload(ctx, userA, "r", domain.LangSwedish, false)
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, running))
for _, id := range []string{queued, running} {
require.NoError(t, s.FailUpload(ctx, userA, id, domain.UploadErrBusy))
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, domain.UploadFailed, got.Status)
}
}
@@ -20,7 +20,7 @@ import (
func startedUpload(t *testing.T, s *store.Store) string { func startedUpload(t *testing.T, s *store.Store) string {
t.Helper() t.Helper()
seedUsers(t, userA, userB) seedUsers(t, userA, userB)
id, err := s.CreateUpload(context.Background(), userA, "m", domain.LangSwedish, false) id, err := s.CreateUpload(context.Background(), userA, "m", false)
require.NoError(t, err) require.NoError(t, err)
return id return id
} }
@@ -68,13 +68,13 @@ func TestUploadProgress_AHeartbeatMovesAliveAtAndNotProgressAt(t *testing.T) {
ctx := context.Background() ctx := context.Background()
at := time.Date(2026, 10, 2, 8, 30, 0, 0, time.UTC) at := time.Date(2026, 10, 2, 8, 30, 0, 0, time.UTC)
require.NoError(t, s.SaveProgress(ctx, userA, id, []domain.ProgressStep{{Key: "audio", Label: "a", State: domain.StepRunning}}, at)) require.NoError(t, s.SaveProgress(ctx, userA, id, []domain.ProgressStep{{Key: "audio", Label: "a", State: domain.StepRunning}}, at))
beatAt := at.Add(90 * time.Second) // deliberately not the database's NOW() before := time.Now().Add(-time.Second)
require.NoError(t, s.Heartbeat(ctx, userA, id, beatAt)) require.NoError(t, s.Heartbeat(ctx, userA, id))
got, err := s.GetUpload(ctx, userA, id) got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err) require.NoError(t, err)
assert.True(t, got.AliveAt.Equal(beatAt), "the heartbeat is stamped with the application's clock, the same one progress_at uses, not the database's NOW() (#51)") assert.True(t, got.AliveAt.After(before), "the heartbeat must stamp the row with now")
assert.True(t, got.ProgressAt.Equal(at), "alive is not progress: a heartbeat must not move progress_at") assert.True(t, got.ProgressAt.Equal(at), "alive is not progress: a heartbeat must not move progress_at")
} }
@@ -99,7 +99,7 @@ func TestUploadProgress_ByAnotherUserChangesNothing(t *testing.T) {
ctx := context.Background() ctx := context.Background()
errSave := s.SaveProgress(ctx, userB, id, []domain.ProgressStep{{Key: "x", Label: "X", State: domain.StepDone}}, time.Now()) errSave := s.SaveProgress(ctx, userB, id, []domain.ProgressStep{{Key: "x", Label: "X", State: domain.StepDone}}, time.Now())
errBeat := s.Heartbeat(ctx, userB, id, time.Now()) errBeat := s.Heartbeat(ctx, userB, id)
assert.ErrorIs(t, errSave, store.ErrUploadNotFound) assert.ErrorIs(t, errSave, store.ErrUploadNotFound)
assert.ErrorIs(t, errBeat, store.ErrUploadNotFound) assert.ErrorIs(t, errBeat, store.ErrUploadNotFound)
@@ -33,7 +33,7 @@ func TestCreateUpload_StartsQueuedAndIsOwnedByTheCreator(t *testing.T) {
seedUsers(t, userA) seedUsers(t, userA)
ctx := context.Background() ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "Teams export", domain.LangSwedish, false) id, err := s.CreateUpload(ctx, userA, "Teams export", false)
require.NoError(t, err) require.NoError(t, err)
got, err := s.GetUpload(ctx, userA, id) got, err := s.GetUpload(ctx, userA, id)
@@ -50,7 +50,7 @@ func TestUploadLifecycle_CompletedKeepsTextSegmentsEndpointAndDuration(t *testin
s := newStore(t) s := newStore(t)
seedUsers(t, userA) seedUsers(t, userA)
ctx := context.Background() ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false) id, err := s.CreateUpload(ctx, userA, "m", false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id)) require.NoError(t, s.StartTranscribing(ctx, userA, id))
@@ -84,7 +84,7 @@ func TestUploadLifecycle_FailedRecordsTypedKindAndNoContent(t *testing.T) {
s := newStore(t) s := newStore(t)
seedUsers(t, userA) seedUsers(t, userA)
ctx := context.Background() ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false) id, err := s.CreateUpload(ctx, userA, "m", false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id)) require.NoError(t, s.StartTranscribing(ctx, userA, id))
@@ -102,7 +102,7 @@ func TestGetUpload_AnotherUsersIDIsNotFound(t *testing.T) {
s := newStore(t) s := newStore(t)
seedUsers(t, userA, userB) seedUsers(t, userA, userB)
ctx := context.Background() ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "A's meeting", domain.LangSwedish, false) id, err := s.CreateUpload(ctx, userA, "A's meeting", false)
require.NoError(t, err) require.NoError(t, err)
_, err = s.GetUpload(ctx, userB, id) _, err = s.GetUpload(ctx, userB, id)
@@ -115,7 +115,7 @@ func TestUploadWrites_ByAnotherUserChangeNothing(t *testing.T) {
s := newStore(t) s := newStore(t)
seedUsers(t, userA, userB) seedUsers(t, userA, userB)
ctx := context.Background() ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "A's meeting", domain.LangSwedish, false) id, err := s.CreateUpload(ctx, userA, "A's meeting", false)
require.NoError(t, err) require.NoError(t, err)
assert.ErrorIs(t, s.StartTranscribing(ctx, userB, id), store.ErrUploadNotFound) assert.ErrorIs(t, s.StartTranscribing(ctx, userB, id), store.ErrUploadNotFound)
@@ -132,11 +132,11 @@ func TestListUploads_ReturnsOnlyTheCallersNewestFirstWithoutTranscriptText(t *te
s := newStore(t) s := newStore(t)
seedUsers(t, userA, userB) seedUsers(t, userA, userB)
ctx := context.Background() ctx := context.Background()
first, err := s.CreateUpload(ctx, userA, "first", domain.LangSwedish, false) first, err := s.CreateUpload(ctx, userA, "first", false)
require.NoError(t, err) require.NoError(t, err)
second, err := s.CreateUpload(ctx, userA, "second", domain.LangSwedish, false) second, err := s.CreateUpload(ctx, userA, "second", false)
require.NoError(t, err) require.NoError(t, err)
_, err = s.CreateUpload(ctx, userB, "B's", domain.LangSwedish, false) _, err = s.CreateUpload(ctx, userB, "B's", false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, first)) require.NoError(t, s.StartTranscribing(ctx, userA, first))
require.NoError(t, s.CompleteUpload(ctx, userA, first, domain.Transcription{Text: "long text body", Endpoint: "e"})) require.NoError(t, s.CompleteUpload(ctx, userA, first, domain.Transcription{Text: "long text body", Endpoint: "e"}))
@@ -160,12 +160,12 @@ func TestFailInterruptedUploads_FailsOnlyUnfinishedRowsForEveryUser(t *testing.T
_, err := super.Exec(ctx, `INSERT INTO user_identities (dex_subject, user_id) VALUES ($1, $2)`, subject, uid) _, err := super.Exec(ctx, `INSERT INTO user_identities (dex_subject, user_id) VALUES ($1, $2)`, subject, uid)
require.NoError(t, err) require.NoError(t, err)
} }
queued, err := s.CreateUpload(ctx, userA, "queued", domain.LangSwedish, false) queued, err := s.CreateUpload(ctx, userA, "queued", false)
require.NoError(t, err) require.NoError(t, err)
running, err := s.CreateUpload(ctx, userB, "running", domain.LangSwedish, false) running, err := s.CreateUpload(ctx, userB, "running", false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userB, running)) require.NoError(t, s.StartTranscribing(ctx, userB, running))
done, err := s.CreateUpload(ctx, userA, "done", domain.LangSwedish, false) done, err := s.CreateUpload(ctx, userA, "done", false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, done)) require.NoError(t, s.StartTranscribing(ctx, userA, done))
require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "kept", Endpoint: "e"})) require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "kept", Endpoint: "e"}))
@@ -191,9 +191,9 @@ func TestCreateUpload_RecordsWhetherTheFileHadAVideoTrack(t *testing.T) {
s := newStore(t) s := newStore(t)
seedUsers(t, userA) seedUsers(t, userA)
ctx := context.Background() ctx := context.Background()
withVideo, err := s.CreateUpload(ctx, userA, "camera off", domain.LangSwedish, true) withVideo, err := s.CreateUpload(ctx, userA, "camera off", true)
require.NoError(t, err) require.NoError(t, err)
audioOnly, err := s.CreateUpload(ctx, userA, "voice only", domain.LangSwedish, false) audioOnly, err := s.CreateUpload(ctx, userA, "voice only", false)
require.NoError(t, err) require.NoError(t, err)
gotVideo, err := s.GetUpload(ctx, userA, withVideo) gotVideo, err := s.GetUpload(ctx, userA, withVideo)
@@ -216,9 +216,9 @@ func TestUploadHadVideo_SurvivesCompletionAndFailure(t *testing.T) {
s := newStore(t) s := newStore(t)
seedUsers(t, userA) seedUsers(t, userA)
ctx := context.Background() ctx := context.Background()
done, err := s.CreateUpload(ctx, userA, "d", domain.LangSwedish, true) done, err := s.CreateUpload(ctx, userA, "d", true)
require.NoError(t, err) require.NoError(t, err)
failed, err := s.CreateUpload(ctx, userA, "f", domain.LangSwedish, true) failed, err := s.CreateUpload(ctx, userA, "f", true)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, done)) require.NoError(t, s.StartTranscribing(ctx, userA, done))
require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "x", Endpoint: "e"})) require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "x", Endpoint: "e"}))
@@ -235,7 +235,7 @@ func TestUploadLifecycle_ReconciliationRoundTripsWithItsContestedSpots(t *testin
s := newStore(t) s := newStore(t)
seedUsers(t, userA) seedUsers(t, userA)
ctx := context.Background() ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false) id, err := s.CreateUpload(ctx, userA, "m", false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id)) require.NoError(t, s.StartTranscribing(ctx, userA, id))
want := &domain.Reconciliation{ want := &domain.Reconciliation{
@@ -258,7 +258,7 @@ func TestUploadLifecycle_ASingleEngineTranscriptHasNoReconciliation(t *testing.T
s := newStore(t) s := newStore(t)
seedUsers(t, userA) seedUsers(t, userA)
ctx := context.Background() ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false) id, err := s.CreateUpload(ctx, userA, "m", false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id)) require.NoError(t, s.StartTranscribing(ctx, userA, id))
require.NoError(t, s.CompleteUpload(ctx, userA, id, domain.Transcription{Text: "x", Endpoint: "e"})) require.NoError(t, s.CompleteUpload(ctx, userA, id, domain.Transcription{Text: "x", Endpoint: "e"}))
@@ -273,7 +273,7 @@ func TestUploadLifecycle_ASkippedReconciliationKeepsItsReason(t *testing.T) {
s := newStore(t) s := newStore(t)
seedUsers(t, userA) seedUsers(t, userA)
ctx := context.Background() ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false) id, err := s.CreateUpload(ctx, userA, "m", false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id)) require.NoError(t, s.StartTranscribing(ctx, userA, id))
rec := &domain.Reconciliation{Status: domain.ReconcileSkipped, Reason: "the model failed 3 times in a row: gateway down", Errors: 3} rec := &domain.Reconciliation{Status: domain.ReconcileSkipped, Reason: "the model failed 3 times in a row: gateway down", Errors: 3}
@@ -284,55 +284,3 @@ func TestUploadLifecycle_ASkippedReconciliationKeepsItsReason(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, rec, got.Reconciliation) assert.Equal(t, rec, got.Reconciliation)
} }
func TestCreateUpload_RecordsTheLanguageTheUserChoseAndTheListCarriesIt(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
swedish, err := s.CreateUpload(ctx, userA, "sv", domain.LangSwedish, false)
require.NoError(t, err)
auto, err := s.CreateUpload(ctx, userA, "auto", domain.LangAuto, false)
require.NoError(t, err)
gotSv, err := s.GetUpload(ctx, userA, swedish)
require.NoError(t, err)
gotAuto, err := s.GetUpload(ctx, userA, auto)
require.NoError(t, err)
listed, err := s.ListUploads(ctx, userA)
require.NoError(t, err)
assert.Equal(t, domain.LangSwedish, gotSv.Language)
assert.Equal(t, domain.LangAuto, gotAuto.Language)
byID := map[string]domain.Language{}
for _, u := range listed {
byID[u.ID] = u.Language
}
assert.Equal(t, map[string]domain.Language{swedish: domain.LangSwedish, auto: domain.LangAuto}, byID)
}
// The closed set is also held by the table, so a bug or a second writer cannot store a
// value the engines are never meant to receive (tapir#61).
func TestUploadedTranscripts_TheTableRefusesALanguageOutsideTheClosedSet(t *testing.T) {
newStore(t)
seedUsers(t, userA)
_, err := rawPool(t).Exec(context.Background(),
`INSERT INTO uploaded_transcripts (user_id, title, language) VALUES ($1, 'x', 'en')`, userA)
require.Error(t, err)
assert.Contains(t, err.Error(), "language", "the CHECK constraint names the column")
}
// A row from before the column existed was transcribed as Swedish; the default says so.
func TestUploadedTranscripts_ARowInsertedWithoutALanguageIsSwedish(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
var id string
require.NoError(t, rawPool(t).QueryRow(context.Background(),
`INSERT INTO uploaded_transcripts (user_id, title) VALUES ($1, 'old') RETURNING id`, userA).Scan(&id))
got, err := s.GetUpload(context.Background(), userA, id)
require.NoError(t, err)
assert.Equal(t, domain.LangSwedish, got.Language)
}
+8 -43
View File
@@ -32,24 +32,14 @@ func NewEnsemble(base, alt, fallback Backend, adj reconcile.Adjudicator, opts re
return &Ensemble{base: base, alt: alt, fallback: fallback, adj: adj, opts: opts} return &Ensemble{base: base, alt: alt, fallback: fallback, adj: adj, opts: opts}
} }
// Plan implements uploads.Planner: the phases of a Swedish job, the two-engine one. The // Plan implements uploads.Planner: the phases this ensemble reports, so the page can
// fallback is a phase only when one is configured. // show them before they run. The fallback is a phase only when one is configured.
func (e *Ensemble) Plan() []progress.Planned { return e.PlanFor(domain.LangSwedish) } func (e *Ensemble) Plan() []progress.Planned {
plan := []progress.Planned{
// PlanFor implements uploads.LanguagePlanner. A recording that is not Swedish never runs the {Key: "engine_a", Label: "Transcribing with " + e.base.Name()},
// base engine, the comparison or the decisions (transcribeOther), so the page lists only {Key: "engine_b", Label: "Transcribing with " + e.alt.Name()},
// what can run: the other engine, then the fallback if one is configured. {Key: "compare", Label: "Comparing the two transcripts"},
func (e *Ensemble) PlanFor(lang domain.Language) []progress.Planned { {Key: "decide", Label: "Deciding the differences"},
var plan []progress.Planned
if lang == domain.LangSwedish || lang == "" {
plan = []progress.Planned{
{Key: "engine_a", Label: "Transcribing with " + e.base.Name()},
{Key: "engine_b", Label: "Transcribing with " + e.alt.Name()},
{Key: "compare", Label: "Comparing the two transcripts"},
{Key: "decide", Label: "Deciding the differences"},
}
} else {
plan = []progress.Planned{{Key: "engine_b", Label: "Transcribing with " + e.alt.Name()}}
} }
if e.fallback != nil { if e.fallback != nil {
plan = append(plan, progress.Planned{Key: "fallback", Label: "Cloud fallback (" + e.fallback.Name() + ")"}) plan = append(plan, progress.Planned{Key: "fallback", Label: "Cloud fallback (" + e.fallback.Name() + ")"})
@@ -61,9 +51,6 @@ func (e *Ensemble) PlanFor(lang domain.Language) []progress.Planned {
// progress.Reporter in ctx (a no-op when nobody listens). // progress.Reporter in ctx (a no-op when nobody listens).
func (e *Ensemble) Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error) { func (e *Ensemble) Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error) {
rep := progress.From(ctx) rep := progress.From(ctx)
if domain.LanguageFrom(ctx) != domain.LangSwedish {
return e.transcribeOther(ctx, rep, wavPath)
}
b, berr := e.run(ctx, rep, "engine_a", e.base, wavPath) b, berr := e.run(ctx, rep, "engine_a", e.base, wavPath)
if err := ctx.Err(); err != nil { if err := ctx.Err(); err != nil {
return domain.Transcription{}, err return domain.Transcription{}, err
@@ -102,28 +89,6 @@ func (e *Ensemble) Transcribe(ctx context.Context, wavPath string) (domain.Trans
return e.single(f, e.fallback.Name(), "both local engines failed"), nil return e.single(f, e.fallback.Name(), "both local engines failed"), nil
} }
// transcribeOther handles a recording that is not Swedish (tapir#61). The base engine is
// Swedish-only, so it never receives the audio, and with one reading there is nothing to
// reconcile: the other engine transcribes alone, then the cloud fallback if it fails and the
// job is still live. The phases that did not run are shown as skipped by the tracker.
func (e *Ensemble) transcribeOther(ctx context.Context, rep progress.Reporter, wavPath string) (domain.Transcription, error) {
t, err := e.run(ctx, rep, "engine_b", e.alt, wavPath)
if err == nil {
t.Endpoint = e.alt.Name()
return t, nil
}
if e.fallback == nil || ctx.Err() != nil {
return domain.Transcription{}, fmt.Errorf("%s: %w", e.alt.Name(), err)
}
f, ferr := e.run(ctx, rep, "fallback", e.fallback, wavPath)
if ferr != nil {
return domain.Transcription{}, errors.Join(
fmt.Errorf("%s: %w", e.alt.Name(), err), fmt.Errorf("%s: %w", e.fallback.Name(), ferr))
}
f.Endpoint = e.fallback.Name()
return f, nil
}
// run is one engine call, reported as a phase. A job that was stopped is not a broken // run is one engine call, reported as a phase. A job that was stopped is not a broken
// engine, so a cancelled context is never reported as a failed phase. // engine, so a cancelled context is never reported as a failed phase.
func (e *Ensemble) run(ctx context.Context, rep progress.Reporter, key string, be Backend, wav string) (domain.Transcription, error) { func (e *Ensemble) run(ctx context.Context, rep progress.Reporter, key string, be Backend, wav string) (domain.Transcription, error) {
@@ -160,25 +160,3 @@ func TestEnsembleProgress_TheReconcileCountsAreNotedForTheOperator(t *testing.T)
assert.Equal(t, 0, n["unsure"]) assert.Equal(t, 0, n["unsure"])
assert.Equal(t, 0, n["errors"]) assert.Equal(t, 0, n["errors"])
} }
// An auto-language upload never runs the base engine, the comparison or the decisions, so the
// page must not list them as coming (tapir#61, PR #62 review).
func TestEnsemblePlanFor_AnAutoLanguageListsOnlyThePhasesThatCanRun(t *testing.T) {
with := newEnsemble("a", "a", nil, nil, true, &verdictAdj{}).e
without := newEnsemble("a", "a", nil, nil, false, &verdictAdj{}).e
keys := func(p []progress.Planned) []string {
var k []string
for _, s := range p {
k = append(k, s.Key)
}
return k
}
assert.Equal(t, []string{"engine_b", "fallback"}, keys(with.PlanFor(domain.LangAuto)))
assert.Equal(t, []string{"engine_b"}, keys(without.PlanFor(domain.LangAuto)), "no fallback configured: none promised")
assert.Equal(t, with.Plan(), with.PlanFor(domain.LangSwedish), "Swedish is the plan it always was")
assert.Contains(t, with.PlanFor(domain.LangAuto)[0].Label, "iguana/kb-whisper")
assert.Equal(t, with.Plan(), with.PlanFor(""), "a job queued before the choice existed is Swedish")
assert.Equal(t, with.PlanFor(domain.LangAuto), with.PlanFor(domain.Language("xx")),
"an unknown language is planned as it is run: not the Swedish phases")
}
@@ -204,52 +204,3 @@ func TestEnsemble_AShutdownDuringTheSecondEngineIsAnErrorNotASingleEngineSuccess
assert.ErrorIs(t, err, context.Canceled, "a cancelled job must not be recorded as 'kb-whisper failed'") assert.ErrorIs(t, err, context.Canceled, "a cancelled job must not be recorded as 'kb-whisper failed'")
assert.Empty(t, got.Text) assert.Empty(t, got.Text)
} }
// The base engine is Swedish-only. A recording that is not Swedish must never reach it, and
// there is no second opinion to reconcile: the other engine transcribes alone (tapir#61).
func autoCtx() context.Context { return domain.WithLanguage(context.Background(), domain.LangAuto) }
func TestEnsemble_AnAutoLanguageUploadNeverReachesTheBaseEngineAndIsNotReconciled(t *testing.T) {
adj := &verdictAdj{v: reconcile.VerdictAlt}
r := newEnsemble("base text", "alt text", nil, nil, true, adj)
got, err := r.e.Transcribe(autoCtx(), "/a.wav")
require.NoError(t, err)
assert.Equal(t, []string{"iguana/kb-whisper"}, *r.calls, "only the other engine is called")
assert.Zero(t, adj.calls, "nothing is adjudicated")
assert.Equal(t, "alt text", got.Text)
assert.Equal(t, "iguana/kb-whisper", got.Endpoint)
assert.Nil(t, got.Reconciliation, "reconciliation does not apply, so none is recorded")
}
func TestEnsemble_AnAutoLanguageUploadFallsBackWhenItsEngineFailsAndStillSkipsTheBaseEngine(t *testing.T) {
r := newEnsemble("base text", "alt text", nil, errors.New("engine down"), true, &verdictAdj{})
got, err := r.e.Transcribe(autoCtx(), "/a.wav")
require.NoError(t, err)
assert.Equal(t, []string{"iguana/kb-whisper", "berget/kb-whisper"}, *r.calls)
assert.Equal(t, "berget/kb-whisper", got.Endpoint)
}
func TestEnsemble_AnAutoLanguageUploadWithNoFallbackFailsWithoutTouchingTheBaseEngine(t *testing.T) {
r := newEnsemble("base text", "alt text", nil, errors.New("engine down"), false, &verdictAdj{})
_, err := r.e.Transcribe(autoCtx(), "/a.wav")
require.Error(t, err)
assert.Equal(t, []string{"iguana/kb-whisper"}, *r.calls)
}
func TestEnsemble_ACancelledAutoLanguageJobDoesNotReachTheFallback(t *testing.T) {
r := newEnsemble("base text", "alt text", nil, errors.New("engine down"), true, &verdictAdj{})
ctx, cancel := context.WithCancel(autoCtx())
cancel()
_, err := r.e.Transcribe(ctx, "/a.wav")
require.Error(t, err)
assert.NotContains(t, *r.calls, "berget/kb-whisper", "a stopped job must not ship its audio to a cloud endpoint")
}
+1 -12
View File
@@ -71,20 +71,9 @@ func (g *Gateway) Transcribe(ctx context.Context, path string) (domain.Transcrip
return domain.Transcription{}, fmt.Errorf("transcribe: stat audio: %w", err) return domain.Transcription{}, fmt.Errorf("transcribe: stat audio: %w", err)
} }
// The language is one request field and a closed set: sv is sent, auto sends nothing so
// the engine detects it, and anything else never leaves this process (tapir#61).
fields := [][2]string{{"model", g.model}, {"response_format", "verbose_json"}}
switch lang := domain.LanguageFrom(ctx); lang {
case domain.LangSwedish:
fields = append(fields, [2]string{"language", string(lang)})
case domain.LangAuto:
default:
return domain.Transcription{}, fmt.Errorf("transcribe: language %q is not one tapir sends", lang)
}
var prefix bytes.Buffer var prefix bytes.Buffer
mw := multipart.NewWriter(&prefix) mw := multipart.NewWriter(&prefix)
for _, kv := range fields { for _, kv := range [][2]string{{"model", g.model}, {"response_format", "verbose_json"}, {"language", "sv"}} {
if err := mw.WriteField(kv[0], kv[1]); err != nil { if err := mw.WriteField(kv[0], kv[1]); err != nil {
return domain.Transcription{}, fmt.Errorf("transcribe: build request: %w", err) return domain.Transcription{}, fmt.Errorf("transcribe: build request: %w", err)
} }
@@ -255,48 +255,3 @@ func TestGateway_AnErrorBodyIsBoundedToTwoHundredBytesInTheStatusError(t *testin
require.ErrorAs(t, err, &se) require.ErrorAs(t, err, &se)
assert.LessOrEqual(t, len(se.Snippet), 200, "a gateway's whole body must not become part of an error") assert.LessOrEqual(t, len(se.Snippet), 200, "a gateway's whole body must not become part of an error")
} }
// The language travels on the context (tapir#61); the request field is the only place it
// takes effect, so these observe the request the gateway fake received.
func TestGateway_SendsLanguageSvWhenTheContextCarriesNoneAndForSwedish(t *testing.T) {
for name, ctx := range map[string]context.Context{
"no language in the context": context.Background(),
"swedish": domain.WithLanguage(context.Background(), domain.LangSwedish),
} {
t.Run(name, func(t *testing.T) {
var got seen
srv := serve(t, 200, verboseJSON, &got)
g := transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client())
_, err := g.Transcribe(ctx, wavFile(t, "RIFF-synthetic-pcm"))
require.NoError(t, err)
assert.Equal(t, "sv", got.fields["language"])
})
}
}
func TestGateway_OmitsTheLanguageFieldForAnAutoContextSoTheEngineDetectsIt(t *testing.T) {
var got seen
srv := serve(t, 200, verboseJSON, &got)
g := transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client())
_, err := g.Transcribe(domain.WithLanguage(context.Background(), domain.LangAuto), wavFile(t, "RIFF-synthetic-pcm"))
require.NoError(t, err)
_, present := got.fields["language"]
assert.False(t, present, "an auto upload must not be forced to Swedish")
assert.Equal(t, "iguana/kb-whisper", got.fields["model"], "the rest of the request is unchanged")
}
func TestGateway_ALanguageOutsideTheClosedSetIsRefusedBeforeAnythingIsSent(t *testing.T) {
var got seen
srv := serve(t, 200, verboseJSON, &got)
g := transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client())
_, err := g.Transcribe(domain.WithLanguage(context.Background(), domain.Language("en")), wavFile(t, "RIFF-synthetic-pcm"))
require.Error(t, err)
assert.Empty(t, got.method, "no request may reach the engine with a language the closed set does not hold")
}
@@ -102,45 +102,3 @@ func TestEnsemble_AHungEngineIsASoftFailureAndTheOtherEnginesTextIsUsed(t *testi
assert.Contains(t, got.Reconciliation.Reason, "timeout") assert.Contains(t, got.Reconciliation.Reason, "timeout")
assert.False(t, errors.Is(err, context.DeadlineExceeded)) assert.False(t, errors.Is(err, context.DeadlineExceeded))
} }
// WithTimeout sits in front of both local engines in production (cmd/tapir/uploads.go). It
// must hand the engine the job's context, not a fresh one: the language and the
// cancellation both travel on it (tapir#61).
func TestWithTimeout_TheEngineStillReceivesTheLanguageOfTheJob(t *testing.T) {
for name, tt := range map[string]struct {
ctx context.Context
wantField bool
}{
"auto leaves the field out": {domain.WithLanguage(context.Background(), domain.LangAuto), false},
"swedish sends sv": {domain.WithLanguage(context.Background(), domain.LangSwedish), true},
} {
t.Run(name, func(t *testing.T) {
var got seen
srv := serve(t, 200, verboseJSON, &got)
b := transcribe.WithTimeout(transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client()), time.Minute)
_, err := b.Transcribe(tt.ctx, wavFile(t, "RIFF-synthetic-pcm"))
require.NoError(t, err)
lang, present := got.fields["language"]
assert.Equal(t, tt.wantField, present)
if tt.wantField {
assert.Equal(t, "sv", lang)
}
})
}
}
func TestWithTimeout_ACancelledJobIsNotRestartedByTheWrapper(t *testing.T) {
var got seen
srv := serve(t, 200, verboseJSON, &got)
b := transcribe.WithTimeout(transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client()), time.Minute)
ctx, cancel := context.WithCancel(context.Background())
cancel()
_, err := b.Transcribe(ctx, wavFile(t, "RIFF-synthetic-pcm"))
require.Error(t, err)
assert.Empty(t, got.method, "a cancelled job must not send its audio anywhere through the wrapper")
}
+3 -6
View File
@@ -243,12 +243,9 @@ const (
// never stored in, or merged with, the shared public-caption Transcript (ADR-021). // never stored in, or merged with, the shared public-caption Transcript (ADR-021).
// Content and Segments are empty until Status is UploadDone. // Content and Segments are empty until Status is UploadDone.
type UploadedTranscript struct { type UploadedTranscript struct {
ID string ID string
UserID string UserID string
Title string Title string
// Language is what the upload was (or will be) transcribed as: sv, or auto for
// "Other" (tapir#61).
Language Language
Status UploadStatus Status UploadStatus
Content string Content string
Segments []Segment Segments []Segment
-47
View File
@@ -1,47 +0,0 @@
package domain
import "context"
// Language is the spoken language an upload is transcribed as. It is chosen by the user on
// the upload form and reaches the engines as one request field, so it is a closed set,
// never free text (tapir#61).
type Language string
const (
// LangSwedish sends language=sv: the Swedish engines' own language, and what every
// upload was transcribed as before the choice existed.
LangSwedish Language = "sv"
// LangAuto sends no language, so the engine detects it. It is what "Other" on the form
// means: the Swedish-only engines are not used for it.
LangAuto Language = "auto"
)
// ParseLanguage reads a form value. An empty value is Swedish (a client that predates the
// field); anything that is not exactly a known value is refused rather than normalised.
func ParseLanguage(s string) (Language, bool) {
switch Language(s) {
case "", LangSwedish:
return LangSwedish, true
case LangAuto:
return LangAuto, true
}
return "", false
}
type languageKey struct{}
// WithLanguage returns a context that carries the language of the job it serves. The
// Transcriber interface does not change: engines read it with LanguageFrom, the same way
// they read the progress Reporter (ADR-035).
func WithLanguage(ctx context.Context, l Language) context.Context {
return context.WithValue(ctx, languageKey{}, l)
}
// LanguageFrom returns the language carried by ctx, or Swedish when it carries none, so a
// caller that sets nothing behaves as it always did.
func LanguageFrom(ctx context.Context) Language {
if l, ok := ctx.Value(languageKey{}).(Language); ok && l != "" {
return l
}
return LangSwedish
}
-41
View File
@@ -1,41 +0,0 @@
package domain_test
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"git.d-ma.be/mathias/tapir/internal/domain"
)
func TestParseLanguage_KnownValuesAndEmptyAreAcceptedEverythingElseIsNot(t *testing.T) {
tests := []struct {
in string
want domain.Language
wantOK bool
}{
{"sv", domain.LangSwedish, true},
{"auto", domain.LangAuto, true},
{"", domain.LangSwedish, true}, // a client that sends no field keeps today's behaviour
{"SV", "", false}, // exact match: the value reaches a request field
{" sv", "", false},
{"sv\n", "", false},
{"en", "", false}, // a specific other language is not offered yet
{"sv,en", "", false},
{"../sv", "", false},
}
for _, tt := range tests {
t.Run(tt.in, func(t *testing.T) {
got, ok := domain.ParseLanguage(tt.in)
assert.Equal(t, tt.wantOK, ok)
assert.Equal(t, tt.want, got)
})
}
}
func TestLanguageFrom_IsSwedishWhenTheContextCarriesNoneAndOtherwiseWhatWasSet(t *testing.T) {
assert.Equal(t, domain.LangSwedish, domain.LanguageFrom(context.Background()))
ctx := domain.WithLanguage(context.Background(), domain.LangAuto)
assert.Equal(t, domain.LangAuto, domain.LanguageFrom(ctx))
}
+6 -22
View File
@@ -37,13 +37,12 @@ type Reporter interface {
// is for people, so a failed write is logged and never stops the job. // is for people, so a failed write is logged and never stops the job.
type Sink interface { type Sink interface {
SaveProgress(ctx context.Context, steps []domain.ProgressStep, at time.Time) error SaveProgress(ctx context.Context, steps []domain.ProgressStep, at time.Time) error
Beat(ctx context.Context, at time.Time) error Beat(ctx context.Context) error
} }
const ( const (
defaultMinSave = 2 * time.Second defaultMinSave = 2 * time.Second
defaultSaveTimeout = 5 * time.Second logAdvanceEach = time.Minute
logAdvanceEach = time.Minute
) )
// Option tunes a Tracker. // Option tunes a Tracker.
@@ -56,14 +55,6 @@ func WithClock(now func() time.Time) Option { return func(t *Tracker) { t.now =
// Done, Fail and Finish always save. // Done, Fail and Finish always save.
func WithMinSave(d time.Duration) Option { return func(t *Tracker) { t.minSave = d } } func WithMinSave(d time.Duration) Option { return func(t *Tracker) { t.minSave = d } }
// WithSaveTimeout bounds one save or heartbeat write. Progress is cosmetic, so a database
// that stops answering costs a dropped write rather than an unbounded stall. It is still a
// delay: saves run under the tracker's lock, which Begin and Advance (called by the engines)
// need, so each transition can wait up to this long, and a hung database slows a job by
// that much per call, out of the job's own time budget (#51). A dropped final save is not
// retried; the row keeps its last saved step.
func WithSaveTimeout(d time.Duration) Option { return func(t *Tracker) { t.saveTimeout = d } }
// Tracker implements Reporter for one job. // Tracker implements Reporter for one job.
type Tracker struct { type Tracker struct {
ctx context.Context ctx context.Context
@@ -72,8 +63,6 @@ type Tracker struct {
now func() time.Time now func() time.Time
minSave time.Duration minSave time.Duration
saveTimeout time.Duration
mu sync.Mutex mu sync.Mutex
steps []domain.ProgressStep steps []domain.ProgressStep
lastAdvanceSave time.Time lastAdvanceSave time.Time
@@ -83,7 +72,7 @@ type Tracker struct {
// NewTracker plans a job's phases. ctx is used for the saves, so it should outlive the // NewTracker plans a job's phases. ctx is used for the saves, so it should outlive the
// job's own timeout: a job that timed out still gets its failure recorded. // job's own timeout: a job that timed out still gets its failure recorded.
func NewTracker(ctx context.Context, sink Sink, log *slog.Logger, plan []Planned, opts ...Option) *Tracker { func NewTracker(ctx context.Context, sink Sink, log *slog.Logger, plan []Planned, opts ...Option) *Tracker {
t := &Tracker{ctx: ctx, sink: sink, log: log, now: time.Now, minSave: defaultMinSave, saveTimeout: defaultSaveTimeout} t := &Tracker{ctx: ctx, sink: sink, log: log, now: time.Now, minSave: defaultMinSave}
for _, o := range opts { for _, o := range opts {
o(t) o(t)
} }
@@ -223,10 +212,7 @@ func (t *Tracker) Heartbeat(ctx context.Context, beatEvery, stillEvery time.Dura
case <-ctx.Done(): case <-ctx.Done():
return return
case <-beat.C: case <-beat.C:
beatCtx, cancel := context.WithTimeout(ctx, t.saveTimeout) if err := t.sink.Beat(ctx); err != nil && ctx.Err() == nil {
err := t.sink.Beat(beatCtx, t.now())
cancel()
if err != nil && ctx.Err() == nil {
t.log.Warn("progress: heartbeat not saved", "err", err) t.log.Warn("progress: heartbeat not saved", "err", err)
} }
case <-still.C: case <-still.C:
@@ -268,9 +254,7 @@ func (t *Tracker) find(key string) *domain.ProgressStep {
// dropped: progress is cosmetic and must never fail the job it describes. // dropped: progress is cosmetic and must never fail the job it describes.
func (t *Tracker) save() { func (t *Tracker) save() {
snapshot := append([]domain.ProgressStep(nil), t.steps...) snapshot := append([]domain.ProgressStep(nil), t.steps...)
ctx, cancel := context.WithTimeout(t.ctx, t.saveTimeout) if err := t.sink.SaveProgress(t.ctx, snapshot, t.now()); err != nil && t.ctx.Err() == nil {
defer cancel()
if err := t.sink.SaveProgress(ctx, snapshot, t.now()); err != nil && t.ctx.Err() == nil {
t.log.Warn("progress: not saved", "err", err) t.log.Warn("progress: not saved", "err", err)
} }
} }
-154
View File
@@ -1,154 +0,0 @@
package progress_test
// Observes what a database that stops answering does to the job it reports on (#51):
// progress is cosmetic, so a save that never returns must not stall the engine calls that
// report through the tracker, nor the heartbeat that says the process is alive.
import (
"bytes"
"context"
"log/slog"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/progress"
)
// stallSink never answers until its context ends, like a database connection that hung.
type stallSink struct{ beats atomic.Int32 }
func (s *stallSink) SaveProgress(ctx context.Context, _ []domain.ProgressStep, _ time.Time) error {
<-ctx.Done()
return ctx.Err()
}
func (s *stallSink) Beat(ctx context.Context, _ time.Time) error {
s.beats.Add(1)
<-ctx.Done()
return ctx.Err()
}
type lockedBuf struct {
mu sync.Mutex
b bytes.Buffer
}
func (l *lockedBuf) Write(p []byte) (int, error) {
l.mu.Lock()
defer l.mu.Unlock()
return l.b.Write(p)
}
func (l *lockedBuf) String() string {
l.mu.Lock()
defer l.mu.Unlock()
return l.b.String()
}
func stallTracker(sink progress.Sink, buf *lockedBuf) *progress.Tracker {
log := slog.New(slog.NewTextHandler(buf, nil))
return progress.NewTracker(context.Background(), sink, log,
[]progress.Planned{{Key: "a", Label: "A"}}, progress.WithSaveTimeout(50*time.Millisecond))
}
func TestTracker_AHungSaveDoesNotBlockTheReporterPastItsTimeout(t *testing.T) {
buf := &lockedBuf{}
tr := stallTracker(&stallSink{}, buf)
done := make(chan struct{})
go func() { tr.Begin("a", 0); tr.Advance("a", 1); tr.Done("a"); close(done) }()
select {
case <-done:
case <-time.After(3 * time.Second):
t.Fatal("the reporter was blocked by a save that never returned")
}
require.Contains(t, buf.String(), "progress: not saved", "a dropped save is logged")
}
func TestTracker_AHungBeatDoesNotStopTheHeartbeat(t *testing.T) {
sink := &stallSink{}
tr := stallTracker(sink, &lockedBuf{})
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go tr.Heartbeat(ctx, 5*time.Millisecond, time.Hour)
require.Eventually(t, func() bool { return sink.beats.Load() >= 3 }, 3*time.Second, 5*time.Millisecond,
"each beat is abandoned at its timeout and the next one is tried")
}
func TestTracker_TheStillRunningLineIsNotHeldUpBehindAHungSave(t *testing.T) {
buf := &lockedBuf{}
tr := stallTracker(&stallSink{}, buf)
go tr.Begin("a", 0) // holds the tracker's lock inside a save that hangs until its timeout
time.Sleep(10 * time.Millisecond)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go tr.Heartbeat(ctx, time.Hour, 5*time.Millisecond)
require.Eventually(t, func() bool { return strings.Contains(buf.String(), "still running") }, 3*time.Second, 5*time.Millisecond)
}
// deadlineSink records how long each write was given, and answers at once.
type deadlineSink struct {
mu sync.Mutex
saves []time.Duration
beats []time.Duration
noDLOK bool
}
func (d *deadlineSink) record(ctx context.Context, into *[]time.Duration) {
d.mu.Lock()
defer d.mu.Unlock()
dl, ok := ctx.Deadline()
if !ok {
d.noDLOK = true
return
}
*into = append(*into, time.Until(dl))
}
func (d *deadlineSink) SaveProgress(ctx context.Context, _ []domain.ProgressStep, _ time.Time) error {
d.record(ctx, &d.saves)
return nil
}
func (d *deadlineSink) Beat(ctx context.Context, _ time.Time) error {
d.record(ctx, &d.beats)
return nil
}
// The production value is the one the worker gets: it builds a Tracker without options.
// A zero or tiny default would drop every write, a huge one would bound nothing.
func TestTracker_TheDefaultSaveAndBeatDeadlineIsAFewSecondsAway(t *testing.T) {
sink := &deadlineSink{}
tr := progress.NewTracker(context.Background(), sink, slog.New(slog.NewTextHandler(&lockedBuf{}, nil)),
[]progress.Planned{{Key: "a", Label: "A"}})
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go tr.Heartbeat(ctx, 5*time.Millisecond, time.Hour)
tr.Begin("a", 0)
require.Eventually(t, func() bool {
sink.mu.Lock()
defer sink.mu.Unlock()
return len(sink.beats) > 0
}, 3*time.Second, 5*time.Millisecond)
sink.mu.Lock()
defer sink.mu.Unlock()
require.False(t, sink.noDLOK, "a write was given no deadline at all")
require.NotEmpty(t, sink.saves)
for _, d := range append(append([]time.Duration{}, sink.saves...), sink.beats...) {
require.Greater(t, d, time.Second, "the deadline is already (nearly) spent when the write starts")
require.LessOrEqual(t, d, 5*time.Second, "the default bound is 5 s")
}
}
+7 -26
View File
@@ -17,12 +17,11 @@ import (
// fakeSink records every saved snapshot and every heartbeat. // fakeSink records every saved snapshot and every heartbeat.
type fakeSink struct { type fakeSink struct {
mu sync.Mutex mu sync.Mutex
saves [][]domain.ProgressStep saves [][]domain.ProgressStep
at []time.Time at []time.Time
beats int beats int
beatAt []time.Time err error
err error
} }
func (f *fakeSink) SaveProgress(_ context.Context, steps []domain.ProgressStep, at time.Time) error { func (f *fakeSink) SaveProgress(_ context.Context, steps []domain.ProgressStep, at time.Time) error {
@@ -33,11 +32,10 @@ func (f *fakeSink) SaveProgress(_ context.Context, steps []domain.ProgressStep,
return f.err return f.err
} }
func (f *fakeSink) Beat(_ context.Context, at time.Time) error { func (f *fakeSink) Beat(context.Context) error {
f.mu.Lock() f.mu.Lock()
defer f.mu.Unlock() defer f.mu.Unlock()
f.beats++ f.beats++
f.beatAt = append(f.beatAt, at)
return f.err return f.err
} }
@@ -299,7 +297,7 @@ func (r *retainingSink) SaveProgress(_ context.Context, steps []domain.ProgressS
r.got = append(r.got, steps) r.got = append(r.got, steps)
return nil return nil
} }
func (r *retainingSink) Beat(context.Context, time.Time) error { return nil } func (r *retainingSink) Beat(context.Context) error { return nil }
func TestTracker_ASnapshotHandedToTheSinkIsNotChangedByLaterTransitions(t *testing.T) { func TestTracker_ASnapshotHandedToTheSinkIsNotChangedByLaterTransitions(t *testing.T) {
sink := &retainingSink{} sink := &retainingSink{}
@@ -402,20 +400,3 @@ func TestTracker_AnnotateAttachesANoteToAStepAndSavesItAtOnce(t *testing.T) {
require.Equal(t, "20 min of the 79 the file claimed", sink.last(t)[0].Note) require.Equal(t, "20 min of the 79 the file claimed", sink.last(t)[0].Note)
require.NotPanics(t, func() { tr.Annotate("no-such-step", "x") }) require.NotPanics(t, func() { tr.Annotate("no-such-step", "x") })
} }
func TestTracker_BeatIsStampedWithTheTrackersClock(t *testing.T) {
// alive_at used the database's NOW() while the page compared it with the application
// clock; a skewed database clock showed a healthy job as lost (#51).
tr, sink, clk, _ := newTracker(t)
clk.tick(42 * time.Second)
want := clk.now()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go tr.Heartbeat(ctx, 5*time.Millisecond, time.Hour)
require.Eventually(t, func() bool { return sink.beatCount() >= 1 }, 2*time.Second, 5*time.Millisecond)
sink.mu.Lock()
defer sink.mu.Unlock()
require.True(t, sink.beatAt[0].Equal(want), "beat stamped %v, the tracker's clock says %v", sink.beatAt[0], want)
}
+4 -60
View File
@@ -6,7 +6,6 @@ import (
"fmt" "fmt"
"io" "io"
"os" "os"
"sync"
"time" "time"
"git.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
@@ -25,7 +24,7 @@ type Prober interface {
// Store is the slice of the private transcript store the intake needs. // Store is the slice of the private transcript store the intake needs.
type Store interface { type Store interface {
CreateUpload(ctx context.Context, userID, title string, lang domain.Language, hadVideo bool) (string, error) CreateUpload(ctx context.Context, userID, title string, hadVideo bool) (string, error)
FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error
} }
@@ -35,9 +34,6 @@ type Job struct {
UserID string UserID string
UploadID string UploadID string
Path string Path string
// Language is what the user chose for this recording; empty means Swedish (a job
// queued before the choice existed).
Language domain.Language
// Duration is what the file CLAIMED when probed. The worker compares it with what // Duration is what the file CLAIMED when probed. The worker compares it with what
// was really converted; zero means unknown (ADR-035 review). // was really converted; zero means unknown (ADR-035 review).
Duration time.Duration Duration time.Duration
@@ -55,53 +51,6 @@ type Intake struct {
sub Submitter sub Submitter
limits Limits limits Limits
tmpDir string tmpDir string
// MaxReceives and MaxReceivesPerUser bound how many request bodies Accept reads at
// once; zero means the defaults. Without them N parallel POSTs each spool up to
// MaxBytes while the queue still looks empty, and the spool volume fills (#47).
MaxReceives int
MaxReceivesPerUser int
mu sync.Mutex
receives int
byUser map[string]int
}
const (
defaultMaxReceives = 4
defaultMaxReceivesPerUser = 2
)
// acquireReceive reserves a receive slot for userID, or reports there is none. Every
// successful call must be paired with releaseReceive.
func (in *Intake) acquireReceive(userID string) bool {
maxAll, maxUser := in.MaxReceives, in.MaxReceivesPerUser
if maxAll <= 0 {
maxAll = defaultMaxReceives
}
if maxUser <= 0 {
maxUser = defaultMaxReceivesPerUser
}
in.mu.Lock()
defer in.mu.Unlock()
if in.receives >= maxAll || in.byUser[userID] >= maxUser {
return false
}
if in.byUser == nil {
in.byUser = map[string]int{}
}
in.receives++
in.byUser[userID]++
return true
}
func (in *Intake) releaseReceive(userID string) {
in.mu.Lock()
defer in.mu.Unlock()
in.receives--
if in.byUser[userID]--; in.byUser[userID] <= 0 {
delete(in.byUser, userID)
}
} }
// NewIntake wires an Intake. tmpDir is where uploads are spooled; it must exist. // NewIntake wires an Intake. tmpDir is where uploads are spooled; it must exist.
@@ -138,18 +87,13 @@ const busyMessage = "Too many recordings are waiting to be transcribed. Please t
// validation yields a *RejectError and creates NO row — rejection is synchronous // validation yields a *RejectError and creates NO row — rejection is synchronous
// and leaves nothing behind. On success the temp file is handed to the job and the // and leaves nothing behind. On success the temp file is handed to the job and the
// worker deletes it; on every failure path Accept deletes it itself. // worker deletes it; on every failure path Accept deletes it itself.
func (in *Intake) Accept(ctx context.Context, userID, title string, lang domain.Language, src io.Reader) (string, error) { func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reader) (string, error) {
// Refuse before spooling when the queue has no room, if the submitter can say so: a full // Refuse before spooling when the queue has no room, if the submitter can say so: a full
// queue otherwise cost up to the upload cap on disk, an ffprobe run and a failed row for // queue otherwise cost up to the upload cap on disk, an ffprobe run and a failed row for
// every refused attempt. // every refused attempt.
if rc, ok := in.sub.(interface{ HasRoom(userID string) bool }); ok && !rc.HasRoom(userID) { if rc, ok := in.sub.(interface{ HasRoom(userID string) bool }); ok && !rc.HasRoom(userID) {
return "", &RejectError{Reason: ReasonBusy, Message: busyMessage} return "", &RejectError{Reason: ReasonBusy, Message: busyMessage}
} }
// HasRoom cannot see bodies still being received; bound those separately.
if !in.acquireReceive(userID) {
return "", &RejectError{Reason: ReasonBusy, Message: busyMessage}
}
defer in.releaseReceive(userID)
f, err := os.CreateTemp(in.tmpDir, "upload-*") f, err := os.CreateTemp(in.tmpDir, "upload-*")
if err != nil { if err != nil {
return "", fmt.Errorf("uploads: spool: %w", err) return "", fmt.Errorf("uploads: spool: %w", err)
@@ -193,11 +137,11 @@ func (in *Intake) Accept(ctx context.Context, userID, title string, lang domain.
return "", err return "", err
} }
id, err := in.store.CreateUpload(ctx, userID, title, lang, probe.HasVideo) id, err := in.store.CreateUpload(ctx, userID, title, probe.HasVideo)
if err != nil { if err != nil {
return "", fmt.Errorf("uploads: create: %w", err) return "", fmt.Errorf("uploads: create: %w", err)
} }
if err := in.sub.Submit(Job{UserID: userID, UploadID: id, Path: path, Language: lang, Duration: probe.Duration}); err != nil { if err := in.sub.Submit(Job{UserID: userID, UploadID: id, Path: path, Duration: probe.Duration}); err != nil {
if errors.Is(err, ErrQueueFull) { if errors.Is(err, ErrQueueFull) {
// The user's to retry, not a server fault: refuse it as such, and do not leave // The user's to retry, not a server fault: refuse it as such, and do not leave
// the row queued -- that is the stuck spinner. // the row queued -- that is the stuck spinner.
+5 -5
View File
@@ -23,7 +23,7 @@ func TestAccept_AFullQueueIsABusyRejectionNotAServerErrorAndLeavesNothingBehind(
r := newRig(t) r := newRig(t)
r.sub.err = uploads.ErrQueueFull r.sub.err = uploads.ErrQueueFull
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data")) _, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
var rej *uploads.RejectError var rej *uploads.RejectError
require.True(t, errors.As(err, &rej), "a full queue must be a rejection the web layer can answer with 429: %v", err) require.True(t, errors.As(err, &rej), "a full queue must be a rejection the web layer can answer with 429: %v", err)
@@ -48,7 +48,7 @@ func TestAccept_ANoRoomQueueIsRefusedBeforeAnythingIsSpooledProbedOrStored(t *te
full := &roomSubmitter{room: false} full := &roomSubmitter{room: false}
in := uploads.NewIntake(r.st, r.pr, full, uploads.Limits{MaxBytes: 1024, MaxDuration: 90 * time.Minute}, r.tmp) in := uploads.NewIntake(r.st, r.pr, full, uploads.Limits{MaxBytes: 1024, MaxDuration: 90 * time.Minute}, r.tmp)
_, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data")) _, err := in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
var rej *uploads.RejectError var rej *uploads.RejectError
require.True(t, errors.As(err, &rej)) require.True(t, errors.As(err, &rej))
@@ -62,7 +62,7 @@ func TestAccept_ANoRoomQueueIsRefusedBeforeAnythingIsSpooledProbedOrStored(t *te
func TestAccept_ASubmitterThatCannotSayStillWorksAsBefore(t *testing.T) { func TestAccept_ASubmitterThatCannotSayStillWorksAsBefore(t *testing.T) {
r := newRig(t) r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data")) _, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
require.NoError(t, err, "a Submitter without HasRoom is accepted: the check at Submit remains") require.NoError(t, err, "a Submitter without HasRoom is accepted: the check at Submit remains")
} }
@@ -81,7 +81,7 @@ func (b *brokenBody) Read(p []byte) (int, error) {
func TestAccept_AnErrorReadingTheClientsBodyIsMarkedAsTheClientsAndAWrappedSubmitErrorIsNot(t *testing.T) { func TestAccept_AnErrorReadingTheClientsBodyIsMarkedAsTheClientsAndAWrappedSubmitErrorIsNot(t *testing.T) {
r := newRig(t) r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, &brokenBody{n: 1}) _, err := r.in.Accept(context.Background(), "user-1", "t", &brokenBody{n: 1})
require.Error(t, err) require.Error(t, err)
assert.ErrorIs(t, err, uploads.ErrClientRead, "a dropped client body is the sender's to retry") assert.ErrorIs(t, err, uploads.ErrClientRead, "a dropped client body is the sender's to retry")
@@ -90,7 +90,7 @@ func TestAccept_AnErrorReadingTheClientsBodyIsMarkedAsTheClientsAndAWrappedSubmi
// A fault that is ours must NOT carry the marker, even if it wraps the same sentinel. // A fault that is ours must NOT carry the marker, even if it wraps the same sentinel.
r2 := newRig(t) r2 := newRig(t)
r2.sub.err = fmt.Errorf("db: %w", io.ErrUnexpectedEOF) r2.sub.err = fmt.Errorf("db: %w", io.ErrUnexpectedEOF)
_, err = r2.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data")) _, err = r2.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
require.Error(t, err) require.Error(t, err)
assert.NotErrorIs(t, err, uploads.ErrClientRead) assert.NotErrorIs(t, err, uploads.ErrClientRead)
} }
-132
View File
@@ -1,132 +0,0 @@
package uploads_test
// Observes how many request bodies Accept will read at once (#47). HasRoom and the
// per-user queue cap act before and after the receive; N parallel POSTs all pass
// HasRoom while the queue is still empty, so the receive itself is bounded.
import (
"context"
"errors"
"io"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
// slowReceive starts an Accept whose body stays open until the returned writer is closed,
// and waits until it holds a spool file, which proves it is inside the receive.
type slowReceive struct {
w *io.PipeWriter
done chan error
finished bool
}
func startSlowReceive(t *testing.T, r *rig, in *uploads.Intake, user string, wantFiles int) *slowReceive {
t.Helper()
pr, pw := io.Pipe()
s := &slowReceive{w: pw, done: make(chan error, 1)}
go func() {
_, err := in.Accept(context.Background(), user, "t", domain.LangSwedish, pr)
s.done <- err
}()
require.Eventually(t, func() bool { return len(r.tmpFiles(t)) == wantFiles }, 2*time.Second, 5*time.Millisecond,
"the receive for %s should have started", user)
// An abandoned receive ends in a read error, so it never reaches the rig's fakes, which
// are not safe for concurrent use. A receive that finish() already ended is a no-op.
t.Cleanup(func() {
if s.finished {
return
}
_ = pw.CloseWithError(errors.New("test over"))
select {
case <-s.done:
case <-time.After(2 * time.Second):
}
})
return s
}
func (s *slowReceive) finish(t *testing.T) error {
t.Helper()
_, _ = s.w.Write([]byte("data"))
require.NoError(t, s.w.Close())
s.finished = true
select {
case err := <-s.done:
return err
case <-time.After(2 * time.Second):
t.Fatal("Accept did not return after its body ended")
return nil
}
}
func requireBusy(t *testing.T, err error) {
t.Helper()
var rej *uploads.RejectError
require.True(t, errors.As(err, &rej), "want a busy rejection, got %v", err)
assert.Equal(t, uploads.ReasonBusy, rej.Reason)
}
func TestAccept_APerUserLimitOnConcurrentReceivesRefusesTheNextBodyBeforeSpoolingIt(t *testing.T) {
r := newRig(t)
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
in.MaxReceivesPerUser = 2
in.MaxReceives = 10
startSlowReceive(t, r, in, "user-1", 1)
startSlowReceive(t, r, in, "user-1", 2)
_, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
requireBusy(t, err)
assert.Len(t, r.tmpFiles(t), 2, "the refused body was never spooled")
// Another user is not blocked by user-1's two receives.
other := startSlowReceive(t, r, in, "user-2", 3)
require.NoError(t, other.finish(t))
}
func TestAccept_AnOverallLimitOnConcurrentReceivesRefusesTheNextUser(t *testing.T) {
r := newRig(t)
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
in.MaxReceivesPerUser = 5
in.MaxReceives = 3
for i, u := range []string{"a", "b", "c"} {
startSlowReceive(t, r, in, u, i+1)
}
_, err := in.Accept(context.Background(), "d", "t", domain.LangSwedish, strings.NewReader("data"))
requireBusy(t, err)
assert.Len(t, r.tmpFiles(t), 3)
}
func TestAccept_AFinishedReceiveFreesItsSlot(t *testing.T) {
r := newRig(t)
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
in.MaxReceivesPerUser = 1
in.MaxReceives = 5
first := startSlowReceive(t, r, in, "user-1", 1)
require.NoError(t, first.finish(t))
_, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
require.NoError(t, err, "the slot is released when Accept returns")
}
func TestAccept_AFailedReceiveFreesItsSlot(t *testing.T) {
r := newRig(t)
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
in.MaxReceivesPerUser = 1
in.MaxReceives = 5
_, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, &brokenBody{n: 1})
require.ErrorIs(t, err, uploads.ErrClientRead)
_, err = in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
require.NoError(t, err, "a dropped client must not leak a slot")
}
-32
View File
@@ -1,32 +0,0 @@
package uploads_test
// The limits production runs on are the DEFAULTS: nothing outside tests sets
// MaxReceives or MaxReceivesPerUser. Pinned here with the literal numbers, so changing a
// default is a visible decision (#47 review). With 4 receives, 20 queued and 1 running at
// the 200 MiB cap, the spool needs ~5 GiB of the 6 Gi emptyDir (infra); raising any of
// them or TAPIR_UPLOAD_MAX_BYTES must revisit that arithmetic.
import (
"context"
"strings"
"testing"
"time"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
func TestNewIntake_DefaultsAllowTwoReceivesPerUserAndFourOverall(t *testing.T) {
r := newRig(t)
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
startSlowReceive(t, r, in, "a", 1)
startSlowReceive(t, r, in, "a", 2)
_, err := in.Accept(context.Background(), "a", "t", domain.LangSwedish, strings.NewReader("x"))
requireBusy(t, err) // a third body from one user
startSlowReceive(t, r, in, "b", 3)
startSlowReceive(t, r, in, "b", 4)
_, err = in.Accept(context.Background(), "c", "t", domain.LangSwedish, strings.NewReader("x"))
requireBusy(t, err) // a fifth body overall
}
+11 -24
View File
@@ -21,15 +21,13 @@ import (
type fakeStore struct { type fakeStore struct {
created []string // titles created []string // titles
langs []domain.Language
hadVideo []bool hadVideo []bool
failed map[string]domain.UploadErrorKind failed map[string]domain.UploadErrorKind
nextID string nextID string
} }
func (f *fakeStore) CreateUpload(_ context.Context, _, title string, lang domain.Language, hadVideo bool) (string, error) { func (f *fakeStore) CreateUpload(_ context.Context, _, title string, hadVideo bool) (string, error) {
f.created = append(f.created, title) f.created = append(f.created, title)
f.langs = append(f.langs, lang)
f.hadVideo = append(f.hadVideo, hadVideo) f.hadVideo = append(f.hadVideo, hadVideo)
return f.nextID, nil return f.nextID, nil
} }
@@ -100,7 +98,7 @@ func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
r := newRig(t) r := newRig(t)
audio := bytes.Repeat([]byte("a"), 500) audio := bytes.Repeat([]byte("a"), 500)
id, err := r.in.Accept(context.Background(), "user-1", "Teams export", domain.LangSwedish, bytes.NewReader(audio)) id, err := r.in.Accept(context.Background(), "user-1", "Teams export", bytes.NewReader(audio))
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, "up-1", id) assert.Equal(t, "up-1", id)
@@ -108,7 +106,7 @@ func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
assert.Equal(t, audio, r.pr.seen, "the prober must see the complete uploaded bytes, on disk") assert.Equal(t, audio, r.pr.seen, "the prober must see the complete uploaded bytes, on disk")
require.Len(t, r.sub.jobs, 1) require.Len(t, r.sub.jobs, 1)
job := r.sub.jobs[0] job := r.sub.jobs[0]
assert.Equal(t, uploads.Job{UserID: "user-1", UploadID: "up-1", Path: job.Path, Duration: 10 * time.Minute, Language: domain.LangSwedish}, job, assert.Equal(t, uploads.Job{UserID: "user-1", UploadID: "up-1", Path: job.Path, Duration: 10 * time.Minute}, job,
"the job carries what the file CLAIMED, so the worker can compare it with what was converted") "the job carries what the file CLAIMED, so the worker can compare it with what was converted")
assert.Equal(t, filepath.Dir(job.Path), r.tmp, "the temp file lives in the configured directory") assert.Equal(t, filepath.Dir(job.Path), r.tmp, "the temp file lives in the configured directory")
got, err := os.ReadFile(job.Path) got, err := os.ReadFile(job.Path)
@@ -119,7 +117,7 @@ func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
func TestAccept_OversizeBodyIsRejectedAndLeavesNothingBehind(t *testing.T) { func TestAccept_OversizeBodyIsRejectedAndLeavesNothingBehind(t *testing.T) {
r := newRig(t) r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "big", domain.LangSwedish, strings.NewReader(strings.Repeat("a", 1025))) _, err := r.in.Accept(context.Background(), "user-1", "big", strings.NewReader(strings.Repeat("a", 1025)))
var rej *uploads.RejectError var rej *uploads.RejectError
require.True(t, errors.As(err, &rej)) require.True(t, errors.As(err, &rej))
@@ -134,7 +132,7 @@ func TestAccept_OversizeBodyIsRejectedAndLeavesNothingBehind(t *testing.T) {
func TestAccept_BodyExactlyAtTheLimitIsAccepted(t *testing.T) { func TestAccept_BodyExactlyAtTheLimitIsAccepted(t *testing.T) {
r := newRig(t) r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "edge", domain.LangSwedish, strings.NewReader(strings.Repeat("a", 1024))) _, err := r.in.Accept(context.Background(), "user-1", "edge", strings.NewReader(strings.Repeat("a", 1024)))
require.NoError(t, err) require.NoError(t, err)
} }
@@ -154,7 +152,7 @@ func TestAccept_ProbeVerdictRejectsWithoutRowAndRemovesTheFile(t *testing.T) {
r := newRig(t) r := newRig(t)
r.pr.probe = tt.probe r.pr.probe = tt.probe
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data")) _, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
var rej *uploads.RejectError var rej *uploads.RejectError
require.True(t, errors.As(err, &rej)) require.True(t, errors.As(err, &rej))
@@ -171,7 +169,7 @@ func TestAccept_UnreadableFileIsARejectionButAProberFaultIsAnError(t *testing.T)
r := newRig(t) r := newRig(t)
r.pr.err = uploads.ErrUnreadable r.pr.err = uploads.ErrUnreadable
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("not media")) _, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("not media"))
var rej *uploads.RejectError var rej *uploads.RejectError
require.True(t, errors.As(err, &rej), "the user's file is the problem: tell them") require.True(t, errors.As(err, &rej), "the user's file is the problem: tell them")
@@ -182,7 +180,7 @@ func TestAccept_UnreadableFileIsARejectionButAProberFaultIsAnError(t *testing.T)
r := newRig(t) r := newRig(t)
r.pr.err = errors.New("exec: ffprobe: executable file not found") r.pr.err = errors.New("exec: ffprobe: executable file not found")
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data")) _, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
require.Error(t, err) require.Error(t, err)
var rej *uploads.RejectError var rej *uploads.RejectError
@@ -196,7 +194,7 @@ func TestAccept_ClientDisconnectMidUploadLeavesNothingBehind(t *testing.T) {
r := newRig(t) r := newRig(t)
body := io.MultiReader(strings.NewReader("partial"), iotest.ErrReader(errors.New("client went away"))) body := io.MultiReader(strings.NewReader("partial"), iotest.ErrReader(errors.New("client went away")))
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, body) _, err := r.in.Accept(context.Background(), "user-1", "t", body)
require.Error(t, err) require.Error(t, err)
assert.Empty(t, r.pr.paths) assert.Empty(t, r.pr.paths)
@@ -208,7 +206,7 @@ func TestAccept_SubmitFailureFailsTheRowHonestlyAndRemovesTheFile(t *testing.T)
r := newRig(t) r := newRig(t)
r.sub.err = errors.New("worker stopped") r.sub.err = errors.New("worker stopped")
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data")) _, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
require.Error(t, err) require.Error(t, err)
assert.Equal(t, domain.UploadErrInterrupted, r.st.failed["up-1"], assert.Equal(t, domain.UploadErrInterrupted, r.st.failed["up-1"],
@@ -230,7 +228,7 @@ func TestAccept_AFileWithAVideoTrackIsAcceptedAndRecordsThatItHadOne(t *testing.
r := newRig(t) r := newRig(t)
r.pr.probe = tt.probe r.pr.probe = tt.probe
id, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data")) id, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, "up-1", id) assert.Equal(t, "up-1", id)
@@ -240,14 +238,3 @@ func TestAccept_AFileWithAVideoTrackIsAcceptedAndRecordsThatItHadOne(t *testing.
}) })
} }
} }
func TestAccept_TheChosenLanguageIsStoredAndHandedToTheJob(t *testing.T) {
r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangAuto, strings.NewReader("data"))
require.NoError(t, err)
assert.Equal(t, []domain.Language{domain.LangAuto}, r.st.langs, "the row records what the user chose")
require.Len(t, r.sub.jobs, 1)
assert.Equal(t, domain.LangAuto, r.sub.jobs[0].Language, "the worker is told, so the engines are")
}
-48
View File
@@ -1,48 +0,0 @@
package uploads
// Observes what jobSink hands the store: the heartbeat's timestamp is the one the tracker
// stamped (its application clock), not a second reading taken here (#51).
import (
"context"
"testing"
"time"
"git.d-ma.be/mathias/tapir/internal/domain"
)
type atStore struct{ beatAt, saveAt time.Time }
func (*atStore) StartTranscribing(context.Context, string, string) error { return nil }
func (*atStore) CompleteUpload(context.Context, string, string, domain.Transcription) error {
return nil
}
func (*atStore) FailUpload(context.Context, string, string, domain.UploadErrorKind) error { return nil }
func (s *atStore) SaveProgress(_ context.Context, _, _ string, _ []domain.ProgressStep, at time.Time) error {
s.saveAt = at
return nil
}
func (s *atStore) Heartbeat(_ context.Context, _, _ string, at time.Time) error {
s.beatAt = at
return nil
}
func TestJobSink_ForwardsTheTrackersTimestampToTheStoreUnchanged(t *testing.T) {
st := &atStore{}
sink := jobSink{st: st, userID: "u", id: "i"}
stamp := time.Date(2020, 1, 2, 3, 4, 5, 0, time.UTC)
if err := sink.Beat(context.Background(), stamp); err != nil {
t.Fatal(err)
}
if err := sink.SaveProgress(context.Background(), nil, stamp); err != nil {
t.Fatal(err)
}
if !st.beatAt.Equal(stamp) {
t.Errorf("heartbeat reached the store stamped %v, want the tracker's %v", st.beatAt, stamp)
}
if !st.saveAt.Equal(stamp) {
t.Errorf("progress save reached the store stamped %v, want the tracker's %v", st.saveAt, stamp)
}
}
+12 -36
View File
@@ -33,13 +33,6 @@ type Transcriber interface {
Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error) Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error)
} }
// LanguagePlanner is a Planner whose phases depend on the language of the job: a recording
// that is not Swedish skips the second engine and the reconcile, and the page should not list
// them as coming (tapir#61). The worker prefers it over Planner.
type LanguagePlanner interface {
PlanFor(lang domain.Language) []progress.Planned
}
// Planner is an optional extra on a Transcriber: it declares its own phases (for // Planner is an optional extra on a Transcriber: it declares its own phases (for
// example one per engine) so the page can show them. A Transcriber that does not // example one per engine) so the page can show them. A Transcriber that does not
// implement it gets a single "transcribing" phase from the worker. Whatever a Planner // implement it gets a single "transcribing" phase from the worker. Whatever a Planner
@@ -54,7 +47,7 @@ type WorkerStore interface {
CompleteUpload(ctx context.Context, userID, id string, t domain.Transcription) error CompleteUpload(ctx context.Context, userID, id string, t domain.Transcription) error
FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error
SaveProgress(ctx context.Context, userID, id string, steps []domain.ProgressStep, at time.Time) error SaveProgress(ctx context.Context, userID, id string, steps []domain.ProgressStep, at time.Time) error
Heartbeat(ctx context.Context, userID, id string, at time.Time) error Heartbeat(ctx context.Context, userID, id string) error
} }
// WorkerOption tunes a Worker. // WorkerOption tunes a Worker.
@@ -98,9 +91,7 @@ func (s jobSink) SaveProgress(ctx context.Context, steps []domain.ProgressStep,
return s.st.SaveProgress(ctx, s.userID, s.id, steps, at) return s.st.SaveProgress(ctx, s.userID, s.id, steps, at)
} }
func (s jobSink) Beat(ctx context.Context, at time.Time) error { func (s jobSink) Beat(ctx context.Context) error { return s.st.Heartbeat(ctx, s.userID, s.id) }
return s.st.Heartbeat(ctx, s.userID, s.id, at)
}
// Worker transcribes accepted uploads in-process, ONE AT A TIME (ADR-033). It is // Worker transcribes accepted uploads in-process, ONE AT A TIME (ADR-033). It is
// not a second deployable: it shares the web process's lifetime, which is why it // not a second deployable: it shares the web process's lifetime, which is why it
@@ -185,10 +176,9 @@ func (w *Worker) Submit(j Job) error {
} }
// Run processes queued jobs until ctx ends, then deletes the audio of anything // Run processes queued jobs until ctx ends, then deletes the audio of anything
// still queued and refuses new jobs. The job running at shutdown is recorded interrupted // still queued and refuses new jobs. The rows of those jobs stay queued or
// by Process; the rows of jobs still queued stay queued and are failed as interrupted by // transcribing and are failed as interrupted by the start-up sweep
// the start-up sweep (store.FailInterruptedUploads): the audio is not retained, so they // (store.FailInterruptedUploads): the audio is not retained, so they cannot resume.
// cannot resume.
func (w *Worker) Run(ctx context.Context) { func (w *Worker) Run(ctx context.Context) {
defer w.stop() defer w.stop()
for { for {
@@ -237,7 +227,7 @@ func (w *Worker) stop() {
// Process runs one job: row -> transcribing, convert, transcribe, store. The // Process runs one job: row -> transcribing, convert, transcribe, store. The
// uploaded file and the converted WAV are deleted on every path — the audio is // uploaded file and the converted WAV are deleted on every path — the audio is
// never retained (ADR-033). A failure is recorded with a TYPED kind; a shutdown // never retained (ADR-033). A failure is recorded with a TYPED kind; a shutdown
// mid-job is recorded as interrupted at once (see fail). // mid-job is not a failure and leaves the row for the start-up sweep.
// //
// Every phase is reported through one progress.Tracker, which saves it to the row for // Every phase is reported through one progress.Tracker, which saves it to the row for
// the page and logs it, and a heartbeat stamps the row on its own while the job runs // the page and logs it, and a heartbeat stamps the row on its own while the job runs
@@ -258,14 +248,8 @@ func (w *Worker) Process(ctx context.Context, job Job) {
} }
plan := []progress.Planned{{Key: "received", Label: "Received"}, {Key: "audio", Label: "Preparing the audio"}} plan := []progress.Planned{{Key: "received", Label: "Received"}, {Key: "audio", Label: "Preparing the audio"}}
lang := job.Language
if lang == "" {
lang = domain.LangSwedish // queued before the choice existed
}
planner, planned := w.tr.(Planner) planner, planned := w.tr.(Planner)
if lp, ok := w.tr.(LanguagePlanner); ok { if planned {
plan = append(plan, lp.PlanFor(lang)...)
} else if planned {
plan = append(plan, planner.Plan()...) plan = append(plan, planner.Plan()...)
} else { } else {
plan = append(plan, progress.Planned{Key: "transcribe", Label: "Transcribing"}) plan = append(plan, progress.Planned{Key: "transcribe", Label: "Transcribing"})
@@ -285,7 +269,7 @@ func (w *Worker) Process(ctx context.Context, job Job) {
defer func() { stopBeat(); <-hbDone }() // no heartbeat outlives its job defer func() { stopBeat(); <-hbDone }() // no heartbeat outlives its job
// abort records where the job stopped. A shutdown (the parent context ended) is not // abort records where the job stopped. A shutdown (the parent context ended) is not
// a failure of the step: it stays as it was, and fail records the row as interrupted. // a failure: the row is left for the start-up sweep, so the step stays as it was.
abort := func(kind domain.UploadErrorKind) { abort := func(kind domain.UploadErrorKind) {
if ctx.Err() == nil { if ctx.Err() == nil {
tr.FailRunning() tr.FailRunning()
@@ -326,7 +310,7 @@ func (w *Worker) Process(ctx context.Context, job Job) {
if !planned { if !planned {
tr.Begin("transcribe", 0) tr.Begin("transcribe", 0)
} }
t, err := w.tr.Transcribe(domain.WithLanguage(progress.With(jobCtx, tr), lang), wav) t, err := w.tr.Transcribe(progress.With(jobCtx, tr), wav)
if err != nil { if err != nil {
log.Error("upload: transcribe", "kind", errkind.Of(err), "elapsed", time.Since(start).Round(time.Second)) log.Error("upload: transcribe", "kind", errkind.Of(err), "elapsed", time.Since(start).Round(time.Second))
abort(domain.UploadErrTranscriptionFailed) abort(domain.UploadErrTranscriptionFailed)
@@ -348,19 +332,11 @@ func (w *Worker) Process(ctx context.Context, job Job) {
"segments", len(t.Segments), "elapsed", time.Since(start).Round(time.Second)) "segments", len(t.Segments), "elapsed", time.Since(start).Round(time.Second))
} }
// interruptedWriteTimeout bounds the one write a dying process makes to record its job. // fail records a transcription failure — unless the PARENT context ended, which
const interruptedWriteTimeout = 3 * time.Second // is a shutdown, not a failure: the row is left for the start-up sweep.
// fail records a transcription failure. When the PARENT context ended the process is
// shutting down: the job is recorded interrupted at once, on a short context of its own,
// because the process may exit before any other pod's start-up sweep would do it (#50).
// The sweep remains the backstop for a process that is killed outright.
func (w *Worker) fail(ctx context.Context, job Job, log *slog.Logger, kind domain.UploadErrorKind) { func (w *Worker) fail(ctx context.Context, job Job, log *slog.Logger, kind domain.UploadErrorKind) {
if ctx.Err() != nil { if ctx.Err() != nil {
kind = domain.UploadErrInterrupted return
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(context.WithoutCancel(ctx), interruptedWriteTimeout)
defer cancel()
} }
if err := w.store.FailUpload(ctx, job.UserID, job.UploadID, kind); err != nil { if err := w.store.FailUpload(ctx, job.UserID, job.UploadID, kind); err != nil {
log.Error("upload: record failure", "kind", errkind.Of(err)) log.Error("upload: record failure", "kind", errkind.Of(err))
-44
View File
@@ -213,47 +213,3 @@ func TestWorkerProgress_AStoreFailureLogsItsSQLStateNeverTheValueItQuotes(t *tes
assert.Contains(t, out, "upload: store transcript") assert.Contains(t, out, "upload: store transcript")
assert.Contains(t, out, "kind=sql_22P02") assert.Contains(t, out, "kind=sql_22P02")
} }
// langPlannedTr plans per language, like the Ensemble: an auto job has no base engine.
type langPlannedTr struct{ *fakeTranscriber }
func (langPlannedTr) Plan() []progress.Planned { return plannedTr{}.Plan() }
func (langPlannedTr) PlanFor(l domain.Language) []progress.Planned {
if l == domain.LangAuto {
return []progress.Planned{{Key: "engine_b", Label: "Transcribing with B"}}
}
return plannedTr{}.Plan()
}
func (p langPlannedTr) Transcribe(ctx context.Context, wav string) (domain.Transcription, error) {
r := progress.From(ctx)
if domain.LanguageFrom(ctx) != domain.LangAuto {
r.Begin("engine_a", 0)
r.Done("engine_a")
}
r.Begin("engine_b", 0)
r.Done("engine_b")
return p.fakeTranscriber.Transcribe(ctx, wav)
}
func TestWorkerProgress_AJobGetsThePlanOfItsLanguage(t *testing.T) {
for name, tt := range map[string]struct {
lang domain.Language
want []string
}{
"auto": {domain.LangAuto, []string{"received:done", "audio:done", "engine_b:done", "save:done"}},
"swedish": {domain.LangSwedish, []string{"received:done", "audio:done", "engine_a:done", "engine_b:done", "save:done"}},
"no language (queued before the choice existed)": {"", []string{"received:done", "audio:done", "engine_a:done", "engine_b:done", "save:done"}},
} {
t.Run(name, func(t *testing.T) {
r := newWRig(t, time.Minute)
w := newProgressWorker(t, r, langPlannedTr{r.tr})
job := r.spool(t, "a")
job.Language = tt.lang
w.Process(context.Background(), job)
assert.Equal(t, tt.want, keysAndStates(r.st.lastSteps()))
})
}
}
+5 -33
View File
@@ -53,10 +53,7 @@ func (s *wStore) CompleteUpload(_ context.Context, _, id string, t domain.Transc
return nil return nil
} }
func (s *wStore) FailUpload(ctx context.Context, _, id string, k domain.UploadErrorKind) error { func (s *wStore) FailUpload(_ context.Context, _, id string, k domain.UploadErrorKind) error {
if err := ctx.Err(); err != nil { // like the real store: a cancelled context cannot write
return err
}
s.mu.Lock() s.mu.Lock()
defer s.mu.Unlock() defer s.mu.Unlock()
s.failed[id] = k s.failed[id] = k
@@ -70,7 +67,7 @@ func (s *wStore) SaveProgress(_ context.Context, _, _ string, steps []domain.Pro
return nil return nil
} }
func (s *wStore) Heartbeat(context.Context, string, string, time.Time) error { func (s *wStore) Heartbeat(context.Context, string, string) error {
time.Sleep(s.beatDelay) time.Sleep(s.beatDelay)
s.mu.Lock() s.mu.Lock()
defer s.mu.Unlock() defer s.mu.Unlock()
@@ -126,7 +123,6 @@ type fakeTranscriber struct {
result domain.Transcription result domain.Transcription
err error err error
paths []string paths []string
langs []domain.Language // what each call's context carried
// block, when non-nil, holds every call until it is closed or ctx ends. // block, when non-nil, holds every call until it is closed or ctx ends.
block chan struct{} block chan struct{}
running atomic.Int32 running atomic.Int32
@@ -135,7 +131,6 @@ type fakeTranscriber struct {
func (f *fakeTranscriber) Transcribe(ctx context.Context, wav string) (domain.Transcription, error) { func (f *fakeTranscriber) Transcribe(ctx context.Context, wav string) (domain.Transcription, error) {
f.paths = append(f.paths, wav) f.paths = append(f.paths, wav)
f.langs = append(f.langs, domain.LanguageFrom(ctx))
n := f.running.Add(1) n := f.running.Add(1)
defer f.running.Add(-1) defer f.running.Add(-1)
for { for {
@@ -247,9 +242,7 @@ func TestWorkerProcess_FailuresAreTypedAndStillDeleteTheAudio(t *testing.T) {
}) })
} }
func TestWorkerProcess_ShutdownMidJobRecordsTheRowInterruptedAtOnce(t *testing.T) { func TestWorkerProcess_ShutdownMidJobLeavesTheRowForTheStartupSweep(t *testing.T) {
// SIGTERM on a deploy used to leave the row "transcribing" until the NEXT pod's
// start-up sweep; the process may exit before any sweep runs (#50).
r := newWRig(t, time.Minute) r := newWRig(t, time.Minute)
r.tr.block = make(chan struct{}) r.tr.block = make(chan struct{})
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
@@ -262,8 +255,8 @@ func TestWorkerProcess_ShutdownMidJobRecordsTheRowInterruptedAtOnce(t *testing.T
<-done <-done
_, f := r.st.snapshot() _, f := r.st.snapshot()
assert.Equal(t, domain.UploadErrInterrupted, f["a"], "the row is recorded interrupted by the dying process, not left for a sweep") assert.Empty(t, f, "a shutdown is not a transcription failure: the row stays for FailInterruptedUploads to mark interrupted")
assert.Empty(t, r.leftovers(t), "and the audio is still deleted") assert.Empty(t, r.leftovers(t), "but the audio is still deleted")
} }
func TestWorkerProcess_AJobThatOutlivesItsTimeoutIsFailedNotStuck(t *testing.T) { func TestWorkerProcess_AJobThatOutlivesItsTimeoutIsFailedNotStuck(t *testing.T) {
@@ -339,24 +332,3 @@ func (dbErr) Error() string {
return `ERROR: invalid input syntax for type json: "zebrakaka secret words"` return `ERROR: invalid input syntax for type json: "zebrakaka secret words"`
} }
func (dbErr) SQLState() string { return "22P02" } func (dbErr) SQLState() string { return "22P02" }
func TestWorker_TheTranscriberSeesTheLanguageOfTheJobOnItsContext(t *testing.T) {
for name, tt := range map[string]struct {
job domain.Language
want domain.Language
}{
"auto": {domain.LangAuto, domain.LangAuto},
"swedish": {domain.LangSwedish, domain.LangSwedish},
"a job with no language": {"", domain.LangSwedish}, // queued before the field existed
} {
t.Run(name, func(t *testing.T) {
r := newWRig(t, time.Minute)
job := r.spool(t, "a")
job.Language = tt.job
r.w.Process(context.Background(), job)
assert.Equal(t, []domain.Language{tt.want}, r.tr.langs)
})
}
}
-151
View File
@@ -1,151 +0,0 @@
package web_test
// Observes the cross-origin gate around Router (#48): SameSite=Lax does not stop a POST
// from a sibling *.d-ma.be origin, so the router itself must refuse a non-safe request
// whose Origin or Sec-Fetch-Site says it did not come from this site's own pages.
import (
"context"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// foreign is a header set a browser sends for a form post made by another page.
var foreign = map[string]map[string]string{
"sibling subdomain, Sec-Fetch-Site": {"Origin": "https://evil.d-ma.be", "Sec-Fetch-Site": "same-site"},
"other site, Sec-Fetch-Site": {"Origin": "https://evil.example", "Sec-Fetch-Site": "cross-site"},
"sibling subdomain, Origin only": {"Origin": "https://evil.d-ma.be"},
"opaque origin": {"Origin": "null"},
}
// own is a header set for the app's own pages (or a non-browser client): must pass.
var own = map[string]map[string]string{
"same-origin Sec-Fetch-Site": {"Sec-Fetch-Site": "same-origin", "Origin": "http://example.com"},
"matching Origin only": {"Origin": "http://example.com"},
"no browser headers": {},
}
func withHeaders(req *http.Request, h map[string]string) *http.Request {
for k, v := range h {
req.Header.Set(k, v)
}
return req
}
func userCount(t *testing.T) int {
t.Helper()
var n int
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT count(*) FROM users WHERE id = $1`, userID).Scan(&n))
return n
}
func TestCrossOrigin_ARefusedDeleteAccountPostLeavesTheAccountInPlace(t *testing.T) {
for name, h := range foreign {
t.Run(name, func(t *testing.T) {
app, _ := newAccountApp(t)
resetDB(t, rawPool(t))
seedConnection(t, app, "youtube", "ada@channel", "ref")
rec := do(t, app, withHeaders(httptest.NewRequest(http.MethodPost, "/account/delete", nil), h))
assert.Equal(t, http.StatusForbidden, rec.Code)
assert.Equal(t, 1, userCount(t), "the account must survive a refused post")
})
}
}
func TestCrossOrigin_ASameOriginOrHeaderlessDeleteAccountPostIsServed(t *testing.T) {
for name, h := range own {
t.Run(name, func(t *testing.T) {
app, _ := newAccountApp(t)
resetDB(t, rawPool(t))
seedConnection(t, app, "youtube", "ada@channel", "ref")
rec := do(t, app, withHeaders(httptest.NewRequest(http.MethodPost, "/account/delete", nil), h))
assert.Equal(t, http.StatusSeeOther, rec.Code)
assert.Equal(t, "/auth/logout", rec.Header().Get("Location"))
})
}
}
func TestCrossOrigin_ARefusedUploadPostSubmitsNoJob(t *testing.T) {
for name, h := range foreign {
t.Run(name, func(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
rec := do(t, app, withHeaders(uploadReq(t, "m", "m.wav", []byte("RIFF....WAVE")), h))
assert.Equal(t, http.StatusForbidden, rec.Code)
assert.Empty(t, sub.jobs, "a refused post must not enqueue transcription")
})
}
}
func TestCrossOrigin_ARefusedUploadChatPostCallsNoModel(t *testing.T) {
for name, h := range foreign {
t.Run(name, func(t *testing.T) {
app, fc := newUploadChatApp(t)
id := seedUpload(t, userID, "Roadmap sync", "done", meetingText)
req := chatPost("/uploads/"+id+"/chat", url.Values{"message": {"hej"}})
rec := do(t, app, withHeaders(req, h))
assert.Equal(t, http.StatusForbidden, rec.Code)
assert.Zero(t, fc.callCount)
})
}
}
func TestCrossOrigin_AForeignPostIsRefusedOnEachListedStateChangingPath(t *testing.T) {
routes := []string{
"/account/disconnect/youtube",
"/account/summarize-mode",
"/register",
"/paste",
"/v/abc/action",
"/v/abc/summarize",
"/v/abc/retry-now",
"/v/abc/chat",
}
for _, path := range routes {
t.Run(path, func(t *testing.T) {
app := newApp(t)
req := httptest.NewRequest(http.MethodPost, path, strings.NewReader(""))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
rec := do(t, app, withHeaders(req, foreign["sibling subdomain, Sec-Fetch-Site"]))
assert.Equal(t, http.StatusForbidden, rec.Code)
})
}
}
func TestCrossOrigin_ARefusedRecordingDeletePostLeavesTheRecordingInPlace(t *testing.T) {
for name, h := range foreign {
t.Run(name, func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
rec := do(t, app, withHeaders(httptest.NewRequest(http.MethodPost, "/uploads/"+id+"/delete", nil), h))
assert.Equal(t, http.StatusForbidden, rec.Code)
assert.True(t, uploadExists(t, id), "a refused post must not delete the recording")
})
}
}
// A login redirect comes back to /auth/callback as a cross-site GET navigation, and
// health probes carry no headers: safe methods must never be refused on origin grounds.
func TestCrossOrigin_ACrossSiteGETIsNotRefused(t *testing.T) {
app := newApp(t)
for _, path := range []string{"/healthz", "/welcome"} {
req := httptest.NewRequest(http.MethodGet, path, nil)
req.Header.Set("Sec-Fetch-Site", "cross-site")
assert.Equal(t, http.StatusOK, do(t, app, req).Code, path)
}
}
+2 -11
View File
@@ -59,9 +59,6 @@ type Store interface {
// non-owner's id is ErrUploadNotFound. // non-owner's id is ErrUploadNotFound.
GetUpload(ctx context.Context, userID, id string) (*domain.UploadedTranscript, error) GetUpload(ctx context.Context, userID, id string) (*domain.UploadedTranscript, error)
ListUploads(ctx context.Context, userID string) ([]domain.UploadedTranscript, error) ListUploads(ctx context.Context, userID string) ([]domain.UploadedTranscript, error)
// DeleteUpload removes a finished upload; ErrUploadInProgress while it is still being
// processed, ErrUploadNotFound for a non-owner.
DeleteUpload(ctx context.Context, userID, id string) error
// SetTranscriptStatus clears or updates a video's transcript backoff state. // SetTranscriptStatus clears or updates a video's transcript backoff state.
// Used by handleRetryNow to clear rate_limited_at before immediate processing. // Used by handleRetryNow to clear rate_limited_at before immediate processing.
@@ -203,7 +200,6 @@ func (a *App) Router() http.Handler {
app.HandleFunc("POST /uploads", a.handleUploadPost) app.HandleFunc("POST /uploads", a.handleUploadPost)
app.HandleFunc("GET /uploads/{id}", a.handleUploadDetail) app.HandleFunc("GET /uploads/{id}", a.handleUploadDetail)
app.HandleFunc("GET /uploads/{id}/status", a.handleUploadStatus) app.HandleFunc("GET /uploads/{id}/status", a.handleUploadStatus)
app.HandleFunc("POST /uploads/{id}/delete", a.handleUploadDelete)
// Chat over the uploaded transcript reuses the ADR-027 Chatter; mounted only // Chat over the uploaded transcript reuses the ADR-027 Chatter; mounted only
// when one is wired. // when one is wired.
if a.Chat != nil { if a.Chat != nil {
@@ -233,12 +229,7 @@ func (a *App) Router() http.Handler {
// resolved user_id. /register lives inside the auth guard but is exempt from // resolved user_id. /register lives inside the auth guard but is exempt from
// the registration gate (you must be able to reach it before you have a user). // the registration gate (you must be able to reach it before you have a user).
root.Handle("/", a.Auth.Middleware(a.registrationGate(app))) root.Handle("/", a.Auth.Middleware(a.registrationGate(app)))
return root
// The session cookie is SameSite=Lax, which still travels on a POST made from a
// sibling *.d-ma.be page (same site, different origin). Refuse any non-safe request
// whose Origin or Sec-Fetch-Site says it did not come from this origin (#48). It wraps
// everything, so a route added later is covered without remembering to ask.
return http.NewCrossOriginProtection().Handler(root)
} }
// handleWelcome renders the public landing page (/welcome). It is mounted outside // handleWelcome renders the public landing page (/welcome). It is mounted outside
@@ -313,7 +304,7 @@ func (a *App) handleList(w http.ResponseWriter, r *http.Request) {
a.render(w, r, summaryList(buckets, hasConnected, autoSummarize)) a.render(w, r, summaryList(buckets, hasConnected, autoSummarize))
return return
} }
a.render(w, r, ListPage(buckets, f, stats, takeFlash(w, r), hasConnected, channels, autoSummarize, a.Uploads != nil)) a.render(w, r, ListPage(buckets, f, stats, takeFlash(w, r), hasConnected, channels, autoSummarize))
} }
// handleDetail renders one summary in full (highlights, takeaways, action group). // handleDetail renders one summary in full (highlights, takeaways, action group).
+1 -1
View File
@@ -66,7 +66,7 @@ func TestParseYouTubeVideoID(t *testing.T) {
func TestListPageShowsPasteFormOnlyWhenConnected(t *testing.T) { func TestListPageShowsPasteFormOnlyWhenConnected(t *testing.T) {
render := func(connected bool) string { render := func(connected bool) string {
var buf bytes.Buffer var buf bytes.Buffer
if err := ListPage(listBuckets{}, Filter{}, PipelineStats{}, "", connected, nil, true, false).Render(context.Background(), &buf); err != nil { if err := ListPage(listBuckets{}, Filter{}, PipelineStats{}, "", connected, nil, true).Render(context.Background(), &buf); err != nil {
t.Fatalf("render: %v", err) t.Fatalf("render: %v", err)
} }
return buf.String() return buf.String()
+2 -88
View File
@@ -18,7 +18,6 @@ import (
"git.d-ma.be/mathias/tapir/internal/adapters/chat" "git.d-ma.be/mathias/tapir/internal/adapters/chat"
"git.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
"git.d-ma.be/mathias/tapir/internal/config"
"git.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/errkind" "git.d-ma.be/mathias/tapir/internal/errkind"
"git.d-ma.be/mathias/tapir/internal/uploads" "git.d-ma.be/mathias/tapir/internal/uploads"
@@ -28,7 +27,7 @@ import (
// *uploads.Intake satisfies it. It is the whole of what the handlers know about // *uploads.Intake satisfies it. It is the whole of what the handlers know about
// validation, spooling and queueing. // validation, spooling and queueing.
type Uploader interface { type Uploader interface {
Accept(ctx context.Context, userID, title string, lang domain.Language, src io.Reader) (id string, err error) Accept(ctx context.Context, userID, title string, src io.Reader) (id string, err error)
} }
const ( const (
@@ -36,8 +35,6 @@ const (
// and the title field; the cap itself is enforced on the file part by Intake. // and the title field; the cap itself is enforced on the file part by Intake.
uploadOverhead = 1 << 20 uploadOverhead = 1 << 20
maxTitleRunes = 200 maxTitleRunes = 200
// maxLanguageBytes bounds what is read of the language field; a valid value is at most four bytes, so a longer one fails the parse.
maxLanguageBytes = 16
// uploadPollEvery is how often an unfinished upload re-asks for its status. // uploadPollEvery is how often an unfinished upload re-asks for its status.
uploadPollEvery = "2s" uploadPollEvery = "2s"
// serviceSilentAfter is how long a job may go without any signal (heartbeat or // serviceSilentAfter is how long a job may go without any signal (heartbeat or
@@ -91,7 +88,6 @@ func (a *App) handleUploadPost(w http.ResponseWriter, r *http.Request) {
return return
} }
title := "" title := ""
lang := domain.LangSwedish
for { for {
part, err := mr.NextPart() part, err := mr.NextPart()
if errors.Is(err, io.EOF) { if errors.Is(err, io.EOF) {
@@ -124,29 +120,11 @@ func (a *App) handleUploadPost(w http.ResponseWriter, r *http.Request) {
return return
} }
title = cleanTitle(string(b)) title = cleanTitle(string(b))
case "language":
// A closed set read before the file: anything else is refused before the file
// is spooled, so a crafted value never reaches an engine (tapir#61).
b, err := io.ReadAll(io.LimitReader(part, maxLanguageBytes+1))
if err != nil {
if a.abandoned(w, r) {
return
}
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "That upload could not be read. Please try again.")
return
}
l, ok := domain.ParseLanguage(string(b))
if !ok {
a.logger().Info("upload rejected", "reason", "language")
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "Choose a language from the list.")
return
}
lang = l
case "file": case "file":
if title == "" { if title == "" {
title = titleFromFilename(part.FileName()) title = titleFromFilename(part.FileName())
} }
id, err := a.Uploads.Accept(r.Context(), userID, title, lang, part) id, err := a.Uploads.Accept(r.Context(), userID, title, part)
if err != nil { if err != nil {
a.uploadFailure(w, r, userID, err) a.uploadFailure(w, r, userID, err)
return return
@@ -231,32 +209,6 @@ func (a *App) handleUploadStatus(w http.ResponseWriter, r *http.Request) {
a.render(w, r, uploadStatus(*u, a.Chat != nil, a.now())) a.render(w, r, uploadStatus(*u, a.Chat != nil, a.now()))
} }
// handleUploadDelete removes one of the caller's finished recordings and returns to the
// list. Another user's id is a 404 like every other upload route; a recording still being
// transcribed is refused (409) rather than deleted under the worker (#37).
func (a *App) handleUploadDelete(w http.ResponseWriter, r *http.Request) {
userID, ok := a.currentUserID(w, r)
if !ok {
return
}
id := r.PathValue("id")
if !uuidShape.MatchString(id) {
http.NotFound(w, r)
return
}
switch err := a.Store.DeleteUpload(r.Context(), userID, id); {
case err == nil:
http.Redirect(w, r, "/uploads", http.StatusSeeOther)
case errors.Is(err, store.ErrUploadNotFound):
http.NotFound(w, r)
case errors.Is(err, store.ErrUploadInProgress):
a.renderUploadsPage(w, r, userID, http.StatusConflict,
"This recording is still being transcribed. You can delete it once it has finished.")
default:
a.serverError(w, r, "delete upload", err)
}
}
func (a *App) loadOwnedUpload(w http.ResponseWriter, r *http.Request) (*domain.UploadedTranscript, bool) { func (a *App) loadOwnedUpload(w http.ResponseWriter, r *http.Request) (*domain.UploadedTranscript, bool) {
userID, ok := a.currentUserID(w, r) userID, ok := a.currentUserID(w, r)
if !ok { if !ok {
@@ -291,44 +243,6 @@ func titleFromFilename(name string) string {
return t return t
} }
// cloudProvider names the cloud service that produced a transcript, or false when it was
// served locally (or by an unknown endpoint). "Local" is config.IsLocalModel: the same
// definition the start-up validation uses, not a second string match here (#52).
//
// The endpoint is one alias, except for the two-engine path (ADR-034), which writes
// "<base> + <alt>, reconciled by <model>" (ensemble.go). That shape is taken apart exactly
// and each alias is judged whole, so a composite of local models is local, and anything
// else (a cloud alias, one with no provider prefix, one with a space or a connector word in
// it) is not. A string that is not that shape is one alias.
func cloudProvider(endpoint string) (string, bool) {
endpoint = strings.TrimSpace(endpoint)
if endpoint == "" {
return "", false
}
aliases := []string{endpoint}
if engines, reconciler, ok := strings.Cut(endpoint, ", reconciled by "); ok {
aliases = append(strings.Split(engines, " + "), reconciler)
}
for _, a := range aliases {
if a = strings.TrimSpace(a); !config.IsLocalModel(a) {
provider, _, _ := strings.Cut(a, "/")
if provider = strings.TrimSpace(provider); provider == "" {
provider = a
}
return provider, true
}
}
return "", false
}
// languageLabel is how the page names the language a recording was transcribed as.
func languageLabel(l domain.Language) string {
if l == domain.LangAuto {
return "Other (detected by the engine)"
}
return "Swedish"
}
// hasNote reports whether any phase carries a note. A note is the only warning for a // hasNote reports whether any phase carries a note. A note is the only warning for a
// recording that is shorter than its file claimed, so a record that has one is shown open. // recording that is shorter than its file claimed, so a record that has one is shown open.
func hasNote(steps []domain.ProgressStep) bool { func hasNote(steps []domain.ProgressStep) bool {
-31
View File
@@ -24,14 +24,6 @@ templ UploadsPage(v uploadsView) {
} }
<form method="post" action="/uploads" enctype="multipart/form-data"> <form method="post" action="/uploads" enctype="multipart/form-data">
<p><label>Title (optional) <input type="text" name="title" maxlength="200"/></label></p> <p><label>Title (optional) <input type="text" name="title" maxlength="200"/></label></p>
<p>
<label>Language
<select name="language">
<option value="sv" selected>Swedish</option>
<option value="auto">Other (detected by the engine)</option>
</select>
</label>
</p>
<p><input type="file" name="file" accept="audio/*,.m4a,.mp3,.wav,.opus,.ogg,.mp4" required/></p> <p><input type="file" name="file" accept="audio/*,.m4a,.mp3,.wav,.opus,.ogg,.mp4" required/></p>
<p><button type="submit">Upload and transcribe</button></p> <p><button type="submit">Upload and transcribe</button></p>
</form> </form>
@@ -57,7 +49,6 @@ templ UploadPage(u domain.UploadedTranscript, chatEnabled bool, now time.Time) {
@Layout("Tapir — " + u.Title) { @Layout("Tapir — " + u.Title) {
<article class="account"> <article class="account">
<h1>{ u.Title }</h1> <h1>{ u.Title }</h1>
<p class="muted">Language: { languageLabel(u.Language) }</p>
@uploadStatus(u, chatEnabled, now) @uploadStatus(u, chatEnabled, now)
<p><a href="/uploads">All recordings</a></p> <p><a href="/uploads">All recordings</a></p>
</article> </article>
@@ -75,12 +66,6 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
Transcribed { uploads.FormatDuration(time.Duration(u.AudioSeconds * float64(time.Second))) } Transcribed { uploads.FormatDuration(time.Duration(u.AudioSeconds * float64(time.Second))) }
of audio by <span class="chip">{ u.Endpoint }</span>. of audio by <span class="chip">{ u.Endpoint }</span>.
</p> </p>
if provider, cloud := cloudProvider(u.Endpoint); cloud {
<p class="cloud-note">
This recording's audio left the homelab: it was transcribed by a cloud service ({ provider })
because the local engine could not.
</p>
}
@reconciliationNote(u.Reconciliation) @reconciliationNote(u.Reconciliation)
if len(u.Progress) > 0 { if len(u.Progress) > 0 {
<details class="what-was-done" if hasNote(u.Progress) { open }> <details class="what-was-done" if hasNote(u.Progress) { open }>
@@ -95,7 +80,6 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
<summary>Transcript</summary> <summary>Transcript</summary>
<pre class="transcript">{ u.Content }</pre> <pre class="transcript">{ u.Content }</pre>
</details> </details>
@deleteRecording(u.ID)
</div> </div>
case domain.UploadFailed: case domain.UploadFailed:
<div class="upload-status upload-failed flash flash-error" role="alert"> <div class="upload-status upload-failed flash flash-error" role="alert">
@@ -107,7 +91,6 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
@progressSteps(u.Progress, now, false) @progressSteps(u.Progress, now, false)
</details> </details>
} }
@deleteRecording(u.ID)
</div> </div>
default: default:
<div <div
@@ -133,20 +116,6 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
} }
} }
// deleteRecording is the confirm-then-delete control, shown only once a recording has
// finished (done or failed): deleting one that is still being processed is refused.
templ deleteRecording(id string) {
<details class="confirm-delete">
<summary class="btn-danger">Delete this recording…</summary>
<div class="confirm-body">
<p>This permanently deletes the recording and its transcript. Are you sure?</p>
<form method="post" action={ templ.SafeURL("/uploads/" + id + "/delete") }>
<button type="submit" class="btn-danger">Yes, permanently delete it</button>
</form>
</div>
</details>
}
// UploadChatPage is the no-JS full page for chatting with an uploaded transcript: // UploadChatPage is the no-JS full page for chatting with an uploaded transcript:
// the same chat panel as a video's (ADR-027), scoped to this recording. // the same chat panel as a video's (ADR-027), scoped to this recording.
templ UploadChatPage(u domain.UploadedTranscript, v chatView) { templ UploadChatPage(u domain.UploadedTranscript, v chatView) {
-140
View File
@@ -1,140 +0,0 @@
package web_test
// A transcript produced by the cloud fallback means this recording's audio left the
// homelab, which the reader is told in plain words (#52). "Local" is config.IsLocalModel:
// one definition, not a second string match in the view.
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func setEndpoint(t *testing.T, id, endpoint string) {
t.Helper()
_, err := rawPool(t).Exec(context.Background(), `UPDATE uploaded_transcripts SET endpoint = $2 WHERE id = $1`, id, endpoint)
require.NoError(t, err)
}
func TestUploadStatus_ACloudServedTranscriptSaysTheAudioLeftTheHomelab(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
setEndpoint(t, id, "berget/kb-whisper")
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.Contains(t, b, "left the homelab")
assert.Contains(t, b, "cloud service (berget)")
}
func TestUploadStatus_ALocallyServedTranscriptCarriesNoCloudNote(t *testing.T) {
for _, endpoint := range []string{"iguana/kb-whisper", "koala/whisper", ""} {
t.Run(endpoint, func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
setEndpoint(t, id, endpoint)
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.NotContains(t, b, "left the homelab")
})
}
}
func TestUploadStatus_AnEndpointThatOnlyLooksLocalIsStillNamedAsCloud(t *testing.T) {
// The prefix alone must not decide it: IsLocalModel refuses path tricks.
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
setEndpoint(t, id, "iguana/../berget/kb-whisper")
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.Contains(t, b, "left the homelab")
}
// The two-engine path (ADR-034) writes one composite endpoint string. Every name in it is
// a local model, so the audio never left the homelab and the note must not say it did.
func TestUploadStatus_AReconciledAllLocalEndpointCarriesNoCloudNote(t *testing.T) {
for _, endpoint := range []string{
"iguana/kb-whisper + koala/whisper, reconciled by iguana/gemma4-26b",
"iguana/kb-whisper + koala/whisper, reconciled by iguana/gemma4-26b\n",
} {
t.Run(endpoint, func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
setEndpoint(t, id, endpoint)
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.NotContains(t, b, "left the homelab")
})
}
}
func TestUploadStatus_AReconciledEndpointNamingACloudEngineStillNamesTheCloudProvider(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
setEndpoint(t, id, "iguana/kb-whisper + berget/kb-whisper, reconciled by iguana/gemma4-26b")
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.Contains(t, b, "cloud service (berget)")
}
// An alias without a provider prefix is not a local model, wherever it sits in the
// composite: it must never be dropped from the judgement (PR #58 re-review).
func TestUploadStatus_AReconciledEndpointWithAnUnprefixedAliasIsNotCalledLocal(t *testing.T) {
for _, endpoint := range []string{
"iguana/kb-whisper + whisper-1, reconciled by iguana/gemma4-26b",
"iguana/kb-whisper + whisper 1, reconciled by iguana/gemma4-26b",
"iguana/kb-whisper + berget / x, reconciled by iguana/gemma4-26b",
} {
t.Run(endpoint, func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
setEndpoint(t, id, endpoint)
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.Contains(t, b, "left the homelab")
assert.NotContains(t, b, "cloud service ()", "the note names a provider")
})
}
}
// "by", "reconciled" and "+" are connectors only inside the composite format. As a whole
// endpoint, or as an alias within the composite, they are model names and are judged.
func TestUploadStatus_AConnectorWordUsedAsAModelNameIsStillJudged(t *testing.T) {
for _, endpoint := range []string{
"by",
"reconciled",
"+",
"iguana/kb-whisper + by, reconciled by iguana/gemma4-26b",
"iguana/kb-whisper + reconciled, reconciled by iguana/gemma4-26b",
"iguana/kb-whisper + +, reconciled by iguana/gemma4-26b",
"/x", // no provider at all: the note must still name something
} {
t.Run(endpoint, func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
setEndpoint(t, id, endpoint)
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.Contains(t, b, "left the homelab")
assert.NotContains(t, b, "cloud service ()")
})
}
}
func TestUploadStatus_AReconcilerThatIsNotLocalNamesTheCloudProvider(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
setEndpoint(t, id, "iguana/kb-whisper + koala/whisper, reconciled by berget/mistral-large")
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.Contains(t, b, "cloud service (berget)")
}
@@ -1,85 +0,0 @@
package web_test
// Deleting one recording (#37). Owner-scoped like every other upload route: another user's
// id is a 404 and the row survives; an upload still being processed is refused.
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func deleteReq(id string) *http.Request {
return httptest.NewRequest(http.MethodPost, "/uploads/"+id+"/delete", nil)
}
func uploadExists(t *testing.T, id string) bool {
t.Helper()
var n int
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT count(*) FROM uploaded_transcripts WHERE id = $1`, id).Scan(&n))
return n == 1
}
func TestUploadDelete_AnOwnerRemovesAFinishedOrFailedRecordingAndLandsOnTheList(t *testing.T) {
for _, status := range []string{"done", "failed"} {
t.Run(status, func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "Roadmap sync", status, "transcription_failed")
rec := do(t, app, deleteReq(id))
assert.Equal(t, http.StatusSeeOther, rec.Code)
assert.Equal(t, "/uploads", rec.Header().Get("Location"))
assert.False(t, uploadExists(t, id))
})
}
}
func TestUploadDelete_AnotherUsersRecordingIs404AndSurvives(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
theirs := seedUpload(t, otherUserID, "Their private meeting", "done", meetingText)
rec := do(t, app, deleteReq(theirs))
assert.Equal(t, http.StatusNotFound, rec.Code)
assert.True(t, uploadExists(t, theirs), "a non-owner's delete must not remove the row")
}
func TestUploadDelete_ARecordingStillBeingTranscribedIsRefusedWith409AndKept(t *testing.T) {
for _, status := range []string{"queued", "transcribing"} {
t.Run(status, func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", status, "")
rec := do(t, app, deleteReq(id))
assert.Equal(t, http.StatusConflict, rec.Code)
assert.Contains(t, body(t, rec), "still being transcribed")
assert.True(t, uploadExists(t, id))
})
}
}
func TestUploadDelete_ANonIDIsA404NotAServerError(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
assert.Equal(t, http.StatusNotFound, do(t, app, deleteReq("not-a-uuid")).Code)
}
func TestUploadPage_OffersDeleteOnlyForAFinishedOrFailedRecording(t *testing.T) {
for status, want := range map[string]bool{"done": true, "failed": true, "queued": false, "transcribing": false} {
t.Run(status, func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", status, "transcription_failed")
b := body(t, do(t, app, get("/uploads/"+id)))
assert.Equal(t, want, strings.Contains(b, `action="/uploads/`+id+`/delete"`))
})
}
}
@@ -14,8 +14,6 @@ import (
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
) )
// A client that hangs up in the middle of an upload is routine, not a server fault. The // A client that hangs up in the middle of an upload is routine, not a server fault. The
@@ -87,7 +85,7 @@ func TestUploadPost_AClientThatHangsUpAnywhereInTheBodyIsLoggedAtInfoAndNeverAsA
// context is cancelled, then Accept fails the way a read from a dead connection does. // context is cancelled, then Accept fails the way a read from a dead connection does.
type goneMidAccept struct{ cancel context.CancelFunc } type goneMidAccept struct{ cancel context.CancelFunc }
func (g goneMidAccept) Accept(context.Context, string, string, domain.Language, io.Reader) (string, error) { func (g goneMidAccept) Accept(context.Context, string, string, io.Reader) (string, error) {
g.cancel() g.cancel()
return "", fmt.Errorf("receive: %w", io.ErrClosedPipe) return "", fmt.Errorf("receive: %w", io.ErrClosedPipe)
} }
+1 -2
View File
@@ -16,7 +16,6 @@ import (
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/uploads" "git.d-ma.be/mathias/tapir/internal/uploads"
"git.d-ma.be/mathias/tapir/internal/web" "git.d-ma.be/mathias/tapir/internal/web"
) )
@@ -133,7 +132,7 @@ func TestUploadPost_AcceptedAudioQueuesARowAndRedirectsToIt(t *testing.T) {
assert.Equal(t, "Sprint review", title) assert.Equal(t, "Sprint review", title)
assert.Equal(t, "/uploads/"+id, rec.Header().Get("Location")) assert.Equal(t, "/uploads/"+id, rec.Header().Get("Location"))
require.Len(t, sub.jobs, 1) require.Len(t, sub.jobs, 1)
assert.Equal(t, uploads.Job{UserID: userID, UploadID: id, Path: sub.jobs[0].Path, Language: domain.LangSwedish, Duration: audioOK.Duration}, sub.jobs[0]) assert.Equal(t, uploads.Job{UserID: userID, UploadID: id, Path: sub.jobs[0].Path, Duration: audioOK.Duration}, sub.jobs[0])
spooled, err := os.ReadFile(sub.jobs[0].Path) spooled, err := os.ReadFile(sub.jobs[0].Path)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, "synthetic audio bytes", string(spooled)) assert.Equal(t, "synthetic audio bytes", string(spooled))
@@ -1,151 +0,0 @@
package web_test
// The language of a recording is chosen on the upload form (tapir#61). These drive the real
// Router, Intake and Postgres store; the submitter is the process boundary and records the
// job the worker would receive.
import (
"bytes"
"context"
"mime/multipart"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
)
// langReq builds an upload whose parts come in the order given: "title", "language" and
// "file" (the file part always carries the same synthetic bytes).
func langReq(t *testing.T, order []string, language string) *http.Request {
t.Helper()
var buf bytes.Buffer
mw := multipart.NewWriter(&buf)
for _, part := range order {
switch part {
case "title":
require.NoError(t, mw.WriteField("title", "m"))
case "language":
require.NoError(t, mw.WriteField("language", language))
case "file":
fw, err := mw.CreateFormFile("file", "m.wav")
require.NoError(t, err)
_, err = fw.Write([]byte("RIFF....WAVE"))
require.NoError(t, err)
}
}
require.NoError(t, mw.Close())
req := httptest.NewRequest(http.MethodPost, "/uploads", &buf)
req.Header.Set("Content-Type", mw.FormDataContentType())
return req
}
func rowLanguage(t *testing.T, location string) domain.Language {
t.Helper()
id := strings.TrimPrefix(location, "/uploads/")
got, err := newStore(t).GetUpload(context.Background(), userID, id)
require.NoError(t, err)
return got.Language
}
func TestUploadPost_TheLanguageFieldReachesTheJobAndTheRow(t *testing.T) {
for _, tt := range []struct {
name string
language string
want domain.Language
}{
{"other", "auto", domain.LangAuto},
{"swedish", "sv", domain.LangSwedish},
} {
t.Run(tt.name, func(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
rec := do(t, app, langReq(t, []string{"title", "language", "file"}, tt.language))
require.Equal(t, http.StatusSeeOther, rec.Code)
require.Len(t, sub.jobs, 1)
assert.Equal(t, tt.want, sub.jobs[0].Language)
assert.Equal(t, tt.want, rowLanguage(t, rec.Header().Get("Location")))
})
}
}
func TestUploadPost_NoLanguageFieldIsSwedishAsBefore(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
rec := do(t, app, langReq(t, []string{"title", "file"}, ""))
require.Equal(t, http.StatusSeeOther, rec.Code)
require.Len(t, sub.jobs, 1)
assert.Equal(t, domain.LangSwedish, sub.jobs[0].Language)
assert.Equal(t, domain.LangSwedish, rowLanguage(t, rec.Header().Get("Location")))
}
// A part after the file is never read (the file part ends the stream), so a client that
// orders its fields wrongly gets Swedish, the old behaviour, not a guess.
func TestUploadPost_ALanguageFieldAfterTheFileIsNotReadAndTheUploadIsSwedish(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
rec := do(t, app, langReq(t, []string{"title", "file", "language"}, "auto"))
require.Equal(t, http.StatusSeeOther, rec.Code)
require.Len(t, sub.jobs, 1)
assert.Equal(t, domain.LangSwedish, sub.jobs[0].Language)
}
func TestUploadPost_ALanguageOutsideTheClosedSetIsRefusedBeforeAnythingIsSpooledOrStored(t *testing.T) {
for _, bad := range []string{"en", "SV", " sv", "sv,auto", "../auto", strings.Repeat("a", 5000)} {
t.Run(bad[:min(len(bad), 12)], func(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
rec := do(t, app, langReq(t, []string{"title", "language", "file"}, bad))
assert.Equal(t, http.StatusBadRequest, rec.Code)
assert.Empty(t, sub.jobs, "a refused post must not enqueue transcription")
rows, err := newStore(t).ListUploads(context.Background(), userID)
require.NoError(t, err)
assert.Empty(t, rows, "and must leave no row")
assert.Contains(t, body(t, rec), "Choose a language from the list.")
})
}
}
func TestUploadPage_TheFormOffersSwedishPreselectedAndOtherBeforeTheFilePart(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
b := body(t, do(t, app, get("/uploads")))
sel := strings.Index(b, `name="language"`)
file := strings.Index(b, `name="file"`)
require.True(t, sel >= 0, "the form has a language field")
assert.True(t, sel < file, "the field must come before the file part: the stream ends at the file")
assert.Regexp(t, `<option value="sv" selected`, b, "Swedish is preselected: today's behaviour")
assert.Contains(t, b, `<option value="auto"`)
}
func TestUploadPage_NamesTheLanguageInEveryState(t *testing.T) {
for _, status := range []string{"queued", "transcribing", "done", "failed"} {
for _, tt := range []struct {
language domain.Language
want string
}{
{domain.LangSwedish, "Swedish"},
{domain.LangAuto, "Other (detected by the engine)"},
} {
t.Run(status+"/"+string(tt.language), func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", status, "Synthetic transcript text.")
_, err := rawPool(t).Exec(context.Background(), `UPDATE uploaded_transcripts SET language = $2 WHERE id = $1`, id, string(tt.language))
require.NoError(t, err)
b := body(t, do(t, app, get("/uploads/"+id)))
assert.Contains(t, b, "Language: "+tt.want)
})
}
}
}
+1 -3
View File
@@ -13,8 +13,6 @@ import (
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
) )
// Error paths of the upload routes log a kind, never the error's text, which can carry // Error paths of the upload routes log a kind, never the error's text, which can carry
@@ -39,7 +37,7 @@ func (l *lockedBuf) String() string {
type failingUploader struct{ err error } type failingUploader struct{ err error }
func (f failingUploader) Accept(context.Context, string, string, domain.Language, io.Reader) (string, error) { func (f failingUploader) Accept(context.Context, string, string, io.Reader) (string, error) {
return "", f.err return "", f.err
} }
-19
View File
@@ -1,19 +0,0 @@
package web_test
// /uploads was mounted but nothing linked to it (#37). The shared Layout has no access to
// the App, so the entry point is the home list, shown only when uploads are mounted.
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestHomeList_LinksToRecordingsOnlyWhenUploadsAreMounted(t *testing.T) {
with, _ := newUploadApp(t, audioOK, 200<<20)
without := newApp(t)
assert.Contains(t, body(t, do(t, with, get("/"))), `href="/uploads"`)
assert.NotContains(t, body(t, do(t, without, get("/"))), `href="/uploads"`,
"a link to a route that is not mounted would be a dead end")
}
File diff suppressed because it is too large Load Diff
+1 -4
View File
@@ -107,12 +107,9 @@ templ flashBanner(code string) {
// #summary-list region; a non-HTMX request renders the whole page. flash carries // #summary-list region; a non-HTMX request renders the whole page. flash carries
// a one-shot notification (e.g. "connected", "registered") surfaced on arrival // a one-shot notification (e.g. "connected", "registered") surfaced on arrival
// after a POST→redirect. // after a POST→redirect.
templ ListPage(b listBuckets, f Filter, stats PipelineStats, flash string, hasConnected bool, channels []string, autoSummarize bool, uploadsEnabled bool) { templ ListPage(b listBuckets, f Filter, stats PipelineStats, flash string, hasConnected bool, channels []string, autoSummarize bool) {
@Layout("Tapir — Summaries") { @Layout("Tapir — Summaries") {
@flashBanner(flash) @flashBanner(flash)
if uploadsEnabled {
<p class="muted"><a href="/uploads">Recordings</a>: upload audio and get a private transcript.</p>
}
if hasConnected { if hasConnected {
@pasteForm() @pasteForm()
} }
File diff suppressed because it is too large Load Diff
-10
View File
@@ -122,16 +122,6 @@ var scenarioCoverage = map[string]string{
"I can ask about my recording and get a grounded answer": "TestUploadChatPost_SendsTheOwnersTranscriptWithTheUploadBudget", "I can ask about my recording and get a grounded answer": "TestUploadChatPost_SendsTheOwnersTranscriptWithTheUploadBudget",
"A long recording is not cut to its first third": "TestAnswerMaxCharsOverridesTheServiceCap", "A long recording is not cut to its first third": "TestAnswerMaxCharsOverridesTheServiceCap",
"A transcript made by the cloud fallback says the audio left the homelab": "TestUploadStatus_ACloudServedTranscriptSaysTheAudioLeftTheHomelab",
"I choose the language of a recording": "TestUploadPost_TheLanguageFieldReachesTheJobAndTheRow",
"Swedish stays the default": "TestUploadPost_NoLanguageFieldIsSwedishAsBefore",
"A language outside the list is refused": "TestUploadPost_ALanguageOutsideTheClosedSetIsRefusedBeforeAnythingIsSpooledOrStored",
"A recording in another language never reaches the Swedish-only engine": "TestEnsemble_AnAutoLanguageUploadNeverReachesTheBaseEngineAndIsNotReconciled",
"I can delete a finished recording": "TestUploadDelete_AnOwnerRemovesAFinishedOrFailedRecordingAndLandsOnTheList",
"A recording still being transcribed cannot be deleted": "TestUploadDelete_ARecordingStillBeingTranscribedIsRefusedWith409AndKept",
"The home list links to my recordings when uploads are on": "TestHomeList_LinksToRecordingsOnlyWhenUploadsAreMounted",
// audio_upload.feature, ADR-034 // audio_upload.feature, ADR-034
"Two engines transcribe the audio and their disagreements are reconciled": "TestEnsemble_BothEnginesRunInOrderAndTheirDisagreementIsReconciled", "Two engines transcribe the audio and their disagreements are reconciled": "TestEnsemble_BothEnginesRunInOrderAndTheirDisagreementIsReconciled",
"Trivial disagreements never reach the model": "TestReconcile_TrivialDisputesAreResolvedWithoutAskingTheModel", "Trivial disagreements never reach the model": "TestReconcile_TrivialDisputesAreResolvedWithoutAskingTheModel",
-143
View File
@@ -1,143 +0,0 @@
package lintgate
// Observes the exit status of `task lint` (the lint step of `task check`), not the lint
// rules themselves. It runs the repo's real Taskfile in a scratch module so a change to the
// lint command that stops it failing on findings (#41) turns this red.
import (
"bytes"
"errors"
"os"
"os/exec"
"path/filepath"
"regexp"
"strings"
"testing"
)
const scratchWithFinding = `package main
import "os"
func main() { os.Remove("x") }
`
func scratchModule(t *testing.T) string {
t.Helper()
taskfile, err := os.ReadFile(filepath.Join("..", "..", "Taskfile.yml"))
if err != nil {
t.Fatalf("read Taskfile.yml: %v", err)
}
dir := t.TempDir()
for name, body := range map[string]string{
"Taskfile.yml": string(taskfile),
"go.mod": "module scratch\n\ngo 1.21\n",
"main.go": scratchWithFinding,
} {
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil {
t.Fatalf("write %s: %v", name, err)
}
}
return dir
}
// ansi matches the colour escape sequences task prints when it thinks it has a terminal.
var ansi = regexp.MustCompile("\x1b\\[[0-9;]*m")
// hasSkipNotice reports whether the skip notice was printed as output. Task echoes the command
// line first, and that line contains the same words, so a substring match would be vacuous.
func hasSkipNotice(out string) bool {
for _, line := range strings.Split(ansi.ReplaceAllString(out, ""), "\n") {
if strings.HasPrefix(line, "golangci-lint not installed") {
return true
}
}
return false
}
// runLint runs `task lint` in dir with the given PATH and returns its combined output and exit code.
func runLint(t *testing.T, dir, path string) (string, int) {
t.Helper()
return runTask(t, dir, path, "lint")
}
func runTask(t *testing.T, dir, path string, args ...string) (string, int) {
t.Helper()
taskBin, err := exec.LookPath("task")
if err != nil {
t.Skip("task not installed")
}
cmd := exec.Command(taskBin, args...)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "PATH="+path)
var out bytes.Buffer
cmd.Stdout, cmd.Stderr = &out, &out
err = cmd.Run()
var exitErr *exec.ExitError
switch {
case err == nil:
return out.String(), 0
case errors.As(err, &exitErr):
return out.String(), exitErr.ExitCode()
default:
t.Fatalf("run task lint: %v", err)
return "", -1
}
}
func TestTaskLint_ExitsNonZeroWhenGolangciLintReportsAFinding(t *testing.T) {
if _, err := exec.LookPath("golangci-lint"); err != nil {
t.Skip("golangci-lint not installed")
}
out, code := runLint(t, scratchModule(t), os.Getenv("PATH"))
if code == 0 {
t.Fatalf("task lint exited 0 on a module with an unchecked error; output:\n%s", out)
}
if hasSkipNotice(out) {
t.Fatalf("task lint blamed a missing binary for a lint failure; output:\n%s", out)
}
if !strings.Contains(out, "errcheck") {
t.Fatalf("a non-zero exit without the finding means the linter crashed, not that it found something; output:\n%s", out)
}
}
// The lint step only gates anything if `task check` runs it.
func TestTaskCheck_RunsTheLintStep(t *testing.T) {
out, code := runTask(t, scratchModule(t), os.Getenv("PATH"), "--dry", "check")
if code != 0 {
t.Fatalf("task --dry check exited %d; output:\n%s", code, out)
}
if !strings.Contains(out, "task: [lint]") {
t.Fatalf("task check no longer runs the lint step; output:\n%s", out)
}
}
func TestTaskLint_SkipsWithANoticeWhenGolangciLintIsAbsent(t *testing.T) {
taskBin, err := exec.LookPath("task")
if err != nil {
t.Skip("task not installed")
}
bin := t.TempDir() // a PATH holding task and nothing else
if err := os.Symlink(taskBin, filepath.Join(bin, "task")); err != nil {
t.Fatalf("symlink task: %v", err)
}
out, code := runLint(t, scratchModule(t), bin)
if code != 0 {
t.Fatalf("task lint exited %d without golangci-lint on PATH; output:\n%s", code, out)
}
if !hasSkipNotice(out) {
t.Fatalf("expected the skip notice; output:\n%s", out)
}
}
// CI runs task with colour on, so the notice line starts with an escape sequence; a bare
// prefix match missed it there while passing on a terminal-less local run.
func TestHasSkipNotice_SeesTheNoticeBehindColourCodes(t *testing.T) {
out := "\x1b[32mtask: [lint] ...\n\x1b[0mgolangci-lint not installed — skipping (CI installs it)\n"
if !hasSkipNotice(out) {
t.Fatalf("notice not found in coloured output: %q", out)
}
if hasSkipNotice("\x1b[32mtask: [lint] if command -v golangci-lint ...; echo \"golangci-lint not installed\"\n") {
t.Fatal("the echoed command line must not count as the notice")
}
}