Compare commits
@@ -53,10 +53,12 @@ jobs:
|
||||
|
||||
- name: Derive image tags
|
||||
id: meta
|
||||
env:
|
||||
# signals#47: a ref name can carry $(...); pass it via env, not ${{ }}.
|
||||
REF: ${{ github.ref }}
|
||||
run: |
|
||||
SHA=$(git rev-parse --short HEAD)
|
||||
echo "sha-tag=${SHA}" >> "$GITHUB_OUTPUT"
|
||||
REF="${{ github.ref }}"
|
||||
if [[ "$REF" == refs/tags/v* ]]; then
|
||||
echo "version-tag=${REF#refs/tags/}" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
@@ -27,6 +27,23 @@ docs it indexes.
|
||||
- **Language is Go (ADR-001).** Do not introduce Python or a second language. If you think you
|
||||
need one, that's a new ADR with a real justification, not a default.
|
||||
|
||||
## Deploying: a push does not deploy
|
||||
|
||||
Tapir runs on the koala k3s cluster under Flux GitOps. Pushing here only builds the image;
|
||||
`kubectl set image` is reverted by Flux drift correction within seconds. The manifest bump is
|
||||
manual: Flux image-automation in `mathias/infra` covers only `supervisor`, not tapir (infra#111
|
||||
is closed, but `k3s/flux/flux-system/image-automation.yaml` has no tapir policy).
|
||||
|
||||
1. CI (`.gitea/workflows/ci.yml`) builds and pushes `localhost:5000/tapir:<short-sha>` on the
|
||||
main push.
|
||||
2. In `mathias/infra`, bump `image:` in `k3s/apps/tapir/deployment.yaml` to that sha; commit to
|
||||
`main` (infra's own rules and full `task check` apply).
|
||||
3. On koala: `flux reconcile kustomization apps --with-source`, then
|
||||
`kubectl -n tapir rollout status deploy/tapir`. Both need per-action approval in a session.
|
||||
4. Confirm the running image is the new sha and the startup log shows the new code.
|
||||
|
||||
Runbook and why: brain `homelab/failures/flux-reverts-kubectl-set-image-deploy-via-git`.
|
||||
|
||||
## Things that look reusable but are NOT — read before "reusing"
|
||||
|
||||
These caused real mistakes that were caught and corrected; the corrections are load-bearing.
|
||||
|
||||
+31
-7
@@ -1434,7 +1434,12 @@ an hour of Swedish *was* an unknown here; it is now measured, see decision 8.)
|
||||
empty disables), recorded per row, and called out in Open items rather than defaulted
|
||||
silently.
|
||||
- Swedish only (`language=sv`). Language detection and routing (#28 decision 2) are not
|
||||
touched.
|
||||
touched. **Amended 2026-10-04 (#61, interim):** the language is now a per-upload choice on
|
||||
the form, `sv` (preselected, and what an upload that names none gets) or `auto` ("Other":
|
||||
`language` is omitted so the engine detects it), a closed set held by the form, the request
|
||||
code and a CHECK constraint on `uploaded_transcripts.language` (migration 021). Automatic
|
||||
detection and a general-purpose engine for other languages are still not built (#61); an
|
||||
`auto` upload goes to the same configured engine, which is not tuned for it.
|
||||
- `-H 'Expect:'`, multipart field `file` — encoded in the adapter and its tests, not rediscovered.
|
||||
|
||||
7. **Jobs run in-process, one at a time (ADR-018 stands).** Transcription is minutes (~7.5 min
|
||||
@@ -1461,10 +1466,19 @@ an hour of Swedish *was* an unknown here; it is now measured, see decision 8.)
|
||||
as busy beyond that. Together with the 20-job queue and the running job that bounds the spool
|
||||
at about (4 + 20 + 1) x 200 MiB = 5 GiB of the 6 Gi `emptyDir` (infra): raising any of those or
|
||||
`TAPIR_UPLOAD_MAX_BYTES` must revisit that sum (pinned by `TestNewIntake_Defaults...`, not
|
||||
enforced against infra). **A slow-drip body holds its slot without a time bound**: the server
|
||||
sets `ReadHeaderTimeout` only, because a `ReadTimeout` would cut legitimate 200 MiB uploads on
|
||||
slow links; two slow bodies take a user's two slots and two such users make everyone's upload
|
||||
busy. It needs a valid session; recorded as a limitation with a follow-up issue.
|
||||
enforced against infra). **A body is bounded by its silence, not its length (#56, amended
|
||||
2026-10-04)**: a `ReadTimeout` would cut legitimate 200 MiB uploads on slow links, so instead
|
||||
every read of the upload body re-arms the connection's read deadline 60 s ahead
|
||||
(`App.UploadIdleTimeout`). A body that stops delivering bytes fails its read, Accept returns
|
||||
and the slot is released; one that keeps moving is never cut. The handler never clears the
|
||||
deadline: an armed deadline also bounds net/http's drain of an unread remainder before it
|
||||
replies, and net/http resets it itself before its background read. A deadline still armed
|
||||
while the file is probed (the multipart part can end before the body's last bytes arrive)
|
||||
costs at most keep-alive reuse of that connection, not the upload. The 60 s default is the
|
||||
deployed value (cmd/tapir does not set it) and is pinned by a test. It needs `http.ResponseController` to reach the connection, so the
|
||||
metrics middleware's writer implements `Unwrap`; where a deadline cannot be set the handler
|
||||
logs `upload idle bound unavailable` rather than silently running unbounded. Still not
|
||||
bounded: a body that drips one byte per 59 s holds its slot for as long as it keeps that up.
|
||||
|
||||
8. **Chat reuses `chat.Service`; it is not forked.** The uploaded-transcript chat handlers
|
||||
look the row up through the RLS-scoped store, so a non-owner gets 404 by the same mechanism
|
||||
@@ -1551,8 +1565,14 @@ deferred. The scope cut — chat first, no rich analysis, no video — is what k
|
||||
disabled by default, a failed iguana job fails honestly and the user retries. *Blocks:*
|
||||
the production default only; the adapter ships either way.
|
||||
- ~~Verify Tapir's ingress `proxy-body-size` against the 200 MB cap (`infra`).~~ **Resolved 2026-10-03:** real unauthenticated bodies of 2, 30, 120 and 210 MiB all pass NPM (TLS, piguard) and Traefik to tapir (a 302 to login); there is no body limit in that range. Uploads were enabled in production the same day, local engines only (infra#520).
|
||||
- Per-transcript deletion is not built; account deletion cascades. A private-content store
|
||||
without a delete button is a gap worth a ticket, not a silent omission.
|
||||
- ~~Per-transcript deletion is not built; account deletion cascades.~~ **Built 2026-10-03
|
||||
(#37):** `POST /uploads/{id}/delete`, owner-scoped (a non-owner gets 404 by the same RLS
|
||||
mechanism), confirm-then-delete on the recording's page. A recording that is **queued or
|
||||
transcribing is refused (409) until it ends**, because the worker would otherwise write to a
|
||||
row that is gone; a job whose service was lost stays undeletable until the next start's sweep
|
||||
marks it failed. The refusal reflects the row's state when the delete was attempted: a
|
||||
recording that finishes in the same instant still gets a 409 once, and a retry deletes it.
|
||||
The home list now links to `/uploads` when uploads are mounted.
|
||||
|
||||
**Reversibility.** Additive: one table (down-migration drops it), one adapter, one upload
|
||||
surface, one image change. Removing the upload route removes the feature; `transcripts`, `videos`
|
||||
@@ -1639,6 +1659,10 @@ on the slow numbers and is reopened: 31B scored 97% against 92%.
|
||||
job timeout rises from 30 to 60 minutes: ~4 min of kb-whisper and ~40 s of pianissimo per hour of
|
||||
audio, plus the adjudication.
|
||||
|
||||
**Amended 2026-10-04 (#61, interim):** reconcile is for Swedish. A recording whose language is
|
||||
`auto` is never sent to the base engine (it is Swedish-only) and is not reconciled: the other
|
||||
engine transcribes alone, then the cloud fallback if that fails and the job is live, and the
|
||||
page does not claim a second opinion.
|
||||
8. **Not decided here.** A cloud adjudicator (rejected: private content); tuning the prompt; making
|
||||
`gemma4-31b` the default; a "reconciling" progress state (the page says "transcribing" for the whole
|
||||
job); feeding the uncertain spots to chat; whether pianissimo should also be a middle tier in the
|
||||
|
||||
@@ -25,7 +25,7 @@ type fakeUploadsStore struct {
|
||||
failedKinds []domain.UploadErrorKind
|
||||
}
|
||||
|
||||
func (f *fakeUploadsStore) CreateUpload(context.Context, string, string, bool) (string, error) {
|
||||
func (f *fakeUploadsStore) CreateUpload(context.Context, string, string, domain.Language, bool) (string, error) {
|
||||
return "id", nil
|
||||
}
|
||||
|
||||
@@ -44,7 +44,7 @@ func (f *fakeUploadsStore) SaveProgress(context.Context, string, string, []domai
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeUploadsStore) Heartbeat(context.Context, string, string) error { return nil }
|
||||
func (f *fakeUploadsStore) Heartbeat(context.Context, string, string, time.Time) error { return nil }
|
||||
|
||||
func (f *fakeUploadsStore) FailInterruptedUploads(context.Context) (int, error) {
|
||||
f.sweeps++
|
||||
|
||||
@@ -14,7 +14,8 @@ it** — endpoints and aliases drift, and this file is a snapshot (2026-06-06),
|
||||
not LiteLLM. Use `30401` for off-cluster LiteLLM. `confirm` resolved.
|
||||
- **Speech-to-text (ADR-033):** the same gateway serves `POST /v1/audio/transcriptions` for
|
||||
`iguana/kb-whisper` (primary, local) and `berget/kb-whisper` (fallback, EU cloud — private audio
|
||||
leaves the homelab if it is used). **Call it via the NodePort / in-cluster address above, NOT
|
||||
leaves the homelab if it is used: when the primary fails **or returns an empty transcript**,
|
||||
e.g. a silent or music-only file; the transcript page then says so, #52). **Call it via the NodePort / in-cluster address above, NOT
|
||||
`https://llm-api.d-ma.be`:** that public ingress (openresty) returns `413` for bodies over
|
||||
roughly 8–10 MB (~5 minutes of 16 kHz WAV), while the NodePort took 174 MB (90 minutes) in one
|
||||
request (measured 2026-10-01, `docs/research/audio-length-ceiling-2026-10-01.md`). Send multipart
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
# ADR-034 second blind check: protocol (ready to run; tapir#33, D9, D10)
|
||||
|
||||
This repo is public and the recordings are private, so this file holds the method only. The
|
||||
sheet, the key and the filled-in verdicts stay outside the repo, and the trial data is deleted
|
||||
after the check. Results go back here as aggregate numbers, as in
|
||||
`reconcile-blind-check-2026-10-02.md`.
|
||||
|
||||
## What the check decides
|
||||
|
||||
ADR-034 (two engines, a local model reconciles their disagreements) is `Proposed` and off in
|
||||
production. Two conditions were set for keeping it. The first run answered the first one on one
|
||||
reading only and did not answer the second:
|
||||
|
||||
1. **The reconciled reading is right at least 85% of the time.** Open flaw in the bar: it did not
|
||||
say how ties count. The first run gave 95% (decided items only) or 70% (all 30), so it was met on
|
||||
one reading and missed on the other.
|
||||
2. **It is no worse than the better single engine.** Not answered: before ADR-035 the spots the
|
||||
model *kept* as the base were not stored, so kb-whisper alone could not be compared.
|
||||
|
||||
ADR-035 now stores every judged spot (`reconciliation.contested`: both readings, which one was
|
||||
used, unsure or not, a few words of left context, a time). That makes condition 2 measurable.
|
||||
|
||||
## Rules to fix BEFORE looking at any result (the owner confirms or changes them)
|
||||
|
||||
| Rule | Proposed |
|
||||
|---|---|
|
||||
| Who judges | The speaker, hearing the audio where the text cannot settle it (D9). Not Opus: it cannot hear, and its agreement with the speaker was only shown on items picked as decidable. |
|
||||
| Sample | 30 spots drawn at random with a fixed seed, from ALL judged spots (changed and kept-base alike), not only the changed ones. Insertions and deletions that cannot be placed are listed separately and not sampled. |
|
||||
| Verdict per spot | `x`, `y` (the two readings, shown in random order, key kept apart), `neither`, or `cannot tell`. |
|
||||
| Condition 1 | Reconciled is right on at least 85% of the spots where the judge could tell (`cannot tell` excluded, reported, and `neither` counts as not right). Also report all-30. Pass needs the decided-only figure at 85% or more AND the all-30 figure above 70%. |
|
||||
| Condition 2 | Over the same spots, the reconciled text is right at least as often as the better single engine (base engine alone is right where the base reading is right; the other engine alone where the alt reading is right). Ties favour the single engine. |
|
||||
| If it fails | Switch reconcile off by configuration; the single-engine path is unchanged (ADR-034 reversibility). |
|
||||
|
||||
Why a fixed all-30 floor: the literal 70% last time sat well under the bar, and "decided only" can
|
||||
hide a model that mostly lands on spelling differences.
|
||||
|
||||
## What a run needs (D10: the owner is present)
|
||||
|
||||
- The same real recording as before (about 60 minutes). Only the owner has it.
|
||||
- A **throwaway** tapir instance with its own Postgres, not the production pod, so production's
|
||||
key pin and deployment are untouched. Settings: `TAPIR_RECONCILE_BASE_MODEL=iguana/pianissimo`,
|
||||
`TAPIR_TRANSCRIBE_MODEL=iguana/kb-whisper`, `TAPIR_RECONCILE_MODEL=iguana/gemma4-31b`, cloud
|
||||
fallbacks empty. Job time was 21 min 49 s per 63.6 min last run.
|
||||
- iguana with Low Power Mode off and gemma4-31b able to load (the production-iguana model load
|
||||
that needs a quiet window; D6).
|
||||
|
||||
## Steps
|
||||
|
||||
1. Start the throwaway instance, upload the recording, wait for `done` (about 22 minutes).
|
||||
2. Read `reconciliation.contested` from the throwaway database. From it build, **outside the repo**:
|
||||
`sheet` (spot number, time, left context, reading x, reading y, random order) and `key` (which
|
||||
of x/y is the base, which is the other engine, which one the reconciler used).
|
||||
3. The speaker fills in the sheet (about 20 to 30 minutes for 30 spots, audio open at the times).
|
||||
The agent does not see the key until the sheet is returned.
|
||||
4. Score with the rules above. Write the aggregate numbers into a dated note next to this one,
|
||||
and amend ADR-034's validation section (kept, or switched off, and the tie rule used).
|
||||
5. Delete the throwaway instance, its database and the sheet, as last time.
|
||||
6. Only if the check passes and the owner wants it in production: enabling reconcile there is
|
||||
the separate D10 step (it needs `iguana/pianissimo` and the reconciler added to the tapir key
|
||||
pin in the infra repo plus the env vars on the deployment, `risk/medium`, owner present).
|
||||
|
||||
## Open and owner-only
|
||||
|
||||
- Confirm or change the rules table above before the run.
|
||||
- The **tool that builds the sheet and key** is the one piece not in the repo (last time it was a
|
||||
one-off script). It can be built test-first on request; until then the agent builds it ad hoc
|
||||
outside the repo.
|
||||
- A Berget adjudicator (D8, tapir#45) is a different question with its own measurement and is
|
||||
not part of this check: this check judges the local setup that ADR-034 describes.
|
||||
@@ -1,7 +1,8 @@
|
||||
# Sprint: audio upload follow-ups (Sweep Sprint, 2026-10)
|
||||
|
||||
Method: Sweep Sprint (phase 1 parallel read-only research per issue: **done**; phase 2 sequential
|
||||
execution, one issue per commit: **not started, waits for the owner's go**). Closing oath below.
|
||||
execution, one issue per commit: **v0.31.1 and v0.32.0 shipped 2026-10-03/04, the owner-held items
|
||||
below remain open**). Closing oath below.
|
||||
|
||||
Milestones: [v0.31.1 - security and robustness](https://git.d-ma.be/mathias/tapir/milestone/10),
|
||||
[v0.32.0 - audio upload follow-ups](https://git.d-ma.be/mathias/tapir/milestone/11).
|
||||
@@ -63,8 +64,8 @@ Release v0.31.1: one adversarial review round over 2-5, then tag, push, observe.
|
||||
| D5 | #38: Berget length measurement needs a throwaway key | Yes, throwaway key | #38 |
|
||||
| D6 | #38: load large models on production iguana | Yes, in a quiet window with the owner present | #38 |
|
||||
| D7 | #46: where the ffmpeg tarball lives | Do nothing now; file an issue | none |
|
||||
| D8 | #45: evaluate Berget as the reconciler | No; stay local-only | #45 |
|
||||
| D9 | Who judges ADR-034's second blind check | The owner; do not reuse the one-off Opus authorisation | ADR-034 |
|
||||
| D8 | #45: evaluate Berget as the reconciler | **Decided 2026-10-04 by the owner: yes, Berget may be used, and each user must be able to keep a recording on local models only** (reverses the recommendation). Opt-out first: #60; #45 stays as the model measurement | #60, #45 |
|
||||
| D9 | Who judges ADR-034's second blind check | The owner; do not reuse the one-off Opus authorisation. Protocol: `docs/research/reconcile-second-check-protocol.md` | ADR-034 |
|
||||
| D10 | When to enable reconcile in production | Needs `iguana/pianissimo` in the key pin and env vars; owner present | ADR-034 |
|
||||
|
||||
## Dropped or reduced, with reasons
|
||||
@@ -86,3 +87,29 @@ Release v0.31.1: one adversarial review round over 2-5, then tag, push, observe.
|
||||
SemVer: v0.31.1 patch (fixes, no new behaviour); v0.32.0 minor (nav link, delete, cloud labelling).
|
||||
Every push to tapir main redeploys the pod and fails any job in flight: push when no upload is
|
||||
running (check the pod log first).
|
||||
|
||||
## Status (2026-10-04)
|
||||
|
||||
**Shipped and deployed:** v0.31.1 (PR #55, merge e037c6b) and v0.32.0 (PR #58, merge 85f4cc3; pod
|
||||
tapir-597dbfccc-ng78q 1/1, 0 restarts, image 85f4cc3). Each went through an adversarial review
|
||||
round by agent with the verdicts recorded on the PR (v0.31.1: BLOCK then APPROVE; v0.32.0: BLOCK,
|
||||
BLOCK, APPROVE, APPROVE; every block was a real defect, fixed test-first).
|
||||
|
||||
**Live evidence:** a foreign-origin `POST` to `/account/delete` and to `/uploads/{id}/delete` is
|
||||
refused (403) on production, a same-origin one reaches the login redirect (302).
|
||||
|
||||
**Not observed live (needs an authenticated session, so it is the owner's look; closed on word or screenshot):**
|
||||
|
||||
- #52, cloud side: a transcript served by the Berget fallback names `berget`; a reconciled all-local
|
||||
one shows no note. The local side was observed on 2026-10-04 (a new clip served by iguana: chip
|
||||
`iguana/…`, no cloud note). The cloud side is carried as an acceptance item in #60.
|
||||
|
||||
**Observed live since:** #37 (owner, 2026-10-04: link, confirm-then-delete, no control while transcribing; closed). #50: at the 2026-10-04 rollout the old pod went `Terminating` then `Completed`
|
||||
(not `Error`), 0 restarts; #50 closed. No job was in flight, so a drained job is still unobserved
|
||||
(a recorded limitation, D2).
|
||||
|
||||
**Still open (owner's call, milestone 11):** #36, #38, #45 (now the Berget adjudicator measurement), #60 (per-recording local-only choice, 4 sub-decisions), #46 remainder, #56 (slow-drip bodies hold
|
||||
a receive slot), #57 (shutdown logging, second signal). Decisions D4-D6, D9, D10 above are unchanged; D8 is decided (above).
|
||||
|
||||
**Deferred and filed from the reviews:** #56, #57, #59 (start-up exits when the identity provider's OIDC
|
||||
discovery returns 503 (seen once at 19:40 on 2026-10-03; the pod restarted itself and recovered)).
|
||||
|
||||
@@ -240,6 +240,24 @@ Feature: Upload an audio recording, get a private Swedish transcript, discuss it
|
||||
Then it is transcribed anyway and its audio step says how long the audio really is
|
||||
And a complete recording whose header over-estimated its length is not refused
|
||||
|
||||
Scenario: A transcript made by the cloud fallback says the audio left the homelab
|
||||
When the transcript was produced by an endpoint that is not local
|
||||
Then its page says in plain words that the audio left the homelab and names the provider
|
||||
And a locally transcribed recording carries no such note
|
||||
|
||||
Scenario: I can delete a finished recording
|
||||
When I confirm deleting a done or failed recording
|
||||
Then the recording and its transcript are gone and I am back on my list
|
||||
And another user's recording is a 404 and is untouched
|
||||
|
||||
Scenario: A recording still being transcribed cannot be deleted
|
||||
When I try to delete a recording that is queued or transcribing
|
||||
Then I am told it is still being transcribed and nothing is removed
|
||||
|
||||
Scenario: The home list links to my recordings when uploads are on
|
||||
Given audio upload is enabled
|
||||
Then the home list links to the recordings page
|
||||
|
||||
Scenario: A full queue is a retry, not an error
|
||||
When too many recordings are already waiting
|
||||
Then the upload is answered as busy, its row is failed honestly and its audio is deleted
|
||||
@@ -256,3 +274,20 @@ Feature: Upload an audio recording, get a private Swedish transcript, discuss it
|
||||
Scenario: Input that cannot be right is a client error
|
||||
When the id is not an id, or a header or title holds control bytes
|
||||
Then the answer is a 4xx or the title is cleaned, never a server error
|
||||
|
||||
Scenario: I choose the language of a recording
|
||||
When I upload a recording and choose Swedish or Other on the form
|
||||
Then the recording is transcribed as that language and its page says which
|
||||
|
||||
Scenario: Swedish stays the default
|
||||
When an upload names no language
|
||||
Then it is transcribed as Swedish, as before the choice existed
|
||||
|
||||
Scenario: A language outside the list is refused
|
||||
When an upload names a language the form does not offer
|
||||
Then it is answered 400 before anything is stored or sent to an engine
|
||||
|
||||
Scenario: A recording in another language never reaches the Swedish-only engine
|
||||
When I choose Other for a recording
|
||||
Then the language is left for the engine to detect and the Swedish-only base engine and the reconcile are not used
|
||||
And the engine that does run is the configured one, which is not tuned for other languages
|
||||
|
||||
@@ -33,9 +33,9 @@ require (
|
||||
github.com/rogpeppe/go-internal v1.15.0 // indirect
|
||||
github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 // indirect
|
||||
go.yaml.in/yaml/v2 v2.4.2 // indirect
|
||||
golang.org/x/sync v0.18.0 // indirect
|
||||
golang.org/x/sync v0.21.0 // indirect
|
||||
golang.org/x/sys v0.41.0 // indirect
|
||||
golang.org/x/text v0.31.0 // indirect
|
||||
golang.org/x/text v0.39.0 // indirect
|
||||
google.golang.org/protobuf v1.36.8 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
)
|
||||
|
||||
@@ -113,12 +113,12 @@ go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=
|
||||
go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
|
||||
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
|
||||
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
|
||||
golang.org/x/sync v0.18.0 h1:kr88TuHDroi+UVf+0hZnirlk8o8T+4MrK6mr60WkH/I=
|
||||
golang.org/x/sync v0.18.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
|
||||
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
|
||||
golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
|
||||
golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
|
||||
golang.org/x/text v0.31.0 h1:aC8ghyu4JhP8VojJ2lEHBnochRno1sgL6nEi9WGFGMM=
|
||||
golang.org/x/text v0.31.0/go.mod h1:tKRAlv61yKIjGGHX/4tP1LTbc13YSec1pxVEWXzfoeM=
|
||||
golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus=
|
||||
golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM=
|
||||
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
|
||||
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
|
||||
google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc=
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
package llm
|
||||
|
||||
// Observes how much of a gateway response the client will read. The body is read before
|
||||
// the status is looked at, so a misbehaving gateway or proxy that streams forever was
|
||||
// bounded only by the client's overall timeout (#53).
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// endlessGateway answers with the given status and streams filler until the client hangs
|
||||
// up; wrote reports how many bytes it managed to send.
|
||||
func endlessGateway(t *testing.T, status int, wrote *atomic.Int64) *httptest.Server {
|
||||
t.Helper()
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(status)
|
||||
chunk := []byte(strings.Repeat("a", 4096))
|
||||
for {
|
||||
n, err := w.Write(chunk)
|
||||
wrote.Add(int64(n))
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if f, ok := w.(http.Flusher); ok {
|
||||
f.Flush()
|
||||
}
|
||||
}
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return srv
|
||||
}
|
||||
|
||||
func TestClient_ASuccessBodyLargerThanTheCapIsAnErrorAndTheClientStopsReading(t *testing.T) {
|
||||
var wrote atomic.Int64
|
||||
srv := endlessGateway(t, http.StatusOK, &wrote)
|
||||
c := New(srv.URL, "", "m", 30*time.Second, WithMaxResponseBytes(8192))
|
||||
|
||||
start := time.Now()
|
||||
_, err := c.Complete(context.Background(), "s", "u")
|
||||
|
||||
if err == nil || !strings.Contains(err.Error(), "exceeds") {
|
||||
t.Fatalf("want a response-too-large error, got %v", err)
|
||||
}
|
||||
if time.Since(start) > 5*time.Second {
|
||||
t.Fatalf("the client kept reading: took %s", time.Since(start))
|
||||
}
|
||||
// Hanging up lets the server's writes fail; allow what the transport had already buffered.
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
if got := wrote.Load(); got > 8192+1<<20 {
|
||||
t.Fatalf("server managed to send %d bytes after a cap of 8192", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClient_AFailingBodyIsReadOnlyUpToTheErrorBound(t *testing.T) {
|
||||
var wrote atomic.Int64
|
||||
srv := endlessGateway(t, http.StatusBadGateway, &wrote)
|
||||
c := New(srv.URL, "", "m", 30*time.Second)
|
||||
|
||||
start := time.Now()
|
||||
_, err := c.Complete(context.Background(), "s", "u")
|
||||
|
||||
he, ok := err.(*HTTPError)
|
||||
if !ok {
|
||||
t.Fatalf("want *HTTPError, got %T: %v", err, err)
|
||||
}
|
||||
if he.Code != http.StatusBadGateway {
|
||||
t.Fatalf("code %d", he.Code)
|
||||
}
|
||||
if len(he.Body) > maxErrorBody+1 {
|
||||
t.Fatalf("kept %d bytes of an error body, bound is %d", len(he.Body), maxErrorBody)
|
||||
}
|
||||
if time.Since(start) > 5*time.Second {
|
||||
t.Fatalf("the client kept reading an endless error body: took %s", time.Since(start))
|
||||
}
|
||||
}
|
||||
|
||||
// sizedGateway answers 200 with a valid completion padded with trailing whitespace to
|
||||
// exactly size bytes, so the boundary of the cap can be probed to the byte.
|
||||
func sizedGateway(t *testing.T, size int) *httptest.Server {
|
||||
t.Helper()
|
||||
const base = `{"choices":[{"message":{"content":"ok"}}]}`
|
||||
payload := base + strings.Repeat(" ", size-len(base))
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(payload))
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return srv
|
||||
}
|
||||
|
||||
func TestClient_ABodyOfExactlyTheCapIsReadAndOneByteMoreIsRefused(t *testing.T) {
|
||||
const capBytes = 4096
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
size int
|
||||
wantErr bool
|
||||
}{
|
||||
{"one under", capBytes - 1, false},
|
||||
{"exactly the cap", capBytes, false},
|
||||
{"one over", capBytes + 1, true},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
c := New(sizedGateway(t, tt.size).URL, "", "m", 5*time.Second, WithMaxResponseBytes(capBytes))
|
||||
|
||||
got, err := c.Complete(context.Background(), "s", "u")
|
||||
|
||||
if tt.wantErr {
|
||||
if err == nil || !strings.Contains(err.Error(), "exceeds") {
|
||||
t.Fatalf("want a response-too-large error, got %q, %v", got, err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil || got != "ok" {
|
||||
t.Fatalf("want ok, got %q, %v", got, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// New is what production calls: no option. The default is therefore the live bound.
|
||||
func TestClient_TheDefaultCapIsEightMiBWhenNoOptionIsGiven(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
size int
|
||||
wantErr bool
|
||||
}{
|
||||
{"exactly 8 MiB", 8 << 20, false},
|
||||
{"8 MiB plus one", 8<<20 + 1, true},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
c := New(sizedGateway(t, tt.size).URL, "", "m", 10*time.Second)
|
||||
|
||||
_, err := c.Complete(context.Background(), "s", "u")
|
||||
|
||||
if (err != nil) != tt.wantErr {
|
||||
t.Fatalf("size %d: err = %v, wantErr %v", tt.size, err, tt.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestClient_ANonPositiveCapOptionKeepsTheDefault(t *testing.T) {
|
||||
for _, n := range []int64{0, -1} {
|
||||
c := New(sizedGateway(t, 1024).URL, "", "m", 5*time.Second, WithMaxResponseBytes(n))
|
||||
|
||||
if got, err := c.Complete(context.Background(), "s", "u"); err != nil || got != "ok" {
|
||||
t.Fatalf("WithMaxResponseBytes(%d): want ok from the default cap, got %q, %v", n, got, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -37,6 +37,21 @@ type Client struct {
|
||||
temp float64
|
||||
httpClient *http.Client
|
||||
usageHook func(model string, prompt, completion int)
|
||||
maxBody int64
|
||||
}
|
||||
|
||||
// defaultMaxResponseBytes bounds a success body. A completion is a few KB of text; this
|
||||
// leaves room for a long one and none for a gateway that streams forever (#53).
|
||||
const defaultMaxResponseBytes = 8 << 20
|
||||
|
||||
// WithMaxResponseBytes overrides the bound on a success response body. A non-positive n is
|
||||
// ignored (keeps the default).
|
||||
func WithMaxResponseBytes(n int64) Option {
|
||||
return func(c *Client) {
|
||||
if n > 0 {
|
||||
c.maxBody = n
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Option configures a Client at construction. Variadic so the existing 4-arg
|
||||
@@ -101,6 +116,7 @@ func New(baseURL, apiKey, model string, timeout time.Duration, opts ...Option) *
|
||||
maxTokens: defaultMaxTokens,
|
||||
temp: defaultTemperature,
|
||||
httpClient: &http.Client{Timeout: timeout},
|
||||
maxBody: defaultMaxResponseBytes,
|
||||
}
|
||||
for _, opt := range opts {
|
||||
opt(c)
|
||||
@@ -184,12 +200,18 @@ func (c *Client) Complete(ctx context.Context, system, user string) (string, err
|
||||
}
|
||||
defer resp.Body.Close() //nolint:errcheck
|
||||
|
||||
out, err := io.ReadAll(resp.Body)
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
// Only the bounded head is ever used (HTTPError.Error truncates to it), so read no more.
|
||||
// A read error here only shortens a diagnostic; the status code below is what callers act on.
|
||||
head, _ := io.ReadAll(io.LimitReader(resp.Body, maxErrorBody+1))
|
||||
return "", &HTTPError{Code: resp.StatusCode, Body: string(head)}
|
||||
}
|
||||
out, err := io.ReadAll(io.LimitReader(resp.Body, c.maxBody+1))
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("read response: %w", err)
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return "", &HTTPError{Code: resp.StatusCode, Body: string(out)}
|
||||
if int64(len(out)) > c.maxBody {
|
||||
return "", fmt.Errorf("LLM response exceeds %d bytes", c.maxBody)
|
||||
}
|
||||
|
||||
var cr chatResponse
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
ALTER TABLE uploaded_transcripts
|
||||
DROP COLUMN language;
|
||||
@@ -0,0 +1,13 @@
|
||||
-- Migration 021: the language an upload is transcribed as (tapir#61).
|
||||
--
|
||||
-- 'sv' Swedish: the engines are sent language=sv. Every row before this migration was
|
||||
-- transcribed that way (the request hard-coded it), so the default is true history.
|
||||
-- 'auto' Other: the engine detects the language and the Swedish-only engine is not used.
|
||||
--
|
||||
-- A closed set held by the table as well as by the form and the request code, so a second
|
||||
-- writer or a bug cannot store a value the engines are never meant to receive. The column
|
||||
-- sits on the already-FORCE'd, RLS-scoped uploaded_transcripts, so the exact-set isolation
|
||||
-- test is unaffected.
|
||||
ALTER TABLE uploaded_transcripts
|
||||
ADD COLUMN language TEXT NOT NULL DEFAULT 'sv'
|
||||
CONSTRAINT uploaded_transcripts_language_check CHECK (language IN ('sv', 'auto'));
|
||||
@@ -0,0 +1,19 @@
|
||||
package store_test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// TestMain's Postgres already holds a port. A second instance started in the
|
||||
// same process must still come up, i.e. the port is not derived from something
|
||||
// two starters share (the PID). Observes: startTestPostgres succeeds while
|
||||
// another instance from this process is listening.
|
||||
func TestStartTestPostgres_SecondInstanceStartsWhileFirstHoldsItsPort(t *testing.T) {
|
||||
pg, _, rt, err := startTestPostgres(fmt.Sprintf("tapir-epg-store-second-%d", os.Getpid()))
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = pg.Stop(); _ = os.RemoveAll(rt) })
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package store_test
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
@@ -25,27 +26,15 @@ var _ ports.Sink = (*store.Store)(nil)
|
||||
var dsn string
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
// Port + runtime/data dirs are per-process (PID-derived) so two concurrent
|
||||
// `go test` invocations — e.g. a push-run and a tag-run firing together in CI —
|
||||
// don't collide on a fixed port or a shared data dir (which silently failed
|
||||
// both runs). CachePath is shared so the PG archive is downloaded once, not
|
||||
// per process. Base 54000 keeps this package's range distinct from web's.
|
||||
port := uint32(54000 + os.Getpid()%1000)
|
||||
dsn = fmt.Sprintf("postgres://postgres:postgres@localhost:%d/postgres?sslmode=disable", port)
|
||||
|
||||
rt := filepath.Join(os.TempDir(), fmt.Sprintf("tapir-epg-store-%d", os.Getpid()))
|
||||
pg := embeddedpostgres.NewDatabase(
|
||||
embeddedpostgres.DefaultConfig().
|
||||
Port(port).
|
||||
RuntimePath(rt).
|
||||
DataPath(filepath.Join(rt, "data")).
|
||||
BinariesPath(filepath.Join(rt, "bin")).
|
||||
CachePath(filepath.Join(os.TempDir(), "tapir-epg-cache")),
|
||||
)
|
||||
if err := pg.Start(); err != nil {
|
||||
// Runtime/data dirs are per-process (PID-named) and the port is OS-assigned,
|
||||
// so concurrent `go test` runs on a shared CI host don't collide on either.
|
||||
// CachePath is shared so the PG archive is downloaded once, not per process.
|
||||
pg, port, rt, err := startTestPostgres(fmt.Sprintf("tapir-epg-store-%d", os.Getpid()))
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "embedded-postgres start: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
dsn = fmt.Sprintf("postgres://postgres:postgres@localhost:%d/postgres?sslmode=disable", port)
|
||||
|
||||
code := m.Run()
|
||||
|
||||
@@ -56,6 +45,44 @@ func TestMain(m *testing.M) {
|
||||
os.Exit(code)
|
||||
}
|
||||
|
||||
// freePort asks the OS for an unused TCP port. The port is released before
|
||||
// return, so another process can grab it first; startTestPostgres retries.
|
||||
func freePort() (uint32, error) {
|
||||
l, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("listen for free port: %w", err)
|
||||
}
|
||||
defer func() { _ = l.Close() }()
|
||||
return uint32(l.Addr().(*net.TCPAddr).Port), nil
|
||||
}
|
||||
|
||||
// startTestPostgres starts an embedded Postgres on an OS-assigned free port and
|
||||
// returns it with that port and its runtime dir (caller removes it). A port
|
||||
// taken between freePort and Start (shared CI host) is retried with a new one.
|
||||
func startTestPostgres(name string) (*embeddedpostgres.EmbeddedPostgres, uint32, string, error) {
|
||||
rt := filepath.Join(os.TempDir(), name)
|
||||
var lastErr error
|
||||
for range 3 {
|
||||
port, err := freePort()
|
||||
if err != nil {
|
||||
return nil, 0, rt, err
|
||||
}
|
||||
pg := embeddedpostgres.NewDatabase(
|
||||
embeddedpostgres.DefaultConfig().
|
||||
Port(port).
|
||||
RuntimePath(rt).
|
||||
DataPath(filepath.Join(rt, "data")).
|
||||
BinariesPath(filepath.Join(rt, "bin")).
|
||||
CachePath(filepath.Join(os.TempDir(), "tapir-epg-cache")),
|
||||
)
|
||||
if lastErr = pg.Start(); lastErr == nil {
|
||||
return pg, port, rt, nil
|
||||
}
|
||||
lastErr = fmt.Errorf("start on port %d: %w", port, lastErr)
|
||||
}
|
||||
return nil, 0, rt, lastErr
|
||||
}
|
||||
|
||||
// uuids — fixed so tests are deterministic. user_id/video_id are UUID columns.
|
||||
const (
|
||||
userA = "11111111-1111-1111-1111-111111111111"
|
||||
|
||||
@@ -17,6 +17,10 @@ import (
|
||||
// nothing about whether someone else's upload exists (ADR-033).
|
||||
var ErrUploadNotFound = errors.New("store: uploaded transcript not found")
|
||||
|
||||
// ErrUploadInProgress is returned by DeleteUpload for an upload that is still queued or
|
||||
// transcribing: the worker would write its result to a row that no longer exists.
|
||||
var ErrUploadInProgress = errors.New("store: uploaded transcript is still being processed")
|
||||
|
||||
// Uploaded transcripts are PRIVATE per-user content, stored in their own
|
||||
// RLS-scoped table (migration 017) — never in the shared `transcripts` (ADR-021).
|
||||
// Every method goes through withUser, so row-level security confines it to the
|
||||
@@ -27,12 +31,12 @@ var ErrUploadNotFound = errors.New("store: uploaded transcript not found")
|
||||
// CreateUpload records a new upload in the queued state and returns its id.
|
||||
// hadVideo records that the file carried a video track (only its audio is used);
|
||||
// it is kept for the life of the row so the page can say so.
|
||||
func (s *Store) CreateUpload(ctx context.Context, userID, title string, hadVideo bool) (string, error) {
|
||||
func (s *Store) CreateUpload(ctx context.Context, userID, title string, lang domain.Language, hadVideo bool) (string, error) {
|
||||
var id string
|
||||
if err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
|
||||
return tx.QueryRow(ctx,
|
||||
`INSERT INTO uploaded_transcripts (user_id, title, had_video) VALUES ($1, $2, $3) RETURNING id`,
|
||||
userID, title, hadVideo).Scan(&id)
|
||||
`INSERT INTO uploaded_transcripts (user_id, title, language, had_video) VALUES ($1, $2, $3, $4) RETURNING id`,
|
||||
userID, title, string(lang), hadVideo).Scan(&id)
|
||||
}); err != nil {
|
||||
return "", fmt.Errorf("store: create upload: %w", err)
|
||||
}
|
||||
@@ -80,12 +84,14 @@ func (s *Store) SaveProgress(ctx context.Context, userID, id string, steps []dom
|
||||
WHERE id = $1 AND user_id = $2`, b, at)
|
||||
}
|
||||
|
||||
// Heartbeat stamps the upload as alive now. It does not touch progress_at: alive says
|
||||
// the service is still reporting, not that work is moving (ADR-035).
|
||||
func (s *Store) Heartbeat(ctx context.Context, userID, id string) error {
|
||||
// Heartbeat stamps the upload as alive at the given time. It does not touch progress_at:
|
||||
// alive says the service is still reporting, not that work is moving (ADR-035). The time
|
||||
// comes from the application clock, the one progress_at and the page's health check use,
|
||||
// so a database clock that disagrees cannot make a healthy job look lost (#51).
|
||||
func (s *Store) Heartbeat(ctx context.Context, userID, id string, at time.Time) error {
|
||||
return s.updateUpload(ctx, userID, id, "heartbeat",
|
||||
`UPDATE uploaded_transcripts SET alive_at = NOW()
|
||||
WHERE id = $1 AND user_id = $2`)
|
||||
`UPDATE uploaded_transcripts SET alive_at = $3
|
||||
WHERE id = $1 AND user_id = $2`, at)
|
||||
}
|
||||
|
||||
// FailUpload moves a queued or transcribing upload to failed with a typed reason. A
|
||||
@@ -101,6 +107,33 @@ func (s *Store) FailUpload(ctx context.Context, userID, id string, kind domain.U
|
||||
string(kind))
|
||||
}
|
||||
|
||||
// DeleteUpload removes one of the caller's finished (done or failed) uploads. An upload
|
||||
// still queued or transcribing is refused with ErrUploadInProgress and left untouched; one
|
||||
// that is not the caller's is ErrUploadNotFound, as for every other write (#37).
|
||||
func (s *Store) DeleteUpload(ctx context.Context, userID, id string) error {
|
||||
return s.withUser(ctx, userID, func(tx pgx.Tx) error {
|
||||
ct, err := tx.Exec(ctx,
|
||||
`DELETE FROM uploaded_transcripts
|
||||
WHERE id = $1 AND user_id = $2 AND status IN ('done', 'failed')`, id, userID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("store: delete upload: %w", err)
|
||||
}
|
||||
if ct.RowsAffected() > 0 {
|
||||
return nil
|
||||
}
|
||||
var exists bool
|
||||
if err := tx.QueryRow(ctx,
|
||||
`SELECT EXISTS (SELECT 1 FROM uploaded_transcripts WHERE id = $1 AND user_id = $2)`,
|
||||
id, userID).Scan(&exists); err != nil {
|
||||
return fmt.Errorf("store: delete upload: %w", err)
|
||||
}
|
||||
if exists {
|
||||
return ErrUploadInProgress
|
||||
}
|
||||
return ErrUploadNotFound
|
||||
})
|
||||
}
|
||||
|
||||
// updateUpload runs one UPDATE keyed on ($1 = id, $2 = userID) plus extra args,
|
||||
// and maps "no row touched" to ErrUploadNotFound — which is what a write to another
|
||||
// user's upload looks like under RLS (the row is invisible, not forbidden).
|
||||
@@ -188,7 +221,7 @@ func (s *Store) FailInterruptedUploads(ctx context.Context) (int, error) {
|
||||
return total, nil
|
||||
}
|
||||
|
||||
const uploadSelect = `SELECT id, user_id, title, status, content, segments, endpoint,
|
||||
const uploadSelect = `SELECT id, user_id, title, language, status, content, segments, endpoint,
|
||||
audio_seconds, error_kind, had_video, reconciliation, progress, progress_at, alive_at,
|
||||
created_at, finished_at FROM uploaded_transcripts`
|
||||
|
||||
@@ -201,11 +234,13 @@ func scanUpload(row pgx.Row, u *domain.UploadedTranscript, withText bool) error
|
||||
seconds *float64
|
||||
finished, progressAt, aliveAt *time.Time
|
||||
)
|
||||
if err := row.Scan(&u.ID, &u.UserID, &u.Title, &status, &content, &segments, &endpoint,
|
||||
var language string
|
||||
if err := row.Scan(&u.ID, &u.UserID, &u.Title, &language, &status, &content, &segments, &endpoint,
|
||||
&seconds, &errKind, &u.HadVideo, &reconciliation, &progress, &progressAt, &aliveAt,
|
||||
&u.CreatedAt, &finished); err != nil {
|
||||
return err
|
||||
}
|
||||
u.Language = domain.Language(language)
|
||||
u.Status = domain.UploadStatus(deref(status))
|
||||
u.Endpoint = deref(endpoint)
|
||||
u.ErrorKind = domain.UploadErrorKind(deref(errKind))
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
package store_test
|
||||
|
||||
// Deleting one recording (#37). A finished or failed upload can be removed by its owner;
|
||||
// one that is queued or transcribing is refused until it ends, because the worker would
|
||||
// otherwise write its result to a row that is gone (D3 in the sprint plan).
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"git.d-ma.be/mathias/tapir/internal/adapters/store"
|
||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||
)
|
||||
|
||||
func TestDeleteUpload_RemovesAFinishedOrFailedUploadOfTheOwner(t *testing.T) {
|
||||
for name, finish := range map[string]func(*store.Store, string) error{
|
||||
"done": func(s *store.Store, id string) error {
|
||||
return s.CompleteUpload(context.Background(), userA, id, domain.Transcription{Text: "synthetic", Endpoint: "iguana/kb-whisper"})
|
||||
},
|
||||
"failed": func(s *store.Store, id string) error {
|
||||
return s.FailUpload(context.Background(), userA, id, domain.UploadErrTranscriptionFailed)
|
||||
},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
s := newStore(t)
|
||||
id := startedUpload(t, s)
|
||||
require.NoError(t, s.StartTranscribing(context.Background(), userA, id))
|
||||
require.NoError(t, finish(s, id))
|
||||
|
||||
require.NoError(t, s.DeleteUpload(context.Background(), userA, id))
|
||||
|
||||
_, err := s.GetUpload(context.Background(), userA, id)
|
||||
assert.ErrorIs(t, err, store.ErrUploadNotFound)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeleteUpload_RefusesAnUploadThatIsStillQueuedOrTranscribingAndKeepsTheRow(t *testing.T) {
|
||||
s := newStore(t)
|
||||
ctx := context.Background()
|
||||
queued := startedUpload(t, s)
|
||||
running, err := s.CreateUpload(ctx, userA, "running", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, s.StartTranscribing(ctx, userA, running))
|
||||
|
||||
for _, id := range []string{queued, running} {
|
||||
assert.ErrorIs(t, s.DeleteUpload(ctx, userA, id), store.ErrUploadInProgress)
|
||||
_, err := s.GetUpload(ctx, userA, id)
|
||||
assert.NoError(t, err, "a refused delete must leave the row")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeleteUpload_AnotherUsersUploadIsNotFoundAndSurvives(t *testing.T) {
|
||||
s := newStore(t)
|
||||
ctx := context.Background()
|
||||
id := startedUpload(t, s)
|
||||
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
||||
require.NoError(t, s.FailUpload(ctx, userA, id, domain.UploadErrTranscriptionFailed))
|
||||
|
||||
assert.ErrorIs(t, s.DeleteUpload(ctx, userB, id), store.ErrUploadNotFound,
|
||||
"existence is not disclosed to a non-owner")
|
||||
|
||||
_, err := s.GetUpload(ctx, userA, id)
|
||||
assert.NoError(t, err, "B's delete must not have removed A's row")
|
||||
}
|
||||
@@ -34,7 +34,7 @@ func TestFailUpload_StillFailsAQueuedOrTranscribingUpload(t *testing.T) {
|
||||
s := newStore(t)
|
||||
ctx := context.Background()
|
||||
queued := startedUpload(t, s)
|
||||
running, err := s.CreateUpload(ctx, userA, "r", false)
|
||||
running, err := s.CreateUpload(ctx, userA, "r", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, s.StartTranscribing(ctx, userA, running))
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@ import (
|
||||
func startedUpload(t *testing.T, s *store.Store) string {
|
||||
t.Helper()
|
||||
seedUsers(t, userA, userB)
|
||||
id, err := s.CreateUpload(context.Background(), userA, "m", false)
|
||||
id, err := s.CreateUpload(context.Background(), userA, "m", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
return id
|
||||
}
|
||||
@@ -68,13 +68,13 @@ func TestUploadProgress_AHeartbeatMovesAliveAtAndNotProgressAt(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
at := time.Date(2026, 10, 2, 8, 30, 0, 0, time.UTC)
|
||||
require.NoError(t, s.SaveProgress(ctx, userA, id, []domain.ProgressStep{{Key: "audio", Label: "a", State: domain.StepRunning}}, at))
|
||||
before := time.Now().Add(-time.Second)
|
||||
beatAt := at.Add(90 * time.Second) // deliberately not the database's NOW()
|
||||
|
||||
require.NoError(t, s.Heartbeat(ctx, userA, id))
|
||||
require.NoError(t, s.Heartbeat(ctx, userA, id, beatAt))
|
||||
|
||||
got, err := s.GetUpload(ctx, userA, id)
|
||||
require.NoError(t, err)
|
||||
assert.True(t, got.AliveAt.After(before), "the heartbeat must stamp the row with now")
|
||||
assert.True(t, got.AliveAt.Equal(beatAt), "the heartbeat is stamped with the application's clock, the same one progress_at uses, not the database's NOW() (#51)")
|
||||
assert.True(t, got.ProgressAt.Equal(at), "alive is not progress: a heartbeat must not move progress_at")
|
||||
}
|
||||
|
||||
@@ -99,7 +99,7 @@ func TestUploadProgress_ByAnotherUserChangesNothing(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
|
||||
errSave := s.SaveProgress(ctx, userB, id, []domain.ProgressStep{{Key: "x", Label: "X", State: domain.StepDone}}, time.Now())
|
||||
errBeat := s.Heartbeat(ctx, userB, id)
|
||||
errBeat := s.Heartbeat(ctx, userB, id, time.Now())
|
||||
|
||||
assert.ErrorIs(t, errSave, store.ErrUploadNotFound)
|
||||
assert.ErrorIs(t, errBeat, store.ErrUploadNotFound)
|
||||
|
||||
@@ -33,7 +33,7 @@ func TestCreateUpload_StartsQueuedAndIsOwnedByTheCreator(t *testing.T) {
|
||||
seedUsers(t, userA)
|
||||
ctx := context.Background()
|
||||
|
||||
id, err := s.CreateUpload(ctx, userA, "Teams export", false)
|
||||
id, err := s.CreateUpload(ctx, userA, "Teams export", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
|
||||
got, err := s.GetUpload(ctx, userA, id)
|
||||
@@ -50,7 +50,7 @@ func TestUploadLifecycle_CompletedKeepsTextSegmentsEndpointAndDuration(t *testin
|
||||
s := newStore(t)
|
||||
seedUsers(t, userA)
|
||||
ctx := context.Background()
|
||||
id, err := s.CreateUpload(ctx, userA, "m", false)
|
||||
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
|
||||
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
||||
@@ -84,7 +84,7 @@ func TestUploadLifecycle_FailedRecordsTypedKindAndNoContent(t *testing.T) {
|
||||
s := newStore(t)
|
||||
seedUsers(t, userA)
|
||||
ctx := context.Background()
|
||||
id, err := s.CreateUpload(ctx, userA, "m", false)
|
||||
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
||||
|
||||
@@ -102,7 +102,7 @@ func TestGetUpload_AnotherUsersIDIsNotFound(t *testing.T) {
|
||||
s := newStore(t)
|
||||
seedUsers(t, userA, userB)
|
||||
ctx := context.Background()
|
||||
id, err := s.CreateUpload(ctx, userA, "A's meeting", false)
|
||||
id, err := s.CreateUpload(ctx, userA, "A's meeting", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = s.GetUpload(ctx, userB, id)
|
||||
@@ -115,7 +115,7 @@ func TestUploadWrites_ByAnotherUserChangeNothing(t *testing.T) {
|
||||
s := newStore(t)
|
||||
seedUsers(t, userA, userB)
|
||||
ctx := context.Background()
|
||||
id, err := s.CreateUpload(ctx, userA, "A's meeting", false)
|
||||
id, err := s.CreateUpload(ctx, userA, "A's meeting", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.ErrorIs(t, s.StartTranscribing(ctx, userB, id), store.ErrUploadNotFound)
|
||||
@@ -132,11 +132,11 @@ func TestListUploads_ReturnsOnlyTheCallersNewestFirstWithoutTranscriptText(t *te
|
||||
s := newStore(t)
|
||||
seedUsers(t, userA, userB)
|
||||
ctx := context.Background()
|
||||
first, err := s.CreateUpload(ctx, userA, "first", false)
|
||||
first, err := s.CreateUpload(ctx, userA, "first", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
second, err := s.CreateUpload(ctx, userA, "second", false)
|
||||
second, err := s.CreateUpload(ctx, userA, "second", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
_, err = s.CreateUpload(ctx, userB, "B's", false)
|
||||
_, err = s.CreateUpload(ctx, userB, "B's", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, s.StartTranscribing(ctx, userA, first))
|
||||
require.NoError(t, s.CompleteUpload(ctx, userA, first, domain.Transcription{Text: "long text body", Endpoint: "e"}))
|
||||
@@ -160,12 +160,12 @@ func TestFailInterruptedUploads_FailsOnlyUnfinishedRowsForEveryUser(t *testing.T
|
||||
_, err := super.Exec(ctx, `INSERT INTO user_identities (dex_subject, user_id) VALUES ($1, $2)`, subject, uid)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
queued, err := s.CreateUpload(ctx, userA, "queued", false)
|
||||
queued, err := s.CreateUpload(ctx, userA, "queued", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
running, err := s.CreateUpload(ctx, userB, "running", false)
|
||||
running, err := s.CreateUpload(ctx, userB, "running", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, s.StartTranscribing(ctx, userB, running))
|
||||
done, err := s.CreateUpload(ctx, userA, "done", false)
|
||||
done, err := s.CreateUpload(ctx, userA, "done", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, s.StartTranscribing(ctx, userA, done))
|
||||
require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "kept", Endpoint: "e"}))
|
||||
@@ -191,9 +191,9 @@ func TestCreateUpload_RecordsWhetherTheFileHadAVideoTrack(t *testing.T) {
|
||||
s := newStore(t)
|
||||
seedUsers(t, userA)
|
||||
ctx := context.Background()
|
||||
withVideo, err := s.CreateUpload(ctx, userA, "camera off", true)
|
||||
withVideo, err := s.CreateUpload(ctx, userA, "camera off", domain.LangSwedish, true)
|
||||
require.NoError(t, err)
|
||||
audioOnly, err := s.CreateUpload(ctx, userA, "voice only", false)
|
||||
audioOnly, err := s.CreateUpload(ctx, userA, "voice only", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
|
||||
gotVideo, err := s.GetUpload(ctx, userA, withVideo)
|
||||
@@ -216,9 +216,9 @@ func TestUploadHadVideo_SurvivesCompletionAndFailure(t *testing.T) {
|
||||
s := newStore(t)
|
||||
seedUsers(t, userA)
|
||||
ctx := context.Background()
|
||||
done, err := s.CreateUpload(ctx, userA, "d", true)
|
||||
done, err := s.CreateUpload(ctx, userA, "d", domain.LangSwedish, true)
|
||||
require.NoError(t, err)
|
||||
failed, err := s.CreateUpload(ctx, userA, "f", true)
|
||||
failed, err := s.CreateUpload(ctx, userA, "f", domain.LangSwedish, true)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, s.StartTranscribing(ctx, userA, done))
|
||||
require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "x", Endpoint: "e"}))
|
||||
@@ -235,7 +235,7 @@ func TestUploadLifecycle_ReconciliationRoundTripsWithItsContestedSpots(t *testin
|
||||
s := newStore(t)
|
||||
seedUsers(t, userA)
|
||||
ctx := context.Background()
|
||||
id, err := s.CreateUpload(ctx, userA, "m", false)
|
||||
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
||||
want := &domain.Reconciliation{
|
||||
@@ -258,7 +258,7 @@ func TestUploadLifecycle_ASingleEngineTranscriptHasNoReconciliation(t *testing.T
|
||||
s := newStore(t)
|
||||
seedUsers(t, userA)
|
||||
ctx := context.Background()
|
||||
id, err := s.CreateUpload(ctx, userA, "m", false)
|
||||
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
||||
require.NoError(t, s.CompleteUpload(ctx, userA, id, domain.Transcription{Text: "x", Endpoint: "e"}))
|
||||
@@ -273,7 +273,7 @@ func TestUploadLifecycle_ASkippedReconciliationKeepsItsReason(t *testing.T) {
|
||||
s := newStore(t)
|
||||
seedUsers(t, userA)
|
||||
ctx := context.Background()
|
||||
id, err := s.CreateUpload(ctx, userA, "m", false)
|
||||
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
||||
rec := &domain.Reconciliation{Status: domain.ReconcileSkipped, Reason: "the model failed 3 times in a row: gateway down", Errors: 3}
|
||||
@@ -284,3 +284,55 @@ func TestUploadLifecycle_ASkippedReconciliationKeepsItsReason(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, rec, got.Reconciliation)
|
||||
}
|
||||
|
||||
func TestCreateUpload_RecordsTheLanguageTheUserChoseAndTheListCarriesIt(t *testing.T) {
|
||||
s := newStore(t)
|
||||
seedUsers(t, userA)
|
||||
ctx := context.Background()
|
||||
swedish, err := s.CreateUpload(ctx, userA, "sv", domain.LangSwedish, false)
|
||||
require.NoError(t, err)
|
||||
auto, err := s.CreateUpload(ctx, userA, "auto", domain.LangAuto, false)
|
||||
require.NoError(t, err)
|
||||
|
||||
gotSv, err := s.GetUpload(ctx, userA, swedish)
|
||||
require.NoError(t, err)
|
||||
gotAuto, err := s.GetUpload(ctx, userA, auto)
|
||||
require.NoError(t, err)
|
||||
listed, err := s.ListUploads(ctx, userA)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, domain.LangSwedish, gotSv.Language)
|
||||
assert.Equal(t, domain.LangAuto, gotAuto.Language)
|
||||
byID := map[string]domain.Language{}
|
||||
for _, u := range listed {
|
||||
byID[u.ID] = u.Language
|
||||
}
|
||||
assert.Equal(t, map[string]domain.Language{swedish: domain.LangSwedish, auto: domain.LangAuto}, byID)
|
||||
}
|
||||
|
||||
// The closed set is also held by the table, so a bug or a second writer cannot store a
|
||||
// value the engines are never meant to receive (tapir#61).
|
||||
func TestUploadedTranscripts_TheTableRefusesALanguageOutsideTheClosedSet(t *testing.T) {
|
||||
newStore(t)
|
||||
seedUsers(t, userA)
|
||||
|
||||
_, err := rawPool(t).Exec(context.Background(),
|
||||
`INSERT INTO uploaded_transcripts (user_id, title, language) VALUES ($1, 'x', 'en')`, userA)
|
||||
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "language", "the CHECK constraint names the column")
|
||||
}
|
||||
|
||||
// A row from before the column existed was transcribed as Swedish; the default says so.
|
||||
func TestUploadedTranscripts_ARowInsertedWithoutALanguageIsSwedish(t *testing.T) {
|
||||
s := newStore(t)
|
||||
seedUsers(t, userA)
|
||||
var id string
|
||||
require.NoError(t, rawPool(t).QueryRow(context.Background(),
|
||||
`INSERT INTO uploaded_transcripts (user_id, title) VALUES ($1, 'old') RETURNING id`, userA).Scan(&id))
|
||||
|
||||
got, err := s.GetUpload(context.Background(), userA, id)
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, domain.LangSwedish, got.Language)
|
||||
}
|
||||
|
||||
@@ -32,15 +32,25 @@ func NewEnsemble(base, alt, fallback Backend, adj reconcile.Adjudicator, opts re
|
||||
return &Ensemble{base: base, alt: alt, fallback: fallback, adj: adj, opts: opts}
|
||||
}
|
||||
|
||||
// Plan implements uploads.Planner: the phases this ensemble reports, so the page can
|
||||
// show them before they run. The fallback is a phase only when one is configured.
|
||||
func (e *Ensemble) Plan() []progress.Planned {
|
||||
plan := []progress.Planned{
|
||||
// Plan implements uploads.Planner: the phases of a Swedish job, the two-engine one. The
|
||||
// fallback is a phase only when one is configured.
|
||||
func (e *Ensemble) Plan() []progress.Planned { return e.PlanFor(domain.LangSwedish) }
|
||||
|
||||
// PlanFor implements uploads.LanguagePlanner. A recording that is not Swedish never runs the
|
||||
// base engine, the comparison or the decisions (transcribeOther), so the page lists only
|
||||
// what can run: the other engine, then the fallback if one is configured.
|
||||
func (e *Ensemble) PlanFor(lang domain.Language) []progress.Planned {
|
||||
var plan []progress.Planned
|
||||
if lang == domain.LangSwedish || lang == "" {
|
||||
plan = []progress.Planned{
|
||||
{Key: "engine_a", Label: "Transcribing with " + e.base.Name()},
|
||||
{Key: "engine_b", Label: "Transcribing with " + e.alt.Name()},
|
||||
{Key: "compare", Label: "Comparing the two transcripts"},
|
||||
{Key: "decide", Label: "Deciding the differences"},
|
||||
}
|
||||
} else {
|
||||
plan = []progress.Planned{{Key: "engine_b", Label: "Transcribing with " + e.alt.Name()}}
|
||||
}
|
||||
if e.fallback != nil {
|
||||
plan = append(plan, progress.Planned{Key: "fallback", Label: "Cloud fallback (" + e.fallback.Name() + ")"})
|
||||
}
|
||||
@@ -51,6 +61,9 @@ func (e *Ensemble) Plan() []progress.Planned {
|
||||
// progress.Reporter in ctx (a no-op when nobody listens).
|
||||
func (e *Ensemble) Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error) {
|
||||
rep := progress.From(ctx)
|
||||
if domain.LanguageFrom(ctx) != domain.LangSwedish {
|
||||
return e.transcribeOther(ctx, rep, wavPath)
|
||||
}
|
||||
b, berr := e.run(ctx, rep, "engine_a", e.base, wavPath)
|
||||
if err := ctx.Err(); err != nil {
|
||||
return domain.Transcription{}, err
|
||||
@@ -89,6 +102,28 @@ func (e *Ensemble) Transcribe(ctx context.Context, wavPath string) (domain.Trans
|
||||
return e.single(f, e.fallback.Name(), "both local engines failed"), nil
|
||||
}
|
||||
|
||||
// transcribeOther handles a recording that is not Swedish (tapir#61). The base engine is
|
||||
// Swedish-only, so it never receives the audio, and with one reading there is nothing to
|
||||
// reconcile: the other engine transcribes alone, then the cloud fallback if it fails and the
|
||||
// job is still live. The phases that did not run are shown as skipped by the tracker.
|
||||
func (e *Ensemble) transcribeOther(ctx context.Context, rep progress.Reporter, wavPath string) (domain.Transcription, error) {
|
||||
t, err := e.run(ctx, rep, "engine_b", e.alt, wavPath)
|
||||
if err == nil {
|
||||
t.Endpoint = e.alt.Name()
|
||||
return t, nil
|
||||
}
|
||||
if e.fallback == nil || ctx.Err() != nil {
|
||||
return domain.Transcription{}, fmt.Errorf("%s: %w", e.alt.Name(), err)
|
||||
}
|
||||
f, ferr := e.run(ctx, rep, "fallback", e.fallback, wavPath)
|
||||
if ferr != nil {
|
||||
return domain.Transcription{}, errors.Join(
|
||||
fmt.Errorf("%s: %w", e.alt.Name(), err), fmt.Errorf("%s: %w", e.fallback.Name(), ferr))
|
||||
}
|
||||
f.Endpoint = e.fallback.Name()
|
||||
return f, nil
|
||||
}
|
||||
|
||||
// run is one engine call, reported as a phase. A job that was stopped is not a broken
|
||||
// engine, so a cancelled context is never reported as a failed phase.
|
||||
func (e *Ensemble) run(ctx context.Context, rep progress.Reporter, key string, be Backend, wav string) (domain.Transcription, error) {
|
||||
|
||||
@@ -160,3 +160,25 @@ func TestEnsembleProgress_TheReconcileCountsAreNotedForTheOperator(t *testing.T)
|
||||
assert.Equal(t, 0, n["unsure"])
|
||||
assert.Equal(t, 0, n["errors"])
|
||||
}
|
||||
|
||||
// An auto-language upload never runs the base engine, the comparison or the decisions, so the
|
||||
// page must not list them as coming (tapir#61, PR #62 review).
|
||||
func TestEnsemblePlanFor_AnAutoLanguageListsOnlyThePhasesThatCanRun(t *testing.T) {
|
||||
with := newEnsemble("a", "a", nil, nil, true, &verdictAdj{}).e
|
||||
without := newEnsemble("a", "a", nil, nil, false, &verdictAdj{}).e
|
||||
keys := func(p []progress.Planned) []string {
|
||||
var k []string
|
||||
for _, s := range p {
|
||||
k = append(k, s.Key)
|
||||
}
|
||||
return k
|
||||
}
|
||||
|
||||
assert.Equal(t, []string{"engine_b", "fallback"}, keys(with.PlanFor(domain.LangAuto)))
|
||||
assert.Equal(t, []string{"engine_b"}, keys(without.PlanFor(domain.LangAuto)), "no fallback configured: none promised")
|
||||
assert.Equal(t, with.Plan(), with.PlanFor(domain.LangSwedish), "Swedish is the plan it always was")
|
||||
assert.Contains(t, with.PlanFor(domain.LangAuto)[0].Label, "iguana/kb-whisper")
|
||||
assert.Equal(t, with.Plan(), with.PlanFor(""), "a job queued before the choice existed is Swedish")
|
||||
assert.Equal(t, with.PlanFor(domain.LangAuto), with.PlanFor(domain.Language("xx")),
|
||||
"an unknown language is planned as it is run: not the Swedish phases")
|
||||
}
|
||||
|
||||
@@ -204,3 +204,52 @@ func TestEnsemble_AShutdownDuringTheSecondEngineIsAnErrorNotASingleEngineSuccess
|
||||
assert.ErrorIs(t, err, context.Canceled, "a cancelled job must not be recorded as 'kb-whisper failed'")
|
||||
assert.Empty(t, got.Text)
|
||||
}
|
||||
|
||||
// The base engine is Swedish-only. A recording that is not Swedish must never reach it, and
|
||||
// there is no second opinion to reconcile: the other engine transcribes alone (tapir#61).
|
||||
|
||||
func autoCtx() context.Context { return domain.WithLanguage(context.Background(), domain.LangAuto) }
|
||||
|
||||
func TestEnsemble_AnAutoLanguageUploadNeverReachesTheBaseEngineAndIsNotReconciled(t *testing.T) {
|
||||
adj := &verdictAdj{v: reconcile.VerdictAlt}
|
||||
r := newEnsemble("base text", "alt text", nil, nil, true, adj)
|
||||
|
||||
got, err := r.e.Transcribe(autoCtx(), "/a.wav")
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{"iguana/kb-whisper"}, *r.calls, "only the other engine is called")
|
||||
assert.Zero(t, adj.calls, "nothing is adjudicated")
|
||||
assert.Equal(t, "alt text", got.Text)
|
||||
assert.Equal(t, "iguana/kb-whisper", got.Endpoint)
|
||||
assert.Nil(t, got.Reconciliation, "reconciliation does not apply, so none is recorded")
|
||||
}
|
||||
|
||||
func TestEnsemble_AnAutoLanguageUploadFallsBackWhenItsEngineFailsAndStillSkipsTheBaseEngine(t *testing.T) {
|
||||
r := newEnsemble("base text", "alt text", nil, errors.New("engine down"), true, &verdictAdj{})
|
||||
|
||||
got, err := r.e.Transcribe(autoCtx(), "/a.wav")
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{"iguana/kb-whisper", "berget/kb-whisper"}, *r.calls)
|
||||
assert.Equal(t, "berget/kb-whisper", got.Endpoint)
|
||||
}
|
||||
|
||||
func TestEnsemble_AnAutoLanguageUploadWithNoFallbackFailsWithoutTouchingTheBaseEngine(t *testing.T) {
|
||||
r := newEnsemble("base text", "alt text", nil, errors.New("engine down"), false, &verdictAdj{})
|
||||
|
||||
_, err := r.e.Transcribe(autoCtx(), "/a.wav")
|
||||
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, []string{"iguana/kb-whisper"}, *r.calls)
|
||||
}
|
||||
|
||||
func TestEnsemble_ACancelledAutoLanguageJobDoesNotReachTheFallback(t *testing.T) {
|
||||
r := newEnsemble("base text", "alt text", nil, errors.New("engine down"), true, &verdictAdj{})
|
||||
ctx, cancel := context.WithCancel(autoCtx())
|
||||
cancel()
|
||||
|
||||
_, err := r.e.Transcribe(ctx, "/a.wav")
|
||||
|
||||
require.Error(t, err)
|
||||
assert.NotContains(t, *r.calls, "berget/kb-whisper", "a stopped job must not ship its audio to a cloud endpoint")
|
||||
}
|
||||
|
||||
@@ -71,9 +71,20 @@ func (g *Gateway) Transcribe(ctx context.Context, path string) (domain.Transcrip
|
||||
return domain.Transcription{}, fmt.Errorf("transcribe: stat audio: %w", err)
|
||||
}
|
||||
|
||||
// The language is one request field and a closed set: sv is sent, auto sends nothing so
|
||||
// the engine detects it, and anything else never leaves this process (tapir#61).
|
||||
fields := [][2]string{{"model", g.model}, {"response_format", "verbose_json"}}
|
||||
switch lang := domain.LanguageFrom(ctx); lang {
|
||||
case domain.LangSwedish:
|
||||
fields = append(fields, [2]string{"language", string(lang)})
|
||||
case domain.LangAuto:
|
||||
default:
|
||||
return domain.Transcription{}, fmt.Errorf("transcribe: language %q is not one tapir sends", lang)
|
||||
}
|
||||
|
||||
var prefix bytes.Buffer
|
||||
mw := multipart.NewWriter(&prefix)
|
||||
for _, kv := range [][2]string{{"model", g.model}, {"response_format", "verbose_json"}, {"language", "sv"}} {
|
||||
for _, kv := range fields {
|
||||
if err := mw.WriteField(kv[0], kv[1]); err != nil {
|
||||
return domain.Transcription{}, fmt.Errorf("transcribe: build request: %w", err)
|
||||
}
|
||||
|
||||
@@ -255,3 +255,48 @@ func TestGateway_AnErrorBodyIsBoundedToTwoHundredBytesInTheStatusError(t *testin
|
||||
require.ErrorAs(t, err, &se)
|
||||
assert.LessOrEqual(t, len(se.Snippet), 200, "a gateway's whole body must not become part of an error")
|
||||
}
|
||||
|
||||
// The language travels on the context (tapir#61); the request field is the only place it
|
||||
// takes effect, so these observe the request the gateway fake received.
|
||||
|
||||
func TestGateway_SendsLanguageSvWhenTheContextCarriesNoneAndForSwedish(t *testing.T) {
|
||||
for name, ctx := range map[string]context.Context{
|
||||
"no language in the context": context.Background(),
|
||||
"swedish": domain.WithLanguage(context.Background(), domain.LangSwedish),
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
var got seen
|
||||
srv := serve(t, 200, verboseJSON, &got)
|
||||
g := transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client())
|
||||
|
||||
_, err := g.Transcribe(ctx, wavFile(t, "RIFF-synthetic-pcm"))
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "sv", got.fields["language"])
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGateway_OmitsTheLanguageFieldForAnAutoContextSoTheEngineDetectsIt(t *testing.T) {
|
||||
var got seen
|
||||
srv := serve(t, 200, verboseJSON, &got)
|
||||
g := transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client())
|
||||
|
||||
_, err := g.Transcribe(domain.WithLanguage(context.Background(), domain.LangAuto), wavFile(t, "RIFF-synthetic-pcm"))
|
||||
|
||||
require.NoError(t, err)
|
||||
_, present := got.fields["language"]
|
||||
assert.False(t, present, "an auto upload must not be forced to Swedish")
|
||||
assert.Equal(t, "iguana/kb-whisper", got.fields["model"], "the rest of the request is unchanged")
|
||||
}
|
||||
|
||||
func TestGateway_ALanguageOutsideTheClosedSetIsRefusedBeforeAnythingIsSent(t *testing.T) {
|
||||
var got seen
|
||||
srv := serve(t, 200, verboseJSON, &got)
|
||||
g := transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client())
|
||||
|
||||
_, err := g.Transcribe(domain.WithLanguage(context.Background(), domain.Language("en")), wavFile(t, "RIFF-synthetic-pcm"))
|
||||
|
||||
require.Error(t, err)
|
||||
assert.Empty(t, got.method, "no request may reach the engine with a language the closed set does not hold")
|
||||
}
|
||||
|
||||
@@ -102,3 +102,45 @@ func TestEnsemble_AHungEngineIsASoftFailureAndTheOtherEnginesTextIsUsed(t *testi
|
||||
assert.Contains(t, got.Reconciliation.Reason, "timeout")
|
||||
assert.False(t, errors.Is(err, context.DeadlineExceeded))
|
||||
}
|
||||
|
||||
// WithTimeout sits in front of both local engines in production (cmd/tapir/uploads.go). It
|
||||
// must hand the engine the job's context, not a fresh one: the language and the
|
||||
// cancellation both travel on it (tapir#61).
|
||||
|
||||
func TestWithTimeout_TheEngineStillReceivesTheLanguageOfTheJob(t *testing.T) {
|
||||
for name, tt := range map[string]struct {
|
||||
ctx context.Context
|
||||
wantField bool
|
||||
}{
|
||||
"auto leaves the field out": {domain.WithLanguage(context.Background(), domain.LangAuto), false},
|
||||
"swedish sends sv": {domain.WithLanguage(context.Background(), domain.LangSwedish), true},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
var got seen
|
||||
srv := serve(t, 200, verboseJSON, &got)
|
||||
b := transcribe.WithTimeout(transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client()), time.Minute)
|
||||
|
||||
_, err := b.Transcribe(tt.ctx, wavFile(t, "RIFF-synthetic-pcm"))
|
||||
|
||||
require.NoError(t, err)
|
||||
lang, present := got.fields["language"]
|
||||
assert.Equal(t, tt.wantField, present)
|
||||
if tt.wantField {
|
||||
assert.Equal(t, "sv", lang)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestWithTimeout_ACancelledJobIsNotRestartedByTheWrapper(t *testing.T) {
|
||||
var got seen
|
||||
srv := serve(t, 200, verboseJSON, &got)
|
||||
b := transcribe.WithTimeout(transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client()), time.Minute)
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
|
||||
_, err := b.Transcribe(ctx, wavFile(t, "RIFF-synthetic-pcm"))
|
||||
|
||||
require.Error(t, err)
|
||||
assert.Empty(t, got.method, "a cancelled job must not send its audio anywhere through the wrapper")
|
||||
}
|
||||
|
||||
@@ -246,6 +246,9 @@ type UploadedTranscript struct {
|
||||
ID string
|
||||
UserID string
|
||||
Title string
|
||||
// Language is what the upload was (or will be) transcribed as: sv, or auto for
|
||||
// "Other" (tapir#61).
|
||||
Language Language
|
||||
Status UploadStatus
|
||||
Content string
|
||||
Segments []Segment
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
package domain
|
||||
|
||||
import "context"
|
||||
|
||||
// Language is the spoken language an upload is transcribed as. It is chosen by the user on
|
||||
// the upload form and reaches the engines as one request field, so it is a closed set,
|
||||
// never free text (tapir#61).
|
||||
type Language string
|
||||
|
||||
const (
|
||||
// LangSwedish sends language=sv: the Swedish engines' own language, and what every
|
||||
// upload was transcribed as before the choice existed.
|
||||
LangSwedish Language = "sv"
|
||||
// LangAuto sends no language, so the engine detects it. It is what "Other" on the form
|
||||
// means: the Swedish-only engines are not used for it.
|
||||
LangAuto Language = "auto"
|
||||
)
|
||||
|
||||
// ParseLanguage reads a form value. An empty value is Swedish (a client that predates the
|
||||
// field); anything that is not exactly a known value is refused rather than normalised.
|
||||
func ParseLanguage(s string) (Language, bool) {
|
||||
switch Language(s) {
|
||||
case "", LangSwedish:
|
||||
return LangSwedish, true
|
||||
case LangAuto:
|
||||
return LangAuto, true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
type languageKey struct{}
|
||||
|
||||
// WithLanguage returns a context that carries the language of the job it serves. The
|
||||
// Transcriber interface does not change: engines read it with LanguageFrom, the same way
|
||||
// they read the progress Reporter (ADR-035).
|
||||
func WithLanguage(ctx context.Context, l Language) context.Context {
|
||||
return context.WithValue(ctx, languageKey{}, l)
|
||||
}
|
||||
|
||||
// LanguageFrom returns the language carried by ctx, or Swedish when it carries none, so a
|
||||
// caller that sets nothing behaves as it always did.
|
||||
func LanguageFrom(ctx context.Context) Language {
|
||||
if l, ok := ctx.Value(languageKey{}).(Language); ok && l != "" {
|
||||
return l
|
||||
}
|
||||
return LangSwedish
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
package domain_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
|
||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||
)
|
||||
|
||||
func TestParseLanguage_KnownValuesAndEmptyAreAcceptedEverythingElseIsNot(t *testing.T) {
|
||||
tests := []struct {
|
||||
in string
|
||||
want domain.Language
|
||||
wantOK bool
|
||||
}{
|
||||
{"sv", domain.LangSwedish, true},
|
||||
{"auto", domain.LangAuto, true},
|
||||
{"", domain.LangSwedish, true}, // a client that sends no field keeps today's behaviour
|
||||
{"SV", "", false}, // exact match: the value reaches a request field
|
||||
{" sv", "", false},
|
||||
{"sv\n", "", false},
|
||||
{"en", "", false}, // a specific other language is not offered yet
|
||||
{"sv,en", "", false},
|
||||
{"../sv", "", false},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.in, func(t *testing.T) {
|
||||
got, ok := domain.ParseLanguage(tt.in)
|
||||
assert.Equal(t, tt.wantOK, ok)
|
||||
assert.Equal(t, tt.want, got)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLanguageFrom_IsSwedishWhenTheContextCarriesNoneAndOtherwiseWhatWasSet(t *testing.T) {
|
||||
assert.Equal(t, domain.LangSwedish, domain.LanguageFrom(context.Background()))
|
||||
ctx := domain.WithLanguage(context.Background(), domain.LangAuto)
|
||||
assert.Equal(t, domain.LangAuto, domain.LanguageFrom(ctx))
|
||||
}
|
||||
@@ -125,6 +125,11 @@ type statusWriter struct {
|
||||
wroteHeader bool
|
||||
}
|
||||
|
||||
// Unwrap exposes the underlying writer to http.ResponseController, so a handler behind this
|
||||
// middleware can still set a connection read deadline (#56: without it the upload idle bound
|
||||
// was silently unavailable in production).
|
||||
func (s *statusWriter) Unwrap() http.ResponseWriter { return s.ResponseWriter }
|
||||
|
||||
func (s *statusWriter) WriteHeader(code int) {
|
||||
if !s.wroteHeader {
|
||||
s.code = code
|
||||
|
||||
@@ -37,11 +37,12 @@ type Reporter interface {
|
||||
// is for people, so a failed write is logged and never stops the job.
|
||||
type Sink interface {
|
||||
SaveProgress(ctx context.Context, steps []domain.ProgressStep, at time.Time) error
|
||||
Beat(ctx context.Context) error
|
||||
Beat(ctx context.Context, at time.Time) error
|
||||
}
|
||||
|
||||
const (
|
||||
defaultMinSave = 2 * time.Second
|
||||
defaultSaveTimeout = 5 * time.Second
|
||||
logAdvanceEach = time.Minute
|
||||
)
|
||||
|
||||
@@ -55,6 +56,14 @@ func WithClock(now func() time.Time) Option { return func(t *Tracker) { t.now =
|
||||
// Done, Fail and Finish always save.
|
||||
func WithMinSave(d time.Duration) Option { return func(t *Tracker) { t.minSave = d } }
|
||||
|
||||
// WithSaveTimeout bounds one save or heartbeat write. Progress is cosmetic, so a database
|
||||
// that stops answering costs a dropped write rather than an unbounded stall. It is still a
|
||||
// delay: saves run under the tracker's lock, which Begin and Advance (called by the engines)
|
||||
// need, so each transition can wait up to this long, and a hung database slows a job by
|
||||
// that much per call, out of the job's own time budget (#51). A dropped final save is not
|
||||
// retried; the row keeps its last saved step.
|
||||
func WithSaveTimeout(d time.Duration) Option { return func(t *Tracker) { t.saveTimeout = d } }
|
||||
|
||||
// Tracker implements Reporter for one job.
|
||||
type Tracker struct {
|
||||
ctx context.Context
|
||||
@@ -63,6 +72,8 @@ type Tracker struct {
|
||||
now func() time.Time
|
||||
minSave time.Duration
|
||||
|
||||
saveTimeout time.Duration
|
||||
|
||||
mu sync.Mutex
|
||||
steps []domain.ProgressStep
|
||||
lastAdvanceSave time.Time
|
||||
@@ -72,7 +83,7 @@ type Tracker struct {
|
||||
// NewTracker plans a job's phases. ctx is used for the saves, so it should outlive the
|
||||
// job's own timeout: a job that timed out still gets its failure recorded.
|
||||
func NewTracker(ctx context.Context, sink Sink, log *slog.Logger, plan []Planned, opts ...Option) *Tracker {
|
||||
t := &Tracker{ctx: ctx, sink: sink, log: log, now: time.Now, minSave: defaultMinSave}
|
||||
t := &Tracker{ctx: ctx, sink: sink, log: log, now: time.Now, minSave: defaultMinSave, saveTimeout: defaultSaveTimeout}
|
||||
for _, o := range opts {
|
||||
o(t)
|
||||
}
|
||||
@@ -212,7 +223,10 @@ func (t *Tracker) Heartbeat(ctx context.Context, beatEvery, stillEvery time.Dura
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-beat.C:
|
||||
if err := t.sink.Beat(ctx); err != nil && ctx.Err() == nil {
|
||||
beatCtx, cancel := context.WithTimeout(ctx, t.saveTimeout)
|
||||
err := t.sink.Beat(beatCtx, t.now())
|
||||
cancel()
|
||||
if err != nil && ctx.Err() == nil {
|
||||
t.log.Warn("progress: heartbeat not saved", "err", err)
|
||||
}
|
||||
case <-still.C:
|
||||
@@ -254,7 +268,9 @@ func (t *Tracker) find(key string) *domain.ProgressStep {
|
||||
// dropped: progress is cosmetic and must never fail the job it describes.
|
||||
func (t *Tracker) save() {
|
||||
snapshot := append([]domain.ProgressStep(nil), t.steps...)
|
||||
if err := t.sink.SaveProgress(t.ctx, snapshot, t.now()); err != nil && t.ctx.Err() == nil {
|
||||
ctx, cancel := context.WithTimeout(t.ctx, t.saveTimeout)
|
||||
defer cancel()
|
||||
if err := t.sink.SaveProgress(ctx, snapshot, t.now()); err != nil && t.ctx.Err() == nil {
|
||||
t.log.Warn("progress: not saved", "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
package progress_test
|
||||
|
||||
// Observes what a database that stops answering does to the job it reports on (#51):
|
||||
// progress is cosmetic, so a save that never returns must not stall the engine calls that
|
||||
// report through the tracker, nor the heartbeat that says the process is alive.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||
"git.d-ma.be/mathias/tapir/internal/progress"
|
||||
)
|
||||
|
||||
// stallSink never answers until its context ends, like a database connection that hung.
|
||||
type stallSink struct{ beats atomic.Int32 }
|
||||
|
||||
func (s *stallSink) SaveProgress(ctx context.Context, _ []domain.ProgressStep, _ time.Time) error {
|
||||
<-ctx.Done()
|
||||
return ctx.Err()
|
||||
}
|
||||
|
||||
func (s *stallSink) Beat(ctx context.Context, _ time.Time) error {
|
||||
s.beats.Add(1)
|
||||
<-ctx.Done()
|
||||
return ctx.Err()
|
||||
}
|
||||
|
||||
type lockedBuf struct {
|
||||
mu sync.Mutex
|
||||
b bytes.Buffer
|
||||
}
|
||||
|
||||
func (l *lockedBuf) Write(p []byte) (int, error) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
return l.b.Write(p)
|
||||
}
|
||||
|
||||
func (l *lockedBuf) String() string {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
return l.b.String()
|
||||
}
|
||||
|
||||
func stallTracker(sink progress.Sink, buf *lockedBuf) *progress.Tracker {
|
||||
log := slog.New(slog.NewTextHandler(buf, nil))
|
||||
return progress.NewTracker(context.Background(), sink, log,
|
||||
[]progress.Planned{{Key: "a", Label: "A"}}, progress.WithSaveTimeout(50*time.Millisecond))
|
||||
}
|
||||
|
||||
func TestTracker_AHungSaveDoesNotBlockTheReporterPastItsTimeout(t *testing.T) {
|
||||
buf := &lockedBuf{}
|
||||
tr := stallTracker(&stallSink{}, buf)
|
||||
|
||||
done := make(chan struct{})
|
||||
go func() { tr.Begin("a", 0); tr.Advance("a", 1); tr.Done("a"); close(done) }()
|
||||
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("the reporter was blocked by a save that never returned")
|
||||
}
|
||||
require.Contains(t, buf.String(), "progress: not saved", "a dropped save is logged")
|
||||
}
|
||||
|
||||
func TestTracker_AHungBeatDoesNotStopTheHeartbeat(t *testing.T) {
|
||||
sink := &stallSink{}
|
||||
tr := stallTracker(sink, &lockedBuf{})
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
go tr.Heartbeat(ctx, 5*time.Millisecond, time.Hour)
|
||||
|
||||
require.Eventually(t, func() bool { return sink.beats.Load() >= 3 }, 3*time.Second, 5*time.Millisecond,
|
||||
"each beat is abandoned at its timeout and the next one is tried")
|
||||
}
|
||||
|
||||
func TestTracker_TheStillRunningLineIsNotHeldUpBehindAHungSave(t *testing.T) {
|
||||
buf := &lockedBuf{}
|
||||
tr := stallTracker(&stallSink{}, buf)
|
||||
go tr.Begin("a", 0) // holds the tracker's lock inside a save that hangs until its timeout
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
go tr.Heartbeat(ctx, time.Hour, 5*time.Millisecond)
|
||||
|
||||
require.Eventually(t, func() bool { return strings.Contains(buf.String(), "still running") }, 3*time.Second, 5*time.Millisecond)
|
||||
}
|
||||
|
||||
// deadlineSink records how long each write was given, and answers at once.
|
||||
type deadlineSink struct {
|
||||
mu sync.Mutex
|
||||
saves []time.Duration
|
||||
beats []time.Duration
|
||||
noDLOK bool
|
||||
}
|
||||
|
||||
func (d *deadlineSink) record(ctx context.Context, into *[]time.Duration) {
|
||||
d.mu.Lock()
|
||||
defer d.mu.Unlock()
|
||||
dl, ok := ctx.Deadline()
|
||||
if !ok {
|
||||
d.noDLOK = true
|
||||
return
|
||||
}
|
||||
*into = append(*into, time.Until(dl))
|
||||
}
|
||||
|
||||
func (d *deadlineSink) SaveProgress(ctx context.Context, _ []domain.ProgressStep, _ time.Time) error {
|
||||
d.record(ctx, &d.saves)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (d *deadlineSink) Beat(ctx context.Context, _ time.Time) error {
|
||||
d.record(ctx, &d.beats)
|
||||
return nil
|
||||
}
|
||||
|
||||
// The production value is the one the worker gets: it builds a Tracker without options.
|
||||
// A zero or tiny default would drop every write, a huge one would bound nothing.
|
||||
func TestTracker_TheDefaultSaveAndBeatDeadlineIsAFewSecondsAway(t *testing.T) {
|
||||
sink := &deadlineSink{}
|
||||
tr := progress.NewTracker(context.Background(), sink, slog.New(slog.NewTextHandler(&lockedBuf{}, nil)),
|
||||
[]progress.Planned{{Key: "a", Label: "A"}})
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
go tr.Heartbeat(ctx, 5*time.Millisecond, time.Hour)
|
||||
|
||||
tr.Begin("a", 0)
|
||||
require.Eventually(t, func() bool {
|
||||
sink.mu.Lock()
|
||||
defer sink.mu.Unlock()
|
||||
return len(sink.beats) > 0
|
||||
}, 3*time.Second, 5*time.Millisecond)
|
||||
|
||||
sink.mu.Lock()
|
||||
defer sink.mu.Unlock()
|
||||
require.False(t, sink.noDLOK, "a write was given no deadline at all")
|
||||
require.NotEmpty(t, sink.saves)
|
||||
for _, d := range append(append([]time.Duration{}, sink.saves...), sink.beats...) {
|
||||
require.Greater(t, d, time.Second, "the deadline is already (nearly) spent when the write starts")
|
||||
require.LessOrEqual(t, d, 5*time.Second, "the default bound is 5 s")
|
||||
}
|
||||
}
|
||||
@@ -21,6 +21,7 @@ type fakeSink struct {
|
||||
saves [][]domain.ProgressStep
|
||||
at []time.Time
|
||||
beats int
|
||||
beatAt []time.Time
|
||||
err error
|
||||
}
|
||||
|
||||
@@ -32,10 +33,11 @@ func (f *fakeSink) SaveProgress(_ context.Context, steps []domain.ProgressStep,
|
||||
return f.err
|
||||
}
|
||||
|
||||
func (f *fakeSink) Beat(context.Context) error {
|
||||
func (f *fakeSink) Beat(_ context.Context, at time.Time) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.beats++
|
||||
f.beatAt = append(f.beatAt, at)
|
||||
return f.err
|
||||
}
|
||||
|
||||
@@ -297,7 +299,7 @@ func (r *retainingSink) SaveProgress(_ context.Context, steps []domain.ProgressS
|
||||
r.got = append(r.got, steps)
|
||||
return nil
|
||||
}
|
||||
func (r *retainingSink) Beat(context.Context) error { return nil }
|
||||
func (r *retainingSink) Beat(context.Context, time.Time) error { return nil }
|
||||
|
||||
func TestTracker_ASnapshotHandedToTheSinkIsNotChangedByLaterTransitions(t *testing.T) {
|
||||
sink := &retainingSink{}
|
||||
@@ -400,3 +402,20 @@ func TestTracker_AnnotateAttachesANoteToAStepAndSavesItAtOnce(t *testing.T) {
|
||||
require.Equal(t, "20 min of the 79 the file claimed", sink.last(t)[0].Note)
|
||||
require.NotPanics(t, func() { tr.Annotate("no-such-step", "x") })
|
||||
}
|
||||
|
||||
func TestTracker_BeatIsStampedWithTheTrackersClock(t *testing.T) {
|
||||
// alive_at used the database's NOW() while the page compared it with the application
|
||||
// clock; a skewed database clock showed a healthy job as lost (#51).
|
||||
tr, sink, clk, _ := newTracker(t)
|
||||
clk.tick(42 * time.Second)
|
||||
want := clk.now()
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
go tr.Heartbeat(ctx, 5*time.Millisecond, time.Hour)
|
||||
|
||||
require.Eventually(t, func() bool { return sink.beatCount() >= 1 }, 2*time.Second, 5*time.Millisecond)
|
||||
sink.mu.Lock()
|
||||
defer sink.mu.Unlock()
|
||||
require.True(t, sink.beatAt[0].Equal(want), "beat stamped %v, the tracker's clock says %v", sink.beatAt[0], want)
|
||||
}
|
||||
|
||||
@@ -25,7 +25,7 @@ type Prober interface {
|
||||
|
||||
// Store is the slice of the private transcript store the intake needs.
|
||||
type Store interface {
|
||||
CreateUpload(ctx context.Context, userID, title string, hadVideo bool) (string, error)
|
||||
CreateUpload(ctx context.Context, userID, title string, lang domain.Language, hadVideo bool) (string, error)
|
||||
FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error
|
||||
}
|
||||
|
||||
@@ -35,6 +35,9 @@ type Job struct {
|
||||
UserID string
|
||||
UploadID string
|
||||
Path string
|
||||
// Language is what the user chose for this recording; empty means Swedish (a job
|
||||
// queued before the choice existed).
|
||||
Language domain.Language
|
||||
// Duration is what the file CLAIMED when probed. The worker compares it with what
|
||||
// was really converted; zero means unknown (ADR-035 review).
|
||||
Duration time.Duration
|
||||
@@ -135,7 +138,7 @@ const busyMessage = "Too many recordings are waiting to be transcribed. Please t
|
||||
// validation yields a *RejectError and creates NO row — rejection is synchronous
|
||||
// and leaves nothing behind. On success the temp file is handed to the job and the
|
||||
// worker deletes it; on every failure path Accept deletes it itself.
|
||||
func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reader) (string, error) {
|
||||
func (in *Intake) Accept(ctx context.Context, userID, title string, lang domain.Language, src io.Reader) (string, error) {
|
||||
// Refuse before spooling when the queue has no room, if the submitter can say so: a full
|
||||
// queue otherwise cost up to the upload cap on disk, an ffprobe run and a failed row for
|
||||
// every refused attempt.
|
||||
@@ -190,11 +193,11 @@ func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reade
|
||||
return "", err
|
||||
}
|
||||
|
||||
id, err := in.store.CreateUpload(ctx, userID, title, probe.HasVideo)
|
||||
id, err := in.store.CreateUpload(ctx, userID, title, lang, probe.HasVideo)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("uploads: create: %w", err)
|
||||
}
|
||||
if err := in.sub.Submit(Job{UserID: userID, UploadID: id, Path: path, Duration: probe.Duration}); err != nil {
|
||||
if err := in.sub.Submit(Job{UserID: userID, UploadID: id, Path: path, Language: lang, Duration: probe.Duration}); err != nil {
|
||||
if errors.Is(err, ErrQueueFull) {
|
||||
// The user's to retry, not a server fault: refuse it as such, and do not leave
|
||||
// the row queued -- that is the stuck spinner.
|
||||
|
||||
@@ -23,7 +23,7 @@ func TestAccept_AFullQueueIsABusyRejectionNotAServerErrorAndLeavesNothingBehind(
|
||||
r := newRig(t)
|
||||
r.sub.err = uploads.ErrQueueFull
|
||||
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
||||
|
||||
var rej *uploads.RejectError
|
||||
require.True(t, errors.As(err, &rej), "a full queue must be a rejection the web layer can answer with 429: %v", err)
|
||||
@@ -48,7 +48,7 @@ func TestAccept_ANoRoomQueueIsRefusedBeforeAnythingIsSpooledProbedOrStored(t *te
|
||||
full := &roomSubmitter{room: false}
|
||||
in := uploads.NewIntake(r.st, r.pr, full, uploads.Limits{MaxBytes: 1024, MaxDuration: 90 * time.Minute}, r.tmp)
|
||||
|
||||
_, err := in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||
_, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
||||
|
||||
var rej *uploads.RejectError
|
||||
require.True(t, errors.As(err, &rej))
|
||||
@@ -62,7 +62,7 @@ func TestAccept_ANoRoomQueueIsRefusedBeforeAnythingIsSpooledProbedOrStored(t *te
|
||||
func TestAccept_ASubmitterThatCannotSayStillWorksAsBefore(t *testing.T) {
|
||||
r := newRig(t)
|
||||
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
||||
|
||||
require.NoError(t, err, "a Submitter without HasRoom is accepted: the check at Submit remains")
|
||||
}
|
||||
@@ -81,7 +81,7 @@ func (b *brokenBody) Read(p []byte) (int, error) {
|
||||
func TestAccept_AnErrorReadingTheClientsBodyIsMarkedAsTheClientsAndAWrappedSubmitErrorIsNot(t *testing.T) {
|
||||
r := newRig(t)
|
||||
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "t", &brokenBody{n: 1})
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, &brokenBody{n: 1})
|
||||
|
||||
require.Error(t, err)
|
||||
assert.ErrorIs(t, err, uploads.ErrClientRead, "a dropped client body is the sender's to retry")
|
||||
@@ -90,7 +90,7 @@ func TestAccept_AnErrorReadingTheClientsBodyIsMarkedAsTheClientsAndAWrappedSubmi
|
||||
// A fault that is ours must NOT carry the marker, even if it wraps the same sentinel.
|
||||
r2 := newRig(t)
|
||||
r2.sub.err = fmt.Errorf("db: %w", io.ErrUnexpectedEOF)
|
||||
_, err = r2.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||
_, err = r2.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
||||
require.Error(t, err)
|
||||
assert.NotErrorIs(t, err, uploads.ErrClientRead)
|
||||
}
|
||||
|
||||
@@ -15,6 +15,7 @@ import (
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||
"git.d-ma.be/mathias/tapir/internal/uploads"
|
||||
)
|
||||
|
||||
@@ -31,7 +32,7 @@ func startSlowReceive(t *testing.T, r *rig, in *uploads.Intake, user string, wan
|
||||
pr, pw := io.Pipe()
|
||||
s := &slowReceive{w: pw, done: make(chan error, 1)}
|
||||
go func() {
|
||||
_, err := in.Accept(context.Background(), user, "t", pr)
|
||||
_, err := in.Accept(context.Background(), user, "t", domain.LangSwedish, pr)
|
||||
s.done <- err
|
||||
}()
|
||||
require.Eventually(t, func() bool { return len(r.tmpFiles(t)) == wantFiles }, 2*time.Second, 5*time.Millisecond,
|
||||
@@ -80,7 +81,7 @@ func TestAccept_APerUserLimitOnConcurrentReceivesRefusesTheNextBodyBeforeSpoolin
|
||||
startSlowReceive(t, r, in, "user-1", 1)
|
||||
startSlowReceive(t, r, in, "user-1", 2)
|
||||
|
||||
_, err := in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||
_, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
||||
|
||||
requireBusy(t, err)
|
||||
assert.Len(t, r.tmpFiles(t), 2, "the refused body was never spooled")
|
||||
@@ -98,7 +99,7 @@ func TestAccept_AnOverallLimitOnConcurrentReceivesRefusesTheNextUser(t *testing.
|
||||
startSlowReceive(t, r, in, u, i+1)
|
||||
}
|
||||
|
||||
_, err := in.Accept(context.Background(), "d", "t", strings.NewReader("data"))
|
||||
_, err := in.Accept(context.Background(), "d", "t", domain.LangSwedish, strings.NewReader("data"))
|
||||
|
||||
requireBusy(t, err)
|
||||
assert.Len(t, r.tmpFiles(t), 3)
|
||||
@@ -112,7 +113,7 @@ func TestAccept_AFinishedReceiveFreesItsSlot(t *testing.T) {
|
||||
first := startSlowReceive(t, r, in, "user-1", 1)
|
||||
require.NoError(t, first.finish(t))
|
||||
|
||||
_, err := in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||
_, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
||||
|
||||
require.NoError(t, err, "the slot is released when Accept returns")
|
||||
}
|
||||
@@ -122,10 +123,10 @@ func TestAccept_AFailedReceiveFreesItsSlot(t *testing.T) {
|
||||
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
|
||||
in.MaxReceivesPerUser = 1
|
||||
in.MaxReceives = 5
|
||||
_, err := in.Accept(context.Background(), "user-1", "t", &brokenBody{n: 1})
|
||||
_, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, &brokenBody{n: 1})
|
||||
require.ErrorIs(t, err, uploads.ErrClientRead)
|
||||
|
||||
_, err = in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||
_, err = in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
||||
|
||||
require.NoError(t, err, "a dropped client must not leak a slot")
|
||||
}
|
||||
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||
"git.d-ma.be/mathias/tapir/internal/uploads"
|
||||
)
|
||||
|
||||
@@ -21,11 +22,11 @@ func TestNewIntake_DefaultsAllowTwoReceivesPerUserAndFourOverall(t *testing.T) {
|
||||
startSlowReceive(t, r, in, "a", 1)
|
||||
startSlowReceive(t, r, in, "a", 2)
|
||||
|
||||
_, err := in.Accept(context.Background(), "a", "t", strings.NewReader("x"))
|
||||
_, err := in.Accept(context.Background(), "a", "t", domain.LangSwedish, strings.NewReader("x"))
|
||||
requireBusy(t, err) // a third body from one user
|
||||
|
||||
startSlowReceive(t, r, in, "b", 3)
|
||||
startSlowReceive(t, r, in, "b", 4)
|
||||
_, err = in.Accept(context.Background(), "c", "t", strings.NewReader("x"))
|
||||
_, err = in.Accept(context.Background(), "c", "t", domain.LangSwedish, strings.NewReader("x"))
|
||||
requireBusy(t, err) // a fifth body overall
|
||||
}
|
||||
|
||||
@@ -21,13 +21,15 @@ import (
|
||||
|
||||
type fakeStore struct {
|
||||
created []string // titles
|
||||
langs []domain.Language
|
||||
hadVideo []bool
|
||||
failed map[string]domain.UploadErrorKind
|
||||
nextID string
|
||||
}
|
||||
|
||||
func (f *fakeStore) CreateUpload(_ context.Context, _, title string, hadVideo bool) (string, error) {
|
||||
func (f *fakeStore) CreateUpload(_ context.Context, _, title string, lang domain.Language, hadVideo bool) (string, error) {
|
||||
f.created = append(f.created, title)
|
||||
f.langs = append(f.langs, lang)
|
||||
f.hadVideo = append(f.hadVideo, hadVideo)
|
||||
return f.nextID, nil
|
||||
}
|
||||
@@ -98,7 +100,7 @@ func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
|
||||
r := newRig(t)
|
||||
audio := bytes.Repeat([]byte("a"), 500)
|
||||
|
||||
id, err := r.in.Accept(context.Background(), "user-1", "Teams export", bytes.NewReader(audio))
|
||||
id, err := r.in.Accept(context.Background(), "user-1", "Teams export", domain.LangSwedish, bytes.NewReader(audio))
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "up-1", id)
|
||||
@@ -106,7 +108,7 @@ func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
|
||||
assert.Equal(t, audio, r.pr.seen, "the prober must see the complete uploaded bytes, on disk")
|
||||
require.Len(t, r.sub.jobs, 1)
|
||||
job := r.sub.jobs[0]
|
||||
assert.Equal(t, uploads.Job{UserID: "user-1", UploadID: "up-1", Path: job.Path, Duration: 10 * time.Minute}, job,
|
||||
assert.Equal(t, uploads.Job{UserID: "user-1", UploadID: "up-1", Path: job.Path, Duration: 10 * time.Minute, Language: domain.LangSwedish}, job,
|
||||
"the job carries what the file CLAIMED, so the worker can compare it with what was converted")
|
||||
assert.Equal(t, filepath.Dir(job.Path), r.tmp, "the temp file lives in the configured directory")
|
||||
got, err := os.ReadFile(job.Path)
|
||||
@@ -117,7 +119,7 @@ func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
|
||||
func TestAccept_OversizeBodyIsRejectedAndLeavesNothingBehind(t *testing.T) {
|
||||
r := newRig(t)
|
||||
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "big", strings.NewReader(strings.Repeat("a", 1025)))
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "big", domain.LangSwedish, strings.NewReader(strings.Repeat("a", 1025)))
|
||||
|
||||
var rej *uploads.RejectError
|
||||
require.True(t, errors.As(err, &rej))
|
||||
@@ -132,7 +134,7 @@ func TestAccept_OversizeBodyIsRejectedAndLeavesNothingBehind(t *testing.T) {
|
||||
func TestAccept_BodyExactlyAtTheLimitIsAccepted(t *testing.T) {
|
||||
r := newRig(t)
|
||||
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "edge", strings.NewReader(strings.Repeat("a", 1024)))
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "edge", domain.LangSwedish, strings.NewReader(strings.Repeat("a", 1024)))
|
||||
|
||||
require.NoError(t, err)
|
||||
}
|
||||
@@ -152,7 +154,7 @@ func TestAccept_ProbeVerdictRejectsWithoutRowAndRemovesTheFile(t *testing.T) {
|
||||
r := newRig(t)
|
||||
r.pr.probe = tt.probe
|
||||
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
||||
|
||||
var rej *uploads.RejectError
|
||||
require.True(t, errors.As(err, &rej))
|
||||
@@ -169,7 +171,7 @@ func TestAccept_UnreadableFileIsARejectionButAProberFaultIsAnError(t *testing.T)
|
||||
r := newRig(t)
|
||||
r.pr.err = uploads.ErrUnreadable
|
||||
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("not media"))
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("not media"))
|
||||
|
||||
var rej *uploads.RejectError
|
||||
require.True(t, errors.As(err, &rej), "the user's file is the problem: tell them")
|
||||
@@ -180,7 +182,7 @@ func TestAccept_UnreadableFileIsARejectionButAProberFaultIsAnError(t *testing.T)
|
||||
r := newRig(t)
|
||||
r.pr.err = errors.New("exec: ffprobe: executable file not found")
|
||||
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
||||
|
||||
require.Error(t, err)
|
||||
var rej *uploads.RejectError
|
||||
@@ -194,7 +196,7 @@ func TestAccept_ClientDisconnectMidUploadLeavesNothingBehind(t *testing.T) {
|
||||
r := newRig(t)
|
||||
body := io.MultiReader(strings.NewReader("partial"), iotest.ErrReader(errors.New("client went away")))
|
||||
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "t", body)
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, body)
|
||||
|
||||
require.Error(t, err)
|
||||
assert.Empty(t, r.pr.paths)
|
||||
@@ -206,7 +208,7 @@ func TestAccept_SubmitFailureFailsTheRowHonestlyAndRemovesTheFile(t *testing.T)
|
||||
r := newRig(t)
|
||||
r.sub.err = errors.New("worker stopped")
|
||||
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
||||
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, domain.UploadErrInterrupted, r.st.failed["up-1"],
|
||||
@@ -228,7 +230,7 @@ func TestAccept_AFileWithAVideoTrackIsAcceptedAndRecordsThatItHadOne(t *testing.
|
||||
r := newRig(t)
|
||||
r.pr.probe = tt.probe
|
||||
|
||||
id, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||
id, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "up-1", id)
|
||||
@@ -238,3 +240,14 @@ func TestAccept_AFileWithAVideoTrackIsAcceptedAndRecordsThatItHadOne(t *testing.
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestAccept_TheChosenLanguageIsStoredAndHandedToTheJob(t *testing.T) {
|
||||
r := newRig(t)
|
||||
|
||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangAuto, strings.NewReader("data"))
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []domain.Language{domain.LangAuto}, r.st.langs, "the row records what the user chose")
|
||||
require.Len(t, r.sub.jobs, 1)
|
||||
assert.Equal(t, domain.LangAuto, r.sub.jobs[0].Language, "the worker is told, so the engines are")
|
||||
}
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
package uploads
|
||||
|
||||
// Observes what jobSink hands the store: the heartbeat's timestamp is the one the tracker
|
||||
// stamped (its application clock), not a second reading taken here (#51).
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||
)
|
||||
|
||||
type atStore struct{ beatAt, saveAt time.Time }
|
||||
|
||||
func (*atStore) StartTranscribing(context.Context, string, string) error { return nil }
|
||||
func (*atStore) CompleteUpload(context.Context, string, string, domain.Transcription) error {
|
||||
return nil
|
||||
}
|
||||
func (*atStore) FailUpload(context.Context, string, string, domain.UploadErrorKind) error { return nil }
|
||||
func (s *atStore) SaveProgress(_ context.Context, _, _ string, _ []domain.ProgressStep, at time.Time) error {
|
||||
s.saveAt = at
|
||||
return nil
|
||||
}
|
||||
func (s *atStore) Heartbeat(_ context.Context, _, _ string, at time.Time) error {
|
||||
s.beatAt = at
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestJobSink_ForwardsTheTrackersTimestampToTheStoreUnchanged(t *testing.T) {
|
||||
st := &atStore{}
|
||||
sink := jobSink{st: st, userID: "u", id: "i"}
|
||||
stamp := time.Date(2020, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||
|
||||
if err := sink.Beat(context.Background(), stamp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := sink.SaveProgress(context.Background(), nil, stamp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if !st.beatAt.Equal(stamp) {
|
||||
t.Errorf("heartbeat reached the store stamped %v, want the tracker's %v", st.beatAt, stamp)
|
||||
}
|
||||
if !st.saveAt.Equal(stamp) {
|
||||
t.Errorf("progress save reached the store stamped %v, want the tracker's %v", st.saveAt, stamp)
|
||||
}
|
||||
}
|
||||
@@ -33,6 +33,13 @@ type Transcriber interface {
|
||||
Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error)
|
||||
}
|
||||
|
||||
// LanguagePlanner is a Planner whose phases depend on the language of the job: a recording
|
||||
// that is not Swedish skips the second engine and the reconcile, and the page should not list
|
||||
// them as coming (tapir#61). The worker prefers it over Planner.
|
||||
type LanguagePlanner interface {
|
||||
PlanFor(lang domain.Language) []progress.Planned
|
||||
}
|
||||
|
||||
// Planner is an optional extra on a Transcriber: it declares its own phases (for
|
||||
// example one per engine) so the page can show them. A Transcriber that does not
|
||||
// implement it gets a single "transcribing" phase from the worker. Whatever a Planner
|
||||
@@ -47,7 +54,7 @@ type WorkerStore interface {
|
||||
CompleteUpload(ctx context.Context, userID, id string, t domain.Transcription) error
|
||||
FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error
|
||||
SaveProgress(ctx context.Context, userID, id string, steps []domain.ProgressStep, at time.Time) error
|
||||
Heartbeat(ctx context.Context, userID, id string) error
|
||||
Heartbeat(ctx context.Context, userID, id string, at time.Time) error
|
||||
}
|
||||
|
||||
// WorkerOption tunes a Worker.
|
||||
@@ -91,7 +98,9 @@ func (s jobSink) SaveProgress(ctx context.Context, steps []domain.ProgressStep,
|
||||
return s.st.SaveProgress(ctx, s.userID, s.id, steps, at)
|
||||
}
|
||||
|
||||
func (s jobSink) Beat(ctx context.Context) error { return s.st.Heartbeat(ctx, s.userID, s.id) }
|
||||
func (s jobSink) Beat(ctx context.Context, at time.Time) error {
|
||||
return s.st.Heartbeat(ctx, s.userID, s.id, at)
|
||||
}
|
||||
|
||||
// Worker transcribes accepted uploads in-process, ONE AT A TIME (ADR-033). It is
|
||||
// not a second deployable: it shares the web process's lifetime, which is why it
|
||||
@@ -249,8 +258,14 @@ func (w *Worker) Process(ctx context.Context, job Job) {
|
||||
}
|
||||
|
||||
plan := []progress.Planned{{Key: "received", Label: "Received"}, {Key: "audio", Label: "Preparing the audio"}}
|
||||
lang := job.Language
|
||||
if lang == "" {
|
||||
lang = domain.LangSwedish // queued before the choice existed
|
||||
}
|
||||
planner, planned := w.tr.(Planner)
|
||||
if planned {
|
||||
if lp, ok := w.tr.(LanguagePlanner); ok {
|
||||
plan = append(plan, lp.PlanFor(lang)...)
|
||||
} else if planned {
|
||||
plan = append(plan, planner.Plan()...)
|
||||
} else {
|
||||
plan = append(plan, progress.Planned{Key: "transcribe", Label: "Transcribing"})
|
||||
@@ -311,7 +326,7 @@ func (w *Worker) Process(ctx context.Context, job Job) {
|
||||
if !planned {
|
||||
tr.Begin("transcribe", 0)
|
||||
}
|
||||
t, err := w.tr.Transcribe(progress.With(jobCtx, tr), wav)
|
||||
t, err := w.tr.Transcribe(domain.WithLanguage(progress.With(jobCtx, tr), lang), wav)
|
||||
if err != nil {
|
||||
log.Error("upload: transcribe", "kind", errkind.Of(err), "elapsed", time.Since(start).Round(time.Second))
|
||||
abort(domain.UploadErrTranscriptionFailed)
|
||||
|
||||
@@ -213,3 +213,47 @@ func TestWorkerProgress_AStoreFailureLogsItsSQLStateNeverTheValueItQuotes(t *tes
|
||||
assert.Contains(t, out, "upload: store transcript")
|
||||
assert.Contains(t, out, "kind=sql_22P02")
|
||||
}
|
||||
|
||||
// langPlannedTr plans per language, like the Ensemble: an auto job has no base engine.
|
||||
type langPlannedTr struct{ *fakeTranscriber }
|
||||
|
||||
func (langPlannedTr) Plan() []progress.Planned { return plannedTr{}.Plan() }
|
||||
func (langPlannedTr) PlanFor(l domain.Language) []progress.Planned {
|
||||
if l == domain.LangAuto {
|
||||
return []progress.Planned{{Key: "engine_b", Label: "Transcribing with B"}}
|
||||
}
|
||||
return plannedTr{}.Plan()
|
||||
}
|
||||
|
||||
func (p langPlannedTr) Transcribe(ctx context.Context, wav string) (domain.Transcription, error) {
|
||||
r := progress.From(ctx)
|
||||
if domain.LanguageFrom(ctx) != domain.LangAuto {
|
||||
r.Begin("engine_a", 0)
|
||||
r.Done("engine_a")
|
||||
}
|
||||
r.Begin("engine_b", 0)
|
||||
r.Done("engine_b")
|
||||
return p.fakeTranscriber.Transcribe(ctx, wav)
|
||||
}
|
||||
|
||||
func TestWorkerProgress_AJobGetsThePlanOfItsLanguage(t *testing.T) {
|
||||
for name, tt := range map[string]struct {
|
||||
lang domain.Language
|
||||
want []string
|
||||
}{
|
||||
"auto": {domain.LangAuto, []string{"received:done", "audio:done", "engine_b:done", "save:done"}},
|
||||
"swedish": {domain.LangSwedish, []string{"received:done", "audio:done", "engine_a:done", "engine_b:done", "save:done"}},
|
||||
"no language (queued before the choice existed)": {"", []string{"received:done", "audio:done", "engine_a:done", "engine_b:done", "save:done"}},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
r := newWRig(t, time.Minute)
|
||||
w := newProgressWorker(t, r, langPlannedTr{r.tr})
|
||||
job := r.spool(t, "a")
|
||||
job.Language = tt.lang
|
||||
|
||||
w.Process(context.Background(), job)
|
||||
|
||||
assert.Equal(t, tt.want, keysAndStates(r.st.lastSteps()))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -70,7 +70,7 @@ func (s *wStore) SaveProgress(_ context.Context, _, _ string, steps []domain.Pro
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *wStore) Heartbeat(context.Context, string, string) error {
|
||||
func (s *wStore) Heartbeat(context.Context, string, string, time.Time) error {
|
||||
time.Sleep(s.beatDelay)
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
@@ -126,6 +126,7 @@ type fakeTranscriber struct {
|
||||
result domain.Transcription
|
||||
err error
|
||||
paths []string
|
||||
langs []domain.Language // what each call's context carried
|
||||
// block, when non-nil, holds every call until it is closed or ctx ends.
|
||||
block chan struct{}
|
||||
running atomic.Int32
|
||||
@@ -134,6 +135,7 @@ type fakeTranscriber struct {
|
||||
|
||||
func (f *fakeTranscriber) Transcribe(ctx context.Context, wav string) (domain.Transcription, error) {
|
||||
f.paths = append(f.paths, wav)
|
||||
f.langs = append(f.langs, domain.LanguageFrom(ctx))
|
||||
n := f.running.Add(1)
|
||||
defer f.running.Add(-1)
|
||||
for {
|
||||
@@ -337,3 +339,24 @@ func (dbErr) Error() string {
|
||||
return `ERROR: invalid input syntax for type json: "zebrakaka secret words"`
|
||||
}
|
||||
func (dbErr) SQLState() string { return "22P02" }
|
||||
|
||||
func TestWorker_TheTranscriberSeesTheLanguageOfTheJobOnItsContext(t *testing.T) {
|
||||
for name, tt := range map[string]struct {
|
||||
job domain.Language
|
||||
want domain.Language
|
||||
}{
|
||||
"auto": {domain.LangAuto, domain.LangAuto},
|
||||
"swedish": {domain.LangSwedish, domain.LangSwedish},
|
||||
"a job with no language": {"", domain.LangSwedish}, // queued before the field existed
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
r := newWRig(t, time.Minute)
|
||||
job := r.spool(t, "a")
|
||||
job.Language = tt.job
|
||||
|
||||
r.w.Process(context.Background(), job)
|
||||
|
||||
assert.Equal(t, []domain.Language{tt.want}, r.tr.langs)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -125,6 +125,20 @@ func TestCrossOrigin_AForeignPostIsRefusedOnEachListedStateChangingPath(t *testi
|
||||
}
|
||||
}
|
||||
|
||||
func TestCrossOrigin_ARefusedRecordingDeletePostLeavesTheRecordingInPlace(t *testing.T) {
|
||||
for name, h := range foreign {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
||||
|
||||
rec := do(t, app, withHeaders(httptest.NewRequest(http.MethodPost, "/uploads/"+id+"/delete", nil), h))
|
||||
|
||||
assert.Equal(t, http.StatusForbidden, rec.Code)
|
||||
assert.True(t, uploadExists(t, id), "a refused post must not delete the recording")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A login redirect comes back to /auth/callback as a cross-site GET navigation, and
|
||||
// health probes carry no headers: safe methods must never be refused on origin grounds.
|
||||
func TestCrossOrigin_ACrossSiteGETIsNotRefused(t *testing.T) {
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
package web
|
||||
|
||||
import "time"
|
||||
|
||||
// DefaultUploadIdleTimeout exposes the production idle bound (#56) to the black-box tests:
|
||||
// cmd/tapir never sets App.UploadIdleTimeout, so this constant IS the deployed value.
|
||||
const DefaultUploadIdleTimeout = defaultUploadIdleTimeout
|
||||
|
||||
// UploadIdleTimeoutForTest exposes the bound an App resolves to (#56), so a test can pin the
|
||||
// value the zero-value App -- production's -- actually uses rather than only the constant.
|
||||
func (a *App) UploadIdleTimeoutForTest() time.Duration { return a.uploadIdleTimeout() }
|
||||
@@ -59,6 +59,9 @@ type Store interface {
|
||||
// non-owner's id is ErrUploadNotFound.
|
||||
GetUpload(ctx context.Context, userID, id string) (*domain.UploadedTranscript, error)
|
||||
ListUploads(ctx context.Context, userID string) ([]domain.UploadedTranscript, error)
|
||||
// DeleteUpload removes a finished upload; ErrUploadInProgress while it is still being
|
||||
// processed, ErrUploadNotFound for a non-owner.
|
||||
DeleteUpload(ctx context.Context, userID, id string) error
|
||||
|
||||
// SetTranscriptStatus clears or updates a video's transcript backoff state.
|
||||
// Used by handleRetryNow to clear rate_limited_at before immediate processing.
|
||||
@@ -113,6 +116,11 @@ type App struct {
|
||||
// front rather than discovered by failing.
|
||||
Uploads Uploader
|
||||
UploadLimits uploads.Limits
|
||||
// UploadIdleTimeout is how long an upload body may go without delivering a byte
|
||||
// before the connection read is cut off and its receive slot freed (#56). Zero
|
||||
// means defaultUploadIdleTimeout. It bounds silence, not the whole body, so a
|
||||
// large upload on a slow but moving link still completes.
|
||||
UploadIdleTimeout time.Duration
|
||||
// UploadChatMaxChars is the transcript budget for chat over an uploaded
|
||||
// recording. The caption default (18,000 chars, ~22 minutes of speech) would
|
||||
// keep only the first third of an hour-long meeting.
|
||||
@@ -200,6 +208,7 @@ func (a *App) Router() http.Handler {
|
||||
app.HandleFunc("POST /uploads", a.handleUploadPost)
|
||||
app.HandleFunc("GET /uploads/{id}", a.handleUploadDetail)
|
||||
app.HandleFunc("GET /uploads/{id}/status", a.handleUploadStatus)
|
||||
app.HandleFunc("POST /uploads/{id}/delete", a.handleUploadDelete)
|
||||
// Chat over the uploaded transcript reuses the ADR-027 Chatter; mounted only
|
||||
// when one is wired.
|
||||
if a.Chat != nil {
|
||||
@@ -309,7 +318,7 @@ func (a *App) handleList(w http.ResponseWriter, r *http.Request) {
|
||||
a.render(w, r, summaryList(buckets, hasConnected, autoSummarize))
|
||||
return
|
||||
}
|
||||
a.render(w, r, ListPage(buckets, f, stats, takeFlash(w, r), hasConnected, channels, autoSummarize))
|
||||
a.render(w, r, ListPage(buckets, f, stats, takeFlash(w, r), hasConnected, channels, autoSummarize, a.Uploads != nil))
|
||||
}
|
||||
|
||||
// handleDetail renders one summary in full (highlights, takeaways, action group).
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
package oidc_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"git.d-ma.be/mathias/tapir/internal/web/oidc"
|
||||
)
|
||||
|
||||
func testConfig(issuer string) oidc.Config {
|
||||
return oidc.Config{
|
||||
Issuer: issuer,
|
||||
ClientID: testClientID,
|
||||
ClientSecret: "test-client-secret",
|
||||
RedirectURL: "http://tapir.test/auth/callback",
|
||||
SessionSecret: "test-session-secret-please-change",
|
||||
}
|
||||
}
|
||||
|
||||
// #59: a discovery 503 at start-up is retried instead of failing New.
|
||||
func TestNewRetriesDiscoveryAfter503UntilIssuerAnswers(t *testing.T) {
|
||||
f := newFakeIssuer(t)
|
||||
f.unavailableFor = 2
|
||||
|
||||
auth, err := oidc.New(context.Background(), testConfig(f.server.URL),
|
||||
oidc.WithDiscoveryRetry(5, time.Millisecond))
|
||||
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, auth)
|
||||
require.Equal(t, int32(3), f.discoveryHits.Load())
|
||||
}
|
||||
|
||||
func TestNewReturnsDiscoveryErrorAfterAttemptsExhausted(t *testing.T) {
|
||||
f := newFakeIssuer(t)
|
||||
f.unavailableFor = 1000
|
||||
|
||||
_, err := oidc.New(context.Background(), testConfig(f.server.URL),
|
||||
oidc.WithDiscoveryRetry(3, time.Millisecond))
|
||||
|
||||
require.ErrorContains(t, err, "503")
|
||||
require.Equal(t, int32(3), f.discoveryHits.Load())
|
||||
}
|
||||
|
||||
func TestNewStopsDiscoveryBackoffWhenContextCancelled(t *testing.T) {
|
||||
f := newFakeIssuer(t)
|
||||
f.unavailableFor = 1000
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
start := time.Now()
|
||||
_, err := oidc.New(ctx, testConfig(f.server.URL),
|
||||
oidc.WithDiscoveryRetry(5, time.Hour))
|
||||
|
||||
require.True(t, errors.Is(err, context.DeadlineExceeded), "err = %v", err)
|
||||
require.Less(t, time.Since(start), 5*time.Second)
|
||||
require.Equal(t, int32(1), f.discoveryHits.Load())
|
||||
}
|
||||
+66
-16
@@ -57,6 +57,10 @@ const (
|
||||
pendingTTL = 10 * time.Minute
|
||||
sessionCookie = "tapir_session"
|
||||
loginPath = "/auth/login"
|
||||
|
||||
// Discovery retry (#59): 5 tries, 2+4+8+16 s back-off ≈ 30 s, then fail.
|
||||
defaultDiscoveryAttempts = 5
|
||||
defaultDiscoveryDelay = 2 * time.Second
|
||||
)
|
||||
|
||||
// DexAuth implements web.Auth against Dex.
|
||||
@@ -70,6 +74,9 @@ type DexAuth struct {
|
||||
sessionTTL time.Duration
|
||||
now func() time.Time
|
||||
insecure bool // tests only: drop the Secure attribute so cookies flow over http
|
||||
|
||||
discoveryAttempts int
|
||||
discoveryDelay time.Duration
|
||||
}
|
||||
|
||||
var _ web.Auth = (*DexAuth)(nil)
|
||||
@@ -96,14 +103,55 @@ func WithSessionTTL(ttl time.Duration) Option {
|
||||
}
|
||||
}
|
||||
|
||||
// WithDiscoveryRetry overrides the discovery attempt count and the initial
|
||||
// back-off delay (doubled after each failed attempt).
|
||||
func WithDiscoveryRetry(attempts int, delay time.Duration) Option {
|
||||
return func(d *DexAuth) {
|
||||
if attempts > 0 {
|
||||
d.discoveryAttempts = attempts
|
||||
}
|
||||
if delay > 0 {
|
||||
d.discoveryDelay = delay
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// WithInsecureCookies drops the Secure cookie attribute. TEST ONLY — it lets
|
||||
// the session cookie travel over plain http; never use it in production.
|
||||
func WithInsecureCookies() Option {
|
||||
return func(d *DexAuth) { d.insecure = true }
|
||||
}
|
||||
|
||||
// New discovers the issuer and returns a configured DexAuth. ctx bounds the
|
||||
// discovery request only.
|
||||
// discover runs OIDC discovery up to attempts times, doubling delay between
|
||||
// tries (#59: an issuer briefly unavailable behind its proxy answered 503 and
|
||||
// serve exited). A cancelled ctx ends the back-off immediately with ctx's error.
|
||||
func discover(ctx context.Context, issuer string, attempts int, delay time.Duration) (*oidc.Provider, error) {
|
||||
var lastErr error
|
||||
for i := range attempts {
|
||||
if i > 0 {
|
||||
t := time.NewTimer(delay)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
t.Stop()
|
||||
return nil, fmt.Errorf("after %d attempts (last: %v): %w", i, lastErr, ctx.Err())
|
||||
case <-t.C:
|
||||
}
|
||||
delay *= 2
|
||||
}
|
||||
p, err := oidc.NewProvider(ctx, issuer)
|
||||
if err == nil {
|
||||
return p, nil
|
||||
}
|
||||
lastErr = err
|
||||
if ctx.Err() != nil {
|
||||
return nil, fmt.Errorf("after %d attempts: %w", i+1, ctx.Err())
|
||||
}
|
||||
}
|
||||
return nil, fmt.Errorf("after %d attempts: %w", attempts, lastErr)
|
||||
}
|
||||
|
||||
// New discovers the issuer and returns a configured DexAuth. ctx bounds discovery,
|
||||
// including its retry back-off.
|
||||
func New(ctx context.Context, cfg Config, opts ...Option) (*DexAuth, error) {
|
||||
for name, val := range map[string]string{
|
||||
"issuer": cfg.Issuer,
|
||||
@@ -117,28 +165,30 @@ func New(ctx context.Context, cfg Config, opts ...Option) (*DexAuth, error) {
|
||||
}
|
||||
}
|
||||
|
||||
provider, err := oidc.NewProvider(ctx, cfg.Issuer)
|
||||
d := &DexAuth{
|
||||
cfg: cfg,
|
||||
pending: newPendingStore(),
|
||||
secret: []byte(cfg.SessionSecret),
|
||||
sessionTTL: defaultSessionTTL,
|
||||
now: time.Now,
|
||||
discoveryAttempts: defaultDiscoveryAttempts,
|
||||
discoveryDelay: defaultDiscoveryDelay,
|
||||
}
|
||||
for _, o := range opts {
|
||||
o(d)
|
||||
}
|
||||
|
||||
provider, err := discover(ctx, cfg.Issuer, d.discoveryAttempts, d.discoveryDelay)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("oidc: discover issuer: %w", err)
|
||||
}
|
||||
|
||||
d := &DexAuth{
|
||||
cfg: cfg,
|
||||
verifier: provider.Verifier(&oidc.Config{ClientID: cfg.ClientID}),
|
||||
oauth: &oauth2.Config{
|
||||
d.verifier = provider.Verifier(&oidc.Config{ClientID: cfg.ClientID})
|
||||
d.oauth = &oauth2.Config{
|
||||
ClientID: cfg.ClientID,
|
||||
ClientSecret: cfg.ClientSecret,
|
||||
Endpoint: provider.Endpoint(),
|
||||
RedirectURL: cfg.RedirectURL,
|
||||
Scopes: []string{oidc.ScopeOpenID, "profile", "email"},
|
||||
},
|
||||
pending: newPendingStore(),
|
||||
secret: []byte(cfg.SessionSecret),
|
||||
sessionTTL: defaultSessionTTL,
|
||||
now: time.Now,
|
||||
}
|
||||
for _, o := range opts {
|
||||
o(d)
|
||||
}
|
||||
return d, nil
|
||||
}
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -34,6 +35,11 @@ type fakeIssuer struct {
|
||||
sub string
|
||||
email string
|
||||
nonce string
|
||||
|
||||
// discoveryHits counts requests to the discovery document; the first
|
||||
// unavailableFor of them answer 503 (issuer briefly down behind its proxy).
|
||||
discoveryHits atomic.Int32
|
||||
unavailableFor int32
|
||||
}
|
||||
|
||||
const testKID = "test-key"
|
||||
@@ -47,6 +53,10 @@ func newFakeIssuer(t *testing.T) *fakeIssuer {
|
||||
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, _ *http.Request) {
|
||||
if f.discoveryHits.Add(1) <= f.unavailableFor {
|
||||
http.Error(w, "no available server", http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
writeJSON(t, w, map[string]any{
|
||||
"issuer": f.server.URL,
|
||||
"authorization_endpoint": f.server.URL + "/authorize",
|
||||
|
||||
@@ -66,7 +66,7 @@ func TestParseYouTubeVideoID(t *testing.T) {
|
||||
func TestListPageShowsPasteFormOnlyWhenConnected(t *testing.T) {
|
||||
render := func(connected bool) string {
|
||||
var buf bytes.Buffer
|
||||
if err := ListPage(listBuckets{}, Filter{}, PipelineStats{}, "", connected, nil, true).Render(context.Background(), &buf); err != nil {
|
||||
if err := ListPage(listBuckets{}, Filter{}, PipelineStats{}, "", connected, nil, true, false).Render(context.Background(), &buf); err != nil {
|
||||
t.Fatalf("render: %v", err)
|
||||
}
|
||||
return buf.String()
|
||||
|
||||
+136
-3
@@ -18,6 +18,7 @@ import (
|
||||
|
||||
"git.d-ma.be/mathias/tapir/internal/adapters/chat"
|
||||
"git.d-ma.be/mathias/tapir/internal/adapters/store"
|
||||
"git.d-ma.be/mathias/tapir/internal/config"
|
||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||
"git.d-ma.be/mathias/tapir/internal/errkind"
|
||||
"git.d-ma.be/mathias/tapir/internal/uploads"
|
||||
@@ -27,7 +28,7 @@ import (
|
||||
// *uploads.Intake satisfies it. It is the whole of what the handlers know about
|
||||
// validation, spooling and queueing.
|
||||
type Uploader interface {
|
||||
Accept(ctx context.Context, userID, title string, src io.Reader) (id string, err error)
|
||||
Accept(ctx context.Context, userID, title string, lang domain.Language, src io.Reader) (id string, err error)
|
||||
}
|
||||
|
||||
const (
|
||||
@@ -35,6 +36,8 @@ const (
|
||||
// and the title field; the cap itself is enforced on the file part by Intake.
|
||||
uploadOverhead = 1 << 20
|
||||
maxTitleRunes = 200
|
||||
// maxLanguageBytes bounds what is read of the language field; a valid value is at most four bytes, so a longer one fails the parse.
|
||||
maxLanguageBytes = 16
|
||||
// uploadPollEvery is how often an unfinished upload re-asks for its status.
|
||||
uploadPollEvery = "2s"
|
||||
// serviceSilentAfter is how long a job may go without any signal (heartbeat or
|
||||
@@ -81,13 +84,14 @@ func (a *App) handleUploadPost(w http.ResponseWriter, r *http.Request) {
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
r.Body = http.MaxBytesReader(w, r.Body, a.UploadLimits.MaxBytes+uploadOverhead)
|
||||
r.Body = http.MaxBytesReader(w, a.idleBoundedBody(w, r.Body), a.UploadLimits.MaxBytes+uploadOverhead)
|
||||
mr, err := r.MultipartReader()
|
||||
if err != nil {
|
||||
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "Choose an audio file to upload.")
|
||||
return
|
||||
}
|
||||
title := ""
|
||||
lang := domain.LangSwedish
|
||||
for {
|
||||
part, err := mr.NextPart()
|
||||
if errors.Is(err, io.EOF) {
|
||||
@@ -120,11 +124,29 @@ func (a *App) handleUploadPost(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
title = cleanTitle(string(b))
|
||||
case "language":
|
||||
// A closed set read before the file: anything else is refused before the file
|
||||
// is spooled, so a crafted value never reaches an engine (tapir#61).
|
||||
b, err := io.ReadAll(io.LimitReader(part, maxLanguageBytes+1))
|
||||
if err != nil {
|
||||
if a.abandoned(w, r) {
|
||||
return
|
||||
}
|
||||
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "That upload could not be read. Please try again.")
|
||||
return
|
||||
}
|
||||
l, ok := domain.ParseLanguage(string(b))
|
||||
if !ok {
|
||||
a.logger().Info("upload rejected", "reason", "language")
|
||||
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "Choose a language from the list.")
|
||||
return
|
||||
}
|
||||
lang = l
|
||||
case "file":
|
||||
if title == "" {
|
||||
title = titleFromFilename(part.FileName())
|
||||
}
|
||||
id, err := a.Uploads.Accept(r.Context(), userID, title, part)
|
||||
id, err := a.Uploads.Accept(r.Context(), userID, title, lang, part)
|
||||
if err != nil {
|
||||
a.uploadFailure(w, r, userID, err)
|
||||
return
|
||||
@@ -136,6 +158,53 @@ func (a *App) handleUploadPost(w http.ResponseWriter, r *http.Request) {
|
||||
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "Choose an audio file to upload.")
|
||||
}
|
||||
|
||||
// defaultUploadIdleTimeout is how long an upload body may deliver nothing before its
|
||||
// connection read is cut off (#56). A stall this long is not a slow link but a dead or
|
||||
// hostile one; a legitimate upload, however slow, keeps delivering bytes.
|
||||
const defaultUploadIdleTimeout = 60 * time.Second
|
||||
|
||||
// uploadIdleTimeout is the idle bound this App applies: UploadIdleTimeout when set, else
|
||||
// defaultUploadIdleTimeout. cmd/tapir leaves it unset, so production runs the default.
|
||||
func (a *App) uploadIdleTimeout() time.Duration {
|
||||
if a.UploadIdleTimeout <= 0 {
|
||||
return defaultUploadIdleTimeout
|
||||
}
|
||||
return a.UploadIdleTimeout
|
||||
}
|
||||
|
||||
// idleBoundedBody bounds the SILENCE in an upload body, not its total duration (#56): every
|
||||
// read first pushes the connection's read deadline idle into the future, so a body that
|
||||
// keeps moving is never cut, and one that stops fails its read with a timeout. That read
|
||||
// error reaches Accept as a client read error, so Accept returns and its receive slot is
|
||||
// released. A whole-request ReadTimeout would instead cut a large upload on a slow link.
|
||||
//
|
||||
// If the writer cannot set a read deadline (a wrapper without Unwrap, or a test recorder)
|
||||
// the body is returned unbounded and that is logged: a control that silently does not
|
||||
// apply is worse than one that says so.
|
||||
func (a *App) idleBoundedBody(w http.ResponseWriter, body io.ReadCloser) io.ReadCloser {
|
||||
idle := a.uploadIdleTimeout()
|
||||
rc := http.NewResponseController(w)
|
||||
if err := rc.SetReadDeadline(time.Now().Add(idle)); err != nil {
|
||||
a.logger().Warn("upload idle bound unavailable", "err", err)
|
||||
return body
|
||||
}
|
||||
return &idleDeadlineBody{ReadCloser: body, rc: rc, idle: idle}
|
||||
}
|
||||
|
||||
type idleDeadlineBody struct {
|
||||
io.ReadCloser
|
||||
rc *http.ResponseController
|
||||
idle time.Duration
|
||||
}
|
||||
|
||||
func (b *idleDeadlineBody) Read(p []byte) (int, error) {
|
||||
_ = b.rc.SetReadDeadline(time.Now().Add(b.idle))
|
||||
// Never cleared here: net/http clears the read deadline itself before its background
|
||||
// read once the handler has the body, and an armed deadline after an error also bounds
|
||||
// net/http's drain of an unread remainder before it replies.
|
||||
return b.ReadCloser.Read(p)
|
||||
}
|
||||
|
||||
// abandoned reports, logs at INFO and answers 499, because the client has already gone: its request
|
||||
// context is cancelled, so there is nobody to answer and rendering a page would run its
|
||||
// database read on a dead context and log a false ERROR. A hang-up mid-upload is routine.
|
||||
@@ -209,6 +278,32 @@ func (a *App) handleUploadStatus(w http.ResponseWriter, r *http.Request) {
|
||||
a.render(w, r, uploadStatus(*u, a.Chat != nil, a.now()))
|
||||
}
|
||||
|
||||
// handleUploadDelete removes one of the caller's finished recordings and returns to the
|
||||
// list. Another user's id is a 404 like every other upload route; a recording still being
|
||||
// transcribed is refused (409) rather than deleted under the worker (#37).
|
||||
func (a *App) handleUploadDelete(w http.ResponseWriter, r *http.Request) {
|
||||
userID, ok := a.currentUserID(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
id := r.PathValue("id")
|
||||
if !uuidShape.MatchString(id) {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
switch err := a.Store.DeleteUpload(r.Context(), userID, id); {
|
||||
case err == nil:
|
||||
http.Redirect(w, r, "/uploads", http.StatusSeeOther)
|
||||
case errors.Is(err, store.ErrUploadNotFound):
|
||||
http.NotFound(w, r)
|
||||
case errors.Is(err, store.ErrUploadInProgress):
|
||||
a.renderUploadsPage(w, r, userID, http.StatusConflict,
|
||||
"This recording is still being transcribed. You can delete it once it has finished.")
|
||||
default:
|
||||
a.serverError(w, r, "delete upload", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (a *App) loadOwnedUpload(w http.ResponseWriter, r *http.Request) (*domain.UploadedTranscript, bool) {
|
||||
userID, ok := a.currentUserID(w, r)
|
||||
if !ok {
|
||||
@@ -243,6 +338,44 @@ func titleFromFilename(name string) string {
|
||||
return t
|
||||
}
|
||||
|
||||
// cloudProvider names the cloud service that produced a transcript, or false when it was
|
||||
// served locally (or by an unknown endpoint). "Local" is config.IsLocalModel: the same
|
||||
// definition the start-up validation uses, not a second string match here (#52).
|
||||
//
|
||||
// The endpoint is one alias, except for the two-engine path (ADR-034), which writes
|
||||
// "<base> + <alt>, reconciled by <model>" (ensemble.go). That shape is taken apart exactly
|
||||
// and each alias is judged whole, so a composite of local models is local, and anything
|
||||
// else (a cloud alias, one with no provider prefix, one with a space or a connector word in
|
||||
// it) is not. A string that is not that shape is one alias.
|
||||
func cloudProvider(endpoint string) (string, bool) {
|
||||
endpoint = strings.TrimSpace(endpoint)
|
||||
if endpoint == "" {
|
||||
return "", false
|
||||
}
|
||||
aliases := []string{endpoint}
|
||||
if engines, reconciler, ok := strings.Cut(endpoint, ", reconciled by "); ok {
|
||||
aliases = append(strings.Split(engines, " + "), reconciler)
|
||||
}
|
||||
for _, a := range aliases {
|
||||
if a = strings.TrimSpace(a); !config.IsLocalModel(a) {
|
||||
provider, _, _ := strings.Cut(a, "/")
|
||||
if provider = strings.TrimSpace(provider); provider == "" {
|
||||
provider = a
|
||||
}
|
||||
return provider, true
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// languageLabel is how the page names the language a recording was transcribed as.
|
||||
func languageLabel(l domain.Language) string {
|
||||
if l == domain.LangAuto {
|
||||
return "Other (detected by the engine)"
|
||||
}
|
||||
return "Swedish"
|
||||
}
|
||||
|
||||
// hasNote reports whether any phase carries a note. A note is the only warning for a
|
||||
// recording that is shorter than its file claimed, so a record that has one is shown open.
|
||||
func hasNote(steps []domain.ProgressStep) bool {
|
||||
|
||||
@@ -24,6 +24,14 @@ templ UploadsPage(v uploadsView) {
|
||||
}
|
||||
<form method="post" action="/uploads" enctype="multipart/form-data">
|
||||
<p><label>Title (optional) <input type="text" name="title" maxlength="200"/></label></p>
|
||||
<p>
|
||||
<label>Language
|
||||
<select name="language">
|
||||
<option value="sv" selected>Swedish</option>
|
||||
<option value="auto">Other (detected by the engine)</option>
|
||||
</select>
|
||||
</label>
|
||||
</p>
|
||||
<p><input type="file" name="file" accept="audio/*,.m4a,.mp3,.wav,.opus,.ogg,.mp4" required/></p>
|
||||
<p><button type="submit">Upload and transcribe</button></p>
|
||||
</form>
|
||||
@@ -49,6 +57,7 @@ templ UploadPage(u domain.UploadedTranscript, chatEnabled bool, now time.Time) {
|
||||
@Layout("Tapir — " + u.Title) {
|
||||
<article class="account">
|
||||
<h1>{ u.Title }</h1>
|
||||
<p class="muted">Language: { languageLabel(u.Language) }</p>
|
||||
@uploadStatus(u, chatEnabled, now)
|
||||
<p><a href="/uploads">All recordings</a></p>
|
||||
</article>
|
||||
@@ -66,6 +75,12 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
|
||||
Transcribed { uploads.FormatDuration(time.Duration(u.AudioSeconds * float64(time.Second))) }
|
||||
of audio by <span class="chip">{ u.Endpoint }</span>.
|
||||
</p>
|
||||
if provider, cloud := cloudProvider(u.Endpoint); cloud {
|
||||
<p class="cloud-note">
|
||||
This recording's audio left the homelab: it was transcribed by a cloud service ({ provider })
|
||||
because the local engine could not.
|
||||
</p>
|
||||
}
|
||||
@reconciliationNote(u.Reconciliation)
|
||||
if len(u.Progress) > 0 {
|
||||
<details class="what-was-done" if hasNote(u.Progress) { open }>
|
||||
@@ -80,6 +95,7 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
|
||||
<summary>Transcript</summary>
|
||||
<pre class="transcript">{ u.Content }</pre>
|
||||
</details>
|
||||
@deleteRecording(u.ID)
|
||||
</div>
|
||||
case domain.UploadFailed:
|
||||
<div class="upload-status upload-failed flash flash-error" role="alert">
|
||||
@@ -91,6 +107,7 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
|
||||
@progressSteps(u.Progress, now, false)
|
||||
</details>
|
||||
}
|
||||
@deleteRecording(u.ID)
|
||||
</div>
|
||||
default:
|
||||
<div
|
||||
@@ -116,6 +133,20 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
|
||||
}
|
||||
}
|
||||
|
||||
// deleteRecording is the confirm-then-delete control, shown only once a recording has
|
||||
// finished (done or failed): deleting one that is still being processed is refused.
|
||||
templ deleteRecording(id string) {
|
||||
<details class="confirm-delete">
|
||||
<summary class="btn-danger">Delete this recording…</summary>
|
||||
<div class="confirm-body">
|
||||
<p>This permanently deletes the recording and its transcript. Are you sure?</p>
|
||||
<form method="post" action={ templ.SafeURL("/uploads/" + id + "/delete") }>
|
||||
<button type="submit" class="btn-danger">Yes, permanently delete it</button>
|
||||
</form>
|
||||
</div>
|
||||
</details>
|
||||
}
|
||||
|
||||
// UploadChatPage is the no-JS full page for chatting with an uploaded transcript:
|
||||
// the same chat panel as a video's (ADR-027), scoped to this recording.
|
||||
templ UploadChatPage(u domain.UploadedTranscript, v chatView) {
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
package web_test
|
||||
|
||||
// A transcript produced by the cloud fallback means this recording's audio left the
|
||||
// homelab, which the reader is told in plain words (#52). "Local" is config.IsLocalModel:
|
||||
// one definition, not a second string match in the view.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func setEndpoint(t *testing.T, id, endpoint string) {
|
||||
t.Helper()
|
||||
_, err := rawPool(t).Exec(context.Background(), `UPDATE uploaded_transcripts SET endpoint = $2 WHERE id = $1`, id, endpoint)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
func TestUploadStatus_ACloudServedTranscriptSaysTheAudioLeftTheHomelab(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
||||
setEndpoint(t, id, "berget/kb-whisper")
|
||||
|
||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||
|
||||
assert.Contains(t, b, "left the homelab")
|
||||
assert.Contains(t, b, "cloud service (berget)")
|
||||
}
|
||||
|
||||
func TestUploadStatus_ALocallyServedTranscriptCarriesNoCloudNote(t *testing.T) {
|
||||
for _, endpoint := range []string{"iguana/kb-whisper", "koala/whisper", ""} {
|
||||
t.Run(endpoint, func(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
||||
setEndpoint(t, id, endpoint)
|
||||
|
||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||
|
||||
assert.NotContains(t, b, "left the homelab")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadStatus_AnEndpointThatOnlyLooksLocalIsStillNamedAsCloud(t *testing.T) {
|
||||
// The prefix alone must not decide it: IsLocalModel refuses path tricks.
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
||||
setEndpoint(t, id, "iguana/../berget/kb-whisper")
|
||||
|
||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||
|
||||
assert.Contains(t, b, "left the homelab")
|
||||
}
|
||||
|
||||
// The two-engine path (ADR-034) writes one composite endpoint string. Every name in it is
|
||||
// a local model, so the audio never left the homelab and the note must not say it did.
|
||||
func TestUploadStatus_AReconciledAllLocalEndpointCarriesNoCloudNote(t *testing.T) {
|
||||
for _, endpoint := range []string{
|
||||
"iguana/kb-whisper + koala/whisper, reconciled by iguana/gemma4-26b",
|
||||
"iguana/kb-whisper + koala/whisper, reconciled by iguana/gemma4-26b\n",
|
||||
} {
|
||||
t.Run(endpoint, func(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
||||
setEndpoint(t, id, endpoint)
|
||||
|
||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||
|
||||
assert.NotContains(t, b, "left the homelab")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadStatus_AReconciledEndpointNamingACloudEngineStillNamesTheCloudProvider(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
||||
setEndpoint(t, id, "iguana/kb-whisper + berget/kb-whisper, reconciled by iguana/gemma4-26b")
|
||||
|
||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||
|
||||
assert.Contains(t, b, "cloud service (berget)")
|
||||
}
|
||||
|
||||
// An alias without a provider prefix is not a local model, wherever it sits in the
|
||||
// composite: it must never be dropped from the judgement (PR #58 re-review).
|
||||
func TestUploadStatus_AReconciledEndpointWithAnUnprefixedAliasIsNotCalledLocal(t *testing.T) {
|
||||
for _, endpoint := range []string{
|
||||
"iguana/kb-whisper + whisper-1, reconciled by iguana/gemma4-26b",
|
||||
"iguana/kb-whisper + whisper 1, reconciled by iguana/gemma4-26b",
|
||||
"iguana/kb-whisper + berget / x, reconciled by iguana/gemma4-26b",
|
||||
} {
|
||||
t.Run(endpoint, func(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
||||
setEndpoint(t, id, endpoint)
|
||||
|
||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||
|
||||
assert.Contains(t, b, "left the homelab")
|
||||
assert.NotContains(t, b, "cloud service ()", "the note names a provider")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// "by", "reconciled" and "+" are connectors only inside the composite format. As a whole
|
||||
// endpoint, or as an alias within the composite, they are model names and are judged.
|
||||
func TestUploadStatus_AConnectorWordUsedAsAModelNameIsStillJudged(t *testing.T) {
|
||||
for _, endpoint := range []string{
|
||||
"by",
|
||||
"reconciled",
|
||||
"+",
|
||||
"iguana/kb-whisper + by, reconciled by iguana/gemma4-26b",
|
||||
"iguana/kb-whisper + reconciled, reconciled by iguana/gemma4-26b",
|
||||
"iguana/kb-whisper + +, reconciled by iguana/gemma4-26b",
|
||||
"/x", // no provider at all: the note must still name something
|
||||
} {
|
||||
t.Run(endpoint, func(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
||||
setEndpoint(t, id, endpoint)
|
||||
|
||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||
|
||||
assert.Contains(t, b, "left the homelab")
|
||||
assert.NotContains(t, b, "cloud service ()")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadStatus_AReconcilerThatIsNotLocalNamesTheCloudProvider(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
||||
setEndpoint(t, id, "iguana/kb-whisper + koala/whisper, reconciled by berget/mistral-large")
|
||||
|
||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||
|
||||
assert.Contains(t, b, "cloud service (berget)")
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package web_test
|
||||
|
||||
// Deleting one recording (#37). Owner-scoped like every other upload route: another user's
|
||||
// id is a 404 and the row survives; an upload still being processed is refused.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func deleteReq(id string) *http.Request {
|
||||
return httptest.NewRequest(http.MethodPost, "/uploads/"+id+"/delete", nil)
|
||||
}
|
||||
|
||||
func uploadExists(t *testing.T, id string) bool {
|
||||
t.Helper()
|
||||
var n int
|
||||
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT count(*) FROM uploaded_transcripts WHERE id = $1`, id).Scan(&n))
|
||||
return n == 1
|
||||
}
|
||||
|
||||
func TestUploadDelete_AnOwnerRemovesAFinishedOrFailedRecordingAndLandsOnTheList(t *testing.T) {
|
||||
for _, status := range []string{"done", "failed"} {
|
||||
t.Run(status, func(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "Roadmap sync", status, "transcription_failed")
|
||||
|
||||
rec := do(t, app, deleteReq(id))
|
||||
|
||||
assert.Equal(t, http.StatusSeeOther, rec.Code)
|
||||
assert.Equal(t, "/uploads", rec.Header().Get("Location"))
|
||||
assert.False(t, uploadExists(t, id))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadDelete_AnotherUsersRecordingIs404AndSurvives(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
theirs := seedUpload(t, otherUserID, "Their private meeting", "done", meetingText)
|
||||
|
||||
rec := do(t, app, deleteReq(theirs))
|
||||
|
||||
assert.Equal(t, http.StatusNotFound, rec.Code)
|
||||
assert.True(t, uploadExists(t, theirs), "a non-owner's delete must not remove the row")
|
||||
}
|
||||
|
||||
func TestUploadDelete_ARecordingStillBeingTranscribedIsRefusedWith409AndKept(t *testing.T) {
|
||||
for _, status := range []string{"queued", "transcribing"} {
|
||||
t.Run(status, func(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", status, "")
|
||||
|
||||
rec := do(t, app, deleteReq(id))
|
||||
|
||||
assert.Equal(t, http.StatusConflict, rec.Code)
|
||||
assert.Contains(t, body(t, rec), "still being transcribed")
|
||||
assert.True(t, uploadExists(t, id))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadDelete_ANonIDIsA404NotAServerError(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
|
||||
assert.Equal(t, http.StatusNotFound, do(t, app, deleteReq("not-a-uuid")).Code)
|
||||
}
|
||||
|
||||
func TestUploadPage_OffersDeleteOnlyForAFinishedOrFailedRecording(t *testing.T) {
|
||||
for status, want := range map[string]bool{"done": true, "failed": true, "queued": false, "transcribing": false} {
|
||||
t.Run(status, func(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", status, "transcription_failed")
|
||||
|
||||
b := body(t, do(t, app, get("/uploads/"+id)))
|
||||
|
||||
assert.Equal(t, want, strings.Contains(b, `action="/uploads/`+id+`/delete"`))
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -14,6 +14,8 @@ import (
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||
)
|
||||
|
||||
// A client that hangs up in the middle of an upload is routine, not a server fault. The
|
||||
@@ -85,7 +87,7 @@ func TestUploadPost_AClientThatHangsUpAnywhereInTheBodyIsLoggedAtInfoAndNeverAsA
|
||||
// context is cancelled, then Accept fails the way a read from a dead connection does.
|
||||
type goneMidAccept struct{ cancel context.CancelFunc }
|
||||
|
||||
func (g goneMidAccept) Accept(context.Context, string, string, io.Reader) (string, error) {
|
||||
func (g goneMidAccept) Accept(context.Context, string, string, domain.Language, io.Reader) (string, error) {
|
||||
g.cancel()
|
||||
return "", fmt.Errorf("receive: %w", io.ErrClosedPipe)
|
||||
}
|
||||
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||
"git.d-ma.be/mathias/tapir/internal/uploads"
|
||||
"git.d-ma.be/mathias/tapir/internal/web"
|
||||
)
|
||||
@@ -132,7 +133,7 @@ func TestUploadPost_AcceptedAudioQueuesARowAndRedirectsToIt(t *testing.T) {
|
||||
assert.Equal(t, "Sprint review", title)
|
||||
assert.Equal(t, "/uploads/"+id, rec.Header().Get("Location"))
|
||||
require.Len(t, sub.jobs, 1)
|
||||
assert.Equal(t, uploads.Job{UserID: userID, UploadID: id, Path: sub.jobs[0].Path, Duration: audioOK.Duration}, sub.jobs[0])
|
||||
assert.Equal(t, uploads.Job{UserID: userID, UploadID: id, Path: sub.jobs[0].Path, Language: domain.LangSwedish, Duration: audioOK.Duration}, sub.jobs[0])
|
||||
spooled, err := os.ReadFile(sub.jobs[0].Path)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "synthetic audio bytes", string(spooled))
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
package web_test
|
||||
|
||||
// #56: a request body that stops arriving must not hold a receive slot forever. The bound is a
|
||||
// read deadline on the connection, re-armed on every read, so it only exercises over a real
|
||||
// socket and through the production handler stack (metrics.HTTPMiddleware wraps the writer).
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"mime/multipart"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"git.d-ma.be/mathias/tapir/internal/metrics"
|
||||
"git.d-ma.be/mathias/tapir/internal/uploads"
|
||||
"git.d-ma.be/mathias/tapir/internal/web"
|
||||
)
|
||||
|
||||
func multipartBody(t *testing.T, content []byte) (body []byte, contentType string) {
|
||||
t.Helper()
|
||||
var buf bytes.Buffer
|
||||
mw := multipart.NewWriter(&buf)
|
||||
fw, err := mw.CreateFormFile("file", "a.m4a")
|
||||
require.NoError(t, err)
|
||||
_, err = fw.Write(content)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, mw.Close())
|
||||
return buf.Bytes(), mw.FormDataContentType()
|
||||
}
|
||||
|
||||
// rawUpload opens a connection and sends the request headers; the caller feeds the body.
|
||||
func rawUpload(t *testing.T, addr, contentType string, length int) net.Conn {
|
||||
t.Helper()
|
||||
c, err := net.Dial("tcp", addr)
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { _ = c.Close() })
|
||||
_, err = fmt.Fprintf(c, "POST /uploads HTTP/1.1\r\nHost: x\r\nContent-Type: %s\r\nContent-Length: %d\r\n\r\n", contentType, length)
|
||||
require.NoError(t, err)
|
||||
return c
|
||||
}
|
||||
|
||||
// serverEnded reports whether the server ended the request (answered it or closed the
|
||||
// connection) before wait elapsed, rather than leaving the client waiting.
|
||||
func serverEnded(t *testing.T, c net.Conn, wait time.Duration) bool {
|
||||
t.Helper()
|
||||
require.NoError(t, c.SetReadDeadline(time.Now().Add(wait)))
|
||||
_, err := http.ReadResponse(bufio.NewReader(c), nil)
|
||||
var ne net.Error
|
||||
return !errors.As(err, &ne) || !ne.Timeout()
|
||||
}
|
||||
|
||||
// readStatus returns the response status code, or 0 if none arrived before wait elapsed.
|
||||
func readStatus(t *testing.T, c net.Conn, wait time.Duration) int {
|
||||
t.Helper()
|
||||
require.NoError(t, c.SetReadDeadline(time.Now().Add(wait)))
|
||||
resp, err := http.ReadResponse(bufio.NewReader(c), nil)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
_ = resp.Body.Close()
|
||||
return resp.StatusCode
|
||||
}
|
||||
|
||||
func newIdleBoundServer(t *testing.T, idle time.Duration) (addr string) {
|
||||
t.Helper()
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
app.UploadIdleTimeout = idle
|
||||
app.Uploads.(*uploads.Intake).MaxReceivesPerUser = 1
|
||||
srv := httptest.NewServer(metrics.HTTPMiddleware(app.Router()))
|
||||
t.Cleanup(srv.Close)
|
||||
return strings.TrimPrefix(srv.URL, "http://")
|
||||
}
|
||||
|
||||
func TestUploadPost_AConnectionReadDeadlineEndsABodyThatStallsAndReleasesItsReceiveSlot(t *testing.T) {
|
||||
addr := newIdleBoundServer(t, 200*time.Millisecond)
|
||||
body, ct := multipartBody(t, bytes.Repeat([]byte("a"), 8000))
|
||||
|
||||
stalled := rawUpload(t, addr, ct, len(body))
|
||||
_, err := stalled.Write(body[:4000]) // half the body, then nothing more
|
||||
require.NoError(t, err)
|
||||
|
||||
require.True(t, serverEnded(t, stalled, 3*time.Second),
|
||||
"the server never ended the stalled request: it still holds its slot")
|
||||
|
||||
next := rawUpload(t, addr, ct, len(body))
|
||||
_, err = next.Write(body)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, http.StatusSeeOther, readStatus(t, next, 3*time.Second),
|
||||
"the same user's next upload gets the slot the stalled one held")
|
||||
}
|
||||
|
||||
func TestUploadPost_TheReadDeadlineIsReArmedOnEveryReadSoASlowButMovingBodyIsAccepted(t *testing.T) {
|
||||
addr := newIdleBoundServer(t, 300*time.Millisecond)
|
||||
body, ct := multipartBody(t, bytes.Repeat([]byte("a"), 2000))
|
||||
|
||||
c := rawUpload(t, addr, ct, len(body))
|
||||
// ~1.2 s in total, four times the idle bound, but never 300 ms without a byte.
|
||||
const chunks = 20
|
||||
step := len(body)/chunks + 1
|
||||
for i := 0; i < len(body); i += step {
|
||||
_, err := c.Write(body[i:min(i+step, len(body))])
|
||||
require.NoError(t, err)
|
||||
time.Sleep(60 * time.Millisecond)
|
||||
}
|
||||
|
||||
require.Equal(t, http.StatusSeeOther, readStatus(t, c, 3*time.Second))
|
||||
}
|
||||
|
||||
// cmd/tapir never sets UploadIdleTimeout, so the default is the value production runs.
|
||||
// Shorter and legitimate uploads on a slow link are cut; much longer and the bound is moot.
|
||||
func TestDefaultUploadIdleTimeoutIsSixtySeconds(t *testing.T) {
|
||||
require.Equal(t, 60*time.Second, web.DefaultUploadIdleTimeout)
|
||||
}
|
||||
|
||||
// The bound an App actually resolves to, not the constant: production's App leaves
|
||||
// UploadIdleTimeout unset, so the zero value must resolve to exactly 60 s, and a set value
|
||||
// must be used as given.
|
||||
func TestAppUploadIdleTimeout_ResolvesUnsetToSixtySecondsAndHonoursASetValue(t *testing.T) {
|
||||
require.Equal(t, 60*time.Second, (&web.App{}).UploadIdleTimeoutForTest())
|
||||
require.Equal(t, 60*time.Second, (&web.App{UploadIdleTimeout: -time.Second}).UploadIdleTimeoutForTest())
|
||||
require.Equal(t, 300*time.Millisecond, (&web.App{UploadIdleTimeout: 300 * time.Millisecond}).UploadIdleTimeoutForTest())
|
||||
}
|
||||
|
||||
// With UploadIdleTimeout unset, a body with 100 ms gaps between reads is not cut (a resolved
|
||||
// bound in milliseconds or nanoseconds would cut it). The exact value is pinned above.
|
||||
func TestUploadPost_AnUnsetIdleTimeoutDoesNotCutABodyWithSubSecondGaps(t *testing.T) {
|
||||
addr := newIdleBoundServer(t, 0)
|
||||
body, ct := multipartBody(t, bytes.Repeat([]byte("a"), 1000))
|
||||
|
||||
c := rawUpload(t, addr, ct, len(body))
|
||||
const chunks = 5
|
||||
step := len(body)/chunks + 1
|
||||
for i := 0; i < len(body); i += step {
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
_, err := c.Write(body[i:min(i+step, len(body))])
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
require.Equal(t, http.StatusSeeOther, readStatus(t, c, 3*time.Second))
|
||||
}
|
||||
|
||||
// A writer that cannot set a read deadline leaves the body unbounded; that must be said in
|
||||
// the log, not happen silently. The in-process recorder is such a writer.
|
||||
func TestUploadPost_AWriterWithoutReadDeadlineSupportLogsThatTheIdleBoundIsUnavailable(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
buf := &lockedBuf{}
|
||||
app.Log = slog.New(slog.NewTextHandler(buf, nil))
|
||||
|
||||
do(t, app, uploadReq(t, "t", "a.m4a", []byte("x")))
|
||||
|
||||
require.Contains(t, buf.String(), "upload idle bound unavailable")
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
package web_test
|
||||
|
||||
// The language of a recording is chosen on the upload form (tapir#61). These drive the real
|
||||
// Router, Intake and Postgres store; the submitter is the process boundary and records the
|
||||
// job the worker would receive.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||
)
|
||||
|
||||
// langReq builds an upload whose parts come in the order given: "title", "language" and
|
||||
// "file" (the file part always carries the same synthetic bytes).
|
||||
func langReq(t *testing.T, order []string, language string) *http.Request {
|
||||
t.Helper()
|
||||
var buf bytes.Buffer
|
||||
mw := multipart.NewWriter(&buf)
|
||||
for _, part := range order {
|
||||
switch part {
|
||||
case "title":
|
||||
require.NoError(t, mw.WriteField("title", "m"))
|
||||
case "language":
|
||||
require.NoError(t, mw.WriteField("language", language))
|
||||
case "file":
|
||||
fw, err := mw.CreateFormFile("file", "m.wav")
|
||||
require.NoError(t, err)
|
||||
_, err = fw.Write([]byte("RIFF....WAVE"))
|
||||
require.NoError(t, err)
|
||||
}
|
||||
}
|
||||
require.NoError(t, mw.Close())
|
||||
req := httptest.NewRequest(http.MethodPost, "/uploads", &buf)
|
||||
req.Header.Set("Content-Type", mw.FormDataContentType())
|
||||
return req
|
||||
}
|
||||
|
||||
func rowLanguage(t *testing.T, location string) domain.Language {
|
||||
t.Helper()
|
||||
id := strings.TrimPrefix(location, "/uploads/")
|
||||
got, err := newStore(t).GetUpload(context.Background(), userID, id)
|
||||
require.NoError(t, err)
|
||||
return got.Language
|
||||
}
|
||||
|
||||
func TestUploadPost_TheLanguageFieldReachesTheJobAndTheRow(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
language string
|
||||
want domain.Language
|
||||
}{
|
||||
{"other", "auto", domain.LangAuto},
|
||||
{"swedish", "sv", domain.LangSwedish},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
app, sub := newUploadApp(t, audioOK, 200<<20)
|
||||
|
||||
rec := do(t, app, langReq(t, []string{"title", "language", "file"}, tt.language))
|
||||
|
||||
require.Equal(t, http.StatusSeeOther, rec.Code)
|
||||
require.Len(t, sub.jobs, 1)
|
||||
assert.Equal(t, tt.want, sub.jobs[0].Language)
|
||||
assert.Equal(t, tt.want, rowLanguage(t, rec.Header().Get("Location")))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadPost_NoLanguageFieldIsSwedishAsBefore(t *testing.T) {
|
||||
app, sub := newUploadApp(t, audioOK, 200<<20)
|
||||
|
||||
rec := do(t, app, langReq(t, []string{"title", "file"}, ""))
|
||||
|
||||
require.Equal(t, http.StatusSeeOther, rec.Code)
|
||||
require.Len(t, sub.jobs, 1)
|
||||
assert.Equal(t, domain.LangSwedish, sub.jobs[0].Language)
|
||||
assert.Equal(t, domain.LangSwedish, rowLanguage(t, rec.Header().Get("Location")))
|
||||
}
|
||||
|
||||
// A part after the file is never read (the file part ends the stream), so a client that
|
||||
// orders its fields wrongly gets Swedish, the old behaviour, not a guess.
|
||||
func TestUploadPost_ALanguageFieldAfterTheFileIsNotReadAndTheUploadIsSwedish(t *testing.T) {
|
||||
app, sub := newUploadApp(t, audioOK, 200<<20)
|
||||
|
||||
rec := do(t, app, langReq(t, []string{"title", "file", "language"}, "auto"))
|
||||
|
||||
require.Equal(t, http.StatusSeeOther, rec.Code)
|
||||
require.Len(t, sub.jobs, 1)
|
||||
assert.Equal(t, domain.LangSwedish, sub.jobs[0].Language)
|
||||
}
|
||||
|
||||
func TestUploadPost_ALanguageOutsideTheClosedSetIsRefusedBeforeAnythingIsSpooledOrStored(t *testing.T) {
|
||||
for _, bad := range []string{"en", "SV", " sv", "sv,auto", "../auto", strings.Repeat("a", 5000)} {
|
||||
t.Run(bad[:min(len(bad), 12)], func(t *testing.T) {
|
||||
app, sub := newUploadApp(t, audioOK, 200<<20)
|
||||
|
||||
rec := do(t, app, langReq(t, []string{"title", "language", "file"}, bad))
|
||||
|
||||
assert.Equal(t, http.StatusBadRequest, rec.Code)
|
||||
assert.Empty(t, sub.jobs, "a refused post must not enqueue transcription")
|
||||
rows, err := newStore(t).ListUploads(context.Background(), userID)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, rows, "and must leave no row")
|
||||
assert.Contains(t, body(t, rec), "Choose a language from the list.")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadPage_TheFormOffersSwedishPreselectedAndOtherBeforeTheFilePart(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
|
||||
b := body(t, do(t, app, get("/uploads")))
|
||||
|
||||
sel := strings.Index(b, `name="language"`)
|
||||
file := strings.Index(b, `name="file"`)
|
||||
require.True(t, sel >= 0, "the form has a language field")
|
||||
assert.True(t, sel < file, "the field must come before the file part: the stream ends at the file")
|
||||
assert.Regexp(t, `<option value="sv" selected`, b, "Swedish is preselected: today's behaviour")
|
||||
assert.Contains(t, b, `<option value="auto"`)
|
||||
}
|
||||
|
||||
func TestUploadPage_NamesTheLanguageInEveryState(t *testing.T) {
|
||||
for _, status := range []string{"queued", "transcribing", "done", "failed"} {
|
||||
for _, tt := range []struct {
|
||||
language domain.Language
|
||||
want string
|
||||
}{
|
||||
{domain.LangSwedish, "Swedish"},
|
||||
{domain.LangAuto, "Other (detected by the engine)"},
|
||||
} {
|
||||
t.Run(status+"/"+string(tt.language), func(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", status, "Synthetic transcript text.")
|
||||
_, err := rawPool(t).Exec(context.Background(), `UPDATE uploaded_transcripts SET language = $2 WHERE id = $1`, id, string(tt.language))
|
||||
require.NoError(t, err)
|
||||
|
||||
b := body(t, do(t, app, get("/uploads/"+id)))
|
||||
|
||||
assert.Contains(t, b, "Language: "+tt.want)
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -13,6 +13,8 @@ import (
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||
)
|
||||
|
||||
// Error paths of the upload routes log a kind, never the error's text, which can carry
|
||||
@@ -37,7 +39,7 @@ func (l *lockedBuf) String() string {
|
||||
|
||||
type failingUploader struct{ err error }
|
||||
|
||||
func (f failingUploader) Accept(context.Context, string, string, io.Reader) (string, error) {
|
||||
func (f failingUploader) Accept(context.Context, string, string, domain.Language, io.Reader) (string, error) {
|
||||
return "", f.err
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
package web_test
|
||||
|
||||
// /uploads was mounted but nothing linked to it (#37). The shared Layout has no access to
|
||||
// the App, so the entry point is the home list, shown only when uploads are mounted.
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
func TestHomeList_LinksToRecordingsOnlyWhenUploadsAreMounted(t *testing.T) {
|
||||
with, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
without := newApp(t)
|
||||
|
||||
assert.Contains(t, body(t, do(t, with, get("/"))), `href="/uploads"`)
|
||||
assert.NotContains(t, body(t, do(t, without, get("/"))), `href="/uploads"`,
|
||||
"a link to a route that is not mounted would be a dead end")
|
||||
}
|
||||
+464
-376
File diff suppressed because it is too large
Load Diff
@@ -107,9 +107,12 @@ templ flashBanner(code string) {
|
||||
// #summary-list region; a non-HTMX request renders the whole page. flash carries
|
||||
// a one-shot notification (e.g. "connected", "registered") surfaced on arrival
|
||||
// after a POST→redirect.
|
||||
templ ListPage(b listBuckets, f Filter, stats PipelineStats, flash string, hasConnected bool, channels []string, autoSummarize bool) {
|
||||
templ ListPage(b listBuckets, f Filter, stats PipelineStats, flash string, hasConnected bool, channels []string, autoSummarize bool, uploadsEnabled bool) {
|
||||
@Layout("Tapir — Summaries") {
|
||||
@flashBanner(flash)
|
||||
if uploadsEnabled {
|
||||
<p class="muted"><a href="/uploads">Recordings</a>: upload audio and get a private transcript.</p>
|
||||
}
|
||||
if hasConnected {
|
||||
@pasteForm()
|
||||
}
|
||||
|
||||
+418
-408
File diff suppressed because it is too large
Load Diff
@@ -122,6 +122,16 @@ var scenarioCoverage = map[string]string{
|
||||
"I can ask about my recording and get a grounded answer": "TestUploadChatPost_SendsTheOwnersTranscriptWithTheUploadBudget",
|
||||
"A long recording is not cut to its first third": "TestAnswerMaxCharsOverridesTheServiceCap",
|
||||
|
||||
"A transcript made by the cloud fallback says the audio left the homelab": "TestUploadStatus_ACloudServedTranscriptSaysTheAudioLeftTheHomelab",
|
||||
"I choose the language of a recording": "TestUploadPost_TheLanguageFieldReachesTheJobAndTheRow",
|
||||
"Swedish stays the default": "TestUploadPost_NoLanguageFieldIsSwedishAsBefore",
|
||||
"A language outside the list is refused": "TestUploadPost_ALanguageOutsideTheClosedSetIsRefusedBeforeAnythingIsSpooledOrStored",
|
||||
"A recording in another language never reaches the Swedish-only engine": "TestEnsemble_AnAutoLanguageUploadNeverReachesTheBaseEngineAndIsNotReconciled",
|
||||
|
||||
"I can delete a finished recording": "TestUploadDelete_AnOwnerRemovesAFinishedOrFailedRecordingAndLandsOnTheList",
|
||||
"A recording still being transcribed cannot be deleted": "TestUploadDelete_ARecordingStillBeingTranscribedIsRefusedWith409AndKept",
|
||||
"The home list links to my recordings when uploads are on": "TestHomeList_LinksToRecordingsOnlyWhenUploadsAreMounted",
|
||||
|
||||
// audio_upload.feature, ADR-034
|
||||
"Two engines transcribe the audio and their disagreements are reconciled": "TestEnsemble_BothEnginesRunInOrderAndTheirDisagreementIsReconciled",
|
||||
"Trivial disagreements never reach the model": "TestReconcile_TrivialDisputesAreResolvedWithoutAskingTheModel",
|
||||
|
||||
Reference in New Issue
Block a user