Compare commits
167
Commits
@@ -127,9 +127,6 @@ The repo is **green and shipping** — last tag `v0.15.0`. `task check` passes (
|
|||||||
implement `uploads.Planner` to declare its own phases. The page tells running / quiet / lost from
|
implement `uploads.Planner` to declare its own phases. The page tells running / quiet / lost from
|
||||||
two timestamps (`alive_at` heartbeat vs `progress_at`) and never shows a percentage that does not
|
two timestamps (`alive_at` heartbeat vs `progress_at`) and never shows a percentage that does not
|
||||||
come from real counts. **Nothing logged may carry transcript text, words from it, or the spool path.**
|
come from real counts. **Nothing logged may carry transcript text, words from it, or the spool path.**
|
||||||
Errors reach a log, a row or a page as `errkind.Of(err)` (a kind such as `timeout` or `http_502`),
|
|
||||||
never `err.Error()`: error text can hold a provider's response body, the spool path or transcript
|
|
||||||
words (ADR-035 decision 8). Local-only for anything that carries transcript text: `config.IsLocalModel`.
|
|
||||||
`TAPIR_SHOT_DIR=… go test -run TestRenderUploadProgressStatesForReview ./internal/web` renders every
|
`TAPIR_SHOT_DIR=… go test -run TestRenderUploadProgressStatesForReview ./internal/web` renders every
|
||||||
state for a screenshot review.
|
state for a screenshot review.
|
||||||
**Lint caveat:** `task check` does not fail on lint (#41: the Taskfile `|| echo` swallows
|
**Lint caveat:** `task check` does not fail on lint (#41: the Taskfile `|| echo` swallows
|
||||||
|
|||||||
+8
-101
@@ -1434,12 +1434,7 @@ an hour of Swedish *was* an unknown here; it is now measured, see decision 8.)
|
|||||||
empty disables), recorded per row, and called out in Open items rather than defaulted
|
empty disables), recorded per row, and called out in Open items rather than defaulted
|
||||||
silently.
|
silently.
|
||||||
- Swedish only (`language=sv`). Language detection and routing (#28 decision 2) are not
|
- Swedish only (`language=sv`). Language detection and routing (#28 decision 2) are not
|
||||||
touched. **Amended 2026-10-04 (#61, interim):** the language is now a per-upload choice on
|
touched.
|
||||||
the form, `sv` (preselected, and what an upload that names none gets) or `auto` ("Other":
|
|
||||||
`language` is omitted so the engine detects it), a closed set held by the form, the request
|
|
||||||
code and a CHECK constraint on `uploaded_transcripts.language` (migration 021). Automatic
|
|
||||||
detection and a general-purpose engine for other languages are still not built (#61); an
|
|
||||||
`auto` upload goes to the same configured engine, which is not tuned for it.
|
|
||||||
- `-H 'Expect:'`, multipart field `file` — encoded in the adapter and its tests, not rediscovered.
|
- `-H 'Expect:'`, multipart field `file` — encoded in the adapter and its tests, not rediscovered.
|
||||||
|
|
||||||
7. **Jobs run in-process, one at a time (ADR-018 stands).** Transcription is minutes (~7.5 min
|
7. **Jobs run in-process, one at a time (ADR-018 stands).** Transcription is minutes (~7.5 min
|
||||||
@@ -1452,25 +1447,6 @@ an hour of Swedish *was* an unknown here; it is now measured, see decision 8.)
|
|||||||
re-uploads. The single-replica assumption gains a second dependent (the semaphore is
|
re-uploads. The single-replica assumption gains a second dependent (the semaphore is
|
||||||
process-local) and is recorded here.
|
process-local) and is recorded here.
|
||||||
|
|
||||||
*Amended 2026-10-03 (#50, #47).* The process now handles SIGTERM (what Kubernetes sends on
|
|
||||||
every deploy; only SIGINT was handled, so shutdown code never ran). On shutdown the running job
|
|
||||||
is recorded `interrupted` at once and the process waits, bounded to 10 s each, for in-flight
|
|
||||||
requests and for that write; the start-up sweep stays the backstop for a kill. **Known
|
|
||||||
limitation, deliberately not built: a deploy still loses a job in flight** (finishing it needs
|
|
||||||
a drain of up to the job time, a ~330 s termination grace and `strategy: Recreate`, i.e.
|
|
||||||
downtime on every deploy; a 90-minute file takes ~4.5 min and a deploy lands on a running job
|
|
||||||
perhaps one time in five at current volume). Jobs still queued at shutdown stay `queued` until
|
|
||||||
the next start's sweep. The user re-uploads, with an honest message.
|
|
||||||
|
|
||||||
*Receive limits (#47).* Request bodies are received at most 4 at a time, 2 per user, answered
|
|
||||||
as busy beyond that. Together with the 20-job queue and the running job that bounds the spool
|
|
||||||
at about (4 + 20 + 1) x 200 MiB = 5 GiB of the 6 Gi `emptyDir` (infra): raising any of those or
|
|
||||||
`TAPIR_UPLOAD_MAX_BYTES` must revisit that sum (pinned by `TestNewIntake_Defaults...`, not
|
|
||||||
enforced against infra). **A slow-drip body holds its slot without a time bound**: the server
|
|
||||||
sets `ReadHeaderTimeout` only, because a `ReadTimeout` would cut legitimate 200 MiB uploads on
|
|
||||||
slow links; two slow bodies take a user's two slots and two such users make everyone's upload
|
|
||||||
busy. It needs a valid session; recorded as a limitation with a follow-up issue.
|
|
||||||
|
|
||||||
8. **Chat reuses `chat.Service`; it is not forked.** The uploaded-transcript chat handlers
|
8. **Chat reuses `chat.Service`; it is not forked.** The uploaded-transcript chat handlers
|
||||||
look the row up through the RLS-scoped store, so a non-owner gets 404 by the same mechanism
|
look the row up through the RLS-scoped store, so a non-owner gets 404 by the same mechanism
|
||||||
as ADR-027's summary gate, **before any model is called**. ADR-027's decisions (ephemeral,
|
as ADR-027's summary gate, **before any model is called**. ADR-027's decisions (ephemeral,
|
||||||
@@ -1485,9 +1461,8 @@ an hour of Swedish *was* an unknown here; it is now measured, see decision 8.)
|
|||||||
- **The opening model.** The offered list leads with the summarizer default `koala/phi4-mini`,
|
- **The opening model.** The offered list leads with the summarizer default `koala/phi4-mini`,
|
||||||
whose 8,192-token context returns HTTP 400 on an hour of speech (~13,000 tokens). Chat over
|
whose 8,192-token context returns HTTP 400 on an hour of speech (~13,000 tokens). Chat over
|
||||||
an upload opens on `TAPIR_UPLOAD_CHAT_MODEL` (default `iguana/gemma4-26b`); the switcher
|
an upload opens on `TAPIR_UPLOAD_CHAT_MODEL` (default `iguana/gemma4-26b`); the switcher
|
||||||
lists **local models only** (amended by ADR-035 after the adversarial review found it listed
|
still lists every model, and picking `phi4-mini` on a long recording fails loudly and
|
||||||
the cloud model too: the whole private transcript goes to the one chosen), and picking
|
inline. A per-model budget is not built.
|
||||||
`phi4-mini` on a long recording fails loudly and inline. A per-model budget is not built.
|
|
||||||
Past ~90,000 chars (≈100 minutes) no offered model can answer by stuffing; see triggers.
|
Past ~90,000 chars (≈100 minutes) no offered model can answer by stuffing; see triggers.
|
||||||
|
|
||||||
9. **Not decided here.** Rich `AVSNITT`/`ALTERNATIV` analysis (gated on #30 — `analyze_srt.py`
|
9. **Not decided here.** Rich `AVSNITT`/`ALTERNATIV` analysis (gated on #30 — `analyze_srt.py`
|
||||||
@@ -1538,32 +1513,19 @@ deferred. The scope cut — chat first, no rich analysis, no video — is what k
|
|||||||
ADR-016 window, it was the interesting thing, not the useful one — remove it.
|
ADR-016 window, it was the interesting thing, not the useful one — remove it.
|
||||||
|
|
||||||
**Open items this ADR creates.**
|
**Open items this ADR creates.**
|
||||||
- **DECIDED (Mathias, 2026-10-02): option A.** Static ffmpeg and ffprobe 7.0.2 are copied into the
|
- **DECISION NEEDED — ffmpeg in the runtime image.** *Options:* (A) static ffmpeg/ffprobe copied
|
||||||
distroless image, pinned by version and sha256 (Dockerfile in this PR; #34). Measured at that time:
|
|
||||||
**about 160 MB** added, not the 70 MB first estimated (80 MB per binary); the build is **GPLv3**;
|
|
||||||
the checksum proves the bytes reviewed, not the publisher; upstream's `releases/` URL moves on, so
|
|
||||||
vendoring the verified tarball is the durable fix (not done). Uploads stay off until the deployment
|
|
||||||
sets `TAPIR_UPLOADS_ENABLED`. *The question as it was put:* *Options:* (A) static ffmpeg/ffprobe copied
|
|
||||||
into distroless **(recommended — the only option that satisfies "accept m4a/mp3/opus/mp4,
|
into distroless **(recommended — the only option that satisfies "accept m4a/mp3/opus/mp4,
|
||||||
validate duration, reject video"; pure-Go decoders cover wav/mp3 only)**; (B) accept WAV
|
validate duration, reject video"; pure-Go decoders cover wav/mp3 only)**; (B) accept WAV
|
||||||
only, convert client-side (fails "I just want to upload it"); (C) a transcode sidecar
|
only, convert client-side (fails "I just want to upload it"); (C) a transcode sidecar
|
||||||
(second deployable, ADR-018's reasoning against). *Blocks:* the Dockerfile slice, not the
|
(second deployable, ADR-018's reasoning against). *Blocks:* the Dockerfile slice, not the
|
||||||
tests, which use the port.
|
tests, which use the port.
|
||||||
- **DECIDED (Mathias, 2026-10-02): option A, enabled by default (#35).** Scope: the audio
|
- **DECISION NEEDED — berget fallback for private audio.** *Options:* (A) enabled by default as
|
||||||
transcription fallback only. Reconciliation text and chat over a recording never go to a cloud
|
|
||||||
model (ADR-034, ADR-035). *The question as it was put:* *Options:* (A) enabled by default as
|
|
||||||
#33 specifies, with the serving endpoint shown **(recommended, matches the issue)**; (B)
|
#33 specifies, with the serving endpoint shown **(recommended, matches the issue)**; (B)
|
||||||
disabled by default, a failed iguana job fails honestly and the user retries. *Blocks:*
|
disabled by default, a failed iguana job fails honestly and the user retries. *Blocks:*
|
||||||
the production default only; the adapter ships either way.
|
the production default only; the adapter ships either way.
|
||||||
- ~~Verify Tapir's ingress `proxy-body-size` against the 200 MB cap (`infra`).~~ **Resolved 2026-10-03:** real unauthenticated bodies of 2, 30, 120 and 210 MiB all pass NPM (TLS, piguard) and Traefik to tapir (a 302 to login); there is no body limit in that range. Uploads were enabled in production the same day, local engines only (infra#520).
|
- Verify Tapir's ingress `proxy-body-size` against the 200 MB cap (`infra`).
|
||||||
- ~~Per-transcript deletion is not built; account deletion cascades.~~ **Built 2026-10-03
|
- Per-transcript deletion is not built; account deletion cascades. A private-content store
|
||||||
(#37):** `POST /uploads/{id}/delete`, owner-scoped (a non-owner gets 404 by the same RLS
|
without a delete button is a gap worth a ticket, not a silent omission.
|
||||||
mechanism), confirm-then-delete on the recording's page. A recording that is **queued or
|
|
||||||
transcribing is refused (409) until it ends**, because the worker would otherwise write to a
|
|
||||||
row that is gone; a job whose service was lost stays undeletable until the next start's sweep
|
|
||||||
marks it failed. The refusal reflects the row's state when the delete was attempted: a
|
|
||||||
recording that finishes in the same instant still gets a 409 once, and a retry deletes it.
|
|
||||||
The home list now links to `/uploads` when uploads are mounted.
|
|
||||||
|
|
||||||
**Reversibility.** Additive: one table (down-migration drops it), one adapter, one upload
|
**Reversibility.** Additive: one table (down-migration drops it), one adapter, one upload
|
||||||
surface, one image change. Removing the upload route removes the feature; `transcripts`, `videos`
|
surface, one image change. Removing the upload route removes the feature; `transcripts`, `videos`
|
||||||
@@ -1650,10 +1612,6 @@ on the slow numbers and is reopened: 31B scored 97% against 92%.
|
|||||||
job timeout rises from 30 to 60 minutes: ~4 min of kb-whisper and ~40 s of pianissimo per hour of
|
job timeout rises from 30 to 60 minutes: ~4 min of kb-whisper and ~40 s of pianissimo per hour of
|
||||||
audio, plus the adjudication.
|
audio, plus the adjudication.
|
||||||
|
|
||||||
**Amended 2026-10-04 (#61, interim):** reconcile is for Swedish. A recording whose language is
|
|
||||||
`auto` is never sent to the base engine (it is Swedish-only) and is not reconciled: the other
|
|
||||||
engine transcribes alone, then the cloud fallback if that fails and the job is live, and the
|
|
||||||
page does not claim a second opinion.
|
|
||||||
8. **Not decided here.** A cloud adjudicator (rejected: private content); tuning the prompt; making
|
8. **Not decided here.** A cloud adjudicator (rejected: private content); tuning the prompt; making
|
||||||
`gemma4-31b` the default; a "reconciling" progress state (the page says "transcribing" for the whole
|
`gemma4-31b` the default; a "reconciling" progress state (the page says "transcribing" for the whole
|
||||||
job); feeding the uncertain spots to chat; whether pianissimo should also be a middle tier in the
|
job); feeding the uncertain spots to chat; whether pianissimo should also be a middle tier in the
|
||||||
@@ -1684,12 +1642,6 @@ recording.
|
|||||||
~30 disputes where the reconciled text differs from pianissimo alone. Keep the feature only if the
|
~30 disputes where the reconciled text differs from pianissimo alone. Keep the feature only if the
|
||||||
reconciled reading is right at least 85% of the time **and** no worse than the better single engine.
|
reconciled reading is right at least 85% of the time **and** no worse than the better single engine.
|
||||||
If not, switch it off (config) and keep the single-engine path.
|
If not, switch it off (config) and keep the single-engine path.
|
||||||
**Decision (Mathias, 2026-10-02), on the first result below: keep it as an opt-in feature**, reading
|
|
||||||
the bar as 95% of decided items, with the second condition to be answered on the next real run.
|
|
||||||
For deployment the reconciler is `iguana/gemma4-31b` (97% against the speaker's answers, about 22
|
|
||||||
minutes for the 63.6-minute recording with Low Power Mode off). A cloud reconciler was raised and is
|
|
||||||
**not adopted**: it would send private transcript text to a cloud API on every upload; recorded as
|
|
||||||
an evaluation question in #45.
|
|
||||||
**First result, 2026-10-02, before merge** (`docs/research/reconcile-blind-check-2026-10-02.md`): 30
|
**First result, 2026-10-02, before merge** (`docs/research/reconcile-blind-check-2026-10-02.md`): 30
|
||||||
random replacement spots on the real 63.6-minute recording, judged blind by Opus from text (not by the
|
random replacement spots on the real 63.6-minute recording, judged blind by Opus from text (not by the
|
||||||
speaker, who delegated it): 21 right, 1 worse than pianissimo, 3 neither reading right, 5 undecidable.
|
speaker, who delegated it): 21 right, 1 worse than pianissimo, 3 neither reading right, 5 undecidable.
|
||||||
@@ -1759,51 +1711,6 @@ wording while only `content` is merged.
|
|||||||
a segment edge), they stay the base's and the result says so in the log. This replaces ADR-034
|
a segment edge), they stay the base's and the result says so in the log. This replaces ADR-034
|
||||||
decision 6's "segments stay untouched".
|
decision 6's "segments stay untouched".
|
||||||
|
|
||||||
8. **Hardening from the adversarial review** (`skills:pr-gatekeeper`, head `7029dd1`: BLOCK; recorded on
|
|
||||||
PR #39). Each finding was reproduced by a failing test first.
|
|
||||||
- **Chat over a recording offers and accepts local models only** (it listed `berget/mistral-small`);
|
|
||||||
`TAPIR_UPLOAD_CHAT_MODEL` must be local at start-up; with no local model, nothing is sent.
|
|
||||||
- **A model alias counts as local only if it is one plain name after `koala/` or `iguana/`**
|
|
||||||
(`koala/../berget/x`, `koala//x`, `koala/x?host=…` were accepted by a bare prefix check).
|
|
||||||
- **An error reaches a log line, the row or the page as a kind** (`timeout`, `canceled`, `http_502`,
|
|
||||||
`network`, `error`), never its text, which can hold a provider's response body, the spool path or
|
|
||||||
a word from the transcript (`internal/errkind`). The stored reconcile reason and an engine's
|
|
||||||
failure reason are kinds; the page no longer shows provider text.
|
|
||||||
- **Every local engine call has its own deadline** (`TAPIR_TRANSCRIBE_ENGINE_TIMEOUT`, default 20
|
|
||||||
minutes: two wedged engines in sequence then leave 20 of the job's 60 minutes for a fallback),
|
|
||||||
so a wedged engine fails while the job lives and the fallback can run; the job's own
|
|
||||||
deadline or a cancel still never reaches the cloud.
|
|
||||||
- **The duration limit is checked against what is converted**, not the file's header: ffmpeg is
|
|
||||||
capped at the limit plus a minute, a longer result fails the job (`audio_too_long`), and a result
|
|
||||||
under 80% of the claimed length and more than two minutes short is **transcribed and noted** on
|
|
||||||
the audio step ("the audio is 20 min long; the file claimed 1 h 19 min"), not refused. The second
|
|
||||||
adversarial review showed a complete variable-bitrate MP3 whose header over-estimated its length
|
|
||||||
by 4x was wrongly refused by the first version of this check; a note keeps a truncated file
|
|
||||||
visible without turning away real recordings.
|
|
||||||
- **The queue is bounded**, 20 waiting in total and 3 per user, and its room is checked before the
|
|
||||||
upload is spooled (not after 200 MB and an ffprobe run); a full queue is a 429 with a typed reason,
|
|
||||||
never a stuck row or a server error.
|
|
||||||
- **The configured upload chat model is callable.** It is added to the shared chat service's model
|
|
||||||
list when uploads are on and it is local, because the service re-maps a model it does not offer
|
|
||||||
(a valid `iguana/gemma4-31b` was being silently swapped for `koala/phi4-mini`). The video chat's own switcher
|
|
||||||
lists that local model too while uploads are on.
|
|
||||||
- **A client that hangs up mid-upload is not an error**: the handler logs INFO and writes nothing
|
|
||||||
(rendering the error page ran a database read on a cancelled context and logged a false ERROR). The
|
|
||||||
400 "cut off" answer is keyed on a marker for errors from *reading the client's body*
|
|
||||||
(`uploads.ErrClientRead`), not on `io.ErrUnexpectedEOF`, which is also what a database drop looks like.
|
|
||||||
- **A note is not hidden**: a finished upload's record opens by default when any step carries a note.
|
|
||||||
- **Bad input is a 4xx**: a non-UUID id is a 404, a malformed or cut-off multipart body is a 400
|
|
||||||
whose text is not logged, and a title with control bytes, bidirectional controls or invalid UTF-8 is
|
|
||||||
cleaned (emoji joiners, Swedish letters and variation selectors are kept), not a 500.
|
|
||||||
- **The image was built and inspected** (buildah on koala, 2026-10-02): the first build FAILED
|
|
||||||
(`tar: xz: Cannot exec`, the golang image has no xz); fixed, rebuilt at 180 MB, ffmpeg and ffprobe
|
|
||||||
run as `nonroot` in distroless. CI does not build the image on a pull request
|
|
||||||
(`if: github.event_name != 'pull_request'`), so the first CI build of it is the merge to main.
|
|
||||||
Not done, filed as follow-ups: a vendored ffmpeg tarball, CSRF cookie review, ffprobe CPU and
|
|
||||||
stream-count limits, mov external references, `alive_at` clock skew, a hung database blocking the
|
|
||||||
tracker, an unbounded gateway response body, a user-facing notice for a recording that is partial but
|
|
||||||
inside the tolerance.
|
|
||||||
|
|
||||||
**Alternatives rejected.** Server-sent events or a websocket: a long-lived connection through the
|
**Alternatives rejected.** Server-sent events or a websocket: a long-lived connection through the
|
||||||
ingress, with buffering to get wrong, for a page that changes every few seconds at best; polling is what
|
ingress, with buffering to get wrong, for a page that changes every few seconds at best; polling is what
|
||||||
the page already does. A percentage from the engine calls: there is no data behind it. A separate
|
the page already does. A percentage from the engine calls: there is no data behind it. A separate
|
||||||
|
|||||||
-40
@@ -16,52 +16,12 @@ COPY . .
|
|||||||
# no libc. Trim symbols/DWARF to shrink the layer.
|
# no libc. Trim symbols/DWARF to shrink the layer.
|
||||||
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags='-s -w' -o /out/tapir ./cmd/tapir
|
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags='-s -w' -o /out/tapir ./cmd/tapir
|
||||||
|
|
||||||
# ── ffmpeg ──────────────────────────────────────────────────────────────────
|
|
||||||
# Audio upload probes files with ffprobe and converts them with ffmpeg (ADR-033,
|
|
||||||
# #34 option A). Static builds, so they run in the distroless image below.
|
|
||||||
#
|
|
||||||
# Pinned by version AND sha256: a changed upstream file fails the build instead of
|
|
||||||
# shipping. The checksum proves these are the bytes that were reviewed (2026-10-02;
|
|
||||||
# the publisher's md5 matched too). It does NOT prove authenticity: the publisher
|
|
||||||
# signs nothing we can check. amd64 only (the cluster is one amd64 node).
|
|
||||||
#
|
|
||||||
# These builds are GPLv3 (--enable-gpl --enable-version3). The service is private and
|
|
||||||
# the image is not distributed; the licence text is copied into the image anyway.
|
|
||||||
#
|
|
||||||
# The release URL moves on when upstream releases a newer version, so the pinned
|
|
||||||
# version is tried in releases/ and then old-releases/. If neither has it the build
|
|
||||||
# fails; vendoring the verified tarball is the durable fix.
|
|
||||||
FROM golang:1.26 AS ffmpeg
|
|
||||||
|
|
||||||
ARG FFMPEG_VERSION=7.0.2
|
|
||||||
ARG FFMPEG_SHA256=abda8d77ce8309141f83ab8edf0596834087c52467f6badf376a6a2a4c87cf67
|
|
||||||
|
|
||||||
# xz is not in the golang image (found by building it: `tar: xz: Cannot exec`).
|
|
||||||
RUN set -eu; \
|
|
||||||
apt-get update; \
|
|
||||||
apt-get install -y --no-install-recommends xz-utils; \
|
|
||||||
rm -rf /var/lib/apt/lists/*; \
|
|
||||||
mkdir -p /out; \
|
|
||||||
for dir in releases old-releases; do \
|
|
||||||
curl -fsSL -o /out/ffmpeg.tar.xz "https://johnvansickle.com/ffmpeg/${dir}/ffmpeg-${FFMPEG_VERSION}-amd64-static.tar.xz" && break; \
|
|
||||||
done; \
|
|
||||||
echo "${FFMPEG_SHA256} /out/ffmpeg.tar.xz" | sha256sum -c -; \
|
|
||||||
tar -xJf /out/ffmpeg.tar.xz -C /out --strip-components=1 --wildcards '*/ffmpeg' '*/ffprobe' '*/GPLv3.txt'; \
|
|
||||||
rm /out/ffmpeg.tar.xz
|
|
||||||
|
|
||||||
# ── runtime ─────────────────────────────────────────────────────────────────
|
# ── runtime ─────────────────────────────────────────────────────────────────
|
||||||
# distroless static + nonroot: no shell, no package manager, runs as uid 65532.
|
# distroless static + nonroot: no shell, no package manager, runs as uid 65532.
|
||||||
# ca-certificates are bundled, which the OIDC/HTTPS clients need.
|
# ca-certificates are bundled, which the OIDC/HTTPS clients need.
|
||||||
FROM gcr.io/distroless/static-debian12:nonroot
|
FROM gcr.io/distroless/static-debian12:nonroot
|
||||||
|
|
||||||
COPY --from=build /out/tapir /tapir
|
COPY --from=build /out/tapir /tapir
|
||||||
COPY --from=ffmpeg /out/ffmpeg /out/ffprobe /usr/local/bin/
|
|
||||||
COPY --from=ffmpeg /out/GPLv3.txt /usr/share/doc/ffmpeg/GPLv3.txt
|
|
||||||
|
|
||||||
# Where the service finds them. Uploads stay OFF until TAPIR_UPLOADS_ENABLED is set
|
|
||||||
# (a deployment decision, not an image one).
|
|
||||||
ENV TAPIR_FFMPEG_BIN=/usr/local/bin/ffmpeg \
|
|
||||||
TAPIR_FFPROBE_BIN=/usr/local/bin/ffprobe
|
|
||||||
|
|
||||||
EXPOSE 8080
|
EXPOSE 8080
|
||||||
USER nonroot:nonroot
|
USER nonroot:nonroot
|
||||||
|
|||||||
+1
-1
@@ -34,7 +34,7 @@ tasks:
|
|||||||
lint:
|
lint:
|
||||||
desc: golangci-lint (installed by CI; skipped locally if absent).
|
desc: golangci-lint (installed by CI; skipped locally if absent).
|
||||||
cmds:
|
cmds:
|
||||||
- 'if command -v golangci-lint >/dev/null 2>&1; then golangci-lint run ./...; else echo "golangci-lint not installed — skipping (CI installs it)"; fi'
|
- 'command -v golangci-lint >/dev/null 2>&1 && golangci-lint run ./... || echo "golangci-lint not installed — skipping (CI installs it)"'
|
||||||
|
|
||||||
test:
|
test:
|
||||||
desc: Run all tests. -p 1 serializes packages so the embedded-postgres tests
|
desc: Run all tests. -p 1 serializes packages so the embedded-postgres tests
|
||||||
|
|||||||
+14
-15
@@ -17,9 +17,9 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net"
|
|
||||||
"net/http"
|
"net/http"
|
||||||
"os"
|
"os"
|
||||||
|
"os/signal"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -41,7 +41,7 @@ func main() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
log := slog.New(slog.NewTextHandler(os.Stderr, nil))
|
log := slog.New(slog.NewTextHandler(os.Stderr, nil))
|
||||||
ctx, stop := shutdownContext(context.Background())
|
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
|
||||||
defer stop()
|
defer stop()
|
||||||
|
|
||||||
var err error
|
var err error
|
||||||
@@ -215,11 +215,9 @@ func cmdServe(ctx context.Context, log *slog.Logger) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
var workerDone chan struct{} // nil without uploads: nothing to wait for at shutdown
|
|
||||||
wireUploads(app, cfg, intake)
|
wireUploads(app, cfg, intake)
|
||||||
if intake != nil {
|
if intake != nil {
|
||||||
workerDone = make(chan struct{})
|
go worker.Run(ctx)
|
||||||
go func() { worker.Run(ctx); close(workerDone) }()
|
|
||||||
log.Info("audio upload enabled", "model", cfg.TranscribeModel, "fallback", cfg.TranscribeFallbackModel,
|
log.Info("audio upload enabled", "model", cfg.TranscribeModel, "fallback", cfg.TranscribeFallbackModel,
|
||||||
"max_mb", cfg.UploadMaxBytes>>20, "max_minutes", int(cfg.UploadMaxDuration.Minutes()))
|
"max_mb", cfg.UploadMaxBytes>>20, "max_minutes", int(cfg.UploadMaxDuration.Minutes()))
|
||||||
log.Warn("audio upload assumes a SINGLE replica: the transcription queue is process-local (ADR-033, ADR-018)")
|
log.Warn("audio upload assumes a SINGLE replica: the transcription queue is process-local (ADR-033, ADR-018)")
|
||||||
@@ -352,19 +350,20 @@ func cmdServe(ctx context.Context, log *slog.Logger) error {
|
|||||||
}()
|
}()
|
||||||
}
|
}
|
||||||
|
|
||||||
// Graceful shutdown on signal: stop accepting, drain in-flight requests, and let the
|
// Graceful shutdown on signal: stop accepting, drain in-flight requests.
|
||||||
// worker record the job it was running as interrupted (#50).
|
go func() {
|
||||||
var extra []*http.Server
|
<-ctx.Done()
|
||||||
|
shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
_ = srv.Shutdown(shutdownCtx)
|
||||||
if metricsSrv != nil {
|
if metricsSrv != nil {
|
||||||
extra = append(extra, metricsSrv)
|
_ = metricsSrv.Shutdown(shutdownCtx)
|
||||||
}
|
|
||||||
ln, err := net.Listen("tcp", srv.Addr)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("serve: listen: %w", err)
|
|
||||||
}
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
log.Info("serving web ui", "addr", cfg.HTTPAddr, "user", cfg.UserID)
|
log.Info("serving web ui", "addr", cfg.HTTPAddr, "user", cfg.UserID)
|
||||||
if err := serveHTTP(ctx, srv, ln, extra, workerDone, shutdownGrace); err != nil {
|
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
||||||
return err
|
return fmt.Errorf("serve: %w", err)
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -134,30 +134,9 @@ func chatModels(cfg config.Config) []string {
|
|||||||
add(cfg.SummarizerModel)
|
add(cfg.SummarizerModel)
|
||||||
add(cfg.FallbackModel)
|
add(cfg.FallbackModel)
|
||||||
add(cfg.CloudFallbackModel)
|
add(cfg.CloudFallbackModel)
|
||||||
// The model chat over an uploaded recording opens on must be one the shared service can
|
|
||||||
// call: it re-maps a model it does not offer, which silently swapped a configured 31B for
|
|
||||||
// phi4-mini, whose context cannot hold an hour of speech. Local only, and only when
|
|
||||||
// uploads are on, so the video chat is unchanged otherwise (second adversarial review).
|
|
||||||
if cfg.UploadsEnabled && config.IsLocalModel(cfg.UploadChatModel) {
|
|
||||||
add(cfg.UploadChatModel)
|
|
||||||
}
|
|
||||||
return models
|
return models
|
||||||
}
|
}
|
||||||
|
|
||||||
// uploadChatModels is the chat switcher for a PRIVATE uploaded recording: the same
|
|
||||||
// ordered set as chatModels, local models only. The whole transcript (up to 90,000
|
|
||||||
// characters) is sent to the one chosen, so a cloud model the video chat offers must not
|
|
||||||
// be offered here; with none local, none is offered (ADR-035 review).
|
|
||||||
func uploadChatModels(cfg config.Config) []string {
|
|
||||||
var local []string
|
|
||||||
for _, m := range chatModels(cfg) {
|
|
||||||
if config.IsLocalModel(m) {
|
|
||||||
local = append(local, m)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return local
|
|
||||||
}
|
|
||||||
|
|
||||||
// buildChat wires the per-video chat service (ADR-027): a Completer factory over
|
// buildChat wires the per-video chat service (ADR-027): a Completer factory over
|
||||||
// the SAME LiteLLM gateway the summarizer uses (a different alias per model, not a
|
// the SAME LiteLLM gateway the summarizer uses (a different alias per model, not a
|
||||||
// second client config) and the same transcript-truncation budget. It returns nil
|
// second client config) and the same transcript-truncation budget. It returns nil
|
||||||
|
|||||||
@@ -1,55 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"net"
|
|
||||||
"net/http"
|
|
||||||
"os"
|
|
||||||
"os/signal"
|
|
||||||
"syscall"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// shutdownGrace bounds each wait at shutdown (in-flight requests, then the worker). Two of
|
|
||||||
// them stay inside Kubernetes' default 30 s termination grace.
|
|
||||||
const shutdownGrace = 10 * time.Second
|
|
||||||
|
|
||||||
// shutdownContext is cancelled by SIGINT (a terminal) or SIGTERM (what Kubernetes sends
|
|
||||||
// on every deploy). Handling only SIGINT meant a deploy killed the process before any
|
|
||||||
// shutdown code ran (#50).
|
|
||||||
func shutdownContext(parent context.Context) (context.Context, context.CancelFunc) {
|
|
||||||
return signal.NotifyContext(parent, os.Interrupt, syscall.SIGTERM)
|
|
||||||
}
|
|
||||||
|
|
||||||
// serveHTTP serves srv on ln until it fails or ctx ends, then shuts every server down and
|
|
||||||
// waits for in-flight requests and for workerDone (nil: no worker) to finish, each for at
|
|
||||||
// most grace. It must not return earlier: Serve returns http.ErrServerClosed as soon as
|
|
||||||
// Shutdown STARTS, and returning from main kills the drain and the worker's last write.
|
|
||||||
func serveHTTP(ctx context.Context, srv *http.Server, ln net.Listener, extra []*http.Server, workerDone <-chan struct{}, grace time.Duration) error {
|
|
||||||
errc := make(chan error, 1)
|
|
||||||
go func() { errc <- srv.Serve(ln) }()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case err := <-errc:
|
|
||||||
if !errors.Is(err, http.ErrServerClosed) {
|
|
||||||
return fmt.Errorf("serve: %w", err)
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
case <-ctx.Done():
|
|
||||||
}
|
|
||||||
|
|
||||||
shutdownCtx, cancel := context.WithTimeout(context.Background(), grace)
|
|
||||||
defer cancel()
|
|
||||||
for _, s := range append([]*http.Server{srv}, extra...) {
|
|
||||||
_ = s.Shutdown(shutdownCtx)
|
|
||||||
}
|
|
||||||
if workerDone != nil {
|
|
||||||
select {
|
|
||||||
case <-workerDone:
|
|
||||||
case <-time.After(grace):
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
@@ -1,124 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"os"
|
|
||||||
"sync"
|
|
||||||
"syscall"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Kubernetes stops a pod with SIGTERM; only SIGINT was handled, so a deploy killed the
|
|
||||||
// process without running any shutdown code (#50).
|
|
||||||
func TestShutdownContext_IsCancelledBySIGTERMAndSIGINT(t *testing.T) {
|
|
||||||
for _, sig := range []syscall.Signal{syscall.SIGTERM, syscall.SIGINT} {
|
|
||||||
t.Run(sig.String(), func(t *testing.T) {
|
|
||||||
ctx, stop := shutdownContext(context.Background())
|
|
||||||
defer stop()
|
|
||||||
|
|
||||||
if err := syscall.Kill(os.Getpid(), sig); err != nil {
|
|
||||||
t.Fatalf("send %v: %v", sig, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
case <-time.After(2 * time.Second):
|
|
||||||
t.Fatalf("context not cancelled by %v", sig)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// serveHTTP must not return while a request is in flight or the worker is still recording
|
|
||||||
// its job: Serve returns http.ErrServerClosed as soon as Shutdown STARTS, and returning
|
|
||||||
// from it let the process exit before either finished (#50). Each half is tested with the
|
|
||||||
// other already satisfied, so neither can hold the call open on the other's behalf.
|
|
||||||
|
|
||||||
func pendingFor(t *testing.T, returned <-chan error, why string) {
|
|
||||||
t.Helper()
|
|
||||||
select {
|
|
||||||
case err := <-returned:
|
|
||||||
t.Fatalf("serveHTTP returned (%v) %s", err, why)
|
|
||||||
case <-time.After(150 * time.Millisecond):
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestServeHTTP_DrainsAnInFlightRequestBeforeReturning(t *testing.T) {
|
|
||||||
release := make(chan struct{})
|
|
||||||
entered := make(chan struct{})
|
|
||||||
srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
close(entered)
|
|
||||||
<-release
|
|
||||||
_, _ = w.Write([]byte("ok"))
|
|
||||||
}))
|
|
||||||
srv.Start()
|
|
||||||
defer srv.Close()
|
|
||||||
var once sync.Once
|
|
||||||
releaseHandler := func() { once.Do(func() { close(release) }) }
|
|
||||||
defer releaseHandler() // runs before srv.Close: a failing test must not hang on the blocked handler
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
workerDone := make(chan struct{})
|
|
||||||
close(workerDone) // the worker is already done: only the request can hold serveHTTP open
|
|
||||||
returned := make(chan error, 1)
|
|
||||||
go func() { returned <- serveHTTP(ctx, srv.Config, srv.Listener, nil, workerDone, 5*time.Second) }()
|
|
||||||
|
|
||||||
go func() { _, _ = http.Get(srv.URL) }()
|
|
||||||
<-entered
|
|
||||||
cancel()
|
|
||||||
|
|
||||||
pendingFor(t, returned, "while a request was still in flight")
|
|
||||||
releaseHandler()
|
|
||||||
select {
|
|
||||||
case err := <-returned:
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("serveHTTP: %v", err)
|
|
||||||
}
|
|
||||||
case <-time.After(2 * time.Second):
|
|
||||||
t.Fatal("serveHTTP did not return once the request finished")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestServeHTTP_WaitsForTheWorkerBeforeReturning(t *testing.T) {
|
|
||||||
srv := httptest.NewUnstartedServer(http.NotFoundHandler())
|
|
||||||
srv.Start()
|
|
||||||
defer srv.Close()
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
workerDone := make(chan struct{})
|
|
||||||
returned := make(chan error, 1)
|
|
||||||
go func() { returned <- serveHTTP(ctx, srv.Config, srv.Listener, nil, workerDone, 5*time.Second) }() // no request in flight
|
|
||||||
|
|
||||||
cancel()
|
|
||||||
|
|
||||||
pendingFor(t, returned, "before the worker finished")
|
|
||||||
close(workerDone)
|
|
||||||
select {
|
|
||||||
case err := <-returned:
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("serveHTTP: %v", err)
|
|
||||||
}
|
|
||||||
case <-time.After(2 * time.Second):
|
|
||||||
t.Fatal("serveHTTP did not return once the worker finished")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestServeHTTP_GivesUpOnAWorkerThatNeverFinishesAfterTheGrace(t *testing.T) {
|
|
||||||
srv := httptest.NewUnstartedServer(http.NotFoundHandler())
|
|
||||||
srv.Start()
|
|
||||||
defer srv.Close()
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
returned := make(chan error, 1)
|
|
||||||
go func() {
|
|
||||||
returned <- serveHTTP(ctx, srv.Config, srv.Listener, nil, make(chan struct{}), 200*time.Millisecond)
|
|
||||||
}()
|
|
||||||
|
|
||||||
cancel()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-returned:
|
|
||||||
case <-time.After(3 * time.Second):
|
|
||||||
t.Fatal("a stuck worker must not hold the process past the grace period")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
+4
-16
@@ -55,8 +55,8 @@ func buildUploads(ctx context.Context, cfg config.Config, st uploadsStore, log *
|
|||||||
log.Warn("uploads: marked interrupted by a restart (audio is not retained; they must be re-uploaded)", "count", n)
|
log.Warn("uploads: marked interrupted by a restart (audio is not retained; they must be re-uploaded)", "count", n)
|
||||||
}
|
}
|
||||||
|
|
||||||
worker := uploads.NewWorker(st, newConverter(cfg), buildTranscriber(cfg),
|
worker := uploads.NewWorker(st, &audio.FFmpeg{Bin: cfg.FFmpegBin}, buildTranscriber(cfg),
|
||||||
cfg.UploadTmpDir, cfg.TranscribeTimeout, uploads.WithMaxDuration(cfg.UploadMaxDuration))
|
cfg.UploadTmpDir, cfg.TranscribeTimeout)
|
||||||
intake := uploads.NewIntake(st, &audio.FFprobe{Bin: cfg.FFprobeBin}, worker,
|
intake := uploads.NewIntake(st, &audio.FFprobe{Bin: cfg.FFprobeBin}, worker,
|
||||||
uploadLimits(cfg), cfg.UploadTmpDir)
|
uploadLimits(cfg), cfg.UploadTmpDir)
|
||||||
return intake, worker, nil
|
return intake, worker, nil
|
||||||
@@ -91,13 +91,6 @@ func wireUploads(app *web.App, cfg config.Config, in *uploads.Intake) {
|
|||||||
app.UploadLimits = uploadLimits(cfg)
|
app.UploadLimits = uploadLimits(cfg)
|
||||||
app.UploadChatMaxChars = cfg.MaxUploadedTranscriptChars
|
app.UploadChatMaxChars = cfg.MaxUploadedTranscriptChars
|
||||||
app.UploadChatModel = cfg.UploadChatModel
|
app.UploadChatModel = cfg.UploadChatModel
|
||||||
app.UploadChatModels = uploadChatModels(cfg)
|
|
||||||
}
|
|
||||||
|
|
||||||
// newConverter is the ffmpeg adapter capped at the configured maximum duration, so a file
|
|
||||||
// whose header lies about its length cannot become hours of WAV (ADR-035 review).
|
|
||||||
func newConverter(cfg config.Config) *audio.FFmpeg {
|
|
||||||
return &audio.FFmpeg{Bin: cfg.FFmpegBin, MaxSeconds: int(cfg.UploadMaxDuration.Seconds())}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// adjudicatorTimeout bounds one adjudication call. A call takes ~2 s on gemma4-26b and ~10 s
|
// adjudicatorTimeout bounds one adjudication call. A call takes ~2 s on gemma4-26b and ~10 s
|
||||||
@@ -114,12 +107,7 @@ func buildTranscriber(cfg config.Config) uploads.Transcriber {
|
|||||||
// No overall client timeout: a 90 minute recording takes minutes. The deadline is
|
// No overall client timeout: a 90 minute recording takes minutes. The deadline is
|
||||||
// the worker's per-job context.
|
// the worker's per-job context.
|
||||||
client := &http.Client{}
|
client := &http.Client{}
|
||||||
// Each LOCAL engine call gets its own deadline: a wedged engine does not answer, and
|
primary := transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, cfg.TranscribeModel, client)
|
||||||
// without one it used the whole job before the fallback could run.
|
|
||||||
local := func(model string) transcribe.Backend {
|
|
||||||
return transcribe.WithTimeout(transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, model, client), cfg.TranscribeEngineTimeout)
|
|
||||||
}
|
|
||||||
primary := local(cfg.TranscribeModel)
|
|
||||||
var fallback transcribe.Backend
|
var fallback transcribe.Backend
|
||||||
if cfg.TranscribeFallbackModel != "" && cfg.TranscribeFallbackModel != cfg.TranscribeModel {
|
if cfg.TranscribeFallbackModel != "" && cfg.TranscribeFallbackModel != cfg.TranscribeModel {
|
||||||
fallback = transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, cfg.TranscribeFallbackModel, client)
|
fallback = transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, cfg.TranscribeFallbackModel, client)
|
||||||
@@ -128,7 +116,7 @@ func buildTranscriber(cfg config.Config) uploads.Transcriber {
|
|||||||
return transcribe.NewChain(primary, fallback)
|
return transcribe.NewChain(primary, fallback)
|
||||||
}
|
}
|
||||||
|
|
||||||
base := local(cfg.ReconcileBaseModel)
|
base := transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, cfg.ReconcileBaseModel, client)
|
||||||
// Temperature 0 and a short reply: the adjudicator's accuracy was measured that way.
|
// Temperature 0 and a short reply: the adjudicator's accuracy was measured that way.
|
||||||
chat := llm.New(cfg.GatewayURL, cfg.GatewayKey, cfg.ReconcileModel, adjudicatorTimeout,
|
chat := llm.New(cfg.GatewayURL, cfg.GatewayKey, cfg.ReconcileModel, adjudicatorTimeout,
|
||||||
llm.WithTemperature(0), llm.WithMaxTokens(200))
|
llm.WithTemperature(0), llm.WithMaxTokens(200))
|
||||||
|
|||||||
@@ -1,85 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"reflect"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/config"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The chat over an uploaded recording sends the whole private transcript to the model
|
|
||||||
// the user picks, so the switcher for it may offer local models only (ADR-035 review).
|
|
||||||
|
|
||||||
func TestUploadChatModels_NeverOffersACloudModelButKeepsTheLocalOrder(t *testing.T) {
|
|
||||||
var cfg config.Config
|
|
||||||
cfg.SummarizerModel = "koala/phi4-mini"
|
|
||||||
cfg.FallbackModel = "iguana/gemma4-26b"
|
|
||||||
cfg.CloudFallbackModel = "berget/mistral-small"
|
|
||||||
|
|
||||||
got := uploadChatModels(cfg)
|
|
||||||
|
|
||||||
want := []string{"koala/phi4-mini", "iguana/gemma4-26b"}
|
|
||||||
if !reflect.DeepEqual(got, want) {
|
|
||||||
t.Errorf("uploadChatModels = %v, want %v: the cloud model that the video chat offers must not be offered for a private recording", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadChatModels_ACloudPrimaryIsDroppedToo(t *testing.T) {
|
|
||||||
var cfg config.Config
|
|
||||||
cfg.SummarizerModel = "berget/mistral-small"
|
|
||||||
cfg.FallbackModel = "iguana/gemma4-26b"
|
|
||||||
|
|
||||||
got := uploadChatModels(cfg)
|
|
||||||
|
|
||||||
if !reflect.DeepEqual(got, []string{"iguana/gemma4-26b"}) {
|
|
||||||
t.Errorf("uploadChatModels = %v", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadChatModels_NoLocalModelMeansNoneOfferedNotACloudDefault(t *testing.T) {
|
|
||||||
var cfg config.Config
|
|
||||||
cfg.SummarizerModel = "berget/mistral-small"
|
|
||||||
|
|
||||||
if got := uploadChatModels(cfg); len(got) != 0 {
|
|
||||||
t.Errorf("uploadChatModels = %v, want none", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// A valid local TAPIR_UPLOAD_CHAT_MODEL that is not in the video chat's chain used to be
|
|
||||||
// accepted by config and then silently ignored: the shared chat service re-maps a model it
|
|
||||||
// does not offer, so the user got koala/phi4-mini, whose 8,192-token context cannot hold an
|
|
||||||
// hour of speech (second adversarial review).
|
|
||||||
|
|
||||||
func TestChatModels_AnUploadChatModelOutsideTheChainIsOfferedWhenUploadsAreOn(t *testing.T) {
|
|
||||||
var cfg config.Config
|
|
||||||
cfg.SummarizerModel = "koala/phi4-mini"
|
|
||||||
cfg.FallbackModel = "iguana/gemma4-26b"
|
|
||||||
cfg.UploadsEnabled = true
|
|
||||||
cfg.UploadChatModel = "iguana/gemma4-31b"
|
|
||||||
|
|
||||||
service := chatModels(cfg)
|
|
||||||
upload := uploadChatModels(cfg)
|
|
||||||
|
|
||||||
if !reflect.DeepEqual(service, []string{"koala/phi4-mini", "iguana/gemma4-26b", "iguana/gemma4-31b"}) {
|
|
||||||
t.Errorf("chatModels = %v: the shared service must be able to call the configured upload model", service)
|
|
||||||
}
|
|
||||||
if !reflect.DeepEqual(upload, service) {
|
|
||||||
t.Errorf("uploadChatModels = %v, want %v", upload, service)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestChatModels_NothingIsAddedWhenUploadsAreOffOrTheModelIsNotLocal(t *testing.T) {
|
|
||||||
var cfg config.Config
|
|
||||||
cfg.SummarizerModel = "koala/phi4-mini"
|
|
||||||
cfg.UploadChatModel = "iguana/gemma4-31b"
|
|
||||||
|
|
||||||
if got := chatModels(cfg); !reflect.DeepEqual(got, []string{"koala/phi4-mini"}) {
|
|
||||||
t.Errorf("uploads off: chatModels = %v, want the video chain unchanged", got)
|
|
||||||
}
|
|
||||||
|
|
||||||
cfg.UploadsEnabled = true
|
|
||||||
cfg.UploadChatModel = "berget/mistral-small"
|
|
||||||
if got := chatModels(cfg); !reflect.DeepEqual(got, []string{"koala/phi4-mini"}) {
|
|
||||||
t.Errorf("a cloud upload chat model was added to the service: %v", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,26 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/config"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The converter is told the configured maximum, so a file whose header lies about its
|
|
||||||
// length cannot become hours of WAV (adversarial review F2).
|
|
||||||
|
|
||||||
func TestNewConverter_IsCappedAtTheConfiguredMaximumDuration(t *testing.T) {
|
|
||||||
var cfg config.Config
|
|
||||||
cfg.FFmpegBin = "/usr/local/bin/ffmpeg"
|
|
||||||
cfg.UploadMaxDuration = 90 * time.Minute
|
|
||||||
|
|
||||||
c := newConverter(cfg)
|
|
||||||
|
|
||||||
if c.Bin != "/usr/local/bin/ffmpeg" {
|
|
||||||
t.Errorf("Bin = %q", c.Bin)
|
|
||||||
}
|
|
||||||
if c.MaxSeconds != 5400 {
|
|
||||||
t.Errorf("MaxSeconds = %v, want 5400", c.MaxSeconds)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -22,15 +22,13 @@ import (
|
|||||||
type fakeUploadsStore struct {
|
type fakeUploadsStore struct {
|
||||||
sweeps int
|
sweeps int
|
||||||
sweepErr error
|
sweepErr error
|
||||||
failedKinds []domain.UploadErrorKind
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *fakeUploadsStore) CreateUpload(context.Context, string, string, domain.Language, bool) (string, error) {
|
func (f *fakeUploadsStore) CreateUpload(context.Context, string, string, bool) (string, error) {
|
||||||
return "id", nil
|
return "id", nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *fakeUploadsStore) FailUpload(_ context.Context, _, _ string, k domain.UploadErrorKind) error {
|
func (f *fakeUploadsStore) FailUpload(context.Context, string, string, domain.UploadErrorKind) error {
|
||||||
f.failedKinds = append(f.failedKinds, k)
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -44,7 +42,7 @@ func (f *fakeUploadsStore) SaveProgress(context.Context, string, string, []domai
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *fakeUploadsStore) Heartbeat(context.Context, string, string, time.Time) error { return nil }
|
func (f *fakeUploadsStore) Heartbeat(context.Context, string, string) error { return nil }
|
||||||
|
|
||||||
func (f *fakeUploadsStore) FailInterruptedUploads(context.Context) (int, error) {
|
func (f *fakeUploadsStore) FailInterruptedUploads(context.Context) (int, error) {
|
||||||
f.sweeps++
|
f.sweeps++
|
||||||
|
|||||||
@@ -1,78 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The wiring of the per-engine deadline: a gateway whose primary model never answers
|
|
||||||
// must still reach the fallback model inside the job's own deadline (adversarial review
|
|
||||||
// F4). The fake gateway answers per model from the multipart `model` field.
|
|
||||||
|
|
||||||
func TestBuildTranscriber_AWedgedPrimaryReachesTheFallbackBeforeTheJobDeadline(t *testing.T) {
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
_ = r.ParseMultipartForm(1 << 20)
|
|
||||||
if r.FormValue("model") == "iguana/kb-whisper" {
|
|
||||||
<-r.Context().Done() // wedged: never answers
|
|
||||||
return
|
|
||||||
}
|
|
||||||
w.Header().Set("Content-Type", "application/json")
|
|
||||||
_, _ = w.Write([]byte(`{"text":"hej","duration":1,"segments":[{"start":0,"end":1,"text":"hej"}]}`))
|
|
||||||
}))
|
|
||||||
defer srv.Close()
|
|
||||||
cfg := uploadCfg(t, true)
|
|
||||||
cfg.GatewayURL = srv.URL
|
|
||||||
cfg.TranscribeModel = "iguana/kb-whisper"
|
|
||||||
cfg.TranscribeFallbackModel = "berget/kb-whisper"
|
|
||||||
cfg.TranscribeEngineTimeout = 150 * time.Millisecond
|
|
||||||
wav := filepath.Join(t.TempDir(), "a.wav")
|
|
||||||
require.NoError(t, os.WriteFile(wav, []byte("RIFF....WAVEfmt "), 0o600))
|
|
||||||
job, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
got, err := buildTranscriber(cfg).Transcribe(job, wav)
|
|
||||||
|
|
||||||
require.NoError(t, err, "the wedged primary used the whole job and the fallback never ran")
|
|
||||||
assert.Equal(t, "berget/kb-whisper", got.Endpoint)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestBuildTranscriber_AWedgedBaseEngineIsASoftFailureInTheEnsemble(t *testing.T) {
|
|
||||||
// The same deadline must guard the ensemble's BASE engine, not only the chain's primary
|
|
||||||
// (second adversarial review: the wiring was tested for the chain alone).
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
||||||
_ = r.ParseMultipartForm(1 << 20)
|
|
||||||
if r.FormValue("model") == "iguana/pianissimo" {
|
|
||||||
<-r.Context().Done()
|
|
||||||
return
|
|
||||||
}
|
|
||||||
w.Header().Set("Content-Type", "application/json")
|
|
||||||
_, _ = w.Write([]byte(`{"text":"hej","duration":1,"segments":[{"start":0,"end":1,"text":"hej"}]}`))
|
|
||||||
}))
|
|
||||||
defer srv.Close()
|
|
||||||
cfg := uploadCfg(t, true)
|
|
||||||
cfg.GatewayURL = srv.URL
|
|
||||||
cfg.TranscribeModel = "iguana/kb-whisper"
|
|
||||||
cfg.ReconcileBaseModel = "iguana/pianissimo"
|
|
||||||
cfg.ReconcileModel = "iguana/gemma4-31b"
|
|
||||||
cfg.TranscribeEngineTimeout = 150 * time.Millisecond
|
|
||||||
wav := filepath.Join(t.TempDir(), "a.wav")
|
|
||||||
require.NoError(t, os.WriteFile(wav, []byte("RIFF....WAVEfmt "), 0o600))
|
|
||||||
job, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
got, err := buildTranscriber(cfg).Transcribe(job, wav)
|
|
||||||
|
|
||||||
require.NoError(t, err, "a wedged base engine must not use up the job")
|
|
||||||
assert.Equal(t, "hej", got.Text)
|
|
||||||
require.NotNil(t, got.Reconciliation)
|
|
||||||
assert.Equal(t, "single_engine", string(got.Reconciliation.Status))
|
|
||||||
assert.Contains(t, got.Reconciliation.Reason, "timeout")
|
|
||||||
}
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/uploads"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The worker buildUploads returns enforces the configured maximum on what was really
|
|
||||||
// converted (adversarial review F2). A stub ffmpeg writes a sparse 100-minute WAV however
|
|
||||||
// short the file claimed to be; the gateway is never reached because the job fails first.
|
|
||||||
|
|
||||||
func TestBuildUploads_TheWorkerRefusesAudioLongerThanTheConfiguredMaximum(t *testing.T) {
|
|
||||||
dir := t.TempDir()
|
|
||||||
stub := filepath.Join(dir, "ffmpeg")
|
|
||||||
require.NoError(t, os.WriteFile(stub, []byte("#!/bin/sh\nfor a; do last=\"$a\"; done\ntruncate -s 192000044 \"$last\"\n"), 0o700))
|
|
||||||
cfg := uploadCfg(t, true)
|
|
||||||
cfg.FFmpegBin = stub
|
|
||||||
cfg.UploadMaxDuration = 90 * time.Minute
|
|
||||||
cfg.TranscribeTimeout = time.Minute
|
|
||||||
st := &fakeUploadsStore{}
|
|
||||||
_, w, err := buildUploads(context.Background(), cfg, st, quietLog())
|
|
||||||
require.NoError(t, err)
|
|
||||||
src := filepath.Join(cfg.UploadTmpDir, "upload-x")
|
|
||||||
require.NoError(t, os.WriteFile(src, []byte("audio"), 0o600))
|
|
||||||
|
|
||||||
w.Process(context.Background(), uploads.Job{UserID: "u", UploadID: "id", Path: src, Duration: 10 * time.Second})
|
|
||||||
|
|
||||||
assert.Equal(t, []domain.UploadErrorKind{domain.UploadErrTooLong}, st.failedKinds,
|
|
||||||
"a file that claimed 10 s but converted to 100 minutes must fail, not be sent to an engine")
|
|
||||||
}
|
|
||||||
@@ -14,8 +14,7 @@ it** — endpoints and aliases drift, and this file is a snapshot (2026-06-06),
|
|||||||
not LiteLLM. Use `30401` for off-cluster LiteLLM. `confirm` resolved.
|
not LiteLLM. Use `30401` for off-cluster LiteLLM. `confirm` resolved.
|
||||||
- **Speech-to-text (ADR-033):** the same gateway serves `POST /v1/audio/transcriptions` for
|
- **Speech-to-text (ADR-033):** the same gateway serves `POST /v1/audio/transcriptions` for
|
||||||
`iguana/kb-whisper` (primary, local) and `berget/kb-whisper` (fallback, EU cloud — private audio
|
`iguana/kb-whisper` (primary, local) and `berget/kb-whisper` (fallback, EU cloud — private audio
|
||||||
leaves the homelab if it is used: when the primary fails **or returns an empty transcript**,
|
leaves the homelab if it is used). **Call it via the NodePort / in-cluster address above, NOT
|
||||||
e.g. a silent or music-only file; the transcript page then says so, #52). **Call it via the NodePort / in-cluster address above, NOT
|
|
||||||
`https://llm-api.d-ma.be`:** that public ingress (openresty) returns `413` for bodies over
|
`https://llm-api.d-ma.be`:** that public ingress (openresty) returns `413` for bodies over
|
||||||
roughly 8–10 MB (~5 minutes of 16 kHz WAV), while the NodePort took 174 MB (90 minutes) in one
|
roughly 8–10 MB (~5 minutes of 16 kHz WAV), while the NodePort took 174 MB (90 minutes) in one
|
||||||
request (measured 2026-10-01, `docs/research/audio-length-ceiling-2026-10-01.md`). Send multipart
|
request (measured 2026-10-01, `docs/research/audio-length-ceiling-2026-10-01.md`). Send multipart
|
||||||
@@ -24,13 +23,15 @@ it** — endpoints and aliases drift, and this file is a snapshot (2026-06-06),
|
|||||||
`{"error":"failed to process audio"}` string — it has had at least five causes. Tapir's knobs:
|
`{"error":"failed to process audio"}` string — it has had at least five causes. Tapir's knobs:
|
||||||
`TAPIR_UPLOADS_ENABLED` (off by default), `TAPIR_UPLOAD_MAX_MB` (200), `TAPIR_UPLOAD_MAX_MINUTES`
|
`TAPIR_UPLOADS_ENABLED` (off by default), `TAPIR_UPLOAD_MAX_MB` (200), `TAPIR_UPLOAD_MAX_MINUTES`
|
||||||
(90), `TAPIR_TRANSCRIBE_MODEL`, `TAPIR_TRANSCRIBE_FALLBACK_MODEL` (empty disables the cloud
|
(90), `TAPIR_TRANSCRIBE_MODEL`, `TAPIR_TRANSCRIBE_FALLBACK_MODEL` (empty disables the cloud
|
||||||
fallback), `TAPIR_TRANSCRIBE_TIMEOUT` (60m, the whole job), `TAPIR_TRANSCRIBE_ENGINE_TIMEOUT` (20m, one engine call: a wedged engine then fails while the job lives and the fallback can run), `TAPIR_UPLOAD_TMP_DIR`, `TAPIR_FFMPEG_BIN`,
|
fallback), `TAPIR_TRANSCRIBE_TIMEOUT` (30m), `TAPIR_UPLOAD_TMP_DIR`, `TAPIR_FFMPEG_BIN`,
|
||||||
`TAPIR_FFPROBE_BIN`, `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS` (90000 — the measured ceiling of
|
`TAPIR_FFPROBE_BIN`, `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS` (90000 — the measured ceiling of
|
||||||
`iguana/gemma4-26b`, which silently drops the head above it), `TAPIR_UPLOAD_CHAT_MODEL`
|
`iguana/gemma4-26b`, which silently drops the head above it), `TAPIR_UPLOAD_CHAT_MODEL`
|
||||||
(`iguana/gemma4-26b`; must be a local `koala/` or `iguana/` alias, since the whole private transcript is sent to it; `koala/phi4-mini`'s 8,192-token context 400s on an hour of speech). `ffmpeg` + `ffprobe` are in the runtime image (static, pinned by sha256; #34 decided A on 2026-10-02); Tapir's own ingress body limit for a 200 MB upload is unverified (infra).
|
(`iguana/gemma4-26b`; `koala/phi4-mini`'s 8,192-token context 400s on an hour of speech). Needs `ffmpeg` + `ffprobe`
|
||||||
|
in the runtime image; Tapir's own ingress body limit for a 200 MB upload is unverified (infra).
|
||||||
**Two-engine mode (ADR-034), off unless `TAPIR_RECONCILE_BASE_MODEL` is set:** it runs
|
**Two-engine mode (ADR-034), off unless `TAPIR_RECONCILE_BASE_MODEL` is set:** it runs
|
||||||
`iguana/pianissimo` (the base text) and `TAPIR_TRANSCRIBE_MODEL` (kb-whisper) on the same audio, then a
|
`iguana/pianissimo` (the base text) and `TAPIR_TRANSCRIBE_MODEL` (kb-whisper) on the same audio, then a
|
||||||
LOCAL chat model (`TAPIR_RECONCILE_MODEL`) chooses between their disagreements. **Deployment choice (2026-10-02): `iguana/gemma4-31b`** (97% against a speaker's blind answers; 92% for `iguana/gemma4-26b`). With iguana's macOS **Low Power Mode off** the 31B decodes at 22 tok/s and reconciled a 63.6-minute recording in 21 min 49 s end to end; with it on it ran at 8 tok/s. Check `pmset -g | grep lowpowermode` before blaming a model or an engine for slowness.
|
LOCAL chat model (`TAPIR_RECONCILE_MODEL`, e.g. `iguana/gemma4-26b`: 92% against a speaker's blind answers
|
||||||
|
at ~2 s a call; `iguana/gemma4-31b` 97% but ~5x slower) chooses between their disagreements.
|
||||||
`TAPIR_RECONCILE_CHECK_ORDER=true` asks twice per dispute with the order swapped (doubles the time).
|
`TAPIR_RECONCILE_CHECK_ORDER=true` asks twice per dispute with the order swapped (doubles the time).
|
||||||
Every model must start `koala/` or `iguana/`: a cloud alias is a start-up error. The default job timeout is
|
Every model must start `koala/` or `iguana/`: a cloud alias is a start-up error. The default job timeout is
|
||||||
60 minutes. `iguana/pianissimo` is KlangAI's Swedish ASR (a Parakeet/FastConformer on MLX, served by a small
|
60 minutes. `iguana/pianissimo` is KlangAI's Swedish ASR (a Parakeet/FastConformer on MLX, served by a small
|
||||||
|
|||||||
@@ -29,7 +29,7 @@ ingress configuration, which lives in `infra`. "About 10 MB" is an inference.
|
|||||||
## Method
|
## Method
|
||||||
|
|
||||||
- **Audio.** No known-good Swedish recording is available that is safe to use: the only real
|
- **Audio.** No known-good Swedish recording is available that is safe to use: the only real
|
||||||
one is a private recording (#31 custody still open) and was
|
one (`IMG_1233.wav`) is the private `/bygge` recording (#31 custody still open) and was
|
||||||
deliberately not touched. Instead: macOS `say -v Alva` (Swedish) on iguana, a 1,171-char
|
deliberately not touched. Instead: macOS `say -v Alva` (Swedish) on iguana, a 1,171-char
|
||||||
five-paragraph meeting script → 82.5 s of 16 kHz mono PCM WAV (`ffmpeg -ar 16000 -ac 1`).
|
five-paragraph meeting script → 82.5 s of 16 kHz mono PCM WAV (`ffmpeg -ar 16000 -ac 1`).
|
||||||
Concatenated ×4, ×8, ×16, ×32, ×44, ×66 with Python `wave` (the issue's own recipe).
|
Concatenated ×4, ×8, ×16, ×32, ×44, ×66 with Python `wave` (the issue's own recipe).
|
||||||
|
|||||||
@@ -1,69 +0,0 @@
|
|||||||
# ADR-034 second blind check: protocol (ready to run; tapir#33, D9, D10)
|
|
||||||
|
|
||||||
This repo is public and the recordings are private, so this file holds the method only. The
|
|
||||||
sheet, the key and the filled-in verdicts stay outside the repo, and the trial data is deleted
|
|
||||||
after the check. Results go back here as aggregate numbers, as in
|
|
||||||
`reconcile-blind-check-2026-10-02.md`.
|
|
||||||
|
|
||||||
## What the check decides
|
|
||||||
|
|
||||||
ADR-034 (two engines, a local model reconciles their disagreements) is `Proposed` and off in
|
|
||||||
production. Two conditions were set for keeping it. The first run answered the first one on one
|
|
||||||
reading only and did not answer the second:
|
|
||||||
|
|
||||||
1. **The reconciled reading is right at least 85% of the time.** Open flaw in the bar: it did not
|
|
||||||
say how ties count. The first run gave 95% (decided items only) or 70% (all 30), so it was met on
|
|
||||||
one reading and missed on the other.
|
|
||||||
2. **It is no worse than the better single engine.** Not answered: before ADR-035 the spots the
|
|
||||||
model *kept* as the base were not stored, so kb-whisper alone could not be compared.
|
|
||||||
|
|
||||||
ADR-035 now stores every judged spot (`reconciliation.contested`: both readings, which one was
|
|
||||||
used, unsure or not, a few words of left context, a time). That makes condition 2 measurable.
|
|
||||||
|
|
||||||
## Rules to fix BEFORE looking at any result (the owner confirms or changes them)
|
|
||||||
|
|
||||||
| Rule | Proposed |
|
|
||||||
|---|---|
|
|
||||||
| Who judges | The speaker, hearing the audio where the text cannot settle it (D9). Not Opus: it cannot hear, and its agreement with the speaker was only shown on items picked as decidable. |
|
|
||||||
| Sample | 30 spots drawn at random with a fixed seed, from ALL judged spots (changed and kept-base alike), not only the changed ones. Insertions and deletions that cannot be placed are listed separately and not sampled. |
|
|
||||||
| Verdict per spot | `x`, `y` (the two readings, shown in random order, key kept apart), `neither`, or `cannot tell`. |
|
|
||||||
| Condition 1 | Reconciled is right on at least 85% of the spots where the judge could tell (`cannot tell` excluded, reported, and `neither` counts as not right). Also report all-30. Pass needs the decided-only figure at 85% or more AND the all-30 figure above 70%. |
|
|
||||||
| Condition 2 | Over the same spots, the reconciled text is right at least as often as the better single engine (base engine alone is right where the base reading is right; the other engine alone where the alt reading is right). Ties favour the single engine. |
|
|
||||||
| If it fails | Switch reconcile off by configuration; the single-engine path is unchanged (ADR-034 reversibility). |
|
|
||||||
|
|
||||||
Why a fixed all-30 floor: the literal 70% last time sat well under the bar, and "decided only" can
|
|
||||||
hide a model that mostly lands on spelling differences.
|
|
||||||
|
|
||||||
## What a run needs (D10: the owner is present)
|
|
||||||
|
|
||||||
- The same real recording as before (about 60 minutes). Only the owner has it.
|
|
||||||
- A **throwaway** tapir instance with its own Postgres, not the production pod, so production's
|
|
||||||
key pin and deployment are untouched. Settings: `TAPIR_RECONCILE_BASE_MODEL=iguana/pianissimo`,
|
|
||||||
`TAPIR_TRANSCRIBE_MODEL=iguana/kb-whisper`, `TAPIR_RECONCILE_MODEL=iguana/gemma4-31b`, cloud
|
|
||||||
fallbacks empty. Job time was 21 min 49 s per 63.6 min last run.
|
|
||||||
- iguana with Low Power Mode off and gemma4-31b able to load (the production-iguana model load
|
|
||||||
that needs a quiet window; D6).
|
|
||||||
|
|
||||||
## Steps
|
|
||||||
|
|
||||||
1. Start the throwaway instance, upload the recording, wait for `done` (about 22 minutes).
|
|
||||||
2. Read `reconciliation.contested` from the throwaway database. From it build, **outside the repo**:
|
|
||||||
`sheet` (spot number, time, left context, reading x, reading y, random order) and `key` (which
|
|
||||||
of x/y is the base, which is the other engine, which one the reconciler used).
|
|
||||||
3. The speaker fills in the sheet (about 20 to 30 minutes for 30 spots, audio open at the times).
|
|
||||||
The agent does not see the key until the sheet is returned.
|
|
||||||
4. Score with the rules above. Write the aggregate numbers into a dated note next to this one,
|
|
||||||
and amend ADR-034's validation section (kept, or switched off, and the tie rule used).
|
|
||||||
5. Delete the throwaway instance, its database and the sheet, as last time.
|
|
||||||
6. Only if the check passes and the owner wants it in production: enabling reconcile there is
|
|
||||||
the separate D10 step (it needs `iguana/pianissimo` and the reconciler added to the tapir key
|
|
||||||
pin in the infra repo plus the env vars on the deployment, `risk/medium`, owner present).
|
|
||||||
|
|
||||||
## Open and owner-only
|
|
||||||
|
|
||||||
- Confirm or change the rules table above before the run.
|
|
||||||
- The **tool that builds the sheet and key** is the one piece not in the repo (last time it was a
|
|
||||||
one-off script). It can be built test-first on request; until then the agent builds it ad hoc
|
|
||||||
outside the repo.
|
|
||||||
- A Berget adjudicator (D8, tapir#45) is a different question with its own measurement and is
|
|
||||||
not part of this check: this check judges the local setup that ADR-034 describes.
|
|
||||||
@@ -1,115 +0,0 @@
|
|||||||
# Sprint: audio upload follow-ups (Sweep Sprint, 2026-10)
|
|
||||||
|
|
||||||
Method: Sweep Sprint (phase 1 parallel read-only research per issue: **done**; phase 2 sequential
|
|
||||||
execution, one issue per commit: **v0.31.1 and v0.32.0 shipped 2026-10-03/04, the owner-held items
|
|
||||||
below remain open**). Closing oath below.
|
|
||||||
|
|
||||||
Milestones: [v0.31.1 - security and robustness](https://git.d-ma.be/mathias/tapir/milestone/10),
|
|
||||||
[v0.32.0 - audio upload follow-ups](https://git.d-ma.be/mathias/tapir/milestone/11).
|
|
||||||
|
|
||||||
Context: v0.31.0 shipped audio upload (ADR-033/034/035) and is live in production behind auth at
|
|
||||||
`/uploads`, with the Berget cloud fallback on for audio (infra#522). A real 63.7 min upload
|
|
||||||
transcribed in 2m53s with no errors. The phase-1 investigation found two pre-existing or new
|
|
||||||
weaknesses an authenticated user can reach, so the first release is a hotfix.
|
|
||||||
|
|
||||||
## Oath (closing gates, every clause must hold; blocked = descoped and tracked, never "done")
|
|
||||||
|
|
||||||
| # | Clause | Evidence |
|
|
||||||
|---|--------|----------|
|
|
||||||
| SG1 | Every issue in the sprint is closed with a linked commit, or moved out with a stated reason | issue list of both milestones |
|
|
||||||
| SG2 | Each fix has a test that was seen red on main first, named after the mechanism it observes | red/green output in the issue |
|
|
||||||
| SG3 | `task check` AND `golangci-lint run ./...` exit 0 (checked by `$?`, not by grep); `task check` does not gate lint until #41 | gate output |
|
|
||||||
| SG4 | Control code (#48, #49, the upload cap in #47) has a recorded adversarial review verified by execution, verdict in the issue | gatekeeper verdict comment |
|
|
||||||
| SG5 | Docs current in the same commit: ADR-033/035 text, feature file, use-case coverage test | diff |
|
|
||||||
| SG6 | Each release tagged (v0.31.1, v0.32.0), pushed, deployed, and observed live (pod running the new image, no restart loop, a real upload still works) | `kubectl`/log + one upload |
|
|
||||||
| SG7 | Learnings written to the brain; deferred work filed as issues; decisions that are the owner's recorded as DECISION NEEDED | brain ids + issue numbers |
|
|
||||||
|
|
||||||
## Work order
|
|
||||||
|
|
||||||
### v0.31.1 (security and robustness; hotfix, do first)
|
|
||||||
|
|
||||||
1. **#41 lint gate.** `task check` swallows lint findings (`|| echo`). Make the gate fail on findings,
|
|
||||||
fix what it surfaces. First, so every later gate means something. (Decision D1.)
|
|
||||||
2. **#48 cross-origin POST protection.** `http.NewCrossOriginProtection()` around `Router()`.
|
|
||||||
Red: foreign-origin POST to `/account/delete`, `/uploads`, `/uploads/{id}/chat` is refused.
|
|
||||||
3. **#49 ffprobe/ffmpeg limits.** `-max_streams 32` + narrower probe output; probe stdout cap;
|
|
||||||
probe deadline. Three commits.
|
|
||||||
4. **#47 concurrent-upload cap (remaining item).** Bound the spool: reservation per accepted upload
|
|
||||||
against the 6Gi emptyDir so concurrent uploads cannot evict the pod.
|
|
||||||
5. **#50 SIGTERM interim.** Handle SIGTERM; record the interrupted job at once; fix the false
|
|
||||||
comment. Full drain recorded as a limitation. (Decision D2.)
|
|
||||||
|
|
||||||
Release v0.31.1: one adversarial review round over 2-5, then tag, push, observe.
|
|
||||||
|
|
||||||
### v0.32.0 (follow-ups)
|
|
||||||
|
|
||||||
6. **#52** label a cloud-served transcript in plain words; document the empty-transcript trigger.
|
|
||||||
7. **#53** bound the LLM client's response body read.
|
|
||||||
8. **#51** tracker: save timeout, then app-clock heartbeat.
|
|
||||||
9. **#37** nav link to uploads and per-recording delete (queued/transcribing: refuse until it
|
|
||||||
ends). (Decision D3.)
|
|
||||||
10. **#36** chat default model after measurement. (Decision D4.)
|
|
||||||
11. **#38** measurements (Berget length limit, large models on production iguana). (D5, D6.)
|
|
||||||
12. **#46** remainder of the hardening list after the items above are split out.
|
|
||||||
13. **#45** evaluate Berget as the reconciler: a question, not work. (D8.)
|
|
||||||
|
|
||||||
## DECISION NEEDED (owner's call; recommendation first)
|
|
||||||
|
|
||||||
| # | Question | Recommendation | Blocks |
|
|
||||||
|---|----------|----------------|--------|
|
|
||||||
| D1 | #41: should any linter be allowed to opt out of the gate? | No opt-out; fix findings | #41 |
|
|
||||||
| D2 | #50: full SIGTERM drain, or record as a limitation and ship the interim fix? | Limitation + interim; a drain costs 4-6 h and downtime on every deploy at current volume | #50 |
|
|
||||||
| D3 | #37: delete of a queued/transcribing recording | Refuse until it ends | #37 |
|
|
||||||
| D4 | #36: make a 31B model the chat default? | Only after measurement, conditional | #36 |
|
|
||||||
| D5 | #38: Berget length measurement needs a throwaway key | Yes, throwaway key | #38 |
|
|
||||||
| D6 | #38: load large models on production iguana | Yes, in a quiet window with the owner present | #38 |
|
|
||||||
| D7 | #46: where the ffmpeg tarball lives | Do nothing now; file an issue | none |
|
|
||||||
| D8 | #45: evaluate Berget as the reconciler | **Decided 2026-10-04 by the owner: yes, Berget may be used, and each user must be able to keep a recording on local models only** (reverses the recommendation). Opt-out first: #60; #45 stays as the model measurement | #60, #45 |
|
|
||||||
| D9 | Who judges ADR-034's second blind check | The owner; do not reuse the one-off Opus authorisation. Protocol: `docs/research/reconcile-second-check-protocol.md` | ADR-034 |
|
|
||||||
| D10 | When to enable reconcile in production | Needs `iguana/pianissimo` in the key pin and env vars; owner present | ADR-034 |
|
|
||||||
|
|
||||||
## Dropped or reduced, with reasons
|
|
||||||
|
|
||||||
- **#46 item 1 (`alis`/`dref` external reference):** checked, not a problem. Under the application's
|
|
||||||
flags ffmpeg skips the external track and writes a 78-byte WAV.
|
|
||||||
- **#46 item 2 / #47 item 2:** merged into #49.
|
|
||||||
- **#47 item 3 (full drain):** reduced to #50 plus a recorded limitation.
|
|
||||||
- **#46 `Beat` taking the tracker mutex:** incorrect in #46; only `logStill` does. Corrected in #51.
|
|
||||||
|
|
||||||
## Human-present items (do not run unattended)
|
|
||||||
|
|
||||||
- Production iguana model loads (#38, D6).
|
|
||||||
- Enabling reconcile in production (D10).
|
|
||||||
- Any key change or `apply` in infra (infra#523, infra#524 are open and not part of this sprint).
|
|
||||||
|
|
||||||
## Release plan
|
|
||||||
|
|
||||||
SemVer: v0.31.1 patch (fixes, no new behaviour); v0.32.0 minor (nav link, delete, cloud labelling).
|
|
||||||
Every push to tapir main redeploys the pod and fails any job in flight: push when no upload is
|
|
||||||
running (check the pod log first).
|
|
||||||
|
|
||||||
## Status (2026-10-04)
|
|
||||||
|
|
||||||
**Shipped and deployed:** v0.31.1 (PR #55, merge e037c6b) and v0.32.0 (PR #58, merge 85f4cc3; pod
|
|
||||||
tapir-597dbfccc-ng78q 1/1, 0 restarts, image 85f4cc3). Each went through an adversarial review
|
|
||||||
round by agent with the verdicts recorded on the PR (v0.31.1: BLOCK then APPROVE; v0.32.0: BLOCK,
|
|
||||||
BLOCK, APPROVE, APPROVE; every block was a real defect, fixed test-first).
|
|
||||||
|
|
||||||
**Live evidence:** a foreign-origin `POST` to `/account/delete` and to `/uploads/{id}/delete` is
|
|
||||||
refused (403) on production, a same-origin one reaches the login redirect (302).
|
|
||||||
|
|
||||||
**Not observed live (needs an authenticated session, so it is the owner's look; closed on word or screenshot):**
|
|
||||||
|
|
||||||
- #52, cloud side: a transcript served by the Berget fallback names `berget`; a reconciled all-local
|
|
||||||
one shows no note. The local side was observed on 2026-10-04 (a new clip served by iguana: chip
|
|
||||||
`iguana/…`, no cloud note). The cloud side is carried as an acceptance item in #60.
|
|
||||||
|
|
||||||
**Observed live since:** #37 (owner, 2026-10-04: link, confirm-then-delete, no control while transcribing; closed). #50: at the 2026-10-04 rollout the old pod went `Terminating` then `Completed`
|
|
||||||
(not `Error`), 0 restarts; #50 closed. No job was in flight, so a drained job is still unobserved
|
|
||||||
(a recorded limitation, D2).
|
|
||||||
|
|
||||||
**Still open (owner's call, milestone 11):** #36, #38, #45 (now the Berget adjudicator measurement), #60 (per-recording local-only choice, 4 sub-decisions), #46 remainder, #56 (slow-drip bodies hold
|
|
||||||
a receive slot), #57 (shutdown logging, second signal). Decisions D4-D6, D9, D10 above are unchanged; D8 is decided (above).
|
|
||||||
|
|
||||||
**Deferred and filed from the reviews:** #56, #57, #59 (start-up exits when the identity provider's OIDC
|
|
||||||
discovery returns 503 (seen once at 19:40 on 2026-10-03; the pod restarted itself and recovered)).
|
|
||||||
@@ -214,80 +214,3 @@ Feature: Upload an audio recording, get a private Swedish transcript, discuss it
|
|||||||
When the model changes a word
|
When the model changes a word
|
||||||
Then the segment containing it carries the reconciled word
|
Then the segment containing it carries the reconciled word
|
||||||
And the segment's timing is unchanged
|
And the segment's timing is unchanged
|
||||||
|
|
||||||
# ADR-035 decision 8: hardening from the adversarial review
|
|
||||||
|
|
||||||
Scenario: Chat about a recording is offered with local models only
|
|
||||||
When I open the chat for my recording
|
|
||||||
Then the model list holds only models that run on the homelab
|
|
||||||
And a cloud model the shared chat service knows is not in it
|
|
||||||
|
|
||||||
Scenario: A forged cloud model is never sent the transcript
|
|
||||||
When a chat request names a cloud model
|
|
||||||
Then a local model answers instead, and nothing was sent to the cloud one
|
|
||||||
|
|
||||||
Scenario: A wedged engine does not use up the whole job
|
|
||||||
When the primary engine never answers
|
|
||||||
Then it fails at its own deadline while the job is still alive
|
|
||||||
And the fallback runs, unless the job itself was cancelled or timed out
|
|
||||||
|
|
||||||
Scenario: A file that lies about its length is refused
|
|
||||||
When a file claims to be short but converts to more than the limit
|
|
||||||
Then the job fails with a plain explanation and no engine receives the audio
|
|
||||||
|
|
||||||
Scenario: A recording much shorter than its file claimed is transcribed and noted
|
|
||||||
When a file converts to under 80 percent of the length it claimed and more than two minutes short
|
|
||||||
Then it is transcribed anyway and its audio step says how long the audio really is
|
|
||||||
And a complete recording whose header over-estimated its length is not refused
|
|
||||||
|
|
||||||
Scenario: A transcript made by the cloud fallback says the audio left the homelab
|
|
||||||
When the transcript was produced by an endpoint that is not local
|
|
||||||
Then its page says in plain words that the audio left the homelab and names the provider
|
|
||||||
And a locally transcribed recording carries no such note
|
|
||||||
|
|
||||||
Scenario: I can delete a finished recording
|
|
||||||
When I confirm deleting a done or failed recording
|
|
||||||
Then the recording and its transcript are gone and I am back on my list
|
|
||||||
And another user's recording is a 404 and is untouched
|
|
||||||
|
|
||||||
Scenario: A recording still being transcribed cannot be deleted
|
|
||||||
When I try to delete a recording that is queued or transcribing
|
|
||||||
Then I am told it is still being transcribed and nothing is removed
|
|
||||||
|
|
||||||
Scenario: The home list links to my recordings when uploads are on
|
|
||||||
Given audio upload is enabled
|
|
||||||
Then the home list links to the recordings page
|
|
||||||
|
|
||||||
Scenario: A full queue is a retry, not an error
|
|
||||||
When too many recordings are already waiting
|
|
||||||
Then the upload is answered as busy, its row is failed honestly and its audio is deleted
|
|
||||||
|
|
||||||
Scenario: One user cannot fill the queue for everyone
|
|
||||||
When one user has several recordings waiting
|
|
||||||
Then further uploads from that user are refused and other users are not affected
|
|
||||||
|
|
||||||
Scenario: Errors are shown as kinds, never their text
|
|
||||||
When a model or an engine fails
|
|
||||||
Then the log, the stored reason and the page carry a kind such as timeout or http_502
|
|
||||||
And never the provider's response, a file path or a word from the transcript
|
|
||||||
|
|
||||||
Scenario: Input that cannot be right is a client error
|
|
||||||
When the id is not an id, or a header or title holds control bytes
|
|
||||||
Then the answer is a 4xx or the title is cleaned, never a server error
|
|
||||||
|
|
||||||
Scenario: I choose the language of a recording
|
|
||||||
When I upload a recording and choose Swedish or Other on the form
|
|
||||||
Then the recording is transcribed as that language and its page says which
|
|
||||||
|
|
||||||
Scenario: Swedish stays the default
|
|
||||||
When an upload names no language
|
|
||||||
Then it is transcribed as Swedish, as before the choice existed
|
|
||||||
|
|
||||||
Scenario: A language outside the list is refused
|
|
||||||
When an upload names a language the form does not offer
|
|
||||||
Then it is answered 400 before anything is stored or sent to an engine
|
|
||||||
|
|
||||||
Scenario: A recording in another language never reaches the Swedish-only engine
|
|
||||||
When I choose Other for a recording
|
|
||||||
Then the language is left for the engine to detect and the Swedish-only base engine and the reconcile are not used
|
|
||||||
And the engine that does run is the configured one, which is not tuned for other languages
|
|
||||||
|
|||||||
@@ -33,9 +33,9 @@ require (
|
|||||||
github.com/rogpeppe/go-internal v1.15.0 // indirect
|
github.com/rogpeppe/go-internal v1.15.0 // indirect
|
||||||
github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 // indirect
|
github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 // indirect
|
||||||
go.yaml.in/yaml/v2 v2.4.2 // indirect
|
go.yaml.in/yaml/v2 v2.4.2 // indirect
|
||||||
golang.org/x/sync v0.21.0 // indirect
|
golang.org/x/sync v0.18.0 // indirect
|
||||||
golang.org/x/sys v0.41.0 // indirect
|
golang.org/x/sys v0.41.0 // indirect
|
||||||
golang.org/x/text v0.39.0 // indirect
|
golang.org/x/text v0.31.0 // indirect
|
||||||
google.golang.org/protobuf v1.36.8 // indirect
|
google.golang.org/protobuf v1.36.8 // indirect
|
||||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -113,12 +113,12 @@ go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=
|
|||||||
go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
|
go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
|
||||||
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
|
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
|
||||||
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
|
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
|
||||||
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
|
golang.org/x/sync v0.18.0 h1:kr88TuHDroi+UVf+0hZnirlk8o8T+4MrK6mr60WkH/I=
|
||||||
golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
golang.org/x/sync v0.18.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
|
||||||
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
|
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
|
||||||
golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
|
golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
|
||||||
golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus=
|
golang.org/x/text v0.31.0 h1:aC8ghyu4JhP8VojJ2lEHBnochRno1sgL6nEi9WGFGMM=
|
||||||
golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM=
|
golang.org/x/text v0.31.0/go.mod h1:tKRAlv61yKIjGGHX/4tP1LTbc13YSec1pxVEWXzfoeM=
|
||||||
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
|
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
|
||||||
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
|
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
|
||||||
google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc=
|
google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc=
|
||||||
|
|||||||
@@ -5,7 +5,6 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"os/exec"
|
"os/exec"
|
||||||
"strconv"
|
|
||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -18,17 +17,8 @@ import (
|
|||||||
type FFmpeg struct {
|
type FFmpeg struct {
|
||||||
// Bin is the ffmpeg executable ("ffmpeg" resolves via PATH).
|
// Bin is the ffmpeg executable ("ffmpeg" resolves via PATH).
|
||||||
Bin string
|
Bin string
|
||||||
// MaxSeconds caps the converted length when positive. The duration limit is a claim
|
|
||||||
// in the file's header and can be lied about (a 2 h file claiming 10 s converted to a
|
|
||||||
// 120-minute WAV); this bounds what a lie can cost. A minute of slack is added so a
|
|
||||||
// file at the limit is never cut, and the worker measures the result.
|
|
||||||
MaxSeconds int
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// lengthSlackSeconds is added to MaxSeconds so ffmpeg's rounding never clips a file that
|
|
||||||
// is really at the limit.
|
|
||||||
const lengthSlackSeconds = 60
|
|
||||||
|
|
||||||
// ToWAV writes src, converted, to dst. Video streams are dropped (-vn): by now the
|
// ToWAV writes src, converted, to dst. Video streams are dropped (-vn): by now the
|
||||||
// file has already been probed as audio-only, but a cover-art stream must not end
|
// file has already been probed as audio-only, but a cover-art stream must not end
|
||||||
// up in the WAV.
|
// up in the WAV.
|
||||||
@@ -39,23 +29,14 @@ func (f *FFmpeg) ToWAV(ctx context.Context, src, dst string) error {
|
|||||||
}
|
}
|
||||||
// Input restrictions go BEFORE -i: placed after it they would apply to the output.
|
// Input restrictions go BEFORE -i: placed after it they would apply to the output.
|
||||||
args := append([]string{"-y", "-v", "error"}, inputRestrictions...)
|
args := append([]string{"-y", "-v", "error"}, inputRestrictions...)
|
||||||
args = append(args, "-i", src, "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le")
|
args = append(args, "-i", src, "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le", "-f", "wav", dst)
|
||||||
if f.MaxSeconds > 0 {
|
|
||||||
// After -i: an OUTPUT cap. Before it, it would limit what is read, not written.
|
|
||||||
args = append(args, "-t", strconv.Itoa(f.MaxSeconds+lengthSlackSeconds))
|
|
||||||
}
|
|
||||||
args = append(args, "-f", "wav", dst)
|
|
||||||
cmd := exec.CommandContext(ctx, bin, args...)
|
cmd := exec.CommandContext(ctx, bin, args...)
|
||||||
cmd.WaitDelay = killGrace
|
cmd.WaitDelay = killGrace
|
||||||
// Only a bounded head of ffmpeg's output is kept: a hostile file made it print ~0.4
|
if out, err := cmd.CombinedOutput(); err != nil {
|
||||||
// bytes per input byte, all of which CombinedOutput would buffer and copy (#49).
|
|
||||||
out := &headWriter{max: maxStderr}
|
|
||||||
cmd.Stdout, cmd.Stderr = out, out
|
|
||||||
if err := cmd.Run(); err != nil {
|
|
||||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||||
return fmt.Errorf("audio: ffmpeg: %w", ctxErr)
|
return fmt.Errorf("audio: ffmpeg: %w", ctxErr)
|
||||||
}
|
}
|
||||||
return fmt.Errorf("audio: ffmpeg: %w: %s", err, strings.TrimSpace(out.buf.String()))
|
return fmt.Errorf("audio: ffmpeg: %w: %s", err, strings.TrimSpace(string(out)))
|
||||||
}
|
}
|
||||||
// A zero exit that wrote nothing is still a failure: never hand whisper a
|
// A zero exit that wrote nothing is still a failure: never hand whisper a
|
||||||
// missing file and let the error surface three layers later.
|
// missing file and let the error surface three layers later.
|
||||||
|
|||||||
@@ -77,64 +77,3 @@ printf data > "$last"`)
|
|||||||
assert.Less(t, pw, in, "input options must come BEFORE -i or ffmpeg applies them to the output")
|
assert.Less(t, pw, in, "input options must come BEFORE -i or ffmpeg applies them to the output")
|
||||||
assert.Less(t, fw, in)
|
assert.Less(t, fw, in)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestFFmpegToWAV_PassesAnOutputLengthCapAfterTheInputWhenAMaximumIsSet(t *testing.T) {
|
|
||||||
// The duration limit is a CLAIM in the file's header; a file that lies about it can
|
|
||||||
// convert to hours of audio (adversarial review F2: a 2 h file claiming 10 s became a
|
|
||||||
// 120-minute WAV). The cap is an OUTPUT option, after -i, with a minute of slack.
|
|
||||||
rec := filepath.Join(t.TempDir(), "argv")
|
|
||||||
bin := stub(t, `printf '%s\n' "$@" > `+rec+`
|
|
||||||
for a; do last="$a"; done
|
|
||||||
printf data > "$last"`)
|
|
||||||
|
|
||||||
err := (&audio.FFmpeg{Bin: bin, MaxSeconds: 5400}).ToWAV(context.Background(), "/spool/in", filepath.Join(t.TempDir(), "out.wav"))
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
raw, err := os.ReadFile(rec)
|
|
||||||
require.NoError(t, err)
|
|
||||||
joined := " " + strings.Join(strings.Split(strings.TrimSpace(string(raw)), "\n"), " ") + " "
|
|
||||||
assert.Contains(t, joined, " -t 5460 ", "limit plus a minute of slack")
|
|
||||||
assert.Less(t, strings.Index(joined, " -i /spool/in "), strings.Index(joined, " -t 5460 "), "-t after -i is an output cap; before it, an input one")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFFmpegToWAV_PassesNoLengthCapWhenNoMaximumIsSet(t *testing.T) {
|
|
||||||
rec := filepath.Join(t.TempDir(), "argv")
|
|
||||||
bin := stub(t, `printf '%s\n' "$@" > `+rec+`
|
|
||||||
for a; do last="$a"; done
|
|
||||||
printf data > "$last"`)
|
|
||||||
|
|
||||||
require.NoError(t, (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/spool/in", filepath.Join(t.TempDir(), "out.wav")))
|
|
||||||
|
|
||||||
raw, err := os.ReadFile(rec)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.NotContains(t, " "+strings.ReplaceAll(string(raw), "\n", " ")+" ", " -t ")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFFmpegToWAV_CapsTheStreamCountBeforeTheInput(t *testing.T) {
|
|
||||||
// ffmpeg also allocated ~570 MB opening the 900-stream file (#49).
|
|
||||||
rec := filepath.Join(t.TempDir(), "argv")
|
|
||||||
bin := stub(t, `printf '%s\n' "$@" > `+rec+`
|
|
||||||
for a; do last="$a"; done
|
|
||||||
printf data > "$last"`)
|
|
||||||
|
|
||||||
err := (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/spool/in", filepath.Join(t.TempDir(), "o.wav"))
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
args := argvOf(t, rec)
|
|
||||||
ms, in := indexOf(args, "-max_streams"), indexOf(args, "-i")
|
|
||||||
require.GreaterOrEqual(t, ms, 0, "argv: %v", args)
|
|
||||||
assert.Equal(t, "32", args[ms+1])
|
|
||||||
assert.Less(t, ms, in, "an input option placed after -i is applied to the output")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFFmpegToWAV_ABoundedPartOfAFloodingStderrIsKept(t *testing.T) {
|
|
||||||
// A hostile mp3 made ffmpeg print ~0.4 bytes of stderr per input byte (83 MB for a
|
|
||||||
// 209 MB file), all of it buffered, then copied twice into the error (#49 review).
|
|
||||||
bin := stub(t, `head -c 5000000 /dev/zero | tr '\0' 'e' >&2; exit 1`)
|
|
||||||
|
|
||||||
err := (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/spool/in", filepath.Join(t.TempDir(), "o.wav"))
|
|
||||||
|
|
||||||
require.Error(t, err)
|
|
||||||
assert.Less(t, len(err.Error()), 16*1024, "the error carries a bounded head of stderr, not all of it")
|
|
||||||
assert.Contains(t, err.Error(), "eeee", "and still carries some of it")
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -4,7 +4,6 @@
|
|||||||
package audio
|
package audio
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
@@ -20,51 +19,6 @@ import (
|
|||||||
type FFprobe struct {
|
type FFprobe struct {
|
||||||
// Bin is the ffprobe executable ("ffprobe" resolves via PATH).
|
// Bin is the ffprobe executable ("ffprobe" resolves via PATH).
|
||||||
Bin string
|
Bin string
|
||||||
// MaxOutput bounds how much of ffprobe's stdout is read; zero means DefaultMaxOutput.
|
|
||||||
MaxOutput int
|
|
||||||
// Timeout bounds one probe; zero means DefaultProbeTimeout. A probe is milliseconds of
|
|
||||||
// work on a real file, so one that runs this long is the FILE's doing (#49).
|
|
||||||
Timeout time.Duration
|
|
||||||
}
|
|
||||||
|
|
||||||
// DefaultProbeTimeout is the probe deadline when FFprobe.Timeout is unset.
|
|
||||||
const DefaultProbeTimeout = 45 * time.Second
|
|
||||||
|
|
||||||
// DefaultMaxOutput is far above the narrowed probe output of a legitimate file (a few
|
|
||||||
// hundred bytes) and far below what a hostile one can make ffprobe print.
|
|
||||||
const DefaultMaxOutput = 1 << 20
|
|
||||||
|
|
||||||
// maxStderr is how much of ffprobe's stderr is kept for the error message.
|
|
||||||
const maxStderr = 4096
|
|
||||||
|
|
||||||
// errOutputCap is the write error that stops ffprobe once it prints past the cap.
|
|
||||||
var errOutputCap = errors.New("audio: ffprobe output exceeds the cap")
|
|
||||||
|
|
||||||
// capWriter collects up to max bytes and then refuses further writes, which closes the
|
|
||||||
// pipe and ends the process instead of buffering whatever it prints.
|
|
||||||
type capWriter struct {
|
|
||||||
buf bytes.Buffer
|
|
||||||
max int
|
|
||||||
}
|
|
||||||
|
|
||||||
func (w *capWriter) Write(p []byte) (int, error) {
|
|
||||||
if w.buf.Len()+len(p) > w.max {
|
|
||||||
return 0, errOutputCap
|
|
||||||
}
|
|
||||||
return w.buf.Write(p)
|
|
||||||
}
|
|
||||||
|
|
||||||
// headWriter keeps the first max bytes and silently drops the rest.
|
|
||||||
type headWriter struct {
|
|
||||||
buf bytes.Buffer
|
|
||||||
max int
|
|
||||||
}
|
|
||||||
|
|
||||||
func (w *headWriter) Write(p []byte) (int, error) {
|
|
||||||
if room := w.max - w.buf.Len(); room > 0 {
|
|
||||||
w.buf.Write(p[:min(room, len(p))])
|
|
||||||
}
|
|
||||||
return len(p), nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// AllowedDemuxers are the input formats an upload may be. The input is
|
// AllowedDemuxers are the input formats an upload may be. The input is
|
||||||
@@ -76,22 +30,9 @@ func (w *headWriter) Write(p []byte) (int, error) {
|
|||||||
// formats. "mov" covers .mp4/.m4a/.3gp; "matroska" covers .webm.
|
// formats. "mov" covers .mp4/.m4a/.3gp; "matroska" covers .webm.
|
||||||
const AllowedDemuxers = "wav,mov,mp4,m4a,3gp,mp3,ogg,matroska,webm,flac,aac,opus,aiff,amr,asf"
|
const AllowedDemuxers = "wav,mov,mp4,m4a,3gp,mp3,ogg,matroska,webm,flac,aac,opus,aiff,amr,asf"
|
||||||
|
|
||||||
// MaxStreams bounds how many streams a file may declare. A 1.3 MB file declaring 900 of
|
|
||||||
// them cost ~570 MB of RSS in each of ffprobe and ffmpeg (#49), against a 512 Mi pod;
|
|
||||||
// real recordings have a handful.
|
|
||||||
const MaxStreams = "32"
|
|
||||||
|
|
||||||
// inputRestrictions are ffmpeg/ffprobe INPUT options and must precede the input.
|
// inputRestrictions are ffmpeg/ffprobe INPUT options and must precede the input.
|
||||||
// Local files and pipes only: no network, no other protocols, and a bounded stream count.
|
// Local files and pipes only: no network, no other protocols.
|
||||||
var inputRestrictions = []string{
|
var inputRestrictions = []string{"-protocol_whitelist", "file,pipe", "-format_whitelist", AllowedDemuxers}
|
||||||
"-protocol_whitelist", "file,pipe",
|
|
||||||
"-format_whitelist", AllowedDemuxers,
|
|
||||||
"-max_streams", MaxStreams,
|
|
||||||
}
|
|
||||||
|
|
||||||
// probeEntries is exactly what ParseProbe reads. -show_format/-show_streams print every
|
|
||||||
// tag too, and a 120 MB WAV made 491 MB of JSON that Output() buffers.
|
|
||||||
const probeEntries = "format=duration:stream=codec_type:stream_disposition=attached_pic"
|
|
||||||
|
|
||||||
// killGrace bounds how long Probe waits for a cancelled ffprobe's pipes to close.
|
// killGrace bounds how long Probe waits for a cancelled ffprobe's pipes to close.
|
||||||
const killGrace = 2 * time.Second
|
const killGrace = 2 * time.Second
|
||||||
@@ -107,38 +48,21 @@ func (f *FFprobe) Probe(ctx context.Context, path string) (uploads.Probe, error)
|
|||||||
bin = "ffprobe"
|
bin = "ffprobe"
|
||||||
}
|
}
|
||||||
args := append([]string{"-v", "error"}, inputRestrictions...)
|
args := append([]string{"-v", "error"}, inputRestrictions...)
|
||||||
args = append(args, "-print_format", "json", "-show_entries", probeEntries, path)
|
args = append(args, "-print_format", "json", "-show_format", "-show_streams", path)
|
||||||
timeout := f.Timeout
|
cmd := exec.CommandContext(ctx, bin, args...)
|
||||||
if timeout <= 0 {
|
|
||||||
timeout = DefaultProbeTimeout
|
|
||||||
}
|
|
||||||
probeCtx, cancel := context.WithTimeout(ctx, timeout)
|
|
||||||
defer cancel()
|
|
||||||
cmd := exec.CommandContext(probeCtx, bin, args...)
|
|
||||||
cmd.WaitDelay = killGrace
|
cmd.WaitDelay = killGrace
|
||||||
maxOut := f.MaxOutput
|
out, err := cmd.Output()
|
||||||
if maxOut <= 0 {
|
if err != nil {
|
||||||
maxOut = DefaultMaxOutput
|
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||||
}
|
|
||||||
stdout, stderr := &capWriter{max: maxOut}, &headWriter{max: maxStderr}
|
|
||||||
cmd.Stdout, cmd.Stderr = stdout, stderr
|
|
||||||
if err := cmd.Run(); err != nil {
|
|
||||||
if ctxErr := ctx.Err(); ctxErr != nil { // the caller gave up: not a verdict on the file
|
|
||||||
return uploads.Probe{}, fmt.Errorf("audio: ffprobe: %w", ctxErr)
|
return uploads.Probe{}, fmt.Errorf("audio: ffprobe: %w", ctxErr)
|
||||||
}
|
}
|
||||||
if probeCtx.Err() != nil { // our own deadline passed while the caller was still waiting
|
|
||||||
return uploads.Probe{}, fmt.Errorf("%w: ffprobe did not finish within %s", uploads.ErrUnreadable, timeout)
|
|
||||||
}
|
|
||||||
if errors.Is(err, errOutputCap) {
|
|
||||||
return uploads.Probe{}, fmt.Errorf("%w: ffprobe printed more than %d bytes", uploads.ErrUnreadable, maxOut)
|
|
||||||
}
|
|
||||||
var exit *exec.ExitError
|
var exit *exec.ExitError
|
||||||
if errors.As(err, &exit) {
|
if errors.As(err, &exit) {
|
||||||
return uploads.Probe{}, fmt.Errorf("%w: ffprobe: %s", uploads.ErrUnreadable, stderr.buf.String())
|
return uploads.Probe{}, fmt.Errorf("%w: ffprobe: %s", uploads.ErrUnreadable, exit.Stderr)
|
||||||
}
|
}
|
||||||
return uploads.Probe{}, fmt.Errorf("audio: run ffprobe: %w", err)
|
return uploads.Probe{}, fmt.Errorf("audio: run ffprobe: %w", err)
|
||||||
}
|
}
|
||||||
p, err := ParseProbe(stdout.buf.Bytes())
|
p, err := ParseProbe(out)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return uploads.Probe{}, fmt.Errorf("audio: ffprobe output: %w", err)
|
return uploads.Probe{}, fmt.Errorf("audio: ffprobe output: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -184,78 +184,3 @@ func TestAllowedDemuxers_ExcludeThePlaylistFormats(t *testing.T) {
|
|||||||
assert.Contains(t, strings.Split(audio.AllowedDemuxers, ","), good)
|
assert.Contains(t, strings.Split(audio.AllowedDemuxers, ","), good)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestFFprobeProbe_CapsTheStreamCountBeforeTheInput(t *testing.T) {
|
|
||||||
// A 1.3 MB file declaring 900 streams cost ~570 MB of RSS in ffprobe (#49) against
|
|
||||||
// a 512 Mi pod. -max_streams is an input option: it must precede the path.
|
|
||||||
bin, argvFile := argvStub(t)
|
|
||||||
|
|
||||||
_, err := (&audio.FFprobe{Bin: bin}).Probe(context.Background(), "/spool/upload-1")
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
args := argvOf(t, argvFile)
|
|
||||||
ms := indexOf(args, "-max_streams")
|
|
||||||
require.GreaterOrEqual(t, ms, 0, "argv: %v", args)
|
|
||||||
assert.Equal(t, "32", args[ms+1])
|
|
||||||
assert.Less(t, ms, len(args)-1, "must come before the path, which stays last")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFFprobeProbe_AsksOnlyForTheFieldsParseProbeReads(t *testing.T) {
|
|
||||||
// -show_format/-show_streams print every tag: a 120 MB WAV produced 491 MB of JSON
|
|
||||||
// that cmd.Output() buffers in Go (#49).
|
|
||||||
bin, argvFile := argvStub(t)
|
|
||||||
|
|
||||||
_, err := (&audio.FFprobe{Bin: bin}).Probe(context.Background(), "/spool/upload-1")
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
args := argvOf(t, argvFile)
|
|
||||||
assert.Equal(t, -1, indexOf(args, "-show_format"), "argv: %v", args)
|
|
||||||
assert.Equal(t, -1, indexOf(args, "-show_streams"), "argv: %v", args)
|
|
||||||
se := indexOf(args, "-show_entries")
|
|
||||||
require.GreaterOrEqual(t, se, 0, "argv: %v", args)
|
|
||||||
assert.Equal(t, "format=duration:stream=codec_type:stream_disposition=attached_pic", args[se+1])
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFFprobeProbe_OutputPastTheCapIsUnreadableNotBuffered(t *testing.T) {
|
|
||||||
// ffprobe's stdout is attacker-influenced (tags, 491 MB observed). Past the cap the
|
|
||||||
// probe is stopped and the file is reported as unreadable (#49).
|
|
||||||
bin := stub(t, `head -c 200000 /dev/zero | tr '\0' 'a'`)
|
|
||||||
|
|
||||||
_, err := (&audio.FFprobe{Bin: bin, MaxOutput: 4096}).Probe(context.Background(), "/x")
|
|
||||||
|
|
||||||
assert.ErrorIs(t, err, uploads.ErrUnreadable)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFFprobeProbe_OutputWithinTheCapIsParsed(t *testing.T) {
|
|
||||||
bin := stub(t, `echo '{"streams":[{"codec_type":"audio"}],"format":{"duration":"90"}}'`)
|
|
||||||
|
|
||||||
got, err := (&audio.FFprobe{Bin: bin, MaxOutput: 4096}).Probe(context.Background(), "/x")
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.True(t, got.HasAudio)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFFprobeProbe_AHungProbeTimesOutOnItsOwnDeadlineAsUnreadableNotAsOurFault(t *testing.T) {
|
|
||||||
// Accept runs the probe on the request context, which has no deadline of its own;
|
|
||||||
// a probe that never returns would pin a request (#49). `exec` so the sleep is the
|
|
||||||
// process itself and not a child holding the pipe open.
|
|
||||||
bin := stub(t, `exec sleep 30`)
|
|
||||||
|
|
||||||
start := time.Now()
|
|
||||||
_, err := (&audio.FFprobe{Bin: bin, Timeout: 200 * time.Millisecond}).Probe(context.Background(), "/x")
|
|
||||||
|
|
||||||
assert.ErrorIs(t, err, uploads.ErrUnreadable)
|
|
||||||
assert.Less(t, time.Since(start), 3*time.Second)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFFprobeProbe_ParentCancellationIsStillNotUnreadableWhenATimeoutIsSet(t *testing.T) {
|
|
||||||
bin := stub(t, `exec sleep 30`)
|
|
||||||
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
_, err := (&audio.FFprobe{Bin: bin, Timeout: time.Minute}).Probe(ctx, "/x")
|
|
||||||
|
|
||||||
require.Error(t, err)
|
|
||||||
assert.False(t, errors.Is(err, uploads.ErrUnreadable), "the caller giving up is not a verdict on the file")
|
|
||||||
assert.ErrorIs(t, err, context.DeadlineExceeded)
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,156 +0,0 @@
|
|||||||
package llm
|
|
||||||
|
|
||||||
// Observes how much of a gateway response the client will read. The body is read before
|
|
||||||
// the status is looked at, so a misbehaving gateway or proxy that streams forever was
|
|
||||||
// bounded only by the client's overall timeout (#53).
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"strings"
|
|
||||||
"sync/atomic"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// endlessGateway answers with the given status and streams filler until the client hangs
|
|
||||||
// up; wrote reports how many bytes it managed to send.
|
|
||||||
func endlessGateway(t *testing.T, status int, wrote *atomic.Int64) *httptest.Server {
|
|
||||||
t.Helper()
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
w.WriteHeader(status)
|
|
||||||
chunk := []byte(strings.Repeat("a", 4096))
|
|
||||||
for {
|
|
||||||
n, err := w.Write(chunk)
|
|
||||||
wrote.Add(int64(n))
|
|
||||||
if err != nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if f, ok := w.(http.Flusher); ok {
|
|
||||||
f.Flush()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}))
|
|
||||||
t.Cleanup(srv.Close)
|
|
||||||
return srv
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestClient_ASuccessBodyLargerThanTheCapIsAnErrorAndTheClientStopsReading(t *testing.T) {
|
|
||||||
var wrote atomic.Int64
|
|
||||||
srv := endlessGateway(t, http.StatusOK, &wrote)
|
|
||||||
c := New(srv.URL, "", "m", 30*time.Second, WithMaxResponseBytes(8192))
|
|
||||||
|
|
||||||
start := time.Now()
|
|
||||||
_, err := c.Complete(context.Background(), "s", "u")
|
|
||||||
|
|
||||||
if err == nil || !strings.Contains(err.Error(), "exceeds") {
|
|
||||||
t.Fatalf("want a response-too-large error, got %v", err)
|
|
||||||
}
|
|
||||||
if time.Since(start) > 5*time.Second {
|
|
||||||
t.Fatalf("the client kept reading: took %s", time.Since(start))
|
|
||||||
}
|
|
||||||
// Hanging up lets the server's writes fail; allow what the transport had already buffered.
|
|
||||||
time.Sleep(100 * time.Millisecond)
|
|
||||||
if got := wrote.Load(); got > 8192+1<<20 {
|
|
||||||
t.Fatalf("server managed to send %d bytes after a cap of 8192", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestClient_AFailingBodyIsReadOnlyUpToTheErrorBound(t *testing.T) {
|
|
||||||
var wrote atomic.Int64
|
|
||||||
srv := endlessGateway(t, http.StatusBadGateway, &wrote)
|
|
||||||
c := New(srv.URL, "", "m", 30*time.Second)
|
|
||||||
|
|
||||||
start := time.Now()
|
|
||||||
_, err := c.Complete(context.Background(), "s", "u")
|
|
||||||
|
|
||||||
he, ok := err.(*HTTPError)
|
|
||||||
if !ok {
|
|
||||||
t.Fatalf("want *HTTPError, got %T: %v", err, err)
|
|
||||||
}
|
|
||||||
if he.Code != http.StatusBadGateway {
|
|
||||||
t.Fatalf("code %d", he.Code)
|
|
||||||
}
|
|
||||||
if len(he.Body) > maxErrorBody+1 {
|
|
||||||
t.Fatalf("kept %d bytes of an error body, bound is %d", len(he.Body), maxErrorBody)
|
|
||||||
}
|
|
||||||
if time.Since(start) > 5*time.Second {
|
|
||||||
t.Fatalf("the client kept reading an endless error body: took %s", time.Since(start))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// sizedGateway answers 200 with a valid completion padded with trailing whitespace to
|
|
||||||
// exactly size bytes, so the boundary of the cap can be probed to the byte.
|
|
||||||
func sizedGateway(t *testing.T, size int) *httptest.Server {
|
|
||||||
t.Helper()
|
|
||||||
const base = `{"choices":[{"message":{"content":"ok"}}]}`
|
|
||||||
payload := base + strings.Repeat(" ", size-len(base))
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
w.Header().Set("Content-Type", "application/json")
|
|
||||||
_, _ = w.Write([]byte(payload))
|
|
||||||
}))
|
|
||||||
t.Cleanup(srv.Close)
|
|
||||||
return srv
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestClient_ABodyOfExactlyTheCapIsReadAndOneByteMoreIsRefused(t *testing.T) {
|
|
||||||
const capBytes = 4096
|
|
||||||
for _, tt := range []struct {
|
|
||||||
name string
|
|
||||||
size int
|
|
||||||
wantErr bool
|
|
||||||
}{
|
|
||||||
{"one under", capBytes - 1, false},
|
|
||||||
{"exactly the cap", capBytes, false},
|
|
||||||
{"one over", capBytes + 1, true},
|
|
||||||
} {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
c := New(sizedGateway(t, tt.size).URL, "", "m", 5*time.Second, WithMaxResponseBytes(capBytes))
|
|
||||||
|
|
||||||
got, err := c.Complete(context.Background(), "s", "u")
|
|
||||||
|
|
||||||
if tt.wantErr {
|
|
||||||
if err == nil || !strings.Contains(err.Error(), "exceeds") {
|
|
||||||
t.Fatalf("want a response-too-large error, got %q, %v", got, err)
|
|
||||||
}
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if err != nil || got != "ok" {
|
|
||||||
t.Fatalf("want ok, got %q, %v", got, err)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// New is what production calls: no option. The default is therefore the live bound.
|
|
||||||
func TestClient_TheDefaultCapIsEightMiBWhenNoOptionIsGiven(t *testing.T) {
|
|
||||||
for _, tt := range []struct {
|
|
||||||
name string
|
|
||||||
size int
|
|
||||||
wantErr bool
|
|
||||||
}{
|
|
||||||
{"exactly 8 MiB", 8 << 20, false},
|
|
||||||
{"8 MiB plus one", 8<<20 + 1, true},
|
|
||||||
} {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
c := New(sizedGateway(t, tt.size).URL, "", "m", 10*time.Second)
|
|
||||||
|
|
||||||
_, err := c.Complete(context.Background(), "s", "u")
|
|
||||||
|
|
||||||
if (err != nil) != tt.wantErr {
|
|
||||||
t.Fatalf("size %d: err = %v, wantErr %v", tt.size, err, tt.wantErr)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestClient_ANonPositiveCapOptionKeepsTheDefault(t *testing.T) {
|
|
||||||
for _, n := range []int64{0, -1} {
|
|
||||||
c := New(sizedGateway(t, 1024).URL, "", "m", 5*time.Second, WithMaxResponseBytes(n))
|
|
||||||
|
|
||||||
if got, err := c.Complete(context.Background(), "s", "u"); err != nil || got != "ok" {
|
|
||||||
t.Fatalf("WithMaxResponseBytes(%d): want ok from the default cap, got %q, %v", n, got, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -37,21 +37,6 @@ type Client struct {
|
|||||||
temp float64
|
temp float64
|
||||||
httpClient *http.Client
|
httpClient *http.Client
|
||||||
usageHook func(model string, prompt, completion int)
|
usageHook func(model string, prompt, completion int)
|
||||||
maxBody int64
|
|
||||||
}
|
|
||||||
|
|
||||||
// defaultMaxResponseBytes bounds a success body. A completion is a few KB of text; this
|
|
||||||
// leaves room for a long one and none for a gateway that streams forever (#53).
|
|
||||||
const defaultMaxResponseBytes = 8 << 20
|
|
||||||
|
|
||||||
// WithMaxResponseBytes overrides the bound on a success response body. A non-positive n is
|
|
||||||
// ignored (keeps the default).
|
|
||||||
func WithMaxResponseBytes(n int64) Option {
|
|
||||||
return func(c *Client) {
|
|
||||||
if n > 0 {
|
|
||||||
c.maxBody = n
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Option configures a Client at construction. Variadic so the existing 4-arg
|
// Option configures a Client at construction. Variadic so the existing 4-arg
|
||||||
@@ -70,29 +55,6 @@ func WithMaxTokens(n int) Option {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// maxErrorBody bounds how much of a failing response ends up in an error message.
|
|
||||||
const maxErrorBody = 512
|
|
||||||
|
|
||||||
// HTTPError is a non-200 answer from the gateway. Its text keeps the form callers that
|
|
||||||
// log it already rely on, with the body bounded; its Code is what code may branch on and
|
|
||||||
// what errkind turns into a kind. The body is provider text and can echo the prompt, so
|
|
||||||
// nothing private may persist or display Error() (ADR-035 review F3).
|
|
||||||
type HTTPError struct {
|
|
||||||
Code int
|
|
||||||
Body string
|
|
||||||
}
|
|
||||||
|
|
||||||
func (e *HTTPError) Error() string {
|
|
||||||
body := e.Body
|
|
||||||
if len(body) > maxErrorBody {
|
|
||||||
body = body[:maxErrorBody] + "…"
|
|
||||||
}
|
|
||||||
return fmt.Sprintf("LLM returned %d: %s", e.Code, body)
|
|
||||||
}
|
|
||||||
|
|
||||||
// HTTPStatus implements errkind.HTTPStatuser.
|
|
||||||
func (e *HTTPError) HTTPStatus() int { return e.Code }
|
|
||||||
|
|
||||||
// WithTemperature overrides the sampling temperature. Zero IS a valid value and is sent
|
// WithTemperature overrides the sampling temperature. Zero IS a valid value and is sent
|
||||||
// as zero: the reconciler's accuracy was measured at temperature 0 and must run there.
|
// as zero: the reconciler's accuracy was measured at temperature 0 and must run there.
|
||||||
func WithTemperature(t float64) Option {
|
func WithTemperature(t float64) Option {
|
||||||
@@ -116,7 +78,6 @@ func New(baseURL, apiKey, model string, timeout time.Duration, opts ...Option) *
|
|||||||
maxTokens: defaultMaxTokens,
|
maxTokens: defaultMaxTokens,
|
||||||
temp: defaultTemperature,
|
temp: defaultTemperature,
|
||||||
httpClient: &http.Client{Timeout: timeout},
|
httpClient: &http.Client{Timeout: timeout},
|
||||||
maxBody: defaultMaxResponseBytes,
|
|
||||||
}
|
}
|
||||||
for _, opt := range opts {
|
for _, opt := range opts {
|
||||||
opt(c)
|
opt(c)
|
||||||
@@ -200,18 +161,12 @@ func (c *Client) Complete(ctx context.Context, system, user string) (string, err
|
|||||||
}
|
}
|
||||||
defer resp.Body.Close() //nolint:errcheck
|
defer resp.Body.Close() //nolint:errcheck
|
||||||
|
|
||||||
if resp.StatusCode != http.StatusOK {
|
out, err := io.ReadAll(resp.Body)
|
||||||
// Only the bounded head is ever used (HTTPError.Error truncates to it), so read no more.
|
|
||||||
// A read error here only shortens a diagnostic; the status code below is what callers act on.
|
|
||||||
head, _ := io.ReadAll(io.LimitReader(resp.Body, maxErrorBody+1))
|
|
||||||
return "", &HTTPError{Code: resp.StatusCode, Body: string(head)}
|
|
||||||
}
|
|
||||||
out, err := io.ReadAll(io.LimitReader(resp.Body, c.maxBody+1))
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", fmt.Errorf("read response: %w", err)
|
return "", fmt.Errorf("read response: %w", err)
|
||||||
}
|
}
|
||||||
if int64(len(out)) > c.maxBody {
|
if resp.StatusCode != http.StatusOK {
|
||||||
return "", fmt.Errorf("LLM response exceeds %d bytes", c.maxBody)
|
return "", fmt.Errorf("LLM returned %d: %s", resp.StatusCode, out)
|
||||||
}
|
}
|
||||||
|
|
||||||
var cr chatResponse
|
var cr chatResponse
|
||||||
|
|||||||
@@ -1,63 +0,0 @@
|
|||||||
package llm
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/errkind"
|
|
||||||
)
|
|
||||||
|
|
||||||
// A non-200 from the gateway is an HTTP status error: its kind can be logged and stored
|
|
||||||
// without its body, which can echo the prompt (adversarial review F3).
|
|
||||||
|
|
||||||
func failingGateway(t *testing.T, status int, body string) *httptest.Server {
|
|
||||||
t.Helper()
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
w.WriteHeader(status)
|
|
||||||
_, _ = w.Write([]byte(body))
|
|
||||||
}))
|
|
||||||
t.Cleanup(srv.Close)
|
|
||||||
return srv
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestClient_ANon200IsAnHTTPStatusErrorWhoseKindCarriesNoBody(t *testing.T) {
|
|
||||||
srv := failingGateway(t, http.StatusBadRequest, `{"error":"prompt was: Anders ansvarar för virket"}`)
|
|
||||||
|
|
||||||
_, err := New(srv.URL, "", "m", 5*time.Second).Complete(context.Background(), "s", "u")
|
|
||||||
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("expected an error")
|
|
||||||
}
|
|
||||||
if got := errkind.Of(err); got != "http_400" {
|
|
||||||
t.Errorf("errkind.Of = %q, want http_400", got)
|
|
||||||
}
|
|
||||||
var he *HTTPError
|
|
||||||
if !errors.As(err, &he) || he.Code != http.StatusBadRequest {
|
|
||||||
t.Errorf("error is not an *HTTPError with the status code: %v", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestClient_TheErrorTextIsUnchangedForCallersThatLogIt(t *testing.T) {
|
|
||||||
srv := failingGateway(t, http.StatusBadGateway, "upstream down")
|
|
||||||
|
|
||||||
_, err := New(srv.URL, "", "m", 5*time.Second).Complete(context.Background(), "s", "u")
|
|
||||||
|
|
||||||
if err == nil || err.Error() != "LLM returned 502: upstream down" {
|
|
||||||
t.Errorf("error text = %v: callers that log it already rely on this form", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestClient_AnEnormousErrorBodyIsBoundedInTheErrorText(t *testing.T) {
|
|
||||||
srv := failingGateway(t, http.StatusInternalServerError, strings.Repeat("x", 100_000))
|
|
||||||
|
|
||||||
_, err := New(srv.URL, "", "m", 5*time.Second).Complete(context.Background(), "s", "u")
|
|
||||||
|
|
||||||
if err == nil || len(err.Error()) > 1024 {
|
|
||||||
t.Errorf("error text is %d bytes: a gateway's whole body must not become an error message", len(err.Error()))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,2 +0,0 @@
|
|||||||
ALTER TABLE uploaded_transcripts
|
|
||||||
DROP COLUMN language;
|
|
||||||
@@ -1,13 +0,0 @@
|
|||||||
-- Migration 021: the language an upload is transcribed as (tapir#61).
|
|
||||||
--
|
|
||||||
-- 'sv' Swedish: the engines are sent language=sv. Every row before this migration was
|
|
||||||
-- transcribed that way (the request hard-coded it), so the default is true history.
|
|
||||||
-- 'auto' Other: the engine detects the language and the Swedish-only engine is not used.
|
|
||||||
--
|
|
||||||
-- A closed set held by the table as well as by the form and the request code, so a second
|
|
||||||
-- writer or a bug cannot store a value the engines are never meant to receive. The column
|
|
||||||
-- sits on the already-FORCE'd, RLS-scoped uploaded_transcripts, so the exact-set isolation
|
|
||||||
-- test is unaffected.
|
|
||||||
ALTER TABLE uploaded_transcripts
|
|
||||||
ADD COLUMN language TEXT NOT NULL DEFAULT 'sv'
|
|
||||||
CONSTRAINT uploaded_transcripts_language_check CHECK (language IN ('sv', 'auto'));
|
|
||||||
@@ -17,10 +17,6 @@ import (
|
|||||||
// nothing about whether someone else's upload exists (ADR-033).
|
// nothing about whether someone else's upload exists (ADR-033).
|
||||||
var ErrUploadNotFound = errors.New("store: uploaded transcript not found")
|
var ErrUploadNotFound = errors.New("store: uploaded transcript not found")
|
||||||
|
|
||||||
// ErrUploadInProgress is returned by DeleteUpload for an upload that is still queued or
|
|
||||||
// transcribing: the worker would write its result to a row that no longer exists.
|
|
||||||
var ErrUploadInProgress = errors.New("store: uploaded transcript is still being processed")
|
|
||||||
|
|
||||||
// Uploaded transcripts are PRIVATE per-user content, stored in their own
|
// Uploaded transcripts are PRIVATE per-user content, stored in their own
|
||||||
// RLS-scoped table (migration 017) — never in the shared `transcripts` (ADR-021).
|
// RLS-scoped table (migration 017) — never in the shared `transcripts` (ADR-021).
|
||||||
// Every method goes through withUser, so row-level security confines it to the
|
// Every method goes through withUser, so row-level security confines it to the
|
||||||
@@ -31,12 +27,12 @@ var ErrUploadInProgress = errors.New("store: uploaded transcript is still being
|
|||||||
// CreateUpload records a new upload in the queued state and returns its id.
|
// CreateUpload records a new upload in the queued state and returns its id.
|
||||||
// hadVideo records that the file carried a video track (only its audio is used);
|
// hadVideo records that the file carried a video track (only its audio is used);
|
||||||
// it is kept for the life of the row so the page can say so.
|
// it is kept for the life of the row so the page can say so.
|
||||||
func (s *Store) CreateUpload(ctx context.Context, userID, title string, lang domain.Language, hadVideo bool) (string, error) {
|
func (s *Store) CreateUpload(ctx context.Context, userID, title string, hadVideo bool) (string, error) {
|
||||||
var id string
|
var id string
|
||||||
if err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
|
if err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
|
||||||
return tx.QueryRow(ctx,
|
return tx.QueryRow(ctx,
|
||||||
`INSERT INTO uploaded_transcripts (user_id, title, language, had_video) VALUES ($1, $2, $3, $4) RETURNING id`,
|
`INSERT INTO uploaded_transcripts (user_id, title, had_video) VALUES ($1, $2, $3) RETURNING id`,
|
||||||
userID, title, string(lang), hadVideo).Scan(&id)
|
userID, title, hadVideo).Scan(&id)
|
||||||
}); err != nil {
|
}); err != nil {
|
||||||
return "", fmt.Errorf("store: create upload: %w", err)
|
return "", fmt.Errorf("store: create upload: %w", err)
|
||||||
}
|
}
|
||||||
@@ -84,56 +80,23 @@ func (s *Store) SaveProgress(ctx context.Context, userID, id string, steps []dom
|
|||||||
WHERE id = $1 AND user_id = $2`, b, at)
|
WHERE id = $1 AND user_id = $2`, b, at)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Heartbeat stamps the upload as alive at the given time. It does not touch progress_at:
|
// Heartbeat stamps the upload as alive now. It does not touch progress_at: alive says
|
||||||
// alive says the service is still reporting, not that work is moving (ADR-035). The time
|
// the service is still reporting, not that work is moving (ADR-035).
|
||||||
// comes from the application clock, the one progress_at and the page's health check use,
|
func (s *Store) Heartbeat(ctx context.Context, userID, id string) error {
|
||||||
// so a database clock that disagrees cannot make a healthy job look lost (#51).
|
|
||||||
func (s *Store) Heartbeat(ctx context.Context, userID, id string, at time.Time) error {
|
|
||||||
return s.updateUpload(ctx, userID, id, "heartbeat",
|
return s.updateUpload(ctx, userID, id, "heartbeat",
|
||||||
`UPDATE uploaded_transcripts SET alive_at = $3
|
`UPDATE uploaded_transcripts SET alive_at = NOW()
|
||||||
WHERE id = $1 AND user_id = $2`, at)
|
WHERE id = $1 AND user_id = $2`)
|
||||||
}
|
}
|
||||||
|
|
||||||
// FailUpload moves a queued or transcribing upload to failed with a typed reason. A
|
// FailUpload moves an upload to failed with a typed reason.
|
||||||
// finished upload is left as it was: a shutdown can record a job as failed after its
|
|
||||||
// transcript was already committed (the commit lands, the caller sees a cancelled
|
|
||||||
// context), and that must not hide the text. Such a call touches no row and returns
|
|
||||||
// ErrUploadNotFound.
|
|
||||||
func (s *Store) FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error {
|
func (s *Store) FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error {
|
||||||
return s.updateUpload(ctx, userID, id, "fail upload",
|
return s.updateUpload(ctx, userID, id, "fail upload",
|
||||||
`UPDATE uploaded_transcripts
|
`UPDATE uploaded_transcripts
|
||||||
SET status = 'failed', error_kind = $3, finished_at = NOW()
|
SET status = 'failed', error_kind = $3, finished_at = NOW()
|
||||||
WHERE id = $1 AND user_id = $2 AND status IN ('queued', 'transcribing')`,
|
WHERE id = $1 AND user_id = $2`,
|
||||||
string(kind))
|
string(kind))
|
||||||
}
|
}
|
||||||
|
|
||||||
// DeleteUpload removes one of the caller's finished (done or failed) uploads. An upload
|
|
||||||
// still queued or transcribing is refused with ErrUploadInProgress and left untouched; one
|
|
||||||
// that is not the caller's is ErrUploadNotFound, as for every other write (#37).
|
|
||||||
func (s *Store) DeleteUpload(ctx context.Context, userID, id string) error {
|
|
||||||
return s.withUser(ctx, userID, func(tx pgx.Tx) error {
|
|
||||||
ct, err := tx.Exec(ctx,
|
|
||||||
`DELETE FROM uploaded_transcripts
|
|
||||||
WHERE id = $1 AND user_id = $2 AND status IN ('done', 'failed')`, id, userID)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("store: delete upload: %w", err)
|
|
||||||
}
|
|
||||||
if ct.RowsAffected() > 0 {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
var exists bool
|
|
||||||
if err := tx.QueryRow(ctx,
|
|
||||||
`SELECT EXISTS (SELECT 1 FROM uploaded_transcripts WHERE id = $1 AND user_id = $2)`,
|
|
||||||
id, userID).Scan(&exists); err != nil {
|
|
||||||
return fmt.Errorf("store: delete upload: %w", err)
|
|
||||||
}
|
|
||||||
if exists {
|
|
||||||
return ErrUploadInProgress
|
|
||||||
}
|
|
||||||
return ErrUploadNotFound
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// updateUpload runs one UPDATE keyed on ($1 = id, $2 = userID) plus extra args,
|
// updateUpload runs one UPDATE keyed on ($1 = id, $2 = userID) plus extra args,
|
||||||
// and maps "no row touched" to ErrUploadNotFound — which is what a write to another
|
// and maps "no row touched" to ErrUploadNotFound — which is what a write to another
|
||||||
// user's upload looks like under RLS (the row is invisible, not forbidden).
|
// user's upload looks like under RLS (the row is invisible, not forbidden).
|
||||||
@@ -221,7 +184,7 @@ func (s *Store) FailInterruptedUploads(ctx context.Context) (int, error) {
|
|||||||
return total, nil
|
return total, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
const uploadSelect = `SELECT id, user_id, title, language, status, content, segments, endpoint,
|
const uploadSelect = `SELECT id, user_id, title, status, content, segments, endpoint,
|
||||||
audio_seconds, error_kind, had_video, reconciliation, progress, progress_at, alive_at,
|
audio_seconds, error_kind, had_video, reconciliation, progress, progress_at, alive_at,
|
||||||
created_at, finished_at FROM uploaded_transcripts`
|
created_at, finished_at FROM uploaded_transcripts`
|
||||||
|
|
||||||
@@ -234,13 +197,11 @@ func scanUpload(row pgx.Row, u *domain.UploadedTranscript, withText bool) error
|
|||||||
seconds *float64
|
seconds *float64
|
||||||
finished, progressAt, aliveAt *time.Time
|
finished, progressAt, aliveAt *time.Time
|
||||||
)
|
)
|
||||||
var language string
|
if err := row.Scan(&u.ID, &u.UserID, &u.Title, &status, &content, &segments, &endpoint,
|
||||||
if err := row.Scan(&u.ID, &u.UserID, &u.Title, &language, &status, &content, &segments, &endpoint,
|
|
||||||
&seconds, &errKind, &u.HadVideo, &reconciliation, &progress, &progressAt, &aliveAt,
|
&seconds, &errKind, &u.HadVideo, &reconciliation, &progress, &progressAt, &aliveAt,
|
||||||
&u.CreatedAt, &finished); err != nil {
|
&u.CreatedAt, &finished); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
u.Language = domain.Language(language)
|
|
||||||
u.Status = domain.UploadStatus(deref(status))
|
u.Status = domain.UploadStatus(deref(status))
|
||||||
u.Endpoint = deref(endpoint)
|
u.Endpoint = deref(endpoint)
|
||||||
u.ErrorKind = domain.UploadErrorKind(deref(errKind))
|
u.ErrorKind = domain.UploadErrorKind(deref(errKind))
|
||||||
|
|||||||
@@ -1,68 +0,0 @@
|
|||||||
package store_test
|
|
||||||
|
|
||||||
// Deleting one recording (#37). A finished or failed upload can be removed by its owner;
|
|
||||||
// one that is queued or transcribing is refused until it ends, because the worker would
|
|
||||||
// otherwise write its result to a row that is gone (D3 in the sprint plan).
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/adapters/store"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestDeleteUpload_RemovesAFinishedOrFailedUploadOfTheOwner(t *testing.T) {
|
|
||||||
for name, finish := range map[string]func(*store.Store, string) error{
|
|
||||||
"done": func(s *store.Store, id string) error {
|
|
||||||
return s.CompleteUpload(context.Background(), userA, id, domain.Transcription{Text: "synthetic", Endpoint: "iguana/kb-whisper"})
|
|
||||||
},
|
|
||||||
"failed": func(s *store.Store, id string) error {
|
|
||||||
return s.FailUpload(context.Background(), userA, id, domain.UploadErrTranscriptionFailed)
|
|
||||||
},
|
|
||||||
} {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
s := newStore(t)
|
|
||||||
id := startedUpload(t, s)
|
|
||||||
require.NoError(t, s.StartTranscribing(context.Background(), userA, id))
|
|
||||||
require.NoError(t, finish(s, id))
|
|
||||||
|
|
||||||
require.NoError(t, s.DeleteUpload(context.Background(), userA, id))
|
|
||||||
|
|
||||||
_, err := s.GetUpload(context.Background(), userA, id)
|
|
||||||
assert.ErrorIs(t, err, store.ErrUploadNotFound)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestDeleteUpload_RefusesAnUploadThatIsStillQueuedOrTranscribingAndKeepsTheRow(t *testing.T) {
|
|
||||||
s := newStore(t)
|
|
||||||
ctx := context.Background()
|
|
||||||
queued := startedUpload(t, s)
|
|
||||||
running, err := s.CreateUpload(ctx, userA, "running", domain.LangSwedish, false)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NoError(t, s.StartTranscribing(ctx, userA, running))
|
|
||||||
|
|
||||||
for _, id := range []string{queued, running} {
|
|
||||||
assert.ErrorIs(t, s.DeleteUpload(ctx, userA, id), store.ErrUploadInProgress)
|
|
||||||
_, err := s.GetUpload(ctx, userA, id)
|
|
||||||
assert.NoError(t, err, "a refused delete must leave the row")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestDeleteUpload_AnotherUsersUploadIsNotFoundAndSurvives(t *testing.T) {
|
|
||||||
s := newStore(t)
|
|
||||||
ctx := context.Background()
|
|
||||||
id := startedUpload(t, s)
|
|
||||||
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
|
||||||
require.NoError(t, s.FailUpload(ctx, userA, id, domain.UploadErrTranscriptionFailed))
|
|
||||||
|
|
||||||
assert.ErrorIs(t, s.DeleteUpload(ctx, userB, id), store.ErrUploadNotFound,
|
|
||||||
"existence is not disclosed to a non-owner")
|
|
||||||
|
|
||||||
_, err := s.GetUpload(ctx, userA, id)
|
|
||||||
assert.NoError(t, err, "B's delete must not have removed A's row")
|
|
||||||
}
|
|
||||||
@@ -1,47 +0,0 @@
|
|||||||
package store_test
|
|
||||||
|
|
||||||
// A shutdown can record a job as failed after its transcript was already committed (the
|
|
||||||
// commit lands, the caller sees a cancelled context). FailUpload must not be able to turn a
|
|
||||||
// finished row into a failed one and hide the text (#50 review).
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestFailUpload_LeavesAFinishedUploadAsItWas(t *testing.T) {
|
|
||||||
s := newStore(t)
|
|
||||||
ctx := context.Background()
|
|
||||||
id := startedUpload(t, s)
|
|
||||||
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
|
||||||
require.NoError(t, s.CompleteUpload(ctx, userA, id, domain.Transcription{Text: "synthetic text", Endpoint: "iguana/kb-whisper"}))
|
|
||||||
|
|
||||||
_ = s.FailUpload(ctx, userA, id, domain.UploadErrInterrupted)
|
|
||||||
|
|
||||||
got, err := s.GetUpload(ctx, userA, id)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Equal(t, domain.UploadDone, got.Status)
|
|
||||||
assert.Equal(t, "synthetic text", got.Content)
|
|
||||||
assert.Empty(t, got.ErrorKind)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFailUpload_StillFailsAQueuedOrTranscribingUpload(t *testing.T) {
|
|
||||||
s := newStore(t)
|
|
||||||
ctx := context.Background()
|
|
||||||
queued := startedUpload(t, s)
|
|
||||||
running, err := s.CreateUpload(ctx, userA, "r", domain.LangSwedish, false)
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NoError(t, s.StartTranscribing(ctx, userA, running))
|
|
||||||
|
|
||||||
for _, id := range []string{queued, running} {
|
|
||||||
require.NoError(t, s.FailUpload(ctx, userA, id, domain.UploadErrBusy))
|
|
||||||
got, err := s.GetUpload(ctx, userA, id)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Equal(t, domain.UploadFailed, got.Status)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -20,7 +20,7 @@ import (
|
|||||||
func startedUpload(t *testing.T, s *store.Store) string {
|
func startedUpload(t *testing.T, s *store.Store) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
seedUsers(t, userA, userB)
|
seedUsers(t, userA, userB)
|
||||||
id, err := s.CreateUpload(context.Background(), userA, "m", domain.LangSwedish, false)
|
id, err := s.CreateUpload(context.Background(), userA, "m", false)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
return id
|
return id
|
||||||
}
|
}
|
||||||
@@ -68,13 +68,13 @@ func TestUploadProgress_AHeartbeatMovesAliveAtAndNotProgressAt(t *testing.T) {
|
|||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
at := time.Date(2026, 10, 2, 8, 30, 0, 0, time.UTC)
|
at := time.Date(2026, 10, 2, 8, 30, 0, 0, time.UTC)
|
||||||
require.NoError(t, s.SaveProgress(ctx, userA, id, []domain.ProgressStep{{Key: "audio", Label: "a", State: domain.StepRunning}}, at))
|
require.NoError(t, s.SaveProgress(ctx, userA, id, []domain.ProgressStep{{Key: "audio", Label: "a", State: domain.StepRunning}}, at))
|
||||||
beatAt := at.Add(90 * time.Second) // deliberately not the database's NOW()
|
before := time.Now().Add(-time.Second)
|
||||||
|
|
||||||
require.NoError(t, s.Heartbeat(ctx, userA, id, beatAt))
|
require.NoError(t, s.Heartbeat(ctx, userA, id))
|
||||||
|
|
||||||
got, err := s.GetUpload(ctx, userA, id)
|
got, err := s.GetUpload(ctx, userA, id)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.True(t, got.AliveAt.Equal(beatAt), "the heartbeat is stamped with the application's clock, the same one progress_at uses, not the database's NOW() (#51)")
|
assert.True(t, got.AliveAt.After(before), "the heartbeat must stamp the row with now")
|
||||||
assert.True(t, got.ProgressAt.Equal(at), "alive is not progress: a heartbeat must not move progress_at")
|
assert.True(t, got.ProgressAt.Equal(at), "alive is not progress: a heartbeat must not move progress_at")
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -99,7 +99,7 @@ func TestUploadProgress_ByAnotherUserChangesNothing(t *testing.T) {
|
|||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
|
|
||||||
errSave := s.SaveProgress(ctx, userB, id, []domain.ProgressStep{{Key: "x", Label: "X", State: domain.StepDone}}, time.Now())
|
errSave := s.SaveProgress(ctx, userB, id, []domain.ProgressStep{{Key: "x", Label: "X", State: domain.StepDone}}, time.Now())
|
||||||
errBeat := s.Heartbeat(ctx, userB, id, time.Now())
|
errBeat := s.Heartbeat(ctx, userB, id)
|
||||||
|
|
||||||
assert.ErrorIs(t, errSave, store.ErrUploadNotFound)
|
assert.ErrorIs(t, errSave, store.ErrUploadNotFound)
|
||||||
assert.ErrorIs(t, errBeat, store.ErrUploadNotFound)
|
assert.ErrorIs(t, errBeat, store.ErrUploadNotFound)
|
||||||
|
|||||||
@@ -33,7 +33,7 @@ func TestCreateUpload_StartsQueuedAndIsOwnedByTheCreator(t *testing.T) {
|
|||||||
seedUsers(t, userA)
|
seedUsers(t, userA)
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
|
|
||||||
id, err := s.CreateUpload(ctx, userA, "Teams export", domain.LangSwedish, false)
|
id, err := s.CreateUpload(ctx, userA, "Teams export", false)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
got, err := s.GetUpload(ctx, userA, id)
|
got, err := s.GetUpload(ctx, userA, id)
|
||||||
@@ -50,7 +50,7 @@ func TestUploadLifecycle_CompletedKeepsTextSegmentsEndpointAndDuration(t *testin
|
|||||||
s := newStore(t)
|
s := newStore(t)
|
||||||
seedUsers(t, userA)
|
seedUsers(t, userA)
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
|
id, err := s.CreateUpload(ctx, userA, "m", false)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
||||||
@@ -84,7 +84,7 @@ func TestUploadLifecycle_FailedRecordsTypedKindAndNoContent(t *testing.T) {
|
|||||||
s := newStore(t)
|
s := newStore(t)
|
||||||
seedUsers(t, userA)
|
seedUsers(t, userA)
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
|
id, err := s.CreateUpload(ctx, userA, "m", false)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
||||||
|
|
||||||
@@ -102,7 +102,7 @@ func TestGetUpload_AnotherUsersIDIsNotFound(t *testing.T) {
|
|||||||
s := newStore(t)
|
s := newStore(t)
|
||||||
seedUsers(t, userA, userB)
|
seedUsers(t, userA, userB)
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
id, err := s.CreateUpload(ctx, userA, "A's meeting", domain.LangSwedish, false)
|
id, err := s.CreateUpload(ctx, userA, "A's meeting", false)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
_, err = s.GetUpload(ctx, userB, id)
|
_, err = s.GetUpload(ctx, userB, id)
|
||||||
@@ -115,7 +115,7 @@ func TestUploadWrites_ByAnotherUserChangeNothing(t *testing.T) {
|
|||||||
s := newStore(t)
|
s := newStore(t)
|
||||||
seedUsers(t, userA, userB)
|
seedUsers(t, userA, userB)
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
id, err := s.CreateUpload(ctx, userA, "A's meeting", domain.LangSwedish, false)
|
id, err := s.CreateUpload(ctx, userA, "A's meeting", false)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
assert.ErrorIs(t, s.StartTranscribing(ctx, userB, id), store.ErrUploadNotFound)
|
assert.ErrorIs(t, s.StartTranscribing(ctx, userB, id), store.ErrUploadNotFound)
|
||||||
@@ -132,11 +132,11 @@ func TestListUploads_ReturnsOnlyTheCallersNewestFirstWithoutTranscriptText(t *te
|
|||||||
s := newStore(t)
|
s := newStore(t)
|
||||||
seedUsers(t, userA, userB)
|
seedUsers(t, userA, userB)
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
first, err := s.CreateUpload(ctx, userA, "first", domain.LangSwedish, false)
|
first, err := s.CreateUpload(ctx, userA, "first", false)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
second, err := s.CreateUpload(ctx, userA, "second", domain.LangSwedish, false)
|
second, err := s.CreateUpload(ctx, userA, "second", false)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
_, err = s.CreateUpload(ctx, userB, "B's", domain.LangSwedish, false)
|
_, err = s.CreateUpload(ctx, userB, "B's", false)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
require.NoError(t, s.StartTranscribing(ctx, userA, first))
|
require.NoError(t, s.StartTranscribing(ctx, userA, first))
|
||||||
require.NoError(t, s.CompleteUpload(ctx, userA, first, domain.Transcription{Text: "long text body", Endpoint: "e"}))
|
require.NoError(t, s.CompleteUpload(ctx, userA, first, domain.Transcription{Text: "long text body", Endpoint: "e"}))
|
||||||
@@ -160,12 +160,12 @@ func TestFailInterruptedUploads_FailsOnlyUnfinishedRowsForEveryUser(t *testing.T
|
|||||||
_, err := super.Exec(ctx, `INSERT INTO user_identities (dex_subject, user_id) VALUES ($1, $2)`, subject, uid)
|
_, err := super.Exec(ctx, `INSERT INTO user_identities (dex_subject, user_id) VALUES ($1, $2)`, subject, uid)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
}
|
}
|
||||||
queued, err := s.CreateUpload(ctx, userA, "queued", domain.LangSwedish, false)
|
queued, err := s.CreateUpload(ctx, userA, "queued", false)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
running, err := s.CreateUpload(ctx, userB, "running", domain.LangSwedish, false)
|
running, err := s.CreateUpload(ctx, userB, "running", false)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
require.NoError(t, s.StartTranscribing(ctx, userB, running))
|
require.NoError(t, s.StartTranscribing(ctx, userB, running))
|
||||||
done, err := s.CreateUpload(ctx, userA, "done", domain.LangSwedish, false)
|
done, err := s.CreateUpload(ctx, userA, "done", false)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
require.NoError(t, s.StartTranscribing(ctx, userA, done))
|
require.NoError(t, s.StartTranscribing(ctx, userA, done))
|
||||||
require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "kept", Endpoint: "e"}))
|
require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "kept", Endpoint: "e"}))
|
||||||
@@ -191,9 +191,9 @@ func TestCreateUpload_RecordsWhetherTheFileHadAVideoTrack(t *testing.T) {
|
|||||||
s := newStore(t)
|
s := newStore(t)
|
||||||
seedUsers(t, userA)
|
seedUsers(t, userA)
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
withVideo, err := s.CreateUpload(ctx, userA, "camera off", domain.LangSwedish, true)
|
withVideo, err := s.CreateUpload(ctx, userA, "camera off", true)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
audioOnly, err := s.CreateUpload(ctx, userA, "voice only", domain.LangSwedish, false)
|
audioOnly, err := s.CreateUpload(ctx, userA, "voice only", false)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
gotVideo, err := s.GetUpload(ctx, userA, withVideo)
|
gotVideo, err := s.GetUpload(ctx, userA, withVideo)
|
||||||
@@ -216,9 +216,9 @@ func TestUploadHadVideo_SurvivesCompletionAndFailure(t *testing.T) {
|
|||||||
s := newStore(t)
|
s := newStore(t)
|
||||||
seedUsers(t, userA)
|
seedUsers(t, userA)
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
done, err := s.CreateUpload(ctx, userA, "d", domain.LangSwedish, true)
|
done, err := s.CreateUpload(ctx, userA, "d", true)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
failed, err := s.CreateUpload(ctx, userA, "f", domain.LangSwedish, true)
|
failed, err := s.CreateUpload(ctx, userA, "f", true)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
require.NoError(t, s.StartTranscribing(ctx, userA, done))
|
require.NoError(t, s.StartTranscribing(ctx, userA, done))
|
||||||
require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "x", Endpoint: "e"}))
|
require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "x", Endpoint: "e"}))
|
||||||
@@ -235,15 +235,15 @@ func TestUploadLifecycle_ReconciliationRoundTripsWithItsContestedSpots(t *testin
|
|||||||
s := newStore(t)
|
s := newStore(t)
|
||||||
seedUsers(t, userA)
|
seedUsers(t, userA)
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
|
id, err := s.CreateUpload(ctx, userA, "m", false)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
||||||
want := &domain.Reconciliation{
|
want := &domain.Reconciliation{
|
||||||
Status: domain.ReconcileDone, Base: "iguana/pianissimo", Alt: "iguana/kb-whisper", Reconciler: "iguana/gemma4-26b",
|
Status: domain.ReconcileDone, Base: "iguana/pianissimo", Alt: "iguana/kb-whisper", Reconciler: "iguana/gemma4-26b",
|
||||||
Disputes: 380, AutoResolved: 90, Adjudicated: 285, Changed: 120, Unsure: 5, Errors: 0,
|
Disputes: 380, AutoResolved: 90, Adjudicated: 285, Changed: 120, Unsure: 5, Errors: 0,
|
||||||
Contested: []domain.ContestedSpot{
|
Contested: []domain.ContestedSpot{
|
||||||
{Seconds: 2210.5, Base: "Axel", Alt: "Alex", Used: "alt"},
|
{Seconds: 2210.5, Base: "SEB", Alt: "SCB", Used: "alt"},
|
||||||
{Seconds: 3400, Base: "A+2", Alt: "en A", Used: "base", Unsure: true},
|
{Seconds: 3400, Base: "T+1", Alt: "en T", Used: "base", Unsure: true},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -258,7 +258,7 @@ func TestUploadLifecycle_ASingleEngineTranscriptHasNoReconciliation(t *testing.T
|
|||||||
s := newStore(t)
|
s := newStore(t)
|
||||||
seedUsers(t, userA)
|
seedUsers(t, userA)
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
|
id, err := s.CreateUpload(ctx, userA, "m", false)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
||||||
require.NoError(t, s.CompleteUpload(ctx, userA, id, domain.Transcription{Text: "x", Endpoint: "e"}))
|
require.NoError(t, s.CompleteUpload(ctx, userA, id, domain.Transcription{Text: "x", Endpoint: "e"}))
|
||||||
@@ -273,7 +273,7 @@ func TestUploadLifecycle_ASkippedReconciliationKeepsItsReason(t *testing.T) {
|
|||||||
s := newStore(t)
|
s := newStore(t)
|
||||||
seedUsers(t, userA)
|
seedUsers(t, userA)
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
|
id, err := s.CreateUpload(ctx, userA, "m", false)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
||||||
rec := &domain.Reconciliation{Status: domain.ReconcileSkipped, Reason: "the model failed 3 times in a row: gateway down", Errors: 3}
|
rec := &domain.Reconciliation{Status: domain.ReconcileSkipped, Reason: "the model failed 3 times in a row: gateway down", Errors: 3}
|
||||||
@@ -284,55 +284,3 @@ func TestUploadLifecycle_ASkippedReconciliationKeepsItsReason(t *testing.T) {
|
|||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Equal(t, rec, got.Reconciliation)
|
assert.Equal(t, rec, got.Reconciliation)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestCreateUpload_RecordsTheLanguageTheUserChoseAndTheListCarriesIt(t *testing.T) {
|
|
||||||
s := newStore(t)
|
|
||||||
seedUsers(t, userA)
|
|
||||||
ctx := context.Background()
|
|
||||||
swedish, err := s.CreateUpload(ctx, userA, "sv", domain.LangSwedish, false)
|
|
||||||
require.NoError(t, err)
|
|
||||||
auto, err := s.CreateUpload(ctx, userA, "auto", domain.LangAuto, false)
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
gotSv, err := s.GetUpload(ctx, userA, swedish)
|
|
||||||
require.NoError(t, err)
|
|
||||||
gotAuto, err := s.GetUpload(ctx, userA, auto)
|
|
||||||
require.NoError(t, err)
|
|
||||||
listed, err := s.ListUploads(ctx, userA)
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
assert.Equal(t, domain.LangSwedish, gotSv.Language)
|
|
||||||
assert.Equal(t, domain.LangAuto, gotAuto.Language)
|
|
||||||
byID := map[string]domain.Language{}
|
|
||||||
for _, u := range listed {
|
|
||||||
byID[u.ID] = u.Language
|
|
||||||
}
|
|
||||||
assert.Equal(t, map[string]domain.Language{swedish: domain.LangSwedish, auto: domain.LangAuto}, byID)
|
|
||||||
}
|
|
||||||
|
|
||||||
// The closed set is also held by the table, so a bug or a second writer cannot store a
|
|
||||||
// value the engines are never meant to receive (tapir#61).
|
|
||||||
func TestUploadedTranscripts_TheTableRefusesALanguageOutsideTheClosedSet(t *testing.T) {
|
|
||||||
newStore(t)
|
|
||||||
seedUsers(t, userA)
|
|
||||||
|
|
||||||
_, err := rawPool(t).Exec(context.Background(),
|
|
||||||
`INSERT INTO uploaded_transcripts (user_id, title, language) VALUES ($1, 'x', 'en')`, userA)
|
|
||||||
|
|
||||||
require.Error(t, err)
|
|
||||||
assert.Contains(t, err.Error(), "language", "the CHECK constraint names the column")
|
|
||||||
}
|
|
||||||
|
|
||||||
// A row from before the column existed was transcribed as Swedish; the default says so.
|
|
||||||
func TestUploadedTranscripts_ARowInsertedWithoutALanguageIsSwedish(t *testing.T) {
|
|
||||||
s := newStore(t)
|
|
||||||
seedUsers(t, userA)
|
|
||||||
var id string
|
|
||||||
require.NoError(t, rawPool(t).QueryRow(context.Background(),
|
|
||||||
`INSERT INTO uploaded_transcripts (user_id, title) VALUES ($1, 'old') RETURNING id`, userA).Scan(&id))
|
|
||||||
|
|
||||||
got, err := s.GetUpload(context.Background(), userA, id)
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Equal(t, domain.LangSwedish, got.Language)
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -6,7 +6,6 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
"git.d-ma.be/mathias/tapir/internal/errkind"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/progress"
|
"git.d-ma.be/mathias/tapir/internal/progress"
|
||||||
"git.d-ma.be/mathias/tapir/internal/reconcile"
|
"git.d-ma.be/mathias/tapir/internal/reconcile"
|
||||||
)
|
)
|
||||||
@@ -32,25 +31,15 @@ func NewEnsemble(base, alt, fallback Backend, adj reconcile.Adjudicator, opts re
|
|||||||
return &Ensemble{base: base, alt: alt, fallback: fallback, adj: adj, opts: opts}
|
return &Ensemble{base: base, alt: alt, fallback: fallback, adj: adj, opts: opts}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Plan implements uploads.Planner: the phases of a Swedish job, the two-engine one. The
|
// Plan implements uploads.Planner: the phases this ensemble reports, so the page can
|
||||||
// fallback is a phase only when one is configured.
|
// show them before they run. The fallback is a phase only when one is configured.
|
||||||
func (e *Ensemble) Plan() []progress.Planned { return e.PlanFor(domain.LangSwedish) }
|
func (e *Ensemble) Plan() []progress.Planned {
|
||||||
|
plan := []progress.Planned{
|
||||||
// PlanFor implements uploads.LanguagePlanner. A recording that is not Swedish never runs the
|
|
||||||
// base engine, the comparison or the decisions (transcribeOther), so the page lists only
|
|
||||||
// what can run: the other engine, then the fallback if one is configured.
|
|
||||||
func (e *Ensemble) PlanFor(lang domain.Language) []progress.Planned {
|
|
||||||
var plan []progress.Planned
|
|
||||||
if lang == domain.LangSwedish || lang == "" {
|
|
||||||
plan = []progress.Planned{
|
|
||||||
{Key: "engine_a", Label: "Transcribing with " + e.base.Name()},
|
{Key: "engine_a", Label: "Transcribing with " + e.base.Name()},
|
||||||
{Key: "engine_b", Label: "Transcribing with " + e.alt.Name()},
|
{Key: "engine_b", Label: "Transcribing with " + e.alt.Name()},
|
||||||
{Key: "compare", Label: "Comparing the two transcripts"},
|
{Key: "compare", Label: "Comparing the two transcripts"},
|
||||||
{Key: "decide", Label: "Deciding the differences"},
|
{Key: "decide", Label: "Deciding the differences"},
|
||||||
}
|
}
|
||||||
} else {
|
|
||||||
plan = []progress.Planned{{Key: "engine_b", Label: "Transcribing with " + e.alt.Name()}}
|
|
||||||
}
|
|
||||||
if e.fallback != nil {
|
if e.fallback != nil {
|
||||||
plan = append(plan, progress.Planned{Key: "fallback", Label: "Cloud fallback (" + e.fallback.Name() + ")"})
|
plan = append(plan, progress.Planned{Key: "fallback", Label: "Cloud fallback (" + e.fallback.Name() + ")"})
|
||||||
}
|
}
|
||||||
@@ -61,9 +50,6 @@ func (e *Ensemble) PlanFor(lang domain.Language) []progress.Planned {
|
|||||||
// progress.Reporter in ctx (a no-op when nobody listens).
|
// progress.Reporter in ctx (a no-op when nobody listens).
|
||||||
func (e *Ensemble) Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error) {
|
func (e *Ensemble) Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error) {
|
||||||
rep := progress.From(ctx)
|
rep := progress.From(ctx)
|
||||||
if domain.LanguageFrom(ctx) != domain.LangSwedish {
|
|
||||||
return e.transcribeOther(ctx, rep, wavPath)
|
|
||||||
}
|
|
||||||
b, berr := e.run(ctx, rep, "engine_a", e.base, wavPath)
|
b, berr := e.run(ctx, rep, "engine_a", e.base, wavPath)
|
||||||
if err := ctx.Err(); err != nil {
|
if err := ctx.Err(); err != nil {
|
||||||
return domain.Transcription{}, err
|
return domain.Transcription{}, err
|
||||||
@@ -86,9 +72,9 @@ func (e *Ensemble) Transcribe(ctx context.Context, wavPath string) (domain.Trans
|
|||||||
}
|
}
|
||||||
return out, nil
|
return out, nil
|
||||||
case berr == nil:
|
case berr == nil:
|
||||||
return e.single(b, e.base.Name(), fmt.Sprintf("%s failed (%s)", e.alt.Name(), errkind.Of(aerr))), nil
|
return e.single(b, e.base.Name(), fmt.Sprintf("%s failed: %v", e.alt.Name(), aerr)), nil
|
||||||
case aerr == nil:
|
case aerr == nil:
|
||||||
return e.single(a, e.alt.Name(), fmt.Sprintf("%s failed (%s)", e.base.Name(), errkind.Of(berr))), nil
|
return e.single(a, e.alt.Name(), fmt.Sprintf("%s failed: %v", e.base.Name(), berr)), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
both := errors.Join(fmt.Errorf("%s: %w", e.base.Name(), berr), fmt.Errorf("%s: %w", e.alt.Name(), aerr))
|
both := errors.Join(fmt.Errorf("%s: %w", e.base.Name(), berr), fmt.Errorf("%s: %w", e.alt.Name(), aerr))
|
||||||
@@ -102,28 +88,6 @@ func (e *Ensemble) Transcribe(ctx context.Context, wavPath string) (domain.Trans
|
|||||||
return e.single(f, e.fallback.Name(), "both local engines failed"), nil
|
return e.single(f, e.fallback.Name(), "both local engines failed"), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// transcribeOther handles a recording that is not Swedish (tapir#61). The base engine is
|
|
||||||
// Swedish-only, so it never receives the audio, and with one reading there is nothing to
|
|
||||||
// reconcile: the other engine transcribes alone, then the cloud fallback if it fails and the
|
|
||||||
// job is still live. The phases that did not run are shown as skipped by the tracker.
|
|
||||||
func (e *Ensemble) transcribeOther(ctx context.Context, rep progress.Reporter, wavPath string) (domain.Transcription, error) {
|
|
||||||
t, err := e.run(ctx, rep, "engine_b", e.alt, wavPath)
|
|
||||||
if err == nil {
|
|
||||||
t.Endpoint = e.alt.Name()
|
|
||||||
return t, nil
|
|
||||||
}
|
|
||||||
if e.fallback == nil || ctx.Err() != nil {
|
|
||||||
return domain.Transcription{}, fmt.Errorf("%s: %w", e.alt.Name(), err)
|
|
||||||
}
|
|
||||||
f, ferr := e.run(ctx, rep, "fallback", e.fallback, wavPath)
|
|
||||||
if ferr != nil {
|
|
||||||
return domain.Transcription{}, errors.Join(
|
|
||||||
fmt.Errorf("%s: %w", e.alt.Name(), err), fmt.Errorf("%s: %w", e.fallback.Name(), ferr))
|
|
||||||
}
|
|
||||||
f.Endpoint = e.fallback.Name()
|
|
||||||
return f, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// run is one engine call, reported as a phase. A job that was stopped is not a broken
|
// run is one engine call, reported as a phase. A job that was stopped is not a broken
|
||||||
// engine, so a cancelled context is never reported as a failed phase.
|
// engine, so a cancelled context is never reported as a failed phase.
|
||||||
func (e *Ensemble) run(ctx context.Context, rep progress.Reporter, key string, be Backend, wav string) (domain.Transcription, error) {
|
func (e *Ensemble) run(ctx context.Context, rep progress.Reporter, key string, be Backend, wav string) (domain.Transcription, error) {
|
||||||
|
|||||||
@@ -160,25 +160,3 @@ func TestEnsembleProgress_TheReconcileCountsAreNotedForTheOperator(t *testing.T)
|
|||||||
assert.Equal(t, 0, n["unsure"])
|
assert.Equal(t, 0, n["unsure"])
|
||||||
assert.Equal(t, 0, n["errors"])
|
assert.Equal(t, 0, n["errors"])
|
||||||
}
|
}
|
||||||
|
|
||||||
// An auto-language upload never runs the base engine, the comparison or the decisions, so the
|
|
||||||
// page must not list them as coming (tapir#61, PR #62 review).
|
|
||||||
func TestEnsemblePlanFor_AnAutoLanguageListsOnlyThePhasesThatCanRun(t *testing.T) {
|
|
||||||
with := newEnsemble("a", "a", nil, nil, true, &verdictAdj{}).e
|
|
||||||
without := newEnsemble("a", "a", nil, nil, false, &verdictAdj{}).e
|
|
||||||
keys := func(p []progress.Planned) []string {
|
|
||||||
var k []string
|
|
||||||
for _, s := range p {
|
|
||||||
k = append(k, s.Key)
|
|
||||||
}
|
|
||||||
return k
|
|
||||||
}
|
|
||||||
|
|
||||||
assert.Equal(t, []string{"engine_b", "fallback"}, keys(with.PlanFor(domain.LangAuto)))
|
|
||||||
assert.Equal(t, []string{"engine_b"}, keys(without.PlanFor(domain.LangAuto)), "no fallback configured: none promised")
|
|
||||||
assert.Equal(t, with.Plan(), with.PlanFor(domain.LangSwedish), "Swedish is the plan it always was")
|
|
||||||
assert.Contains(t, with.PlanFor(domain.LangAuto)[0].Label, "iguana/kb-whisper")
|
|
||||||
assert.Equal(t, with.Plan(), with.PlanFor(""), "a job queued before the choice existed is Swedish")
|
|
||||||
assert.Equal(t, with.PlanFor(domain.LangAuto), with.PlanFor(domain.Language("xx")),
|
|
||||||
"an unknown language is planned as it is run: not the Swedish phases")
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,36 +0,0 @@
|
|||||||
package transcribe_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/reconcile"
|
|
||||||
)
|
|
||||||
|
|
||||||
// When one engine is down the page says so and why; the "why" is a kind, not the
|
|
||||||
// engine's error text, which can carry provider output (adversarial review F3).
|
|
||||||
|
|
||||||
func TestEnsemble_AFailedEnginesReasonIsAKindNotItsErrorText(t *testing.T) {
|
|
||||||
r := newEnsemble("text", "text", nil, errors.New("kb-whisper said: zebrakaka secret words"), false, &verdictAdj{v: reconcile.VerdictAlt})
|
|
||||||
|
|
||||||
got, err := r.e.Transcribe(context.Background(), "/a.wav")
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.NotContains(t, got.Reconciliation.Reason, "zebrakaka")
|
|
||||||
assert.Contains(t, got.Reconciliation.Reason, "iguana/kb-whisper failed", "which engine failed is still said")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestEnsemble_AnHTTPFailureShowsItsStatusCodeAsTheKind(t *testing.T) {
|
|
||||||
r := newEnsemble("text", "text", &transcribe.StatusError{Code: 502, Snippet: "zebrakaka"}, nil, false, &verdictAdj{})
|
|
||||||
|
|
||||||
got, err := r.e.Transcribe(context.Background(), "/a.wav")
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Contains(t, got.Reconciliation.Reason, "http_502")
|
|
||||||
assert.NotContains(t, got.Reconciliation.Reason, "zebrakaka")
|
|
||||||
}
|
|
||||||
@@ -204,52 +204,3 @@ func TestEnsemble_AShutdownDuringTheSecondEngineIsAnErrorNotASingleEngineSuccess
|
|||||||
assert.ErrorIs(t, err, context.Canceled, "a cancelled job must not be recorded as 'kb-whisper failed'")
|
assert.ErrorIs(t, err, context.Canceled, "a cancelled job must not be recorded as 'kb-whisper failed'")
|
||||||
assert.Empty(t, got.Text)
|
assert.Empty(t, got.Text)
|
||||||
}
|
}
|
||||||
|
|
||||||
// The base engine is Swedish-only. A recording that is not Swedish must never reach it, and
|
|
||||||
// there is no second opinion to reconcile: the other engine transcribes alone (tapir#61).
|
|
||||||
|
|
||||||
func autoCtx() context.Context { return domain.WithLanguage(context.Background(), domain.LangAuto) }
|
|
||||||
|
|
||||||
func TestEnsemble_AnAutoLanguageUploadNeverReachesTheBaseEngineAndIsNotReconciled(t *testing.T) {
|
|
||||||
adj := &verdictAdj{v: reconcile.VerdictAlt}
|
|
||||||
r := newEnsemble("base text", "alt text", nil, nil, true, adj)
|
|
||||||
|
|
||||||
got, err := r.e.Transcribe(autoCtx(), "/a.wav")
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Equal(t, []string{"iguana/kb-whisper"}, *r.calls, "only the other engine is called")
|
|
||||||
assert.Zero(t, adj.calls, "nothing is adjudicated")
|
|
||||||
assert.Equal(t, "alt text", got.Text)
|
|
||||||
assert.Equal(t, "iguana/kb-whisper", got.Endpoint)
|
|
||||||
assert.Nil(t, got.Reconciliation, "reconciliation does not apply, so none is recorded")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestEnsemble_AnAutoLanguageUploadFallsBackWhenItsEngineFailsAndStillSkipsTheBaseEngine(t *testing.T) {
|
|
||||||
r := newEnsemble("base text", "alt text", nil, errors.New("engine down"), true, &verdictAdj{})
|
|
||||||
|
|
||||||
got, err := r.e.Transcribe(autoCtx(), "/a.wav")
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Equal(t, []string{"iguana/kb-whisper", "berget/kb-whisper"}, *r.calls)
|
|
||||||
assert.Equal(t, "berget/kb-whisper", got.Endpoint)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestEnsemble_AnAutoLanguageUploadWithNoFallbackFailsWithoutTouchingTheBaseEngine(t *testing.T) {
|
|
||||||
r := newEnsemble("base text", "alt text", nil, errors.New("engine down"), false, &verdictAdj{})
|
|
||||||
|
|
||||||
_, err := r.e.Transcribe(autoCtx(), "/a.wav")
|
|
||||||
|
|
||||||
require.Error(t, err)
|
|
||||||
assert.Equal(t, []string{"iguana/kb-whisper"}, *r.calls)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestEnsemble_ACancelledAutoLanguageJobDoesNotReachTheFallback(t *testing.T) {
|
|
||||||
r := newEnsemble("base text", "alt text", nil, errors.New("engine down"), true, &verdictAdj{})
|
|
||||||
ctx, cancel := context.WithCancel(autoCtx())
|
|
||||||
cancel()
|
|
||||||
|
|
||||||
_, err := r.e.Transcribe(ctx, "/a.wav")
|
|
||||||
|
|
||||||
require.Error(t, err)
|
|
||||||
assert.NotContains(t, *r.calls, "berget/kb-whisper", "a stopped job must not ship its audio to a cloud endpoint")
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -31,9 +31,6 @@ type StatusError struct {
|
|||||||
Snippet string
|
Snippet string
|
||||||
}
|
}
|
||||||
|
|
||||||
// HTTPStatus implements errkind.HTTPStatuser: the code, never the snippet.
|
|
||||||
func (e *StatusError) HTTPStatus() int { return e.Code }
|
|
||||||
|
|
||||||
func (e *StatusError) Error() string {
|
func (e *StatusError) Error() string {
|
||||||
return fmt.Sprintf("transcribe: gateway returned HTTP %d: %s", e.Code, e.Snippet)
|
return fmt.Sprintf("transcribe: gateway returned HTTP %d: %s", e.Code, e.Snippet)
|
||||||
}
|
}
|
||||||
@@ -71,20 +68,9 @@ func (g *Gateway) Transcribe(ctx context.Context, path string) (domain.Transcrip
|
|||||||
return domain.Transcription{}, fmt.Errorf("transcribe: stat audio: %w", err)
|
return domain.Transcription{}, fmt.Errorf("transcribe: stat audio: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// The language is one request field and a closed set: sv is sent, auto sends nothing so
|
|
||||||
// the engine detects it, and anything else never leaves this process (tapir#61).
|
|
||||||
fields := [][2]string{{"model", g.model}, {"response_format", "verbose_json"}}
|
|
||||||
switch lang := domain.LanguageFrom(ctx); lang {
|
|
||||||
case domain.LangSwedish:
|
|
||||||
fields = append(fields, [2]string{"language", string(lang)})
|
|
||||||
case domain.LangAuto:
|
|
||||||
default:
|
|
||||||
return domain.Transcription{}, fmt.Errorf("transcribe: language %q is not one tapir sends", lang)
|
|
||||||
}
|
|
||||||
|
|
||||||
var prefix bytes.Buffer
|
var prefix bytes.Buffer
|
||||||
mw := multipart.NewWriter(&prefix)
|
mw := multipart.NewWriter(&prefix)
|
||||||
for _, kv := range fields {
|
for _, kv := range [][2]string{{"model", g.model}, {"response_format", "verbose_json"}, {"language", "sv"}} {
|
||||||
if err := mw.WriteField(kv[0], kv[1]); err != nil {
|
if err := mw.WriteField(kv[0], kv[1]); err != nil {
|
||||||
return domain.Transcription{}, fmt.Errorf("transcribe: build request: %w", err)
|
return domain.Transcription{}, fmt.Errorf("transcribe: build request: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,13 +3,11 @@ package transcribe_test
|
|||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"fmt"
|
|
||||||
"io"
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
@@ -17,7 +15,6 @@ import (
|
|||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
|
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
"git.d-ma.be/mathias/tapir/internal/errkind"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// The httptest server stands in for the LiteLLM gateway: it is the process
|
// The httptest server stands in for the LiteLLM gateway: it is the process
|
||||||
@@ -233,70 +230,3 @@ func TestGateway_WithNoSegmentsTheTextFieldIsUsedWithWhitespaceCollapsed(t *test
|
|||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Equal(t, "Hej allihopa. Vi börjar.", got.Text)
|
assert.Equal(t, "Hej allihopa. Vi börjar.", got.Text)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestStatusError_ExposesItsCodeAsAKindAndNotItsSnippet(t *testing.T) {
|
|
||||||
err := fmt.Errorf("transcribe: %w", &transcribe.StatusError{Code: 502, Snippet: "echoed: Anders ansvarar"})
|
|
||||||
|
|
||||||
assert.Equal(t, "http_502", errkind.Of(err))
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestGateway_AnErrorBodyIsBoundedToTwoHundredBytesInTheStatusError(t *testing.T) {
|
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
|
||||||
w.WriteHeader(http.StatusInternalServerError)
|
|
||||||
_, _ = w.Write([]byte(strings.Repeat("x", 100_000)))
|
|
||||||
}))
|
|
||||||
defer srv.Close()
|
|
||||||
wav := filepath.Join(t.TempDir(), "a.wav")
|
|
||||||
require.NoError(t, os.WriteFile(wav, []byte("RIFF....WAVEfmt "), 0o600))
|
|
||||||
|
|
||||||
_, err := transcribe.NewGateway(srv.URL, "", "iguana/kb-whisper", srv.Client()).Transcribe(context.Background(), wav)
|
|
||||||
|
|
||||||
var se *transcribe.StatusError
|
|
||||||
require.ErrorAs(t, err, &se)
|
|
||||||
assert.LessOrEqual(t, len(se.Snippet), 200, "a gateway's whole body must not become part of an error")
|
|
||||||
}
|
|
||||||
|
|
||||||
// The language travels on the context (tapir#61); the request field is the only place it
|
|
||||||
// takes effect, so these observe the request the gateway fake received.
|
|
||||||
|
|
||||||
func TestGateway_SendsLanguageSvWhenTheContextCarriesNoneAndForSwedish(t *testing.T) {
|
|
||||||
for name, ctx := range map[string]context.Context{
|
|
||||||
"no language in the context": context.Background(),
|
|
||||||
"swedish": domain.WithLanguage(context.Background(), domain.LangSwedish),
|
|
||||||
} {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
var got seen
|
|
||||||
srv := serve(t, 200, verboseJSON, &got)
|
|
||||||
g := transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client())
|
|
||||||
|
|
||||||
_, err := g.Transcribe(ctx, wavFile(t, "RIFF-synthetic-pcm"))
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Equal(t, "sv", got.fields["language"])
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestGateway_OmitsTheLanguageFieldForAnAutoContextSoTheEngineDetectsIt(t *testing.T) {
|
|
||||||
var got seen
|
|
||||||
srv := serve(t, 200, verboseJSON, &got)
|
|
||||||
g := transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client())
|
|
||||||
|
|
||||||
_, err := g.Transcribe(domain.WithLanguage(context.Background(), domain.LangAuto), wavFile(t, "RIFF-synthetic-pcm"))
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
_, present := got.fields["language"]
|
|
||||||
assert.False(t, present, "an auto upload must not be forced to Swedish")
|
|
||||||
assert.Equal(t, "iguana/kb-whisper", got.fields["model"], "the rest of the request is unchanged")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestGateway_ALanguageOutsideTheClosedSetIsRefusedBeforeAnythingIsSent(t *testing.T) {
|
|
||||||
var got seen
|
|
||||||
srv := serve(t, 200, verboseJSON, &got)
|
|
||||||
g := transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client())
|
|
||||||
|
|
||||||
_, err := g.Transcribe(domain.WithLanguage(context.Background(), domain.Language("en")), wavFile(t, "RIFF-synthetic-pcm"))
|
|
||||||
|
|
||||||
require.Error(t, err)
|
|
||||||
assert.Empty(t, got.method, "no request may reach the engine with a language the closed set does not hold")
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,33 +0,0 @@
|
|||||||
package transcribe
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
|
||||||
)
|
|
||||||
|
|
||||||
// WithTimeout gives every call to b its own deadline. A wedged engine does not answer at
|
|
||||||
// all (iguana after a Metal OOM, infra#273), and with only the job's deadline it used up
|
|
||||||
// the whole job before a fallback could run -- and Chain then refuses the fallback because
|
|
||||||
// the job context has expired. With its own deadline the call fails while the job is still
|
|
||||||
// alive, which is exactly when the fallback, or the other engine, can take over.
|
|
||||||
//
|
|
||||||
// A zero or negative d returns b unchanged. The wrapper is invisible: Name is b's.
|
|
||||||
func WithTimeout(b Backend, d time.Duration) Backend {
|
|
||||||
if d <= 0 {
|
|
||||||
return b
|
|
||||||
}
|
|
||||||
return timeoutBackend{Backend: b, d: d}
|
|
||||||
}
|
|
||||||
|
|
||||||
type timeoutBackend struct {
|
|
||||||
Backend
|
|
||||||
d time.Duration
|
|
||||||
}
|
|
||||||
|
|
||||||
func (t timeoutBackend) Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error) {
|
|
||||||
ctx, cancel := context.WithTimeout(ctx, t.d)
|
|
||||||
defer cancel()
|
|
||||||
return t.Backend.Transcribe(ctx, wavPath)
|
|
||||||
}
|
|
||||||
@@ -1,146 +0,0 @@
|
|||||||
package transcribe_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/errkind"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/reconcile"
|
|
||||||
)
|
|
||||||
|
|
||||||
// A wedged engine (iguana after a Metal OOM, infra#273) does not answer at all. With only
|
|
||||||
// the job's deadline, the primary used up the whole job before any fallback could run, and
|
|
||||||
// the chain then refused the fallback because the job context had expired. Each engine call
|
|
||||||
// gets its own deadline (adversarial review F4).
|
|
||||||
|
|
||||||
type hangingBackend struct{ name string }
|
|
||||||
|
|
||||||
func (h hangingBackend) Name() string { return h.name }
|
|
||||||
func (hangingBackend) Transcribe(ctx context.Context, _ string) (domain.Transcription, error) {
|
|
||||||
<-ctx.Done()
|
|
||||||
return domain.Transcription{}, ctx.Err()
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWithTimeout_AHungBackendFailsAtItsOwnDeadlineWhileTheJobIsStillAlive(t *testing.T) {
|
|
||||||
job, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
|
||||||
defer cancel()
|
|
||||||
b := transcribe.WithTimeout(hangingBackend{name: "iguana/kb-whisper"}, 40*time.Millisecond)
|
|
||||||
|
|
||||||
start := time.Now()
|
|
||||||
_, err := b.Transcribe(job, "/a.wav")
|
|
||||||
|
|
||||||
require.Error(t, err)
|
|
||||||
assert.Equal(t, "timeout", errkind.Of(err))
|
|
||||||
assert.Less(t, time.Since(start), 2*time.Second, "it must give up at its own deadline, not wait for the job's")
|
|
||||||
assert.NoError(t, job.Err(), "the job context is untouched, which is what lets a fallback run")
|
|
||||||
assert.Equal(t, "iguana/kb-whisper", b.Name(), "the wrapper is invisible in the recorded endpoint")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWithTimeout_ASuccessPassesStraightThrough(t *testing.T) {
|
|
||||||
calls := &[]string{}
|
|
||||||
inner := &orderedBackend{name: "iguana/kb-whisper", text: "hej", calls: calls}
|
|
||||||
|
|
||||||
got, err := transcribe.WithTimeout(inner, time.Minute).Transcribe(context.Background(), "/a.wav")
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Equal(t, "hej", got.Text)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWithTimeout_ZeroMeansNoDeadlineOfItsOwn(t *testing.T) {
|
|
||||||
inner := &orderedBackend{name: "x", text: "hej", calls: &[]string{}}
|
|
||||||
|
|
||||||
assert.Same(t, inner, transcribe.WithTimeout(inner, 0), "a zero timeout must not wrap or change anything")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestChain_AHungPrimaryReachesTheFallbackWhenItHasItsOwnDeadline(t *testing.T) {
|
|
||||||
calls := &[]string{}
|
|
||||||
fb := &orderedBackend{name: "berget/kb-whisper", text: "fallback text", calls: calls}
|
|
||||||
c := transcribe.NewChain(transcribe.WithTimeout(hangingBackend{name: "iguana/kb-whisper"}, 40*time.Millisecond), fb)
|
|
||||||
job, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
got, err := c.Transcribe(job, "/a.wav")
|
|
||||||
|
|
||||||
require.NoError(t, err, "a wedged primary is exactly what the fallback exists for")
|
|
||||||
assert.Equal(t, "berget/kb-whisper", got.Endpoint)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestChain_AJobThatEndedWhileThePrimaryHungStillNeverReachesTheFallback(t *testing.T) {
|
|
||||||
calls := &[]string{}
|
|
||||||
fb := &orderedBackend{name: "berget/kb-whisper", text: "fallback text", calls: calls}
|
|
||||||
c := transcribe.NewChain(transcribe.WithTimeout(hangingBackend{name: "iguana/kb-whisper"}, time.Minute), fb)
|
|
||||||
job, cancel := context.WithTimeout(context.Background(), 40*time.Millisecond)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
_, err := c.Transcribe(job, "/a.wav")
|
|
||||||
|
|
||||||
require.Error(t, err)
|
|
||||||
assert.Empty(t, *calls, "the job's own deadline or a cancel must still stop the audio from reaching a cloud endpoint")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestEnsemble_AHungEngineIsASoftFailureAndTheOtherEnginesTextIsUsed(t *testing.T) {
|
|
||||||
calls := &[]string{}
|
|
||||||
alt := &orderedBackend{name: "iguana/kb-whisper", text: "text from the other engine", calls: calls}
|
|
||||||
e := transcribe.NewEnsemble(
|
|
||||||
transcribe.WithTimeout(hangingBackend{name: "iguana/pianissimo"}, 40*time.Millisecond), alt, nil,
|
|
||||||
&verdictAdj{v: reconcile.VerdictAlt}, ensembleOpts)
|
|
||||||
job, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
got, err := e.Transcribe(job, "/a.wav")
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Equal(t, "text from the other engine", got.Text)
|
|
||||||
assert.Equal(t, domain.ReconcileSingle, got.Reconciliation.Status)
|
|
||||||
assert.Contains(t, got.Reconciliation.Reason, "timeout")
|
|
||||||
assert.False(t, errors.Is(err, context.DeadlineExceeded))
|
|
||||||
}
|
|
||||||
|
|
||||||
// WithTimeout sits in front of both local engines in production (cmd/tapir/uploads.go). It
|
|
||||||
// must hand the engine the job's context, not a fresh one: the language and the
|
|
||||||
// cancellation both travel on it (tapir#61).
|
|
||||||
|
|
||||||
func TestWithTimeout_TheEngineStillReceivesTheLanguageOfTheJob(t *testing.T) {
|
|
||||||
for name, tt := range map[string]struct {
|
|
||||||
ctx context.Context
|
|
||||||
wantField bool
|
|
||||||
}{
|
|
||||||
"auto leaves the field out": {domain.WithLanguage(context.Background(), domain.LangAuto), false},
|
|
||||||
"swedish sends sv": {domain.WithLanguage(context.Background(), domain.LangSwedish), true},
|
|
||||||
} {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
var got seen
|
|
||||||
srv := serve(t, 200, verboseJSON, &got)
|
|
||||||
b := transcribe.WithTimeout(transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client()), time.Minute)
|
|
||||||
|
|
||||||
_, err := b.Transcribe(tt.ctx, wavFile(t, "RIFF-synthetic-pcm"))
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
lang, present := got.fields["language"]
|
|
||||||
assert.Equal(t, tt.wantField, present)
|
|
||||||
if tt.wantField {
|
|
||||||
assert.Equal(t, "sv", lang)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWithTimeout_ACancelledJobIsNotRestartedByTheWrapper(t *testing.T) {
|
|
||||||
var got seen
|
|
||||||
srv := serve(t, 200, verboseJSON, &got)
|
|
||||||
b := transcribe.WithTimeout(transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client()), time.Minute)
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
cancel()
|
|
||||||
|
|
||||||
_, err := b.Transcribe(ctx, wavFile(t, "RIFF-synthetic-pcm"))
|
|
||||||
|
|
||||||
require.Error(t, err)
|
|
||||||
assert.Empty(t, got.method, "a cancelled job must not send its audio anywhere through the wrapper")
|
|
||||||
}
|
|
||||||
+12
-30
@@ -3,7 +3,6 @@ package config
|
|||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"regexp"
|
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
@@ -28,10 +27,6 @@ type UploadConfig struct {
|
|||||||
// TranscribeTimeout bounds one job (conversion + transcription); a 90 minute
|
// TranscribeTimeout bounds one job (conversion + transcription); a 90 minute
|
||||||
// recording took ~7.5 minutes on iguana.
|
// recording took ~7.5 minutes on iguana.
|
||||||
TranscribeTimeout time.Duration
|
TranscribeTimeout time.Duration
|
||||||
// TranscribeEngineTimeout bounds ONE engine call, so a wedged engine fails while the
|
|
||||||
// job is still alive and the fallback (or the other engine) can take over. The job
|
|
||||||
// timeout alone cannot do that (ADR-035 review).
|
|
||||||
TranscribeEngineTimeout time.Duration
|
|
||||||
// UploadTmpDir is where uploads are spooled and converted. Audio is deleted
|
// UploadTmpDir is where uploads are spooled and converted. Audio is deleted
|
||||||
// after transcription, so this holds at most the in-flight files.
|
// after transcription, so this holds at most the in-flight files.
|
||||||
UploadTmpDir string
|
UploadTmpDir string
|
||||||
@@ -66,7 +61,6 @@ const (
|
|||||||
defaultTranscribeModel = "iguana/kb-whisper"
|
defaultTranscribeModel = "iguana/kb-whisper"
|
||||||
defaultTranscribeFallbackModel = "berget/kb-whisper"
|
defaultTranscribeFallbackModel = "berget/kb-whisper"
|
||||||
defaultTranscribeTimeout = 60 * time.Minute
|
defaultTranscribeTimeout = 60 * time.Minute
|
||||||
defaultTranscribeEngineTimeout = 20 * time.Minute
|
|
||||||
defaultMaxUploadedChars = 90000
|
defaultMaxUploadedChars = 90000
|
||||||
defaultUploadChatModel = "iguana/gemma4-26b"
|
defaultUploadChatModel = "iguana/gemma4-26b"
|
||||||
)
|
)
|
||||||
@@ -112,15 +106,6 @@ func loadUploadConfig() (UploadConfig, error) {
|
|||||||
}
|
}
|
||||||
u.TranscribeTimeout = timeout
|
u.TranscribeTimeout = timeout
|
||||||
|
|
||||||
engineTimeout, err := durationOr("TAPIR_TRANSCRIBE_ENGINE_TIMEOUT", defaultTranscribeEngineTimeout)
|
|
||||||
if err != nil {
|
|
||||||
return UploadConfig{}, err
|
|
||||||
}
|
|
||||||
if engineTimeout <= 0 {
|
|
||||||
return UploadConfig{}, fmt.Errorf("config: TAPIR_TRANSCRIBE_ENGINE_TIMEOUT must be positive, got %v", engineTimeout)
|
|
||||||
}
|
|
||||||
u.TranscribeEngineTimeout = engineTimeout
|
|
||||||
|
|
||||||
if v := os.Getenv("TAPIR_RECONCILE_CHECK_ORDER"); v != "" {
|
if v := os.Getenv("TAPIR_RECONCILE_CHECK_ORDER"); v != "" {
|
||||||
on, err := strconv.ParseBool(v)
|
on, err := strconv.ParseBool(v)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -131,9 +116,6 @@ func loadUploadConfig() (UploadConfig, error) {
|
|||||||
if err := u.validateReconcile(); err != nil {
|
if err := u.validateReconcile(); err != nil {
|
||||||
return UploadConfig{}, err
|
return UploadConfig{}, err
|
||||||
}
|
}
|
||||||
if !IsLocalModel(u.UploadChatModel) {
|
|
||||||
return UploadConfig{}, fmt.Errorf("config: TAPIR_UPLOAD_CHAT_MODEL=%q is not a local model (koala/ or iguana/): the whole private transcript is sent to it", u.UploadChatModel)
|
|
||||||
}
|
|
||||||
|
|
||||||
chars, err := intOr("TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS", defaultMaxUploadedChars)
|
chars, err := intOr("TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS", defaultMaxUploadedChars)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -159,18 +141,18 @@ func positiveInt(key string, fallback int) (int, error) {
|
|||||||
return n, nil
|
return n, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// localAlias is the shape of a model alias that runs on the homelab: a local host
|
// localPrefixes are the host prefixes of the models that run on the homelab. Private
|
||||||
// prefix, then one plain name. A bare prefix check let through aliases that are shaped
|
// meeting content may be sent to these and to nothing else for reconciliation
|
||||||
// like an escape (koala/../berget/x, koala//x, koala/x?host=berget): whether the
|
// (ADR-034): a cloud alias is refused at start-up rather than discovered later.
|
||||||
// gateway would route them elsewhere is unknown, so they are refused (ADR-035 review).
|
var localPrefixes = []string{"koala/", "iguana/"}
|
||||||
var localAlias = regexp.MustCompile(`^(koala|iguana)/[A-Za-z0-9][A-Za-z0-9._:-]*$`)
|
|
||||||
|
|
||||||
// IsLocalModel reports whether alias is a local model. Private meeting content may be
|
func isLocalModel(alias string) bool {
|
||||||
// sent to these and to nothing else: reconciliation (ADR-034) and chat over an uploaded
|
for _, p := range localPrefixes {
|
||||||
// recording (ADR-035) are both held to it, and a cloud alias is refused at start-up
|
if strings.HasPrefix(alias, p) {
|
||||||
// rather than discovered later.
|
return true
|
||||||
func IsLocalModel(alias string) bool {
|
}
|
||||||
return localAlias.MatchString(alias) && !strings.Contains(alias, "..")
|
}
|
||||||
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
// validateReconcile checks the two-engine settings. They are inert while no base
|
// validateReconcile checks the two-engine settings. They are inert while no base
|
||||||
@@ -186,7 +168,7 @@ func (u UploadConfig) validateReconcile() error {
|
|||||||
{"TAPIR_RECONCILE_BASE_MODEL", u.ReconcileBaseModel},
|
{"TAPIR_RECONCILE_BASE_MODEL", u.ReconcileBaseModel},
|
||||||
{"TAPIR_RECONCILE_MODEL", u.ReconcileModel},
|
{"TAPIR_RECONCILE_MODEL", u.ReconcileModel},
|
||||||
} {
|
} {
|
||||||
if !IsLocalModel(m.alias) {
|
if !isLocalModel(m.alias) {
|
||||||
return fmt.Errorf("config: %s=%q is not a local model (koala/ or iguana/): private meeting content must not leave the homelab for reconciliation", m.name, m.alias)
|
return fmt.Errorf("config: %s=%q is not a local model (koala/ or iguana/): private meeting content must not leave the homelab for reconciliation", m.name, m.alias)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,101 +0,0 @@
|
|||||||
package config
|
|
||||||
|
|
||||||
import (
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
// What counts as a LOCAL model alias, and where that is demanded (ADR-034, ADR-035 review).
|
|
||||||
// A private transcript may be sent to a local model and to nothing else; the check is a
|
|
||||||
// guard against a misconfiguration or a crafted alias, so it must be strict about shape,
|
|
||||||
// not just about a prefix.
|
|
||||||
|
|
||||||
func TestIsLocalModel_AcceptsRealLocalAliasesAndNothingShapedLikeAnEscape(t *testing.T) {
|
|
||||||
tests := []struct {
|
|
||||||
alias string
|
|
||||||
want bool
|
|
||||||
}{
|
|
||||||
{"koala/phi4-mini", true},
|
|
||||||
{"iguana/gemma4-26b", true},
|
|
||||||
{"iguana/gemma4-31b-think", true},
|
|
||||||
{"koala/qwen36-35b-a3b", true},
|
|
||||||
{"iguana/kb-whisper", true},
|
|
||||||
{"iguana/hf.co-model_v1:q4", true},
|
|
||||||
{"", false},
|
|
||||||
{"koala/", false},
|
|
||||||
{"iguana/", false},
|
|
||||||
{"gemma4-26b", false},
|
|
||||||
{"berget/mistral-small", false},
|
|
||||||
{"berget/koala/x", false},
|
|
||||||
{"xkoala/x", false},
|
|
||||||
{"Koala/x", false},
|
|
||||||
{" koala/x", false},
|
|
||||||
{"koala/x ", false},
|
|
||||||
{"koala/x y", false},
|
|
||||||
{"koala/x\n", false},
|
|
||||||
{"koala/../berget/kb", false},
|
|
||||||
{"koala//berget/mistral-small", false},
|
|
||||||
{"koala/x/y", false},
|
|
||||||
{"koala/x?host=berget", false},
|
|
||||||
{"koala/x#frag", false},
|
|
||||||
{"koala/x@evil", false},
|
|
||||||
{"koalа/x", false}, // Cyrillic а
|
|
||||||
}
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.alias, func(t *testing.T) {
|
|
||||||
if got := IsLocalModel(tt.alias); got != tt.want {
|
|
||||||
t.Errorf("IsLocalModel(%q) = %v, want %v", tt.alias, got, tt.want)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLoad_UploadChatModelMustBeLocalBecauseTheWholeTranscriptIsSentToIt(t *testing.T) {
|
|
||||||
for _, alias := range []string{"berget/mistral-small", "gemini/gemini-2.5-pro", "gemma4-26b", "koala/../berget/kb"} {
|
|
||||||
t.Run(alias, func(t *testing.T) {
|
|
||||||
clearUploadEnv(t)
|
|
||||||
setEnv(t, map[string]string{"TAPIR_UPLOAD_CHAT_MODEL": alias})
|
|
||||||
|
|
||||||
if _, err := Load(); err == nil {
|
|
||||||
t.Errorf("Load accepted TAPIR_UPLOAD_CHAT_MODEL=%q: a private transcript of up to 90,000 characters would be sent to it", alias)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLoad_ALocalUploadChatModelIsAccepted(t *testing.T) {
|
|
||||||
clearUploadEnv(t)
|
|
||||||
setEnv(t, map[string]string{"TAPIR_UPLOAD_CHAT_MODEL": "iguana/gemma4-31b"})
|
|
||||||
|
|
||||||
cfg, err := Load()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Load: %v", err)
|
|
||||||
}
|
|
||||||
if cfg.UploadChatModel != "iguana/gemma4-31b" {
|
|
||||||
t.Errorf("UploadChatModel = %q", cfg.UploadChatModel)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLoad_TheEngineTimeoutDefaultsToTwentyMinutesAndMustBePositive(t *testing.T) {
|
|
||||||
clearUploadEnv(t)
|
|
||||||
cfg, err := Load()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Load: %v", err)
|
|
||||||
}
|
|
||||||
if cfg.TranscribeEngineTimeout != 20*time.Minute {
|
|
||||||
t.Errorf("TranscribeEngineTimeout = %v, want 20m: a 90-minute recording takes about 8 minutes on kb-whisper, and two wedged engines in sequence must leave the job time to reach a fallback", cfg.TranscribeEngineTimeout)
|
|
||||||
}
|
|
||||||
|
|
||||||
setEnv(t, map[string]string{"TAPIR_TRANSCRIBE_ENGINE_TIMEOUT": "7m"})
|
|
||||||
cfg, err = Load()
|
|
||||||
if err != nil || cfg.TranscribeEngineTimeout != 7*time.Minute {
|
|
||||||
t.Errorf("override: %v, %v", cfg.TranscribeEngineTimeout, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, bad := range []string{"0", "-1m", "soon"} {
|
|
||||||
setEnv(t, map[string]string{"TAPIR_TRANSCRIBE_ENGINE_TIMEOUT": bad})
|
|
||||||
if _, err := Load(); err == nil {
|
|
||||||
t.Errorf("Load accepted TAPIR_TRANSCRIBE_ENGINE_TIMEOUT=%q", bad)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -12,7 +12,7 @@ func clearUploadEnv(t *testing.T) {
|
|||||||
t.Helper()
|
t.Helper()
|
||||||
for _, k := range []string{
|
for _, k := range []string{
|
||||||
"TAPIR_UPLOADS_ENABLED", "TAPIR_UPLOAD_MAX_MB", "TAPIR_UPLOAD_MAX_MINUTES",
|
"TAPIR_UPLOADS_ENABLED", "TAPIR_UPLOAD_MAX_MB", "TAPIR_UPLOAD_MAX_MINUTES",
|
||||||
"TAPIR_TRANSCRIBE_MODEL", "TAPIR_TRANSCRIBE_FALLBACK_MODEL", "TAPIR_TRANSCRIBE_TIMEOUT", "TAPIR_TRANSCRIBE_ENGINE_TIMEOUT",
|
"TAPIR_TRANSCRIBE_MODEL", "TAPIR_TRANSCRIBE_FALLBACK_MODEL", "TAPIR_TRANSCRIBE_TIMEOUT",
|
||||||
"TAPIR_UPLOAD_TMP_DIR", "TAPIR_FFMPEG_BIN", "TAPIR_FFPROBE_BIN",
|
"TAPIR_UPLOAD_TMP_DIR", "TAPIR_FFMPEG_BIN", "TAPIR_FFPROBE_BIN",
|
||||||
"TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS", "TAPIR_UPLOAD_CHAT_MODEL",
|
"TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS", "TAPIR_UPLOAD_CHAT_MODEL",
|
||||||
"TAPIR_RECONCILE_BASE_MODEL", "TAPIR_RECONCILE_MODEL", "TAPIR_RECONCILE_CHECK_ORDER",
|
"TAPIR_RECONCILE_BASE_MODEL", "TAPIR_RECONCILE_MODEL", "TAPIR_RECONCILE_CHECK_ORDER",
|
||||||
|
|||||||
@@ -203,10 +203,6 @@ type ProgressStep struct {
|
|||||||
FinishedAt time.Time `json:"finished_at,omitzero"`
|
FinishedAt time.Time `json:"finished_at,omitzero"`
|
||||||
Done int `json:"done,omitempty"`
|
Done int `json:"done,omitempty"`
|
||||||
Total int `json:"total,omitempty"`
|
Total int `json:"total,omitempty"`
|
||||||
// Note is a short plain remark about the phase that the reader should see even though
|
|
||||||
// it did not fail, for example that the audio is much shorter than the file claimed.
|
|
||||||
// Durations and counts only: never transcript text, a name or a path.
|
|
||||||
Note string `json:"note,omitempty"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// UploadStatus is the state of an uploaded-audio transcription job.
|
// UploadStatus is the state of an uploaded-audio transcription job.
|
||||||
@@ -230,13 +226,6 @@ const (
|
|||||||
UploadErrInterrupted UploadErrorKind = "interrupted"
|
UploadErrInterrupted UploadErrorKind = "interrupted"
|
||||||
// UploadErrTranscriptionFailed: every configured transcription endpoint failed.
|
// UploadErrTranscriptionFailed: every configured transcription endpoint failed.
|
||||||
UploadErrTranscriptionFailed UploadErrorKind = "transcription_failed"
|
UploadErrTranscriptionFailed UploadErrorKind = "transcription_failed"
|
||||||
// UploadErrTooLong: the audio, once converted, is longer than the limit. The limit is
|
|
||||||
// first checked against the file's header, which a hostile or damaged file can lie
|
|
||||||
// about (ADR-035 review).
|
|
||||||
UploadErrTooLong UploadErrorKind = "audio_too_long"
|
|
||||||
// UploadErrBusy: too many recordings were already waiting, so this one was not
|
|
||||||
// queued. The user's to retry later (ADR-035 review).
|
|
||||||
UploadErrBusy UploadErrorKind = "queue_full"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// UploadedTranscript is a user's PRIVATE transcript of audio they uploaded. It is
|
// UploadedTranscript is a user's PRIVATE transcript of audio they uploaded. It is
|
||||||
@@ -246,9 +235,6 @@ type UploadedTranscript struct {
|
|||||||
ID string
|
ID string
|
||||||
UserID string
|
UserID string
|
||||||
Title string
|
Title string
|
||||||
// Language is what the upload was (or will be) transcribed as: sv, or auto for
|
|
||||||
// "Other" (tapir#61).
|
|
||||||
Language Language
|
|
||||||
Status UploadStatus
|
Status UploadStatus
|
||||||
Content string
|
Content string
|
||||||
Segments []Segment
|
Segments []Segment
|
||||||
|
|||||||
@@ -1,47 +0,0 @@
|
|||||||
package domain
|
|
||||||
|
|
||||||
import "context"
|
|
||||||
|
|
||||||
// Language is the spoken language an upload is transcribed as. It is chosen by the user on
|
|
||||||
// the upload form and reaches the engines as one request field, so it is a closed set,
|
|
||||||
// never free text (tapir#61).
|
|
||||||
type Language string
|
|
||||||
|
|
||||||
const (
|
|
||||||
// LangSwedish sends language=sv: the Swedish engines' own language, and what every
|
|
||||||
// upload was transcribed as before the choice existed.
|
|
||||||
LangSwedish Language = "sv"
|
|
||||||
// LangAuto sends no language, so the engine detects it. It is what "Other" on the form
|
|
||||||
// means: the Swedish-only engines are not used for it.
|
|
||||||
LangAuto Language = "auto"
|
|
||||||
)
|
|
||||||
|
|
||||||
// ParseLanguage reads a form value. An empty value is Swedish (a client that predates the
|
|
||||||
// field); anything that is not exactly a known value is refused rather than normalised.
|
|
||||||
func ParseLanguage(s string) (Language, bool) {
|
|
||||||
switch Language(s) {
|
|
||||||
case "", LangSwedish:
|
|
||||||
return LangSwedish, true
|
|
||||||
case LangAuto:
|
|
||||||
return LangAuto, true
|
|
||||||
}
|
|
||||||
return "", false
|
|
||||||
}
|
|
||||||
|
|
||||||
type languageKey struct{}
|
|
||||||
|
|
||||||
// WithLanguage returns a context that carries the language of the job it serves. The
|
|
||||||
// Transcriber interface does not change: engines read it with LanguageFrom, the same way
|
|
||||||
// they read the progress Reporter (ADR-035).
|
|
||||||
func WithLanguage(ctx context.Context, l Language) context.Context {
|
|
||||||
return context.WithValue(ctx, languageKey{}, l)
|
|
||||||
}
|
|
||||||
|
|
||||||
// LanguageFrom returns the language carried by ctx, or Swedish when it carries none, so a
|
|
||||||
// caller that sets nothing behaves as it always did.
|
|
||||||
func LanguageFrom(ctx context.Context) Language {
|
|
||||||
if l, ok := ctx.Value(languageKey{}).(Language); ok && l != "" {
|
|
||||||
return l
|
|
||||||
}
|
|
||||||
return LangSwedish
|
|
||||||
}
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
package domain_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestParseLanguage_KnownValuesAndEmptyAreAcceptedEverythingElseIsNot(t *testing.T) {
|
|
||||||
tests := []struct {
|
|
||||||
in string
|
|
||||||
want domain.Language
|
|
||||||
wantOK bool
|
|
||||||
}{
|
|
||||||
{"sv", domain.LangSwedish, true},
|
|
||||||
{"auto", domain.LangAuto, true},
|
|
||||||
{"", domain.LangSwedish, true}, // a client that sends no field keeps today's behaviour
|
|
||||||
{"SV", "", false}, // exact match: the value reaches a request field
|
|
||||||
{" sv", "", false},
|
|
||||||
{"sv\n", "", false},
|
|
||||||
{"en", "", false}, // a specific other language is not offered yet
|
|
||||||
{"sv,en", "", false},
|
|
||||||
{"../sv", "", false},
|
|
||||||
}
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.in, func(t *testing.T) {
|
|
||||||
got, ok := domain.ParseLanguage(tt.in)
|
|
||||||
assert.Equal(t, tt.wantOK, ok)
|
|
||||||
assert.Equal(t, tt.want, got)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestLanguageFrom_IsSwedishWhenTheContextCarriesNoneAndOtherwiseWhatWasSet(t *testing.T) {
|
|
||||||
assert.Equal(t, domain.LangSwedish, domain.LanguageFrom(context.Background()))
|
|
||||||
ctx := domain.WithLanguage(context.Background(), domain.LangAuto)
|
|
||||||
assert.Equal(t, domain.LangAuto, domain.LanguageFrom(ctx))
|
|
||||||
}
|
|
||||||
@@ -1,51 +0,0 @@
|
|||||||
// Package errkind reduces an error to a short kind that is safe to log, store and show.
|
|
||||||
//
|
|
||||||
// An error's own text can hold a provider's response body (a gateway that echoes the
|
|
||||||
// prompt), a file path (ffmpeg names the spool file), or a word from a private
|
|
||||||
// transcript. None of that may reach a log line, a database column or a page about a
|
|
||||||
// private recording (ADR-033: "typed, never a provider's string"; ADR-035). Of returns
|
|
||||||
// only a fixed vocabulary, so it can never carry any.
|
|
||||||
package errkind
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"net"
|
|
||||||
)
|
|
||||||
|
|
||||||
// HTTPStatuser is implemented by errors that are a non-2xx answer from a service.
|
|
||||||
type HTTPStatuser interface{ HTTPStatus() int }
|
|
||||||
|
|
||||||
// SQLStater is implemented by database errors that carry a SQLSTATE (pgx's PgError does).
|
|
||||||
// The five-character code is a fixed vocabulary; the error's message is not: Postgres quotes
|
|
||||||
// the offending value, which for a transcript row is transcript text.
|
|
||||||
type SQLStater interface{ SQLState() string }
|
|
||||||
|
|
||||||
// Of is the kind of err: "timeout", "canceled", "http_<code>", "sql_<state>", "network",
|
|
||||||
// or "error".
|
|
||||||
// It is empty for nil.
|
|
||||||
func Of(err error) string {
|
|
||||||
if err == nil {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
var hs HTTPStatuser
|
|
||||||
var ss SQLStater
|
|
||||||
var ne net.Error
|
|
||||||
switch {
|
|
||||||
case errors.Is(err, context.DeadlineExceeded):
|
|
||||||
return "timeout"
|
|
||||||
case errors.Is(err, context.Canceled):
|
|
||||||
return "canceled"
|
|
||||||
case errors.As(err, &hs):
|
|
||||||
return fmt.Sprintf("http_%d", hs.HTTPStatus())
|
|
||||||
case errors.As(err, &ss) && ss.SQLState() != "":
|
|
||||||
return "sql_" + ss.SQLState()
|
|
||||||
case errors.As(err, &ne):
|
|
||||||
if ne.Timeout() {
|
|
||||||
return "timeout"
|
|
||||||
}
|
|
||||||
return "network"
|
|
||||||
}
|
|
||||||
return "error"
|
|
||||||
}
|
|
||||||
@@ -1,82 +0,0 @@
|
|||||||
package errkind_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"net"
|
|
||||||
"net/url"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/errkind"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Of reduces an error to a short kind that is safe to log, store and show: it never
|
|
||||||
// carries the error's own text, because that text can hold a provider's response body, a
|
|
||||||
// file path or a word from a private transcript (ADR-035 review F3, F5).
|
|
||||||
|
|
||||||
type statusErr struct{ code int }
|
|
||||||
|
|
||||||
func (e statusErr) Error() string {
|
|
||||||
return fmt.Sprintf("gateway said: %d and echoed the transcript", e.code)
|
|
||||||
}
|
|
||||||
func (e statusErr) HTTPStatus() int { return e.code }
|
|
||||||
|
|
||||||
type timeoutNet struct{}
|
|
||||||
|
|
||||||
func (timeoutNet) Error() string { return "i/o timeout to 10.0.0.1" }
|
|
||||||
func (timeoutNet) Timeout() bool { return true }
|
|
||||||
func (timeoutNet) Temporary() bool { return true }
|
|
||||||
|
|
||||||
func TestOf_ReducesAnErrorToAKindAndNeverItsText(t *testing.T) {
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
err error
|
|
||||||
want string
|
|
||||||
}{
|
|
||||||
{"nothing", nil, ""},
|
|
||||||
{"a deadline", context.DeadlineExceeded, "timeout"},
|
|
||||||
{"a wrapped deadline", fmt.Errorf("transcribe: %w", context.DeadlineExceeded), "timeout"},
|
|
||||||
{"a cancellation", fmt.Errorf("job: %w", context.Canceled), "canceled"},
|
|
||||||
{"an HTTP status", statusErr{code: 502}, "http_502"},
|
|
||||||
{"a wrapped HTTP status", fmt.Errorf("model: %w", statusErr{code: 400}), "http_400"},
|
|
||||||
{"a network timeout", timeoutNet{}, "timeout"},
|
|
||||||
{"a network failure", &net.OpError{Op: "dial", Err: errors.New("connection refused")}, "network"},
|
|
||||||
{"a url error wrapping a refusal", &url.Error{Op: "Post", URL: "http://gateway/secret-path", Err: errors.New("refused")}, "network"},
|
|
||||||
{"anything else", errors.New("moov atom not found in /spool/upload-555"), "error"},
|
|
||||||
}
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
assert.Equal(t, tt.want, errkind.Of(tt.err))
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
type sqlErr struct{}
|
|
||||||
|
|
||||||
func (sqlErr) Error() string {
|
|
||||||
return `ERROR: invalid input syntax for type json: "Anders ansvarar för virket"`
|
|
||||||
}
|
|
||||||
func (sqlErr) SQLState() string { return "22P02" }
|
|
||||||
|
|
||||||
func TestOf_ADatabaseErrorIsReducedToItsSQLStateNeverItsMessage(t *testing.T) {
|
|
||||||
// A Postgres error can quote the offending value, which for a transcript row is
|
|
||||||
// transcript text. The five-character SQLSTATE is a fixed vocabulary (fourth review).
|
|
||||||
err := fmt.Errorf("store: complete upload: %w", sqlErr{})
|
|
||||||
|
|
||||||
got := errkind.Of(err)
|
|
||||||
|
|
||||||
assert.Equal(t, "sql_22P02", got)
|
|
||||||
assert.NotContains(t, got, "Anders")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestOf_AKindCarriesNoWordFromTheError(t *testing.T) {
|
|
||||||
err := fmt.Errorf("convert /spool/upload-555: %w", errors.New("Anders ansvarar för virket"))
|
|
||||||
|
|
||||||
got := errkind.Of(err)
|
|
||||||
|
|
||||||
assert.NotContains(t, got, "Anders")
|
|
||||||
assert.NotContains(t, got, "spool")
|
|
||||||
}
|
|
||||||
@@ -12,7 +12,6 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
"git.d-ma.be/mathias/tapir/internal/errkind"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// Planned is one phase of a job, declared up front so the page can show what is coming.
|
// Planned is one phase of a job, declared up front so the page can show what is coming.
|
||||||
@@ -37,12 +36,11 @@ type Reporter interface {
|
|||||||
// is for people, so a failed write is logged and never stops the job.
|
// is for people, so a failed write is logged and never stops the job.
|
||||||
type Sink interface {
|
type Sink interface {
|
||||||
SaveProgress(ctx context.Context, steps []domain.ProgressStep, at time.Time) error
|
SaveProgress(ctx context.Context, steps []domain.ProgressStep, at time.Time) error
|
||||||
Beat(ctx context.Context, at time.Time) error
|
Beat(ctx context.Context) error
|
||||||
}
|
}
|
||||||
|
|
||||||
const (
|
const (
|
||||||
defaultMinSave = 2 * time.Second
|
defaultMinSave = 2 * time.Second
|
||||||
defaultSaveTimeout = 5 * time.Second
|
|
||||||
logAdvanceEach = time.Minute
|
logAdvanceEach = time.Minute
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -56,14 +54,6 @@ func WithClock(now func() time.Time) Option { return func(t *Tracker) { t.now =
|
|||||||
// Done, Fail and Finish always save.
|
// Done, Fail and Finish always save.
|
||||||
func WithMinSave(d time.Duration) Option { return func(t *Tracker) { t.minSave = d } }
|
func WithMinSave(d time.Duration) Option { return func(t *Tracker) { t.minSave = d } }
|
||||||
|
|
||||||
// WithSaveTimeout bounds one save or heartbeat write. Progress is cosmetic, so a database
|
|
||||||
// that stops answering costs a dropped write rather than an unbounded stall. It is still a
|
|
||||||
// delay: saves run under the tracker's lock, which Begin and Advance (called by the engines)
|
|
||||||
// need, so each transition can wait up to this long, and a hung database slows a job by
|
|
||||||
// that much per call, out of the job's own time budget (#51). A dropped final save is not
|
|
||||||
// retried; the row keeps its last saved step.
|
|
||||||
func WithSaveTimeout(d time.Duration) Option { return func(t *Tracker) { t.saveTimeout = d } }
|
|
||||||
|
|
||||||
// Tracker implements Reporter for one job.
|
// Tracker implements Reporter for one job.
|
||||||
type Tracker struct {
|
type Tracker struct {
|
||||||
ctx context.Context
|
ctx context.Context
|
||||||
@@ -72,8 +62,6 @@ type Tracker struct {
|
|||||||
now func() time.Time
|
now func() time.Time
|
||||||
minSave time.Duration
|
minSave time.Duration
|
||||||
|
|
||||||
saveTimeout time.Duration
|
|
||||||
|
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
steps []domain.ProgressStep
|
steps []domain.ProgressStep
|
||||||
lastAdvanceSave time.Time
|
lastAdvanceSave time.Time
|
||||||
@@ -83,7 +71,7 @@ type Tracker struct {
|
|||||||
// NewTracker plans a job's phases. ctx is used for the saves, so it should outlive the
|
// NewTracker plans a job's phases. ctx is used for the saves, so it should outlive the
|
||||||
// job's own timeout: a job that timed out still gets its failure recorded.
|
// job's own timeout: a job that timed out still gets its failure recorded.
|
||||||
func NewTracker(ctx context.Context, sink Sink, log *slog.Logger, plan []Planned, opts ...Option) *Tracker {
|
func NewTracker(ctx context.Context, sink Sink, log *slog.Logger, plan []Planned, opts ...Option) *Tracker {
|
||||||
t := &Tracker{ctx: ctx, sink: sink, log: log, now: time.Now, minSave: defaultMinSave, saveTimeout: defaultSaveTimeout}
|
t := &Tracker{ctx: ctx, sink: sink, log: log, now: time.Now, minSave: defaultMinSave}
|
||||||
for _, o := range opts {
|
for _, o := range opts {
|
||||||
o(t)
|
o(t)
|
||||||
}
|
}
|
||||||
@@ -158,7 +146,7 @@ func (t *Tracker) Fail(key string, err error) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
t.close(s, domain.StepFailed)
|
t.close(s, domain.StepFailed)
|
||||||
t.log.Warn("step failed", "step", key, "elapsed", s.FinishedAt.Sub(s.StartedAt), "kind", errkind.Of(err))
|
t.log.Warn("step failed", "step", key, "elapsed", s.FinishedAt.Sub(s.StartedAt), "err", err)
|
||||||
t.save()
|
t.save()
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -197,20 +185,6 @@ func (t *Tracker) Finish() {
|
|||||||
t.save()
|
t.save()
|
||||||
}
|
}
|
||||||
|
|
||||||
// Annotate attaches a short plain note to a phase and saves it at once. It is for a
|
|
||||||
// remark the reader should see although nothing failed. Durations and counts only: a note
|
|
||||||
// is shown on a page about a private recording.
|
|
||||||
func (t *Tracker) Annotate(key, note string) {
|
|
||||||
t.mu.Lock()
|
|
||||||
defer t.mu.Unlock()
|
|
||||||
s := t.find(key)
|
|
||||||
if s == nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
s.Note = note
|
|
||||||
t.save()
|
|
||||||
}
|
|
||||||
|
|
||||||
// Heartbeat stamps the job's row every beatEvery and logs which phase is still running
|
// Heartbeat stamps the job's row every beatEvery and logs which phase is still running
|
||||||
// every stillEvery, until ctx ends. It runs on its own, independent of any engine call:
|
// every stillEvery, until ctx ends. It runs on its own, independent of any engine call:
|
||||||
// a heartbeat that stopped means the process is gone, not that a request is slow.
|
// a heartbeat that stopped means the process is gone, not that a request is slow.
|
||||||
@@ -223,10 +197,7 @@ func (t *Tracker) Heartbeat(ctx context.Context, beatEvery, stillEvery time.Dura
|
|||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
return
|
return
|
||||||
case <-beat.C:
|
case <-beat.C:
|
||||||
beatCtx, cancel := context.WithTimeout(ctx, t.saveTimeout)
|
if err := t.sink.Beat(ctx); err != nil && ctx.Err() == nil {
|
||||||
err := t.sink.Beat(beatCtx, t.now())
|
|
||||||
cancel()
|
|
||||||
if err != nil && ctx.Err() == nil {
|
|
||||||
t.log.Warn("progress: heartbeat not saved", "err", err)
|
t.log.Warn("progress: heartbeat not saved", "err", err)
|
||||||
}
|
}
|
||||||
case <-still.C:
|
case <-still.C:
|
||||||
@@ -268,9 +239,7 @@ func (t *Tracker) find(key string) *domain.ProgressStep {
|
|||||||
// dropped: progress is cosmetic and must never fail the job it describes.
|
// dropped: progress is cosmetic and must never fail the job it describes.
|
||||||
func (t *Tracker) save() {
|
func (t *Tracker) save() {
|
||||||
snapshot := append([]domain.ProgressStep(nil), t.steps...)
|
snapshot := append([]domain.ProgressStep(nil), t.steps...)
|
||||||
ctx, cancel := context.WithTimeout(t.ctx, t.saveTimeout)
|
if err := t.sink.SaveProgress(t.ctx, snapshot, t.now()); err != nil && t.ctx.Err() == nil {
|
||||||
defer cancel()
|
|
||||||
if err := t.sink.SaveProgress(ctx, snapshot, t.now()); err != nil && t.ctx.Err() == nil {
|
|
||||||
t.log.Warn("progress: not saved", "err", err)
|
t.log.Warn("progress: not saved", "err", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,154 +0,0 @@
|
|||||||
package progress_test
|
|
||||||
|
|
||||||
// Observes what a database that stops answering does to the job it reports on (#51):
|
|
||||||
// progress is cosmetic, so a save that never returns must not stall the engine calls that
|
|
||||||
// report through the tracker, nor the heartbeat that says the process is alive.
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"context"
|
|
||||||
"log/slog"
|
|
||||||
"strings"
|
|
||||||
"sync"
|
|
||||||
"sync/atomic"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/progress"
|
|
||||||
)
|
|
||||||
|
|
||||||
// stallSink never answers until its context ends, like a database connection that hung.
|
|
||||||
type stallSink struct{ beats atomic.Int32 }
|
|
||||||
|
|
||||||
func (s *stallSink) SaveProgress(ctx context.Context, _ []domain.ProgressStep, _ time.Time) error {
|
|
||||||
<-ctx.Done()
|
|
||||||
return ctx.Err()
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *stallSink) Beat(ctx context.Context, _ time.Time) error {
|
|
||||||
s.beats.Add(1)
|
|
||||||
<-ctx.Done()
|
|
||||||
return ctx.Err()
|
|
||||||
}
|
|
||||||
|
|
||||||
type lockedBuf struct {
|
|
||||||
mu sync.Mutex
|
|
||||||
b bytes.Buffer
|
|
||||||
}
|
|
||||||
|
|
||||||
func (l *lockedBuf) Write(p []byte) (int, error) {
|
|
||||||
l.mu.Lock()
|
|
||||||
defer l.mu.Unlock()
|
|
||||||
return l.b.Write(p)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (l *lockedBuf) String() string {
|
|
||||||
l.mu.Lock()
|
|
||||||
defer l.mu.Unlock()
|
|
||||||
return l.b.String()
|
|
||||||
}
|
|
||||||
|
|
||||||
func stallTracker(sink progress.Sink, buf *lockedBuf) *progress.Tracker {
|
|
||||||
log := slog.New(slog.NewTextHandler(buf, nil))
|
|
||||||
return progress.NewTracker(context.Background(), sink, log,
|
|
||||||
[]progress.Planned{{Key: "a", Label: "A"}}, progress.WithSaveTimeout(50*time.Millisecond))
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTracker_AHungSaveDoesNotBlockTheReporterPastItsTimeout(t *testing.T) {
|
|
||||||
buf := &lockedBuf{}
|
|
||||||
tr := stallTracker(&stallSink{}, buf)
|
|
||||||
|
|
||||||
done := make(chan struct{})
|
|
||||||
go func() { tr.Begin("a", 0); tr.Advance("a", 1); tr.Done("a"); close(done) }()
|
|
||||||
|
|
||||||
select {
|
|
||||||
case <-done:
|
|
||||||
case <-time.After(3 * time.Second):
|
|
||||||
t.Fatal("the reporter was blocked by a save that never returned")
|
|
||||||
}
|
|
||||||
require.Contains(t, buf.String(), "progress: not saved", "a dropped save is logged")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTracker_AHungBeatDoesNotStopTheHeartbeat(t *testing.T) {
|
|
||||||
sink := &stallSink{}
|
|
||||||
tr := stallTracker(sink, &lockedBuf{})
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
go tr.Heartbeat(ctx, 5*time.Millisecond, time.Hour)
|
|
||||||
|
|
||||||
require.Eventually(t, func() bool { return sink.beats.Load() >= 3 }, 3*time.Second, 5*time.Millisecond,
|
|
||||||
"each beat is abandoned at its timeout and the next one is tried")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTracker_TheStillRunningLineIsNotHeldUpBehindAHungSave(t *testing.T) {
|
|
||||||
buf := &lockedBuf{}
|
|
||||||
tr := stallTracker(&stallSink{}, buf)
|
|
||||||
go tr.Begin("a", 0) // holds the tracker's lock inside a save that hangs until its timeout
|
|
||||||
time.Sleep(10 * time.Millisecond)
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
go tr.Heartbeat(ctx, time.Hour, 5*time.Millisecond)
|
|
||||||
|
|
||||||
require.Eventually(t, func() bool { return strings.Contains(buf.String(), "still running") }, 3*time.Second, 5*time.Millisecond)
|
|
||||||
}
|
|
||||||
|
|
||||||
// deadlineSink records how long each write was given, and answers at once.
|
|
||||||
type deadlineSink struct {
|
|
||||||
mu sync.Mutex
|
|
||||||
saves []time.Duration
|
|
||||||
beats []time.Duration
|
|
||||||
noDLOK bool
|
|
||||||
}
|
|
||||||
|
|
||||||
func (d *deadlineSink) record(ctx context.Context, into *[]time.Duration) {
|
|
||||||
d.mu.Lock()
|
|
||||||
defer d.mu.Unlock()
|
|
||||||
dl, ok := ctx.Deadline()
|
|
||||||
if !ok {
|
|
||||||
d.noDLOK = true
|
|
||||||
return
|
|
||||||
}
|
|
||||||
*into = append(*into, time.Until(dl))
|
|
||||||
}
|
|
||||||
|
|
||||||
func (d *deadlineSink) SaveProgress(ctx context.Context, _ []domain.ProgressStep, _ time.Time) error {
|
|
||||||
d.record(ctx, &d.saves)
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (d *deadlineSink) Beat(ctx context.Context, _ time.Time) error {
|
|
||||||
d.record(ctx, &d.beats)
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// The production value is the one the worker gets: it builds a Tracker without options.
|
|
||||||
// A zero or tiny default would drop every write, a huge one would bound nothing.
|
|
||||||
func TestTracker_TheDefaultSaveAndBeatDeadlineIsAFewSecondsAway(t *testing.T) {
|
|
||||||
sink := &deadlineSink{}
|
|
||||||
tr := progress.NewTracker(context.Background(), sink, slog.New(slog.NewTextHandler(&lockedBuf{}, nil)),
|
|
||||||
[]progress.Planned{{Key: "a", Label: "A"}})
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
defer cancel()
|
|
||||||
go tr.Heartbeat(ctx, 5*time.Millisecond, time.Hour)
|
|
||||||
|
|
||||||
tr.Begin("a", 0)
|
|
||||||
require.Eventually(t, func() bool {
|
|
||||||
sink.mu.Lock()
|
|
||||||
defer sink.mu.Unlock()
|
|
||||||
return len(sink.beats) > 0
|
|
||||||
}, 3*time.Second, 5*time.Millisecond)
|
|
||||||
|
|
||||||
sink.mu.Lock()
|
|
||||||
defer sink.mu.Unlock()
|
|
||||||
require.False(t, sink.noDLOK, "a write was given no deadline at all")
|
|
||||||
require.NotEmpty(t, sink.saves)
|
|
||||||
for _, d := range append(append([]time.Duration{}, sink.saves...), sink.beats...) {
|
|
||||||
require.Greater(t, d, time.Second, "the deadline is already (nearly) spent when the write starts")
|
|
||||||
require.LessOrEqual(t, d, 5*time.Second, "the default bound is 5 s")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -21,7 +21,6 @@ type fakeSink struct {
|
|||||||
saves [][]domain.ProgressStep
|
saves [][]domain.ProgressStep
|
||||||
at []time.Time
|
at []time.Time
|
||||||
beats int
|
beats int
|
||||||
beatAt []time.Time
|
|
||||||
err error
|
err error
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -33,11 +32,10 @@ func (f *fakeSink) SaveProgress(_ context.Context, steps []domain.ProgressStep,
|
|||||||
return f.err
|
return f.err
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *fakeSink) Beat(_ context.Context, at time.Time) error {
|
func (f *fakeSink) Beat(context.Context) error {
|
||||||
f.mu.Lock()
|
f.mu.Lock()
|
||||||
defer f.mu.Unlock()
|
defer f.mu.Unlock()
|
||||||
f.beats++
|
f.beats++
|
||||||
f.beatAt = append(f.beatAt, at)
|
|
||||||
return f.err
|
return f.err
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -199,7 +197,7 @@ func TestTracker_FailRunningMarksTheRunningStepFailedAndLeavesTheRestPending(t *
|
|||||||
require.Equal(t, domain.StepPending, stateOf(steps, "save"), "what never started stays pending: it shows where the job stopped")
|
require.Equal(t, domain.StepPending, stateOf(steps, "save"), "what never started stays pending: it shows where the job stopped")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestTracker_AKeyedFailMarksOneStepFailedLogsItsKindAndTheJobGoesOn(t *testing.T) {
|
func TestTracker_AKeyedFailMarksOneStepFailedLogsWhyAndTheJobGoesOn(t *testing.T) {
|
||||||
tr, sink, _, buf := newTracker(t)
|
tr, sink, _, buf := newTracker(t)
|
||||||
tr.Start()
|
tr.Start()
|
||||||
tr.Begin("audio", 0)
|
tr.Begin("audio", 0)
|
||||||
@@ -211,8 +209,7 @@ func TestTracker_AKeyedFailMarksOneStepFailedLogsItsKindAndTheJobGoesOn(t *testi
|
|||||||
require.Equal(t, domain.StepFailed, stateOf(steps, "audio"))
|
require.Equal(t, domain.StepFailed, stateOf(steps, "audio"))
|
||||||
require.Equal(t, domain.StepRunning, stateOf(steps, "decide"), "a soft failure must not stop the next step from running")
|
require.Equal(t, domain.StepRunning, stateOf(steps, "decide"), "a soft failure must not stop the next step from running")
|
||||||
require.Contains(t, buf.String(), "level=WARN")
|
require.Contains(t, buf.String(), "level=WARN")
|
||||||
require.NotContains(t, buf.String(), "gateway said no", "an error's text can carry provider output or a path: the log gets its kind")
|
require.Contains(t, buf.String(), "gateway said no")
|
||||||
require.Contains(t, buf.String(), "kind=error")
|
|
||||||
require.Contains(t, buf.String(), "step=audio")
|
require.Contains(t, buf.String(), "step=audio")
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -262,56 +259,12 @@ func TestTracker_EveryTransitionIsLoggedByTheSameCall(t *testing.T) {
|
|||||||
tr.Begin("audio", 0)
|
tr.Begin("audio", 0)
|
||||||
clk.tick(3 * time.Second)
|
clk.tick(3 * time.Second)
|
||||||
tr.Done("audio")
|
tr.Done("audio")
|
||||||
tr.Begin("decide", 5)
|
|
||||||
clk.tick(2 * time.Second)
|
|
||||||
tr.FailRunning()
|
|
||||||
tr.Finish()
|
|
||||||
|
|
||||||
out := buf.String()
|
out := buf.String()
|
||||||
for _, want := range []string{
|
require.Contains(t, out, "step=audio")
|
||||||
`msg="job started"`,
|
require.Contains(t, out, "started")
|
||||||
`msg="step started" step=audio`,
|
require.Contains(t, out, "finished")
|
||||||
`msg="step finished" step=audio elapsed=3s`,
|
require.Contains(t, out, "elapsed=3s")
|
||||||
`msg="step started" step=decide total=5`,
|
|
||||||
`msg="step failed" step=decide elapsed=2s`,
|
|
||||||
`msg="job finished"`,
|
|
||||||
} {
|
|
||||||
require.Contains(t, out, want, "a transition with no log line is invisible to the operator")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTracker_AKeyedFailIsSavedAtOnceNotOnTheNextTransition(t *testing.T) {
|
|
||||||
tr, sink, _, _ := newTracker(t)
|
|
||||||
tr.Start()
|
|
||||||
tr.Begin("audio", 0)
|
|
||||||
before := sink.saveCount()
|
|
||||||
|
|
||||||
tr.Fail("audio", errors.New("down"))
|
|
||||||
|
|
||||||
require.Equal(t, before+1, sink.saveCount(), "a soft-failed phase must show as failed on the page now")
|
|
||||||
require.Equal(t, domain.StepFailed, stateOf(sink.last(t), "audio"))
|
|
||||||
}
|
|
||||||
|
|
||||||
// retainingSink keeps the very slice it is handed, like a sink that queues it for later.
|
|
||||||
type retainingSink struct{ got [][]domain.ProgressStep }
|
|
||||||
|
|
||||||
func (r *retainingSink) SaveProgress(_ context.Context, steps []domain.ProgressStep, _ time.Time) error {
|
|
||||||
r.got = append(r.got, steps)
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
func (r *retainingSink) Beat(context.Context, time.Time) error { return nil }
|
|
||||||
|
|
||||||
func TestTracker_ASnapshotHandedToTheSinkIsNotChangedByLaterTransitions(t *testing.T) {
|
|
||||||
sink := &retainingSink{}
|
|
||||||
tr := progress.NewTracker(context.Background(), sink, slog.New(slog.NewTextHandler(&syncBuf{}, nil)), plan(),
|
|
||||||
progress.WithClock(newClock().now))
|
|
||||||
tr.Start()
|
|
||||||
first := sink.got[0]
|
|
||||||
|
|
||||||
tr.Begin("audio", 0)
|
|
||||||
tr.Done("audio")
|
|
||||||
|
|
||||||
require.Equal(t, domain.StepPending, first[0].State, "a sink that keeps a snapshot must not see it change under it")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestTracker_AdvanceIsLoggedAtMostOncePerMinute(t *testing.T) {
|
func TestTracker_AdvanceIsLoggedAtMostOncePerMinute(t *testing.T) {
|
||||||
@@ -389,33 +342,3 @@ func TestWithAndFrom_ACarriedReporterIsTheOneReturned(t *testing.T) {
|
|||||||
|
|
||||||
require.Equal(t, domain.StepRunning, stateOf(sink.last(t), "audio"))
|
require.Equal(t, domain.StepRunning, stateOf(sink.last(t), "audio"))
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestTracker_AnnotateAttachesANoteToAStepAndSavesItAtOnce(t *testing.T) {
|
|
||||||
tr, sink, _, _ := newTracker(t)
|
|
||||||
tr.Start()
|
|
||||||
tr.Begin("audio", 0)
|
|
||||||
before := sink.saveCount()
|
|
||||||
|
|
||||||
tr.Annotate("audio", "20 min of the 79 the file claimed")
|
|
||||||
|
|
||||||
require.Equal(t, before+1, sink.saveCount())
|
|
||||||
require.Equal(t, "20 min of the 79 the file claimed", sink.last(t)[0].Note)
|
|
||||||
require.NotPanics(t, func() { tr.Annotate("no-such-step", "x") })
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestTracker_BeatIsStampedWithTheTrackersClock(t *testing.T) {
|
|
||||||
// alive_at used the database's NOW() while the page compared it with the application
|
|
||||||
// clock; a skewed database clock showed a healthy job as lost (#51).
|
|
||||||
tr, sink, clk, _ := newTracker(t)
|
|
||||||
clk.tick(42 * time.Second)
|
|
||||||
want := clk.now()
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
go tr.Heartbeat(ctx, 5*time.Millisecond, time.Hour)
|
|
||||||
|
|
||||||
require.Eventually(t, func() bool { return sink.beatCount() >= 1 }, 2*time.Second, 5*time.Millisecond)
|
|
||||||
sink.mu.Lock()
|
|
||||||
defer sink.mu.Unlock()
|
|
||||||
require.True(t, sink.beatAt[0].Equal(want), "beat stamped %v, the tracker's clock says %v", sink.beatAt[0], want)
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ func (f *fakeCompleter) Complete(_ context.Context, system, user string) (string
|
|||||||
return f.replies[i], nil
|
return f.replies[i], nil
|
||||||
}
|
}
|
||||||
|
|
||||||
var req = reconcile.Request{Before: "grannarna ska då", After: "De ska kunna måla", Base: "målade", Alt: "mölade"}
|
var req = reconcile.Request{Before: "kunderna ska då", After: "De ska kunna betala", Base: "onboardade", Alt: "omvårdade"}
|
||||||
|
|
||||||
func TestLLMAdjudicator_SendsTheMeasuredPromptWithTheReadingsInTheRequestedOrder(t *testing.T) {
|
func TestLLMAdjudicator_SendsTheMeasuredPromptWithTheReadingsInTheRequestedOrder(t *testing.T) {
|
||||||
fc := &fakeCompleter{replies: []string{"ANSWER: 1"}}
|
fc := &fakeCompleter{replies: []string{"ANSWER: 1"}}
|
||||||
@@ -45,10 +45,10 @@ func TestLLMAdjudicator_SendsTheMeasuredPromptWithTheReadingsInTheRequestedOrder
|
|||||||
assert.Contains(t, fc.systems[0], "two automatic transcriptions of the same Swedish speech")
|
assert.Contains(t, fc.systems[0], "two automatic transcriptions of the same Swedish speech")
|
||||||
assert.Contains(t, fc.systems[0], "ANSWER: 1")
|
assert.Contains(t, fc.systems[0], "ANSWER: 1")
|
||||||
u := fc.users[0]
|
u := fc.users[0]
|
||||||
assert.Contains(t, u, "Text before the spot:\ngrannarna ska då")
|
assert.Contains(t, u, "Text before the spot:\nkunderna ska då")
|
||||||
assert.Contains(t, u, "[[ THE SPOT ]]")
|
assert.Contains(t, u, "[[ THE SPOT ]]")
|
||||||
assert.Contains(t, u, "Text after the spot:\nDe ska kunna måla")
|
assert.Contains(t, u, "Text after the spot:\nDe ska kunna betala")
|
||||||
assert.Contains(t, u, "Reading 1: målade\nReading 2: mölade", "base first when AltFirst is false")
|
assert.Contains(t, u, "Reading 1: onboardade\nReading 2: omvårdade", "base first when AltFirst is false")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestLLMAdjudicator_AltFirstSwapsWhichReadingIsNumberOne(t *testing.T) {
|
func TestLLMAdjudicator_AltFirstSwapsWhichReadingIsNumberOne(t *testing.T) {
|
||||||
@@ -59,7 +59,7 @@ func TestLLMAdjudicator_AltFirstSwapsWhichReadingIsNumberOne(t *testing.T) {
|
|||||||
_, err := reconcile.NewLLMAdjudicator(fc).Adjudicate(context.Background(), r)
|
_, err := reconcile.NewLLMAdjudicator(fc).Adjudicate(context.Background(), r)
|
||||||
|
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Contains(t, fc.users[0], "Reading 1: mölade\nReading 2: målade")
|
assert.Contains(t, fc.users[0], "Reading 1: omvårdade\nReading 2: onboardade")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestLLMAdjudicator_MapsTheAnswerBackToBaseOrAltWhicheverOrderWasShown(t *testing.T) {
|
func TestLLMAdjudicator_MapsTheAnswerBackToBaseOrAltWhicheverOrderWasShown(t *testing.T) {
|
||||||
@@ -134,8 +134,8 @@ func TestOrderChecked_AsksTwiceWithTheOrderSwappedAndKeepsOnlyAgreement(t *testi
|
|||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Equal(t, tt.want, got)
|
assert.Equal(t, tt.want, got)
|
||||||
require.Len(t, fc.users, 2)
|
require.Len(t, fc.users, 2)
|
||||||
assert.True(t, strings.Contains(fc.users[0], "Reading 1: målade"), "first call shows the base first")
|
assert.True(t, strings.Contains(fc.users[0], "Reading 1: onboardade"), "first call shows the base first")
|
||||||
assert.True(t, strings.Contains(fc.users[1], "Reading 1: mölade"), "second call shows the alternative first")
|
assert.True(t, strings.Contains(fc.users[1], "Reading 1: omvårdade"), "second call shows the alternative first")
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -19,12 +19,12 @@ func TestFindDisputes_NoneWhenTheTextsAgreeWordForWord(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFindDisputes_OneReplacedWordIsOneDisputeCarryingBothReadings(t *testing.T) {
|
func TestFindDisputes_OneReplacedWordIsOneDisputeCarryingBothReadings(t *testing.T) {
|
||||||
got := disputes("Vi ska målade grannarna.", "Vi ska mölade grannarna.")
|
got := disputes("Vi ska onboardade kunderna.", "Vi ska omvårdade kunderna.")
|
||||||
|
|
||||||
require.Len(t, got, 1)
|
require.Len(t, got, 1)
|
||||||
d := got[0]
|
d := got[0]
|
||||||
assert.Equal(t, "målade", d.Base)
|
assert.Equal(t, "onboardade", d.Base)
|
||||||
assert.Equal(t, "mölade", d.Alt)
|
assert.Equal(t, "omvårdade", d.Alt)
|
||||||
assert.Equal(t, 2, d.BaseStart)
|
assert.Equal(t, 2, d.BaseStart)
|
||||||
assert.Equal(t, 3, d.BaseEnd)
|
assert.Equal(t, 3, d.BaseEnd)
|
||||||
assert.Equal(t, 2, d.AltStart)
|
assert.Equal(t, 2, d.AltStart)
|
||||||
@@ -32,11 +32,11 @@ func TestFindDisputes_OneReplacedWordIsOneDisputeCarryingBothReadings(t *testing
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFindDisputes_ReadingsKeepTheirOwnSpellingAndPunctuation(t *testing.T) {
|
func TestFindDisputes_ReadingsKeepTheirOwnSpellingAndPunctuation(t *testing.T) {
|
||||||
got := disputes("Det blir A+2, alltså.", "Det blir A 2, alltså.")
|
got := disputes("Det blir T+1, alltså.", "Det blir T 1, alltså.")
|
||||||
|
|
||||||
require.Len(t, got, 1)
|
require.Len(t, got, 1)
|
||||||
assert.Equal(t, "A+2,", got[0].Base)
|
assert.Equal(t, "T+1,", got[0].Base)
|
||||||
assert.Equal(t, "A 2,", got[0].Alt, "the alternative's words replace the span as the engine wrote them")
|
assert.Equal(t, "T 1,", got[0].Alt, "the alternative's words replace the span as the engine wrote them")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestFindDisputes_AWordOnlyOneEngineHeardHasAnEmptyOtherSide(t *testing.T) {
|
func TestFindDisputes_AWordOnlyOneEngineHeardHasAnEmptyOtherSide(t *testing.T) {
|
||||||
@@ -90,12 +90,12 @@ func TestAutoResolve(t *testing.T) {
|
|||||||
base, alt string
|
base, alt string
|
||||||
resolved bool
|
resolved bool
|
||||||
}{
|
}{
|
||||||
{"a compound written as two words or one", "i takrenns delen", "i takrennsdelen", true},
|
{"a compound written as two words or one", "i crossborder delen", "i crossborderdelen", true},
|
||||||
{"a compound with a hyphen", "takrenns-delen", "takrenns delen", true},
|
{"a compound with a hyphen", "crossborder-delen", "crossborder delen", true},
|
||||||
{"a filler only the alternative heard", "vi ska ju betala", "vi ska betala", true},
|
{"a filler only the alternative heard", "vi ska ju betala", "vi ska betala", true},
|
||||||
{"a filler only the base heard", "vi ska betala", "vi ska ju betala", true},
|
{"a filler only the base heard", "vi ska betala", "vi ska ju betala", true},
|
||||||
{"two fillers in a row", "vi ska ja ju betala", "vi ska betala", true},
|
{"two fillers in a row", "vi ska ja ju betala", "vi ska betala", true},
|
||||||
{"a different word is for the model", "vi ska målade", "vi ska mölade", false},
|
{"a different word is for the model", "vi ska onboardade", "vi ska omvårdade", false},
|
||||||
{"a different number is for the model", "totalt 1050 kronor", "totalt 2050 kronor", false},
|
{"a different number is for the model", "totalt 1050 kronor", "totalt 2050 kronor", false},
|
||||||
{"a content word only one side heard is for the model", "ett förslag", "ett coolt förslag", false},
|
{"a content word only one side heard is for the model", "ett förslag", "ett coolt förslag", false},
|
||||||
{"three filler words are too many to wave through", "vi ska ju då liksom betala", "vi ska betala", false},
|
{"three filler words are too many to wave through", "vi ska ju då liksom betala", "vi ska betala", false},
|
||||||
|
|||||||
@@ -6,7 +6,6 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
"git.d-ma.be/mathias/tapir/internal/errkind"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/progress"
|
"git.d-ma.be/mathias/tapir/internal/progress"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -102,8 +101,9 @@ func Reconcile(ctx context.Context, base, alt domain.Transcription, adj Adjudica
|
|||||||
rec.Errors++
|
rec.Errors++
|
||||||
consecutive++
|
consecutive++
|
||||||
if consecutive >= breakAfter {
|
if consecutive >= breakAfter {
|
||||||
rep.Fail("decide", err)
|
failure := fmt.Errorf("the model failed %d times in a row: %w", consecutive, err)
|
||||||
return skipped(base, rec, fmt.Sprintf("the model failed %d times in a row (%s)", consecutive, errkind.Of(err))), nil
|
rep.Fail("decide", failure)
|
||||||
|
return skipped(base, rec, failure.Error()), nil
|
||||||
}
|
}
|
||||||
v = VerdictUnsure
|
v = VerdictUnsure
|
||||||
} else {
|
} else {
|
||||||
|
|||||||
@@ -48,8 +48,8 @@ func reconcileWith(t *testing.T, ctx context.Context, base, alt string, adj reco
|
|||||||
|
|
||||||
// two compound/filler disputes the model is never asked about, then two it is.
|
// two compound/filler disputes the model is never asked about, then two it is.
|
||||||
const (
|
const (
|
||||||
progBase = "i takrenns delen ska vi ju betala totalt 1015 kronor till Axel idag"
|
progBase = "i crossborder delen ska vi ju betala totalt 1015 kronor till SEB idag"
|
||||||
progAlt = "i takrennsdelen ska vi betala totalt 1050 kronor till Alex idag"
|
progAlt = "i crossborderdelen ska vi betala totalt 1050 kronor till SCB idag"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestReconcileProgress_TheDecidingPhaseTotalIsTheModelDecisionsNotTheDisputes(t *testing.T) {
|
func TestReconcileProgress_TheDecidingPhaseTotalIsTheModelDecisionsNotTheDisputes(t *testing.T) {
|
||||||
|
|||||||
@@ -1,33 +0,0 @@
|
|||||||
package reconcile_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/reconcile"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The reason a reconciliation was skipped is stored and shown on a page about a private
|
|
||||||
// recording, so it carries a kind, never the model's error text, which can echo the
|
|
||||||
// prompt (adversarial review F3).
|
|
||||||
|
|
||||||
func TestReconcile_TheBreakerReasonCarriesAKindNotTheModelsText(t *testing.T) {
|
|
||||||
adj := &scripted{fn: func(int, reconcile.Request) (reconcile.Verdict, error) {
|
|
||||||
return reconcile.VerdictBase, errors.New("gateway echoed the prompt: zebrakaka secret words")
|
|
||||||
}}
|
|
||||||
small := opts
|
|
||||||
small.BreakAfter = 2
|
|
||||||
|
|
||||||
out, err := reconcile.Reconcile(context.Background(), tr(progBase), tr(progAlt), adj, small)
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.Equal(t, domain.ReconcileSkipped, out.Reconciliation.Status)
|
|
||||||
assert.NotContains(t, out.Reconciliation.Reason, "zebrakaka", "provider text must not be stored or shown")
|
|
||||||
assert.Contains(t, out.Reconciliation.Reason, "2 times in a row", "the reason still says what happened")
|
|
||||||
assert.Contains(t, out.Reconciliation.Reason, "error", "and carries a kind")
|
|
||||||
}
|
|
||||||
@@ -31,16 +31,16 @@ func TestReconcileSpots_AKeptSpotAndAChangedSpotAreBothStoredWithTheirVerdict(t
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestReconcileSpots_AutoResolvedDisputesAreNotStored(t *testing.T) {
|
func TestReconcileSpots_AutoResolvedDisputesAreNotStored(t *testing.T) {
|
||||||
out := merge(t, tr("i takrenns delen ska vi ju betala"), tr("i takrennsdelen ska vi betala"), always(reconcile.VerdictAlt))
|
out := merge(t, tr("i crossborder delen ska vi ju betala"), tr("i crossborderdelen ska vi betala"), always(reconcile.VerdictAlt))
|
||||||
|
|
||||||
assert.Empty(t, out.Reconciliation.Contested, "compounds and fillers were never judged; they are counted, not listed")
|
assert.Empty(t, out.Reconciliation.Contested, "compounds and fillers were never judged; they are counted, not listed")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestReconcileSpots_ASpotCarriesTheWordsJustBeforeIt(t *testing.T) {
|
func TestReconcileSpots_ASpotCarriesTheWordsJustBeforeIt(t *testing.T) {
|
||||||
out := merge(t, tr("grannen målar ett vanligt staket mot gatan"), tr("grannen målar ett KORT staket mot gatan"), always(reconcile.VerdictAlt))
|
out := merge(t, tr("vi skickar in ett special meddelande till taxsystemet"), tr("vi skickar in ett CAMT meddelande till taxsystemet"), always(reconcile.VerdictAlt))
|
||||||
|
|
||||||
require.Len(t, out.Reconciliation.Contested, 1)
|
require.Len(t, out.Reconciliation.Contested, 1)
|
||||||
assert.Equal(t, "grannen målar ett", out.Reconciliation.Contested[0].Before)
|
assert.Equal(t, "vi skickar in ett", out.Reconciliation.Contested[0].Before)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestReconcileSpots_TheContextIsAFewWordsNotTheWholeMeeting(t *testing.T) {
|
func TestReconcileSpots_TheContextIsAFewWordsNotTheWholeMeeting(t *testing.T) {
|
||||||
|
|||||||
@@ -58,10 +58,10 @@ func TestReconcile_IdenticalTextsNeedNoModelAndComeBackUnchanged(t *testing.T) {
|
|||||||
func TestReconcile_TrivialDisputesAreResolvedWithoutAskingTheModel(t *testing.T) {
|
func TestReconcile_TrivialDisputesAreResolvedWithoutAskingTheModel(t *testing.T) {
|
||||||
adj := always(reconcile.VerdictAlt)
|
adj := always(reconcile.VerdictAlt)
|
||||||
|
|
||||||
out := merge(t, tr("i takrenns delen ska vi ju betala"), tr("i takrennsdelen ska vi betala"), adj)
|
out := merge(t, tr("i crossborder delen ska vi ju betala"), tr("i crossborderdelen ska vi betala"), adj)
|
||||||
|
|
||||||
assert.Empty(t, adj.calls, "a compound split and a filler particle are not worth a model call")
|
assert.Empty(t, adj.calls, "a compound split and a filler particle are not worth a model call")
|
||||||
assert.Equal(t, "i takrenns delen ska vi ju betala", out.Text, "the base reading is kept")
|
assert.Equal(t, "i crossborder delen ska vi ju betala", out.Text, "the base reading is kept")
|
||||||
assert.Equal(t, 2, out.Reconciliation.Disputes)
|
assert.Equal(t, 2, out.Reconciliation.Disputes)
|
||||||
assert.Equal(t, 2, out.Reconciliation.AutoResolved)
|
assert.Equal(t, 2, out.Reconciliation.AutoResolved)
|
||||||
assert.Zero(t, out.Reconciliation.Adjudicated)
|
assert.Zero(t, out.Reconciliation.Adjudicated)
|
||||||
@@ -76,13 +76,13 @@ func TestReconcile_AltVerdictReplacesTheSpanWithTheAlternativesOwnWords(t *testi
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestReconcile_BaseAndUnsureVerdictsKeepTheBaseButOnlyUnsureIsFlagged(t *testing.T) {
|
func TestReconcile_BaseAndUnsureVerdictsKeepTheBaseButOnlyUnsureIsFlagged(t *testing.T) {
|
||||||
base := tr("Totalt 1015 kronor till Axel.")
|
base := tr("Totalt 1015 kronor till SEB.")
|
||||||
alt := tr("Totalt 1050 kronor till Alex.")
|
alt := tr("Totalt 1050 kronor till SCB.")
|
||||||
|
|
||||||
keptBase := merge(t, base, alt, always(reconcile.VerdictBase))
|
keptBase := merge(t, base, alt, always(reconcile.VerdictBase))
|
||||||
unsure := merge(t, base, alt, always(reconcile.VerdictUnsure))
|
unsure := merge(t, base, alt, always(reconcile.VerdictUnsure))
|
||||||
|
|
||||||
assert.Equal(t, "Totalt 1015 kronor till Axel.", keptBase.Text)
|
assert.Equal(t, "Totalt 1015 kronor till SEB.", keptBase.Text)
|
||||||
assert.Zero(t, keptBase.Reconciliation.Unsure)
|
assert.Zero(t, keptBase.Reconciliation.Unsure)
|
||||||
require.Len(t, keptBase.Reconciliation.Contested, 2,
|
require.Len(t, keptBase.Reconciliation.Contested, 2,
|
||||||
"every judged spot is stored, kept ones too: without them the reconciler cannot be compared with either engine alone (ADR-035)")
|
"every judged spot is stored, kept ones too: without them the reconciler cannot be compared with either engine alone (ADR-035)")
|
||||||
@@ -90,7 +90,7 @@ func TestReconcile_BaseAndUnsureVerdictsKeepTheBaseButOnlyUnsureIsFlagged(t *tes
|
|||||||
assert.Equal(t, "base", c.Used)
|
assert.Equal(t, "base", c.Used)
|
||||||
assert.False(t, c.Unsure, "a confident 'the base is right' is stored but is not an uncertain spot")
|
assert.False(t, c.Unsure, "a confident 'the base is right' is stored but is not an uncertain spot")
|
||||||
}
|
}
|
||||||
assert.Equal(t, "Totalt 1015 kronor till Axel.", unsure.Text, "unsure keeps the base")
|
assert.Equal(t, "Totalt 1015 kronor till SEB.", unsure.Text, "unsure keeps the base")
|
||||||
assert.Equal(t, 2, unsure.Reconciliation.Unsure)
|
assert.Equal(t, 2, unsure.Reconciliation.Unsure)
|
||||||
require.Len(t, unsure.Reconciliation.Contested, 2)
|
require.Len(t, unsure.Reconciliation.Contested, 2)
|
||||||
assert.True(t, unsure.Reconciliation.Contested[0].Unsure)
|
assert.True(t, unsure.Reconciliation.Contested[0].Unsure)
|
||||||
@@ -98,12 +98,12 @@ func TestReconcile_BaseAndUnsureVerdictsKeepTheBaseButOnlyUnsureIsFlagged(t *tes
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestReconcile_ChangedSpotsAreRecordedWithBothReadings(t *testing.T) {
|
func TestReconcile_ChangedSpotsAreRecordedWithBothReadings(t *testing.T) {
|
||||||
out := merge(t, tr("det blir A+2 imorgon"), tr("det blir en A imorgon"), always(reconcile.VerdictAlt))
|
out := merge(t, tr("det blir T+1 imorgon"), tr("det blir en T imorgon"), always(reconcile.VerdictAlt))
|
||||||
|
|
||||||
require.Len(t, out.Reconciliation.Contested, 1)
|
require.Len(t, out.Reconciliation.Contested, 1)
|
||||||
c := out.Reconciliation.Contested[0]
|
c := out.Reconciliation.Contested[0]
|
||||||
assert.Equal(t, "A+2", c.Base)
|
assert.Equal(t, "T+1", c.Base)
|
||||||
assert.Equal(t, "en A", c.Alt)
|
assert.Equal(t, "en T", c.Alt)
|
||||||
assert.Equal(t, "alt", c.Used)
|
assert.Equal(t, "alt", c.Used)
|
||||||
assert.False(t, c.Unsure)
|
assert.False(t, c.Unsure)
|
||||||
}
|
}
|
||||||
@@ -122,16 +122,16 @@ func TestReconcile_AWordOnlyTheBaseHeardCanBeRemoved(t *testing.T) {
|
|||||||
|
|
||||||
func TestReconcile_SeveralDisputesAreAppliedIndependentlyAndInOrder(t *testing.T) {
|
func TestReconcile_SeveralDisputesAreAppliedIndependentlyAndInOrder(t *testing.T) {
|
||||||
adj := &scripted{fn: func(i int, r reconcile.Request) (reconcile.Verdict, error) {
|
adj := &scripted{fn: func(i int, r reconcile.Request) (reconcile.Verdict, error) {
|
||||||
if r.Alt == "mölade" || r.Alt == "Alex" {
|
if r.Alt == "omvårdade" || r.Alt == "SCB" {
|
||||||
return reconcile.VerdictAlt, nil
|
return reconcile.VerdictAlt, nil
|
||||||
}
|
}
|
||||||
return reconcile.VerdictBase, nil
|
return reconcile.VerdictBase, nil
|
||||||
}}
|
}}
|
||||||
|
|
||||||
out := merge(t, tr("kunderna målade nu och sen betalas till Axel via ombud med 1050 kronor"),
|
out := merge(t, tr("kunderna onboardade nu och sen betalas till SEB via ombud med 1050 kronor"),
|
||||||
tr("kunderna mölade nu och sen betalas till Alex via ombud med 2050 kronor"), adj)
|
tr("kunderna omvårdade nu och sen betalas till SCB via ombud med 2050 kronor"), adj)
|
||||||
|
|
||||||
assert.Equal(t, "kunderna mölade nu och sen betalas till Alex via ombud med 1050 kronor", out.Text)
|
assert.Equal(t, "kunderna omvårdade nu och sen betalas till SCB via ombud med 1050 kronor", out.Text)
|
||||||
assert.Equal(t, 3, out.Reconciliation.Disputes)
|
assert.Equal(t, 3, out.Reconciliation.Disputes)
|
||||||
assert.Equal(t, 2, out.Reconciliation.Changed)
|
assert.Equal(t, 2, out.Reconciliation.Changed)
|
||||||
}
|
}
|
||||||
@@ -193,8 +193,7 @@ func TestReconcile_ConsecutiveModelErrorsStopTheWholeReconciliationInsteadOfWait
|
|||||||
assert.Len(t, adj.calls, 3, "three consecutive failures and it stops asking")
|
assert.Len(t, adj.calls, 3, "three consecutive failures and it stops asking")
|
||||||
assert.Equal(t, strings.Join(a, " "), out.Text, "the base text, with none of the partial changes")
|
assert.Equal(t, strings.Join(a, " "), out.Text, "the base text, with none of the partial changes")
|
||||||
assert.Equal(t, domain.ReconcileSkipped, out.Reconciliation.Status)
|
assert.Equal(t, domain.ReconcileSkipped, out.Reconciliation.Status)
|
||||||
assert.Contains(t, out.Reconciliation.Reason, "3 times in a row")
|
assert.Contains(t, out.Reconciliation.Reason, "gateway down")
|
||||||
assert.NotContains(t, out.Reconciliation.Reason, "gateway down", "the reason is shown on the page: a kind, not the model's own text")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestReconcile_ASuccessBetweenFailuresResetsTheBreaker(t *testing.T) {
|
func TestReconcile_ASuccessBetweenFailuresResetsTheBreaker(t *testing.T) {
|
||||||
|
|||||||
@@ -34,9 +34,9 @@ func TestTokenize_KeepsTheWordAsWrittenAndKeysItWithoutCaseOrPunctuation(t *test
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestTokenize_KeysIgnoreHyphensAndPlusSoOnlyWordsDecideEquality(t *testing.T) {
|
func TestTokenize_KeysIgnoreHyphensAndPlusSoOnlyWordsDecideEquality(t *testing.T) {
|
||||||
got := reconcile.Tokenize("end-to-end A+2 5 000 kr")
|
got := reconcile.Tokenize("end-to-end T+1 5 000 kr")
|
||||||
|
|
||||||
assert.Equal(t, []string{"endtoend", "a2", "5", "000", "kr"}, keys(got))
|
assert.Equal(t, []string{"endtoend", "t1", "5", "000", "kr"}, keys(got))
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestTokenize_APunctuationOnlyWordJoinsThePreviousWordInsteadOfBecomingOne(t *testing.T) {
|
func TestTokenize_APunctuationOnlyWordJoinsThePreviousWordInsteadOfBecomingOne(t *testing.T) {
|
||||||
|
|||||||
+5
-107
@@ -6,8 +6,6 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
"sync"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
)
|
)
|
||||||
@@ -25,7 +23,7 @@ type Prober interface {
|
|||||||
|
|
||||||
// Store is the slice of the private transcript store the intake needs.
|
// Store is the slice of the private transcript store the intake needs.
|
||||||
type Store interface {
|
type Store interface {
|
||||||
CreateUpload(ctx context.Context, userID, title string, lang domain.Language, hadVideo bool) (string, error)
|
CreateUpload(ctx context.Context, userID, title string, hadVideo bool) (string, error)
|
||||||
FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error
|
FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -35,12 +33,6 @@ type Job struct {
|
|||||||
UserID string
|
UserID string
|
||||||
UploadID string
|
UploadID string
|
||||||
Path string
|
Path string
|
||||||
// Language is what the user chose for this recording; empty means Swedish (a job
|
|
||||||
// queued before the choice existed).
|
|
||||||
Language domain.Language
|
|
||||||
// Duration is what the file CLAIMED when probed. The worker compares it with what
|
|
||||||
// was really converted; zero means unknown (ADR-035 review).
|
|
||||||
Duration time.Duration
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Submitter hands accepted jobs to the in-process worker (ADR-033).
|
// Submitter hands accepted jobs to the in-process worker (ADR-033).
|
||||||
@@ -55,53 +47,6 @@ type Intake struct {
|
|||||||
sub Submitter
|
sub Submitter
|
||||||
limits Limits
|
limits Limits
|
||||||
tmpDir string
|
tmpDir string
|
||||||
|
|
||||||
// MaxReceives and MaxReceivesPerUser bound how many request bodies Accept reads at
|
|
||||||
// once; zero means the defaults. Without them N parallel POSTs each spool up to
|
|
||||||
// MaxBytes while the queue still looks empty, and the spool volume fills (#47).
|
|
||||||
MaxReceives int
|
|
||||||
MaxReceivesPerUser int
|
|
||||||
|
|
||||||
mu sync.Mutex
|
|
||||||
receives int
|
|
||||||
byUser map[string]int
|
|
||||||
}
|
|
||||||
|
|
||||||
const (
|
|
||||||
defaultMaxReceives = 4
|
|
||||||
defaultMaxReceivesPerUser = 2
|
|
||||||
)
|
|
||||||
|
|
||||||
// acquireReceive reserves a receive slot for userID, or reports there is none. Every
|
|
||||||
// successful call must be paired with releaseReceive.
|
|
||||||
func (in *Intake) acquireReceive(userID string) bool {
|
|
||||||
maxAll, maxUser := in.MaxReceives, in.MaxReceivesPerUser
|
|
||||||
if maxAll <= 0 {
|
|
||||||
maxAll = defaultMaxReceives
|
|
||||||
}
|
|
||||||
if maxUser <= 0 {
|
|
||||||
maxUser = defaultMaxReceivesPerUser
|
|
||||||
}
|
|
||||||
in.mu.Lock()
|
|
||||||
defer in.mu.Unlock()
|
|
||||||
if in.receives >= maxAll || in.byUser[userID] >= maxUser {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
if in.byUser == nil {
|
|
||||||
in.byUser = map[string]int{}
|
|
||||||
}
|
|
||||||
in.receives++
|
|
||||||
in.byUser[userID]++
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|
||||||
func (in *Intake) releaseReceive(userID string) {
|
|
||||||
in.mu.Lock()
|
|
||||||
defer in.mu.Unlock()
|
|
||||||
in.receives--
|
|
||||||
if in.byUser[userID]--; in.byUser[userID] <= 0 {
|
|
||||||
delete(in.byUser, userID)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewIntake wires an Intake. tmpDir is where uploads are spooled; it must exist.
|
// NewIntake wires an Intake. tmpDir is where uploads are spooled; it must exist.
|
||||||
@@ -109,47 +54,12 @@ func NewIntake(s Store, p Prober, sub Submitter, l Limits, tmpDir string) *Intak
|
|||||||
return &Intake{store: s, prober: p, sub: sub, limits: l, tmpDir: tmpDir}
|
return &Intake{store: s, prober: p, sub: sub, limits: l, tmpDir: tmpDir}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ErrClientRead marks an error that came from READING the client's body: a connection that
|
|
||||||
// dropped, a body cut short. It is the sender's to retry, unlike a spool, probe or database
|
|
||||||
// fault, which is ours. It is a marker on the error, not a sentinel that others may wrap:
|
|
||||||
// io.ErrUnexpectedEOF alone is also what a database drop looks like.
|
|
||||||
var ErrClientRead = errors.New("uploads: client body read failed")
|
|
||||||
|
|
||||||
// clientReader remembers the last non-EOF error its source returned, so Accept can tell
|
|
||||||
// a failure reading the client from a failure writing the spool file.
|
|
||||||
type clientReader struct {
|
|
||||||
r io.Reader
|
|
||||||
err error
|
|
||||||
}
|
|
||||||
|
|
||||||
func (c *clientReader) Read(p []byte) (int, error) {
|
|
||||||
n, err := c.r.Read(p)
|
|
||||||
if err != nil && !errors.Is(err, io.EOF) {
|
|
||||||
c.err = err
|
|
||||||
}
|
|
||||||
return n, err
|
|
||||||
}
|
|
||||||
|
|
||||||
// busyMessage is what a user is told when the queue is full.
|
|
||||||
const busyMessage = "Too many recordings are waiting to be transcribed. Please try again in a few minutes."
|
|
||||||
|
|
||||||
// Accept spools src to disk (bounded by MaxBytes), probes it, validates it, records
|
// Accept spools src to disk (bounded by MaxBytes), probes it, validates it, records
|
||||||
// a queued upload and submits the job. It returns the upload id. A file that fails
|
// a queued upload and submits the job. It returns the upload id. A file that fails
|
||||||
// validation yields a *RejectError and creates NO row — rejection is synchronous
|
// validation yields a *RejectError and creates NO row — rejection is synchronous
|
||||||
// and leaves nothing behind. On success the temp file is handed to the job and the
|
// and leaves nothing behind. On success the temp file is handed to the job and the
|
||||||
// worker deletes it; on every failure path Accept deletes it itself.
|
// worker deletes it; on every failure path Accept deletes it itself.
|
||||||
func (in *Intake) Accept(ctx context.Context, userID, title string, lang domain.Language, src io.Reader) (string, error) {
|
func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reader) (string, error) {
|
||||||
// Refuse before spooling when the queue has no room, if the submitter can say so: a full
|
|
||||||
// queue otherwise cost up to the upload cap on disk, an ffprobe run and a failed row for
|
|
||||||
// every refused attempt.
|
|
||||||
if rc, ok := in.sub.(interface{ HasRoom(userID string) bool }); ok && !rc.HasRoom(userID) {
|
|
||||||
return "", &RejectError{Reason: ReasonBusy, Message: busyMessage}
|
|
||||||
}
|
|
||||||
// HasRoom cannot see bodies still being received; bound those separately.
|
|
||||||
if !in.acquireReceive(userID) {
|
|
||||||
return "", &RejectError{Reason: ReasonBusy, Message: busyMessage}
|
|
||||||
}
|
|
||||||
defer in.releaseReceive(userID)
|
|
||||||
f, err := os.CreateTemp(in.tmpDir, "upload-*")
|
f, err := os.CreateTemp(in.tmpDir, "upload-*")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", fmt.Errorf("uploads: spool: %w", err)
|
return "", fmt.Errorf("uploads: spool: %w", err)
|
||||||
@@ -163,15 +73,11 @@ func (in *Intake) Accept(ctx context.Context, userID, title string, lang domain.
|
|||||||
}()
|
}()
|
||||||
|
|
||||||
// Read one byte past the cap so "exactly at the limit" and "over" are told apart.
|
// Read one byte past the cap so "exactly at the limit" and "over" are told apart.
|
||||||
cr := &clientReader{r: io.LimitReader(src, in.limits.MaxBytes+1)}
|
n, err := io.Copy(f, io.LimitReader(src, in.limits.MaxBytes+1))
|
||||||
n, err := io.Copy(f, cr)
|
|
||||||
if cerr := f.Close(); err == nil {
|
if cerr := f.Close(); err == nil {
|
||||||
err = cerr
|
err = cerr
|
||||||
}
|
}
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if cr.err != nil && errors.Is(err, cr.err) {
|
|
||||||
return "", fmt.Errorf("%w: %w", ErrClientRead, err)
|
|
||||||
}
|
|
||||||
return "", fmt.Errorf("uploads: receive: %w", err)
|
return "", fmt.Errorf("uploads: receive: %w", err)
|
||||||
}
|
}
|
||||||
if n > in.limits.MaxBytes {
|
if n > in.limits.MaxBytes {
|
||||||
@@ -193,19 +99,11 @@ func (in *Intake) Accept(ctx context.Context, userID, title string, lang domain.
|
|||||||
return "", err
|
return "", err
|
||||||
}
|
}
|
||||||
|
|
||||||
id, err := in.store.CreateUpload(ctx, userID, title, lang, probe.HasVideo)
|
id, err := in.store.CreateUpload(ctx, userID, title, probe.HasVideo)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", fmt.Errorf("uploads: create: %w", err)
|
return "", fmt.Errorf("uploads: create: %w", err)
|
||||||
}
|
}
|
||||||
if err := in.sub.Submit(Job{UserID: userID, UploadID: id, Path: path, Language: lang, Duration: probe.Duration}); err != nil {
|
if err := in.sub.Submit(Job{UserID: userID, UploadID: id, Path: path}); err != nil {
|
||||||
if errors.Is(err, ErrQueueFull) {
|
|
||||||
// The user's to retry, not a server fault: refuse it as such, and do not leave
|
|
||||||
// the row queued -- that is the stuck spinner.
|
|
||||||
if ferr := in.store.FailUpload(ctx, userID, id, domain.UploadErrBusy); ferr != nil {
|
|
||||||
return "", fmt.Errorf("uploads: submit: %w (and failing the row: %v)", err, ferr)
|
|
||||||
}
|
|
||||||
return "", &RejectError{Reason: ReasonBusy, Message: busyMessage}
|
|
||||||
}
|
|
||||||
// A row that will never run must not sit queued: that is the stuck spinner.
|
// A row that will never run must not sit queued: that is the stuck spinner.
|
||||||
if ferr := in.store.FailUpload(ctx, userID, id, domain.UploadErrInterrupted); ferr != nil {
|
if ferr := in.store.FailUpload(ctx, userID, id, domain.UploadErrInterrupted); ferr != nil {
|
||||||
return "", fmt.Errorf("uploads: submit: %w (and failing the row: %v)", err, ferr)
|
return "", fmt.Errorf("uploads: submit: %w (and failing the row: %v)", err, ferr)
|
||||||
|
|||||||
@@ -1,96 +0,0 @@
|
|||||||
package uploads_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/uploads"
|
|
||||||
)
|
|
||||||
|
|
||||||
// A full queue is the user's to retry, not a server fault: a typed rejection, the row
|
|
||||||
// failed honestly, the audio removed (adversarial review F6).
|
|
||||||
|
|
||||||
func TestAccept_AFullQueueIsABusyRejectionNotAServerErrorAndLeavesNothingBehind(t *testing.T) {
|
|
||||||
r := newRig(t)
|
|
||||||
r.sub.err = uploads.ErrQueueFull
|
|
||||||
|
|
||||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
|
||||||
|
|
||||||
var rej *uploads.RejectError
|
|
||||||
require.True(t, errors.As(err, &rej), "a full queue must be a rejection the web layer can answer with 429: %v", err)
|
|
||||||
assert.Equal(t, uploads.ReasonBusy, rej.Reason)
|
|
||||||
assert.Contains(t, rej.Message, "waiting")
|
|
||||||
assert.Equal(t, domain.UploadErrBusy, r.st.failed["up-1"], "the row must not sit queued: that is the stuck spinner")
|
|
||||||
assert.Empty(t, r.tmpFiles(t), "the audio is deleted")
|
|
||||||
}
|
|
||||||
|
|
||||||
// roomSubmitter is a Submitter that can say, before anything is spooled, that there is no room.
|
|
||||||
type roomSubmitter struct {
|
|
||||||
fakeSubmitter
|
|
||||||
room bool
|
|
||||||
}
|
|
||||||
|
|
||||||
func (r *roomSubmitter) HasRoom(string) bool { return r.room }
|
|
||||||
|
|
||||||
func TestAccept_ANoRoomQueueIsRefusedBeforeAnythingIsSpooledProbedOrStored(t *testing.T) {
|
|
||||||
// Checking only at Submit meant 200 MB was spooled and ffprobe run for every refused
|
|
||||||
// attempt, and a failed row was left behind each time (second adversarial review).
|
|
||||||
r := newRig(t)
|
|
||||||
full := &roomSubmitter{room: false}
|
|
||||||
in := uploads.NewIntake(r.st, r.pr, full, uploads.Limits{MaxBytes: 1024, MaxDuration: 90 * time.Minute}, r.tmp)
|
|
||||||
|
|
||||||
_, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
|
||||||
|
|
||||||
var rej *uploads.RejectError
|
|
||||||
require.True(t, errors.As(err, &rej))
|
|
||||||
assert.Equal(t, uploads.ReasonBusy, rej.Reason)
|
|
||||||
assert.Empty(t, r.st.created, "no row for an upload that was never accepted")
|
|
||||||
assert.Empty(t, r.pr.paths, "ffprobe was never run on it")
|
|
||||||
assert.Empty(t, r.tmpFiles(t), "nothing was spooled")
|
|
||||||
assert.Empty(t, full.jobs)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestAccept_ASubmitterThatCannotSayStillWorksAsBefore(t *testing.T) {
|
|
||||||
r := newRig(t)
|
|
||||||
|
|
||||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
|
||||||
|
|
||||||
require.NoError(t, err, "a Submitter without HasRoom is accepted: the check at Submit remains")
|
|
||||||
}
|
|
||||||
|
|
||||||
// brokenBody is a client body that fails mid-read, like a connection that dropped.
|
|
||||||
type brokenBody struct{ n int }
|
|
||||||
|
|
||||||
func (b *brokenBody) Read(p []byte) (int, error) {
|
|
||||||
if b.n > 0 {
|
|
||||||
b.n = 0
|
|
||||||
return copy(p, "partial audio"), nil
|
|
||||||
}
|
|
||||||
return 0, io.ErrUnexpectedEOF
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestAccept_AnErrorReadingTheClientsBodyIsMarkedAsTheClientsAndAWrappedSubmitErrorIsNot(t *testing.T) {
|
|
||||||
r := newRig(t)
|
|
||||||
|
|
||||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, &brokenBody{n: 1})
|
|
||||||
|
|
||||||
require.Error(t, err)
|
|
||||||
assert.ErrorIs(t, err, uploads.ErrClientRead, "a dropped client body is the sender's to retry")
|
|
||||||
assert.Empty(t, r.tmpFiles(t))
|
|
||||||
|
|
||||||
// A fault that is ours must NOT carry the marker, even if it wraps the same sentinel.
|
|
||||||
r2 := newRig(t)
|
|
||||||
r2.sub.err = fmt.Errorf("db: %w", io.ErrUnexpectedEOF)
|
|
||||||
_, err = r2.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
|
||||||
require.Error(t, err)
|
|
||||||
assert.NotErrorIs(t, err, uploads.ErrClientRead)
|
|
||||||
}
|
|
||||||
@@ -1,132 +0,0 @@
|
|||||||
package uploads_test
|
|
||||||
|
|
||||||
// Observes how many request bodies Accept will read at once (#47). HasRoom and the
|
|
||||||
// per-user queue cap act before and after the receive; N parallel POSTs all pass
|
|
||||||
// HasRoom while the queue is still empty, so the receive itself is bounded.
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"io"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/uploads"
|
|
||||||
)
|
|
||||||
|
|
||||||
// slowReceive starts an Accept whose body stays open until the returned writer is closed,
|
|
||||||
// and waits until it holds a spool file, which proves it is inside the receive.
|
|
||||||
type slowReceive struct {
|
|
||||||
w *io.PipeWriter
|
|
||||||
done chan error
|
|
||||||
finished bool
|
|
||||||
}
|
|
||||||
|
|
||||||
func startSlowReceive(t *testing.T, r *rig, in *uploads.Intake, user string, wantFiles int) *slowReceive {
|
|
||||||
t.Helper()
|
|
||||||
pr, pw := io.Pipe()
|
|
||||||
s := &slowReceive{w: pw, done: make(chan error, 1)}
|
|
||||||
go func() {
|
|
||||||
_, err := in.Accept(context.Background(), user, "t", domain.LangSwedish, pr)
|
|
||||||
s.done <- err
|
|
||||||
}()
|
|
||||||
require.Eventually(t, func() bool { return len(r.tmpFiles(t)) == wantFiles }, 2*time.Second, 5*time.Millisecond,
|
|
||||||
"the receive for %s should have started", user)
|
|
||||||
// An abandoned receive ends in a read error, so it never reaches the rig's fakes, which
|
|
||||||
// are not safe for concurrent use. A receive that finish() already ended is a no-op.
|
|
||||||
t.Cleanup(func() {
|
|
||||||
if s.finished {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
_ = pw.CloseWithError(errors.New("test over"))
|
|
||||||
select {
|
|
||||||
case <-s.done:
|
|
||||||
case <-time.After(2 * time.Second):
|
|
||||||
}
|
|
||||||
})
|
|
||||||
return s
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *slowReceive) finish(t *testing.T) error {
|
|
||||||
t.Helper()
|
|
||||||
_, _ = s.w.Write([]byte("data"))
|
|
||||||
require.NoError(t, s.w.Close())
|
|
||||||
s.finished = true
|
|
||||||
select {
|
|
||||||
case err := <-s.done:
|
|
||||||
return err
|
|
||||||
case <-time.After(2 * time.Second):
|
|
||||||
t.Fatal("Accept did not return after its body ended")
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func requireBusy(t *testing.T, err error) {
|
|
||||||
t.Helper()
|
|
||||||
var rej *uploads.RejectError
|
|
||||||
require.True(t, errors.As(err, &rej), "want a busy rejection, got %v", err)
|
|
||||||
assert.Equal(t, uploads.ReasonBusy, rej.Reason)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestAccept_APerUserLimitOnConcurrentReceivesRefusesTheNextBodyBeforeSpoolingIt(t *testing.T) {
|
|
||||||
r := newRig(t)
|
|
||||||
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
|
|
||||||
in.MaxReceivesPerUser = 2
|
|
||||||
in.MaxReceives = 10
|
|
||||||
startSlowReceive(t, r, in, "user-1", 1)
|
|
||||||
startSlowReceive(t, r, in, "user-1", 2)
|
|
||||||
|
|
||||||
_, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
|
||||||
|
|
||||||
requireBusy(t, err)
|
|
||||||
assert.Len(t, r.tmpFiles(t), 2, "the refused body was never spooled")
|
|
||||||
// Another user is not blocked by user-1's two receives.
|
|
||||||
other := startSlowReceive(t, r, in, "user-2", 3)
|
|
||||||
require.NoError(t, other.finish(t))
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestAccept_AnOverallLimitOnConcurrentReceivesRefusesTheNextUser(t *testing.T) {
|
|
||||||
r := newRig(t)
|
|
||||||
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
|
|
||||||
in.MaxReceivesPerUser = 5
|
|
||||||
in.MaxReceives = 3
|
|
||||||
for i, u := range []string{"a", "b", "c"} {
|
|
||||||
startSlowReceive(t, r, in, u, i+1)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err := in.Accept(context.Background(), "d", "t", domain.LangSwedish, strings.NewReader("data"))
|
|
||||||
|
|
||||||
requireBusy(t, err)
|
|
||||||
assert.Len(t, r.tmpFiles(t), 3)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestAccept_AFinishedReceiveFreesItsSlot(t *testing.T) {
|
|
||||||
r := newRig(t)
|
|
||||||
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
|
|
||||||
in.MaxReceivesPerUser = 1
|
|
||||||
in.MaxReceives = 5
|
|
||||||
first := startSlowReceive(t, r, in, "user-1", 1)
|
|
||||||
require.NoError(t, first.finish(t))
|
|
||||||
|
|
||||||
_, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
|
||||||
|
|
||||||
require.NoError(t, err, "the slot is released when Accept returns")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestAccept_AFailedReceiveFreesItsSlot(t *testing.T) {
|
|
||||||
r := newRig(t)
|
|
||||||
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
|
|
||||||
in.MaxReceivesPerUser = 1
|
|
||||||
in.MaxReceives = 5
|
|
||||||
_, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, &brokenBody{n: 1})
|
|
||||||
require.ErrorIs(t, err, uploads.ErrClientRead)
|
|
||||||
|
|
||||||
_, err = in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
|
||||||
|
|
||||||
require.NoError(t, err, "a dropped client must not leak a slot")
|
|
||||||
}
|
|
||||||
@@ -1,32 +0,0 @@
|
|||||||
package uploads_test
|
|
||||||
|
|
||||||
// The limits production runs on are the DEFAULTS: nothing outside tests sets
|
|
||||||
// MaxReceives or MaxReceivesPerUser. Pinned here with the literal numbers, so changing a
|
|
||||||
// default is a visible decision (#47 review). With 4 receives, 20 queued and 1 running at
|
|
||||||
// the 200 MiB cap, the spool needs ~5 GiB of the 6 Gi emptyDir (infra); raising any of
|
|
||||||
// them or TAPIR_UPLOAD_MAX_BYTES must revisit that arithmetic.
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/uploads"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestNewIntake_DefaultsAllowTwoReceivesPerUserAndFourOverall(t *testing.T) {
|
|
||||||
r := newRig(t)
|
|
||||||
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
|
|
||||||
startSlowReceive(t, r, in, "a", 1)
|
|
||||||
startSlowReceive(t, r, in, "a", 2)
|
|
||||||
|
|
||||||
_, err := in.Accept(context.Background(), "a", "t", domain.LangSwedish, strings.NewReader("x"))
|
|
||||||
requireBusy(t, err) // a third body from one user
|
|
||||||
|
|
||||||
startSlowReceive(t, r, in, "b", 3)
|
|
||||||
startSlowReceive(t, r, in, "b", 4)
|
|
||||||
_, err = in.Accept(context.Background(), "c", "t", domain.LangSwedish, strings.NewReader("x"))
|
|
||||||
requireBusy(t, err) // a fifth body overall
|
|
||||||
}
|
|
||||||
@@ -21,15 +21,13 @@ import (
|
|||||||
|
|
||||||
type fakeStore struct {
|
type fakeStore struct {
|
||||||
created []string // titles
|
created []string // titles
|
||||||
langs []domain.Language
|
|
||||||
hadVideo []bool
|
hadVideo []bool
|
||||||
failed map[string]domain.UploadErrorKind
|
failed map[string]domain.UploadErrorKind
|
||||||
nextID string
|
nextID string
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *fakeStore) CreateUpload(_ context.Context, _, title string, lang domain.Language, hadVideo bool) (string, error) {
|
func (f *fakeStore) CreateUpload(_ context.Context, _, title string, hadVideo bool) (string, error) {
|
||||||
f.created = append(f.created, title)
|
f.created = append(f.created, title)
|
||||||
f.langs = append(f.langs, lang)
|
|
||||||
f.hadVideo = append(f.hadVideo, hadVideo)
|
f.hadVideo = append(f.hadVideo, hadVideo)
|
||||||
return f.nextID, nil
|
return f.nextID, nil
|
||||||
}
|
}
|
||||||
@@ -100,7 +98,7 @@ func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
|
|||||||
r := newRig(t)
|
r := newRig(t)
|
||||||
audio := bytes.Repeat([]byte("a"), 500)
|
audio := bytes.Repeat([]byte("a"), 500)
|
||||||
|
|
||||||
id, err := r.in.Accept(context.Background(), "user-1", "Teams export", domain.LangSwedish, bytes.NewReader(audio))
|
id, err := r.in.Accept(context.Background(), "user-1", "Teams export", bytes.NewReader(audio))
|
||||||
|
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Equal(t, "up-1", id)
|
assert.Equal(t, "up-1", id)
|
||||||
@@ -108,8 +106,7 @@ func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
|
|||||||
assert.Equal(t, audio, r.pr.seen, "the prober must see the complete uploaded bytes, on disk")
|
assert.Equal(t, audio, r.pr.seen, "the prober must see the complete uploaded bytes, on disk")
|
||||||
require.Len(t, r.sub.jobs, 1)
|
require.Len(t, r.sub.jobs, 1)
|
||||||
job := r.sub.jobs[0]
|
job := r.sub.jobs[0]
|
||||||
assert.Equal(t, uploads.Job{UserID: "user-1", UploadID: "up-1", Path: job.Path, Duration: 10 * time.Minute, Language: domain.LangSwedish}, job,
|
assert.Equal(t, uploads.Job{UserID: "user-1", UploadID: "up-1", Path: job.Path}, job)
|
||||||
"the job carries what the file CLAIMED, so the worker can compare it with what was converted")
|
|
||||||
assert.Equal(t, filepath.Dir(job.Path), r.tmp, "the temp file lives in the configured directory")
|
assert.Equal(t, filepath.Dir(job.Path), r.tmp, "the temp file lives in the configured directory")
|
||||||
got, err := os.ReadFile(job.Path)
|
got, err := os.ReadFile(job.Path)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
@@ -119,7 +116,7 @@ func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
|
|||||||
func TestAccept_OversizeBodyIsRejectedAndLeavesNothingBehind(t *testing.T) {
|
func TestAccept_OversizeBodyIsRejectedAndLeavesNothingBehind(t *testing.T) {
|
||||||
r := newRig(t)
|
r := newRig(t)
|
||||||
|
|
||||||
_, err := r.in.Accept(context.Background(), "user-1", "big", domain.LangSwedish, strings.NewReader(strings.Repeat("a", 1025)))
|
_, err := r.in.Accept(context.Background(), "user-1", "big", strings.NewReader(strings.Repeat("a", 1025)))
|
||||||
|
|
||||||
var rej *uploads.RejectError
|
var rej *uploads.RejectError
|
||||||
require.True(t, errors.As(err, &rej))
|
require.True(t, errors.As(err, &rej))
|
||||||
@@ -134,7 +131,7 @@ func TestAccept_OversizeBodyIsRejectedAndLeavesNothingBehind(t *testing.T) {
|
|||||||
func TestAccept_BodyExactlyAtTheLimitIsAccepted(t *testing.T) {
|
func TestAccept_BodyExactlyAtTheLimitIsAccepted(t *testing.T) {
|
||||||
r := newRig(t)
|
r := newRig(t)
|
||||||
|
|
||||||
_, err := r.in.Accept(context.Background(), "user-1", "edge", domain.LangSwedish, strings.NewReader(strings.Repeat("a", 1024)))
|
_, err := r.in.Accept(context.Background(), "user-1", "edge", strings.NewReader(strings.Repeat("a", 1024)))
|
||||||
|
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
}
|
}
|
||||||
@@ -154,7 +151,7 @@ func TestAccept_ProbeVerdictRejectsWithoutRowAndRemovesTheFile(t *testing.T) {
|
|||||||
r := newRig(t)
|
r := newRig(t)
|
||||||
r.pr.probe = tt.probe
|
r.pr.probe = tt.probe
|
||||||
|
|
||||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||||
|
|
||||||
var rej *uploads.RejectError
|
var rej *uploads.RejectError
|
||||||
require.True(t, errors.As(err, &rej))
|
require.True(t, errors.As(err, &rej))
|
||||||
@@ -171,7 +168,7 @@ func TestAccept_UnreadableFileIsARejectionButAProberFaultIsAnError(t *testing.T)
|
|||||||
r := newRig(t)
|
r := newRig(t)
|
||||||
r.pr.err = uploads.ErrUnreadable
|
r.pr.err = uploads.ErrUnreadable
|
||||||
|
|
||||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("not media"))
|
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("not media"))
|
||||||
|
|
||||||
var rej *uploads.RejectError
|
var rej *uploads.RejectError
|
||||||
require.True(t, errors.As(err, &rej), "the user's file is the problem: tell them")
|
require.True(t, errors.As(err, &rej), "the user's file is the problem: tell them")
|
||||||
@@ -182,7 +179,7 @@ func TestAccept_UnreadableFileIsARejectionButAProberFaultIsAnError(t *testing.T)
|
|||||||
r := newRig(t)
|
r := newRig(t)
|
||||||
r.pr.err = errors.New("exec: ffprobe: executable file not found")
|
r.pr.err = errors.New("exec: ffprobe: executable file not found")
|
||||||
|
|
||||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||||
|
|
||||||
require.Error(t, err)
|
require.Error(t, err)
|
||||||
var rej *uploads.RejectError
|
var rej *uploads.RejectError
|
||||||
@@ -196,7 +193,7 @@ func TestAccept_ClientDisconnectMidUploadLeavesNothingBehind(t *testing.T) {
|
|||||||
r := newRig(t)
|
r := newRig(t)
|
||||||
body := io.MultiReader(strings.NewReader("partial"), iotest.ErrReader(errors.New("client went away")))
|
body := io.MultiReader(strings.NewReader("partial"), iotest.ErrReader(errors.New("client went away")))
|
||||||
|
|
||||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, body)
|
_, err := r.in.Accept(context.Background(), "user-1", "t", body)
|
||||||
|
|
||||||
require.Error(t, err)
|
require.Error(t, err)
|
||||||
assert.Empty(t, r.pr.paths)
|
assert.Empty(t, r.pr.paths)
|
||||||
@@ -208,7 +205,7 @@ func TestAccept_SubmitFailureFailsTheRowHonestlyAndRemovesTheFile(t *testing.T)
|
|||||||
r := newRig(t)
|
r := newRig(t)
|
||||||
r.sub.err = errors.New("worker stopped")
|
r.sub.err = errors.New("worker stopped")
|
||||||
|
|
||||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||||
|
|
||||||
require.Error(t, err)
|
require.Error(t, err)
|
||||||
assert.Equal(t, domain.UploadErrInterrupted, r.st.failed["up-1"],
|
assert.Equal(t, domain.UploadErrInterrupted, r.st.failed["up-1"],
|
||||||
@@ -230,7 +227,7 @@ func TestAccept_AFileWithAVideoTrackIsAcceptedAndRecordsThatItHadOne(t *testing.
|
|||||||
r := newRig(t)
|
r := newRig(t)
|
||||||
r.pr.probe = tt.probe
|
r.pr.probe = tt.probe
|
||||||
|
|
||||||
id, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
|
id, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||||
|
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Equal(t, "up-1", id)
|
assert.Equal(t, "up-1", id)
|
||||||
@@ -240,14 +237,3 @@ func TestAccept_AFileWithAVideoTrackIsAcceptedAndRecordsThatItHadOne(t *testing.
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestAccept_TheChosenLanguageIsStoredAndHandedToTheJob(t *testing.T) {
|
|
||||||
r := newRig(t)
|
|
||||||
|
|
||||||
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangAuto, strings.NewReader("data"))
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Equal(t, []domain.Language{domain.LangAuto}, r.st.langs, "the row records what the user chose")
|
|
||||||
require.Len(t, r.sub.jobs, 1)
|
|
||||||
assert.Equal(t, domain.LangAuto, r.sub.jobs[0].Language, "the worker is told, so the engines are")
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,48 +0,0 @@
|
|||||||
package uploads
|
|
||||||
|
|
||||||
// Observes what jobSink hands the store: the heartbeat's timestamp is the one the tracker
|
|
||||||
// stamped (its application clock), not a second reading taken here (#51).
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
|
||||||
)
|
|
||||||
|
|
||||||
type atStore struct{ beatAt, saveAt time.Time }
|
|
||||||
|
|
||||||
func (*atStore) StartTranscribing(context.Context, string, string) error { return nil }
|
|
||||||
func (*atStore) CompleteUpload(context.Context, string, string, domain.Transcription) error {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
func (*atStore) FailUpload(context.Context, string, string, domain.UploadErrorKind) error { return nil }
|
|
||||||
func (s *atStore) SaveProgress(_ context.Context, _, _ string, _ []domain.ProgressStep, at time.Time) error {
|
|
||||||
s.saveAt = at
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
func (s *atStore) Heartbeat(_ context.Context, _, _ string, at time.Time) error {
|
|
||||||
s.beatAt = at
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestJobSink_ForwardsTheTrackersTimestampToTheStoreUnchanged(t *testing.T) {
|
|
||||||
st := &atStore{}
|
|
||||||
sink := jobSink{st: st, userID: "u", id: "i"}
|
|
||||||
stamp := time.Date(2020, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
|
|
||||||
if err := sink.Beat(context.Background(), stamp); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if err := sink.SaveProgress(context.Background(), nil, stamp); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !st.beatAt.Equal(stamp) {
|
|
||||||
t.Errorf("heartbeat reached the store stamped %v, want the tracker's %v", st.beatAt, stamp)
|
|
||||||
}
|
|
||||||
if !st.saveAt.Equal(stamp) {
|
|
||||||
t.Errorf("progress save reached the store stamped %v, want the tracker's %v", st.saveAt, stamp)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -34,8 +34,6 @@ const (
|
|||||||
ReasonTooLong Reason = "too_long"
|
ReasonTooLong Reason = "too_long"
|
||||||
ReasonTooLarge Reason = "too_large"
|
ReasonTooLarge Reason = "too_large"
|
||||||
ReasonUnreadable Reason = "unreadable"
|
ReasonUnreadable Reason = "unreadable"
|
||||||
// ReasonBusy: the queue is full. Not about the file: the user retries later.
|
|
||||||
ReasonBusy Reason = "busy"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// RejectError is a synchronous, user-presentable refusal. Message is written for
|
// RejectError is a synchronous, user-presentable refusal. Message is written for
|
||||||
|
|||||||
+27
-155
@@ -10,18 +10,12 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
"git.d-ma.be/mathias/tapir/internal/errkind"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/progress"
|
"git.d-ma.be/mathias/tapir/internal/progress"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ErrStopped is returned by Submit once the worker has stopped.
|
// ErrStopped is returned by Submit once the worker has stopped.
|
||||||
var ErrStopped = errors.New("uploads: worker stopped")
|
var ErrStopped = errors.New("uploads: worker stopped")
|
||||||
|
|
||||||
// ErrQueueFull is returned by Submit when the queue, or one user's share of it, is at
|
|
||||||
// its cap. Every queued job holds up to the upload cap on disk until its turn, so an
|
|
||||||
// unbounded queue is a disk-filling lever (ADR-035 review).
|
|
||||||
var ErrQueueFull = errors.New("uploads: queue full")
|
|
||||||
|
|
||||||
// Converter turns the uploaded file into the 16 kHz mono PCM WAV whisper wants.
|
// Converter turns the uploaded file into the 16 kHz mono PCM WAV whisper wants.
|
||||||
type Converter interface {
|
type Converter interface {
|
||||||
ToWAV(ctx context.Context, src, dst string) error
|
ToWAV(ctx context.Context, src, dst string) error
|
||||||
@@ -33,13 +27,6 @@ type Transcriber interface {
|
|||||||
Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error)
|
Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
// LanguagePlanner is a Planner whose phases depend on the language of the job: a recording
|
|
||||||
// that is not Swedish skips the second engine and the reconcile, and the page should not list
|
|
||||||
// them as coming (tapir#61). The worker prefers it over Planner.
|
|
||||||
type LanguagePlanner interface {
|
|
||||||
PlanFor(lang domain.Language) []progress.Planned
|
|
||||||
}
|
|
||||||
|
|
||||||
// Planner is an optional extra on a Transcriber: it declares its own phases (for
|
// Planner is an optional extra on a Transcriber: it declares its own phases (for
|
||||||
// example one per engine) so the page can show them. A Transcriber that does not
|
// example one per engine) so the page can show them. A Transcriber that does not
|
||||||
// implement it gets a single "transcribing" phase from the worker. Whatever a Planner
|
// implement it gets a single "transcribing" phase from the worker. Whatever a Planner
|
||||||
@@ -54,7 +41,7 @@ type WorkerStore interface {
|
|||||||
CompleteUpload(ctx context.Context, userID, id string, t domain.Transcription) error
|
CompleteUpload(ctx context.Context, userID, id string, t domain.Transcription) error
|
||||||
FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error
|
FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error
|
||||||
SaveProgress(ctx context.Context, userID, id string, steps []domain.ProgressStep, at time.Time) error
|
SaveProgress(ctx context.Context, userID, id string, steps []domain.ProgressStep, at time.Time) error
|
||||||
Heartbeat(ctx context.Context, userID, id string, at time.Time) error
|
Heartbeat(ctx context.Context, userID, id string) error
|
||||||
}
|
}
|
||||||
|
|
||||||
// WorkerOption tunes a Worker.
|
// WorkerOption tunes a Worker.
|
||||||
@@ -66,24 +53,10 @@ func WithHeartbeat(beat, still time.Duration) WorkerOption {
|
|||||||
return func(w *Worker) { w.beatEvery, w.stillEvery = beat, still }
|
return func(w *Worker) { w.beatEvery, w.stillEvery = beat, still }
|
||||||
}
|
}
|
||||||
|
|
||||||
// WithQueueLimits bounds how many jobs may WAIT: in total, and per user. A job being
|
|
||||||
// processed is not waiting. The defaults are 20 and 3 (one instance, one job at a time).
|
|
||||||
func WithQueueLimits(total, perUser int) WorkerOption {
|
|
||||||
return func(w *Worker) { w.maxQueued, w.maxPerUser = total, perUser }
|
|
||||||
}
|
|
||||||
|
|
||||||
// WithMaxDuration makes the worker measure the converted audio and fail a job whose real
|
|
||||||
// length exceeds the limit, or falls far short of what the file claimed. Zero (the
|
|
||||||
// default) measures nothing. The limit is first checked against the file's header, which
|
|
||||||
// can lie (ADR-035 review).
|
|
||||||
func WithMaxDuration(d time.Duration) WorkerOption { return func(w *Worker) { w.maxDuration = d } }
|
|
||||||
|
|
||||||
// WithLogger replaces slog.Default() as the worker's logger.
|
// WithLogger replaces slog.Default() as the worker's logger.
|
||||||
func WithLogger(l *slog.Logger) WorkerOption { return func(w *Worker) { w.log = l } }
|
func WithLogger(l *slog.Logger) WorkerOption { return func(w *Worker) { w.log = l } }
|
||||||
|
|
||||||
const (
|
const (
|
||||||
defaultMaxQueued = 20
|
|
||||||
defaultMaxPerUser = 3
|
|
||||||
defaultBeatEvery = 10 * time.Second
|
defaultBeatEvery = 10 * time.Second
|
||||||
defaultStillEvery = 5 * time.Minute
|
defaultStillEvery = 5 * time.Minute
|
||||||
)
|
)
|
||||||
@@ -98,9 +71,7 @@ func (s jobSink) SaveProgress(ctx context.Context, steps []domain.ProgressStep,
|
|||||||
return s.st.SaveProgress(ctx, s.userID, s.id, steps, at)
|
return s.st.SaveProgress(ctx, s.userID, s.id, steps, at)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s jobSink) Beat(ctx context.Context, at time.Time) error {
|
func (s jobSink) Beat(ctx context.Context) error { return s.st.Heartbeat(ctx, s.userID, s.id) }
|
||||||
return s.st.Heartbeat(ctx, s.userID, s.id, at)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Worker transcribes accepted uploads in-process, ONE AT A TIME (ADR-033). It is
|
// Worker transcribes accepted uploads in-process, ONE AT A TIME (ADR-033). It is
|
||||||
// not a second deployable: it shares the web process's lifetime, which is why it
|
// not a second deployable: it shares the web process's lifetime, which is why it
|
||||||
@@ -116,8 +87,6 @@ type Worker struct {
|
|||||||
|
|
||||||
log *slog.Logger
|
log *slog.Logger
|
||||||
beatEvery, stillEvery time.Duration
|
beatEvery, stillEvery time.Duration
|
||||||
maxDuration time.Duration
|
|
||||||
maxQueued, maxPerUser int
|
|
||||||
|
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
queue []Job
|
queue []Job
|
||||||
@@ -131,7 +100,6 @@ func NewWorker(s WorkerStore, c Converter, t Transcriber, tmpDir string, timeout
|
|||||||
w := &Worker{
|
w := &Worker{
|
||||||
store: s, conv: c, tr: t, tmpDir: tmpDir, timeout: timeout, wake: make(chan struct{}, 1),
|
store: s, conv: c, tr: t, tmpDir: tmpDir, timeout: timeout, wake: make(chan struct{}, 1),
|
||||||
beatEvery: defaultBeatEvery, stillEvery: defaultStillEvery,
|
beatEvery: defaultBeatEvery, stillEvery: defaultStillEvery,
|
||||||
maxQueued: defaultMaxQueued, maxPerUser: defaultMaxPerUser,
|
|
||||||
}
|
}
|
||||||
for _, o := range opts {
|
for _, o := range opts {
|
||||||
o(w)
|
o(w)
|
||||||
@@ -139,24 +107,6 @@ func NewWorker(s WorkerStore, c Converter, t Transcriber, tmpDir string, timeout
|
|||||||
return w
|
return w
|
||||||
}
|
}
|
||||||
|
|
||||||
// HasRoom reports whether a job from this user would be accepted right now. It lets the
|
|
||||||
// intake refuse BEFORE spooling up to the upload cap and running ffprobe; Submit still
|
|
||||||
// enforces the same limits atomically, so a race costs at worst one refused submit.
|
|
||||||
func (w *Worker) HasRoom(userID string) bool {
|
|
||||||
w.mu.Lock()
|
|
||||||
defer w.mu.Unlock()
|
|
||||||
if w.stopped || len(w.queue) >= w.maxQueued {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
mine := 0
|
|
||||||
for _, q := range w.queue {
|
|
||||||
if q.UserID == userID {
|
|
||||||
mine++
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return mine < w.maxPerUser
|
|
||||||
}
|
|
||||||
|
|
||||||
// Submit queues a job. It never blocks. It satisfies Submitter.
|
// Submit queues a job. It never blocks. It satisfies Submitter.
|
||||||
func (w *Worker) Submit(j Job) error {
|
func (w *Worker) Submit(j Job) error {
|
||||||
w.mu.Lock()
|
w.mu.Lock()
|
||||||
@@ -164,18 +114,6 @@ func (w *Worker) Submit(j Job) error {
|
|||||||
if w.stopped {
|
if w.stopped {
|
||||||
return ErrStopped
|
return ErrStopped
|
||||||
}
|
}
|
||||||
if len(w.queue) >= w.maxQueued {
|
|
||||||
return ErrQueueFull
|
|
||||||
}
|
|
||||||
mine := 0
|
|
||||||
for _, q := range w.queue {
|
|
||||||
if q.UserID == j.UserID {
|
|
||||||
mine++
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if mine >= w.maxPerUser {
|
|
||||||
return ErrQueueFull
|
|
||||||
}
|
|
||||||
w.queue = append(w.queue, j)
|
w.queue = append(w.queue, j)
|
||||||
select {
|
select {
|
||||||
case w.wake <- struct{}{}:
|
case w.wake <- struct{}{}:
|
||||||
@@ -185,10 +123,9 @@ func (w *Worker) Submit(j Job) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Run processes queued jobs until ctx ends, then deletes the audio of anything
|
// Run processes queued jobs until ctx ends, then deletes the audio of anything
|
||||||
// still queued and refuses new jobs. The job running at shutdown is recorded interrupted
|
// still queued and refuses new jobs. The rows of those jobs stay queued or
|
||||||
// by Process; the rows of jobs still queued stay queued and are failed as interrupted by
|
// transcribing and are failed as interrupted by the start-up sweep
|
||||||
// the start-up sweep (store.FailInterruptedUploads): the audio is not retained, so they
|
// (store.FailInterruptedUploads): the audio is not retained, so they cannot resume.
|
||||||
// cannot resume.
|
|
||||||
func (w *Worker) Run(ctx context.Context) {
|
func (w *Worker) Run(ctx context.Context) {
|
||||||
defer w.stop()
|
defer w.stop()
|
||||||
for {
|
for {
|
||||||
@@ -237,7 +174,7 @@ func (w *Worker) stop() {
|
|||||||
// Process runs one job: row -> transcribing, convert, transcribe, store. The
|
// Process runs one job: row -> transcribing, convert, transcribe, store. The
|
||||||
// uploaded file and the converted WAV are deleted on every path — the audio is
|
// uploaded file and the converted WAV are deleted on every path — the audio is
|
||||||
// never retained (ADR-033). A failure is recorded with a TYPED kind; a shutdown
|
// never retained (ADR-033). A failure is recorded with a TYPED kind; a shutdown
|
||||||
// mid-job is recorded as interrupted at once (see fail).
|
// mid-job is not a failure and leaves the row for the start-up sweep.
|
||||||
//
|
//
|
||||||
// Every phase is reported through one progress.Tracker, which saves it to the row for
|
// Every phase is reported through one progress.Tracker, which saves it to the row for
|
||||||
// the page and logs it, and a heartbeat stamps the row on its own while the job runs
|
// the page and logs it, and a heartbeat stamps the row on its own while the job runs
|
||||||
@@ -252,20 +189,14 @@ func (w *Worker) Process(ctx context.Context, job Job) {
|
|||||||
start := time.Now()
|
start := time.Now()
|
||||||
|
|
||||||
if err := w.store.StartTranscribing(ctx, job.UserID, job.UploadID); err != nil {
|
if err := w.store.StartTranscribing(ctx, job.UserID, job.UploadID); err != nil {
|
||||||
log.Error("upload: start transcribing", "kind", errkind.Of(err))
|
log.Error("upload: start transcribing", "err", err)
|
||||||
w.fail(ctx, job, log, domain.UploadErrTranscriptionFailed)
|
w.fail(ctx, job, log)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
plan := []progress.Planned{{Key: "received", Label: "Received"}, {Key: "audio", Label: "Preparing the audio"}}
|
plan := []progress.Planned{{Key: "received", Label: "Received"}, {Key: "audio", Label: "Preparing the audio"}}
|
||||||
lang := job.Language
|
|
||||||
if lang == "" {
|
|
||||||
lang = domain.LangSwedish // queued before the choice existed
|
|
||||||
}
|
|
||||||
planner, planned := w.tr.(Planner)
|
planner, planned := w.tr.(Planner)
|
||||||
if lp, ok := w.tr.(LanguagePlanner); ok {
|
if planned {
|
||||||
plan = append(plan, lp.PlanFor(lang)...)
|
|
||||||
} else if planned {
|
|
||||||
plan = append(plan, planner.Plan()...)
|
plan = append(plan, planner.Plan()...)
|
||||||
} else {
|
} else {
|
||||||
plan = append(plan, progress.Planned{Key: "transcribe", Label: "Transcribing"})
|
plan = append(plan, progress.Planned{Key: "transcribe", Label: "Transcribing"})
|
||||||
@@ -285,18 +216,18 @@ func (w *Worker) Process(ctx context.Context, job Job) {
|
|||||||
defer func() { stopBeat(); <-hbDone }() // no heartbeat outlives its job
|
defer func() { stopBeat(); <-hbDone }() // no heartbeat outlives its job
|
||||||
|
|
||||||
// abort records where the job stopped. A shutdown (the parent context ended) is not
|
// abort records where the job stopped. A shutdown (the parent context ended) is not
|
||||||
// a failure of the step: it stays as it was, and fail records the row as interrupted.
|
// a failure: the row is left for the start-up sweep, so the step stays as it was.
|
||||||
abort := func(kind domain.UploadErrorKind) {
|
abort := func() {
|
||||||
if ctx.Err() == nil {
|
if ctx.Err() == nil {
|
||||||
tr.FailRunning()
|
tr.FailRunning()
|
||||||
}
|
}
|
||||||
w.fail(ctx, job, log, kind)
|
w.fail(ctx, job, log)
|
||||||
}
|
}
|
||||||
|
|
||||||
wav, err := w.tempWAV()
|
wav, err := w.tempWAV()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Error("upload: temp wav", "kind", errkind.Of(err))
|
log.Error("upload: temp wav", "err", err)
|
||||||
abort(domain.UploadErrTranscriptionFailed)
|
abort()
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
defer func() { _ = os.Remove(wav) }()
|
defer func() { _ = os.Remove(wav) }()
|
||||||
@@ -305,31 +236,19 @@ func (w *Worker) Process(ctx context.Context, job Job) {
|
|||||||
defer cancel()
|
defer cancel()
|
||||||
tr.Begin("audio", 0)
|
tr.Begin("audio", 0)
|
||||||
if err := w.conv.ToWAV(jobCtx, job.Path, wav); err != nil {
|
if err := w.conv.ToWAV(jobCtx, job.Path, wav); err != nil {
|
||||||
log.Error("upload: convert", "kind", errkind.Of(err))
|
log.Error("upload: convert", "err", err)
|
||||||
abort(domain.UploadErrTranscriptionFailed)
|
abort()
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
got, tooLong := w.audioLengthProblem(wav)
|
|
||||||
if tooLong {
|
|
||||||
abort(domain.UploadErrTooLong)
|
|
||||||
log.Error("upload: converted audio is over the limit", "audio_s", int(got.Seconds()), "limit_s", int(w.maxDuration.Seconds()))
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if w.maxDuration > 0 && job.Duration > 0 {
|
|
||||||
if note := shortfallNote(got, job.Duration); note != "" {
|
|
||||||
log.Warn("upload: audio is much shorter than the file claimed", "audio_s", int(got.Seconds()), "claimed_s", int(job.Duration.Seconds()))
|
|
||||||
tr.Annotate("audio", note)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
tr.Done("audio")
|
tr.Done("audio")
|
||||||
|
|
||||||
if !planned {
|
if !planned {
|
||||||
tr.Begin("transcribe", 0)
|
tr.Begin("transcribe", 0)
|
||||||
}
|
}
|
||||||
t, err := w.tr.Transcribe(domain.WithLanguage(progress.With(jobCtx, tr), lang), wav)
|
t, err := w.tr.Transcribe(progress.With(jobCtx, tr), wav)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Error("upload: transcribe", "kind", errkind.Of(err), "elapsed", time.Since(start).Round(time.Second))
|
log.Error("upload: transcribe", "err", err, "elapsed", time.Since(start).Round(time.Second))
|
||||||
abort(domain.UploadErrTranscriptionFailed)
|
abort()
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if !planned {
|
if !planned {
|
||||||
@@ -338,8 +257,8 @@ func (w *Worker) Process(ctx context.Context, job Job) {
|
|||||||
|
|
||||||
tr.Begin("save", 0)
|
tr.Begin("save", 0)
|
||||||
if err := w.store.CompleteUpload(ctx, job.UserID, job.UploadID, t); err != nil {
|
if err := w.store.CompleteUpload(ctx, job.UserID, job.UploadID, t); err != nil {
|
||||||
log.Error("upload: store transcript", "kind", errkind.Of(err))
|
log.Error("upload: store transcript", "err", err)
|
||||||
abort(domain.UploadErrTranscriptionFailed)
|
abort()
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
tr.Done("save")
|
tr.Done("save")
|
||||||
@@ -348,22 +267,14 @@ func (w *Worker) Process(ctx context.Context, job Job) {
|
|||||||
"segments", len(t.Segments), "elapsed", time.Since(start).Round(time.Second))
|
"segments", len(t.Segments), "elapsed", time.Since(start).Round(time.Second))
|
||||||
}
|
}
|
||||||
|
|
||||||
// interruptedWriteTimeout bounds the one write a dying process makes to record its job.
|
// fail records a transcription failure — unless the PARENT context ended, which
|
||||||
const interruptedWriteTimeout = 3 * time.Second
|
// is a shutdown, not a failure: the row is left for the start-up sweep.
|
||||||
|
func (w *Worker) fail(ctx context.Context, job Job, log *slog.Logger) {
|
||||||
// fail records a transcription failure. When the PARENT context ended the process is
|
|
||||||
// shutting down: the job is recorded interrupted at once, on a short context of its own,
|
|
||||||
// because the process may exit before any other pod's start-up sweep would do it (#50).
|
|
||||||
// The sweep remains the backstop for a process that is killed outright.
|
|
||||||
func (w *Worker) fail(ctx context.Context, job Job, log *slog.Logger, kind domain.UploadErrorKind) {
|
|
||||||
if ctx.Err() != nil {
|
if ctx.Err() != nil {
|
||||||
kind = domain.UploadErrInterrupted
|
return
|
||||||
var cancel context.CancelFunc
|
|
||||||
ctx, cancel = context.WithTimeout(context.WithoutCancel(ctx), interruptedWriteTimeout)
|
|
||||||
defer cancel()
|
|
||||||
}
|
}
|
||||||
if err := w.store.FailUpload(ctx, job.UserID, job.UploadID, kind); err != nil {
|
if err := w.store.FailUpload(ctx, job.UserID, job.UploadID, domain.UploadErrTranscriptionFailed); err != nil {
|
||||||
log.Error("upload: record failure", "kind", errkind.Of(err))
|
log.Error("upload: record failure", "err", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -379,42 +290,3 @@ func (w *Worker) tempWAV() (string, error) {
|
|||||||
}
|
}
|
||||||
return name, nil
|
return name, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
const (
|
|
||||||
// tooLongSlack is how far over the limit the converted audio may be before the claim
|
|
||||||
// in the file's header is called a lie: ffmpeg rounds.
|
|
||||||
tooLongSlack = 30 * time.Second
|
|
||||||
// A file counts as truncated when it converts to under 80% of the length it claimed
|
|
||||||
// AND more than two minutes short. Probe estimates for variable-bitrate files are
|
|
||||||
// imprecise; refusing real recordings would be worse than the problem.
|
|
||||||
incompleteBelow = 0.8
|
|
||||||
incompleteShortfall = 2 * time.Minute
|
|
||||||
)
|
|
||||||
|
|
||||||
// audioLengthProblem measures the converted WAV (16 kHz mono 16-bit PCM is 32,000 bytes a
|
|
||||||
// second after a 44-byte header) against the limit. It returns the real length and whether
|
|
||||||
// that is over the limit by more than the tolerance.
|
|
||||||
func (w *Worker) audioLengthProblem(wav string) (got time.Duration, tooLong bool) {
|
|
||||||
if w.maxDuration <= 0 {
|
|
||||||
return 0, false
|
|
||||||
}
|
|
||||||
info, err := os.Stat(wav)
|
|
||||||
if err != nil {
|
|
||||||
return 0, false
|
|
||||||
}
|
|
||||||
got = time.Duration(float64(max(info.Size()-44, 0)) / 32000 * float64(time.Second))
|
|
||||||
return got, got > w.maxDuration+tooLongSlack
|
|
||||||
}
|
|
||||||
|
|
||||||
// shortfallNote is a remark for the audio step when the converted audio is much shorter
|
|
||||||
// than the file claimed, or "" when it is not. It is a NOTE and never a failure: a
|
|
||||||
// container header can over-estimate a complete file (a variable-bitrate MP3 without a Xing
|
|
||||||
// header claimed 79 minutes for 20 real ones), so refusing would turn away real
|
|
||||||
// recordings, while saying nothing about a truncated one would hide a partial transcript.
|
|
||||||
func shortfallNote(got, claimed time.Duration) string {
|
|
||||||
if float64(got) >= incompleteBelow*float64(claimed) || claimed-got <= incompleteShortfall {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
return fmt.Sprintf("The audio is %s long; the file claimed %s. If the recording was cut off, upload the complete one.",
|
|
||||||
FormatDuration(got), FormatDuration(claimed))
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,187 +0,0 @@
|
|||||||
package uploads_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"os"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/uploads"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The duration limit is a claim in the file's header. The worker measures what was
|
|
||||||
// really converted and refuses both a recording that is longer than the limit (the
|
|
||||||
// header lied low) and one that ends far earlier than the file claimed (it was truncated
|
|
||||||
// and would otherwise yield a partial transcript with no warning) -- adversarial review F2.
|
|
||||||
|
|
||||||
// sizedConverter writes a (sparse) 16 kHz mono PCM WAV of the given length, without
|
|
||||||
// actually allocating it: 32,000 bytes per second plus a 44-byte header.
|
|
||||||
type sizedConverter struct{ seconds float64 }
|
|
||||||
|
|
||||||
func (c sizedConverter) ToWAV(_ context.Context, _, dst string) error {
|
|
||||||
f, err := os.Create(dst)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
defer func() { _ = f.Close() }()
|
|
||||||
return f.Truncate(44 + int64(c.seconds*32000))
|
|
||||||
}
|
|
||||||
|
|
||||||
func durationWorker(t *testing.T, r *wrig, wavSeconds float64, max time.Duration) *uploads.Worker {
|
|
||||||
t.Helper()
|
|
||||||
opts := []uploads.WorkerOption{}
|
|
||||||
if max > 0 {
|
|
||||||
opts = append(opts, uploads.WithMaxDuration(max))
|
|
||||||
}
|
|
||||||
return uploads.NewWorker(r.st, sizedConverter{seconds: wavSeconds}, r.tr, r.tmp, time.Minute, opts...)
|
|
||||||
}
|
|
||||||
|
|
||||||
func jobWith(r *wrig, t *testing.T, claimed time.Duration) uploads.Job {
|
|
||||||
j := r.spool(t, "a")
|
|
||||||
j.Duration = claimed
|
|
||||||
return j
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWorkerDuration_AConvertedLengthOverTheLimitFailsTheJobBeforeAnyEngineSeesIt(t *testing.T) {
|
|
||||||
r := newWRig(t, time.Minute)
|
|
||||||
w := durationWorker(t, r, 100*60, 90*time.Minute)
|
|
||||||
|
|
||||||
w.Process(context.Background(), jobWith(r, t, 10*time.Second))
|
|
||||||
|
|
||||||
_, failed := r.st.snapshot()
|
|
||||||
assert.Equal(t, domain.UploadErrTooLong, failed["a"], "a header that claimed 10 s must not let 100 minutes through")
|
|
||||||
assert.Empty(t, r.tr.paths, "no engine may be sent audio that is over the limit")
|
|
||||||
assert.Empty(t, r.leftovers(t), "the audio is still deleted")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWorkerDuration_JustOverTheLimitWithinTheSlackStillTranscribes(t *testing.T) {
|
|
||||||
r := newWRig(t, time.Minute)
|
|
||||||
w := durationWorker(t, r, 90*60+20, 90*time.Minute)
|
|
||||||
|
|
||||||
w.Process(context.Background(), jobWith(r, t, 90*time.Minute))
|
|
||||||
|
|
||||||
completed, failed := r.st.snapshot()
|
|
||||||
assert.Empty(t, failed)
|
|
||||||
assert.Contains(t, completed, "a", "ffmpeg rounds: 20 s over a 90-minute limit is not a lie")
|
|
||||||
}
|
|
||||||
|
|
||||||
func audioStepNote(r *wrig) string {
|
|
||||||
for _, st := range r.st.lastSteps() {
|
|
||||||
if st.Key == "audio" {
|
|
||||||
return st.Note
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWorkerDuration_AFileThatEndsFarEarlierThanItClaimedIsStillTranscribedAndSaysSo(t *testing.T) {
|
|
||||||
// A container header can over-estimate a complete file (VBR MP3 without a Xing header
|
|
||||||
// claimed 79 minutes for 20 real ones). Refusing it was wrong; staying silent about a
|
|
||||||
// truncated one is wrong too. So it is transcribed, and the audio step carries a note.
|
|
||||||
r := newWRig(t, time.Minute)
|
|
||||||
w := durationWorker(t, r, 20*60, 90*time.Minute)
|
|
||||||
|
|
||||||
w.Process(context.Background(), jobWith(r, t, 79*time.Minute))
|
|
||||||
|
|
||||||
completed, failed := r.st.snapshot()
|
|
||||||
assert.Empty(t, failed, "a complete recording with a bad header estimate must not be refused")
|
|
||||||
assert.Contains(t, completed, "a")
|
|
||||||
note := audioStepNote(r)
|
|
||||||
assert.Contains(t, note, "20 min", "what was really transcribed")
|
|
||||||
assert.Contains(t, note, "1 h 19 min", "what the file claimed")
|
|
||||||
assert.Contains(t, note, "cut off", "and what to do if that is not a surprise")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWorkerDuration_TheShortfallNoteIsPinnedAtEightyPercentAndTwoMinutes(t *testing.T) {
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
claimed time.Duration
|
|
||||||
wav float64
|
|
||||||
noted bool
|
|
||||||
}{
|
|
||||||
{"79 percent of an hour", 60 * time.Minute, 0.79 * 3600, true},
|
|
||||||
{"exactly 80 percent of an hour", 60 * time.Minute, 2880, false},
|
|
||||||
{"one second under 80 percent of an hour", 60 * time.Minute, 2879, true},
|
|
||||||
{"81 percent of an hour", 60 * time.Minute, 0.81 * 3600, false},
|
|
||||||
{"exactly two minutes short and under 80 percent", 6 * time.Minute, 4 * 60, false},
|
|
||||||
{"two minutes and a second short and under 80 percent", 6 * time.Minute, 4*60 - 1, true},
|
|
||||||
{"unknown claimed length", 0, 20 * 60, false},
|
|
||||||
}
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
r := newWRig(t, time.Minute)
|
|
||||||
w := durationWorker(t, r, tt.wav, 90*time.Minute)
|
|
||||||
|
|
||||||
w.Process(context.Background(), jobWith(r, t, tt.claimed))
|
|
||||||
|
|
||||||
assert.Equal(t, tt.noted, audioStepNote(r) != "")
|
|
||||||
completed, _ := r.st.snapshot()
|
|
||||||
assert.Contains(t, completed, "a", "a note never stops the job")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWorkerDuration_TheTooLongToleranceIsExactlyThirtySeconds(t *testing.T) {
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
wav float64
|
|
||||||
fail bool
|
|
||||||
}{
|
|
||||||
{"at the limit plus 30 s", 90*60 + 30, false},
|
|
||||||
{"at the limit plus 31 s", 90*60 + 31, true},
|
|
||||||
}
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
r := newWRig(t, time.Minute)
|
|
||||||
w := durationWorker(t, r, tt.wav, 90*time.Minute)
|
|
||||||
|
|
||||||
w.Process(context.Background(), jobWith(r, t, 90*time.Minute))
|
|
||||||
|
|
||||||
_, failed := r.st.snapshot()
|
|
||||||
if tt.fail {
|
|
||||||
assert.Equal(t, domain.UploadErrTooLong, failed["a"])
|
|
||||||
} else {
|
|
||||||
assert.Empty(t, failed)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWorkerDuration_AnEstimateThatIsMerelyALittleOffIsNotTruncation(t *testing.T) {
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
claimed time.Duration
|
|
||||||
wav float64
|
|
||||||
}{
|
|
||||||
{"five percent short", 60 * time.Minute, 57 * 60},
|
|
||||||
{"under two minutes short of a long file", 60 * time.Minute, 59 * 60},
|
|
||||||
{"unknown claimed length", 0, 20 * 60},
|
|
||||||
}
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
r := newWRig(t, time.Minute)
|
|
||||||
w := durationWorker(t, r, tt.wav, 90*time.Minute)
|
|
||||||
|
|
||||||
w.Process(context.Background(), jobWith(r, t, tt.claimed))
|
|
||||||
|
|
||||||
completed, failed := r.st.snapshot()
|
|
||||||
assert.Empty(t, failed)
|
|
||||||
assert.Contains(t, completed, "a")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWorkerDuration_WithNoMaximumConfiguredNothingIsMeasured(t *testing.T) {
|
|
||||||
r := newWRig(t, time.Minute)
|
|
||||||
w := durationWorker(t, r, 100*60, 0)
|
|
||||||
|
|
||||||
w.Process(context.Background(), jobWith(r, t, 0))
|
|
||||||
|
|
||||||
completed, _ := r.st.snapshot()
|
|
||||||
assert.Contains(t, completed, "a")
|
|
||||||
require.NotNil(t, w)
|
|
||||||
}
|
|
||||||
@@ -146,7 +146,7 @@ func TestWorkerProgress_TheHeartbeatBeatsWhileAnEngineCallIsStuckAndStopsWhenThe
|
|||||||
assert.Equal(t, after, r.st.beatCount(), "no heartbeat may outlive its job")
|
assert.Equal(t, after, r.st.beatCount(), "no heartbeat may outlive its job")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestWorkerProgress_OnASuccessfulJobTheLogNamesEveryPhaseInOrderAndNeverTheTranscriptOrThePath(t *testing.T) {
|
func TestWorkerProgress_TheLogNamesEveryPhaseInOrderAndNeverTheTranscriptOrThePath(t *testing.T) {
|
||||||
r := newWRig(t, time.Minute)
|
r := newWRig(t, time.Minute)
|
||||||
r.tr.result = domain.Transcription{Text: "zebrakaka secret meeting words", Endpoint: "iguana/kb-whisper", AudioSeconds: 3}
|
r.tr.result = domain.Transcription{Text: "zebrakaka secret meeting words", Endpoint: "iguana/kb-whisper", AudioSeconds: 3}
|
||||||
buf := &syncLog{}
|
buf := &syncLog{}
|
||||||
@@ -162,98 +162,4 @@ func TestWorkerProgress_OnASuccessfulJobTheLogNamesEveryPhaseInOrderAndNeverTheT
|
|||||||
assert.Contains(t, out, "upload=a", "every line says which upload it is about")
|
assert.Contains(t, out, "upload=a", "every line says which upload it is about")
|
||||||
assert.NotContains(t, out, "zebrakaka", "no transcript text in the log")
|
assert.NotContains(t, out, "zebrakaka", "no transcript text in the log")
|
||||||
assert.NotContains(t, out, r.tmp, "no spool path, which carries the upload's identity, in the log")
|
assert.NotContains(t, out, r.tmp, "no spool path, which carries the upload's identity, in the log")
|
||||||
assert.Contains(t, out, "upload transcribed", "the job's own summary line")
|
|
||||||
assert.Contains(t, out, "endpoint=iguana/kb-whisper", "which endpoint served it is a quality fact")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWorkerProgress_AFailureLogsAKindNeverTheErrorTextOrThePath(t *testing.T) {
|
|
||||||
tests := []struct {
|
|
||||||
name string
|
|
||||||
setup func(*wrig)
|
|
||||||
line string
|
|
||||||
}{
|
|
||||||
{"conversion fails and names the spool file", func(r *wrig) {
|
|
||||||
r.conv.err = errors.New("Error opening input file " + r.tmp + "/upload-a: Invalid data zebrakaka")
|
|
||||||
}, "upload: convert"},
|
|
||||||
{"every engine fails and the gateway echoed a word", func(r *wrig) {
|
|
||||||
r.tr.err = errors.New("gateway echoed: zebrakaka secret words")
|
|
||||||
}, "upload: transcribe"},
|
|
||||||
}
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
r := newWRig(t, time.Minute)
|
|
||||||
tt.setup(r)
|
|
||||||
buf := &syncLog{}
|
|
||||||
w := newProgressWorker(t, r, r.tr, uploads.WithLogger(slog.New(slog.NewTextHandler(buf, nil))))
|
|
||||||
|
|
||||||
w.Process(context.Background(), r.spool(t, "a"))
|
|
||||||
|
|
||||||
out := buf.String()
|
|
||||||
assert.NotContains(t, out, "zebrakaka", "provider text or file contents in the log")
|
|
||||||
assert.NotContains(t, out, r.tmp, "the spool path in the log")
|
|
||||||
assert.Contains(t, out, tt.line, "the failure must be logged at all, not only scrubbed")
|
|
||||||
assert.Contains(t, out, "kind=error")
|
|
||||||
assert.Contains(t, out, "upload=a")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWorkerProgress_AStoreFailureLogsItsSQLStateNeverTheValueItQuotes(t *testing.T) {
|
|
||||||
// A Postgres error can quote the offending value; for a transcript row that is the
|
|
||||||
// transcript (fourth adversarial review). The log gets the SQLSTATE.
|
|
||||||
r := newWRig(t, time.Minute)
|
|
||||||
r.st.completeErr = dbErr{}
|
|
||||||
buf := &syncLog{}
|
|
||||||
w := newProgressWorker(t, r, r.tr, uploads.WithLogger(slog.New(slog.NewTextHandler(buf, nil))))
|
|
||||||
|
|
||||||
w.Process(context.Background(), r.spool(t, "a"))
|
|
||||||
|
|
||||||
out := buf.String()
|
|
||||||
assert.NotContains(t, out, "zebrakaka")
|
|
||||||
assert.Contains(t, out, "upload: store transcript")
|
|
||||||
assert.Contains(t, out, "kind=sql_22P02")
|
|
||||||
}
|
|
||||||
|
|
||||||
// langPlannedTr plans per language, like the Ensemble: an auto job has no base engine.
|
|
||||||
type langPlannedTr struct{ *fakeTranscriber }
|
|
||||||
|
|
||||||
func (langPlannedTr) Plan() []progress.Planned { return plannedTr{}.Plan() }
|
|
||||||
func (langPlannedTr) PlanFor(l domain.Language) []progress.Planned {
|
|
||||||
if l == domain.LangAuto {
|
|
||||||
return []progress.Planned{{Key: "engine_b", Label: "Transcribing with B"}}
|
|
||||||
}
|
|
||||||
return plannedTr{}.Plan()
|
|
||||||
}
|
|
||||||
|
|
||||||
func (p langPlannedTr) Transcribe(ctx context.Context, wav string) (domain.Transcription, error) {
|
|
||||||
r := progress.From(ctx)
|
|
||||||
if domain.LanguageFrom(ctx) != domain.LangAuto {
|
|
||||||
r.Begin("engine_a", 0)
|
|
||||||
r.Done("engine_a")
|
|
||||||
}
|
|
||||||
r.Begin("engine_b", 0)
|
|
||||||
r.Done("engine_b")
|
|
||||||
return p.fakeTranscriber.Transcribe(ctx, wav)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWorkerProgress_AJobGetsThePlanOfItsLanguage(t *testing.T) {
|
|
||||||
for name, tt := range map[string]struct {
|
|
||||||
lang domain.Language
|
|
||||||
want []string
|
|
||||||
}{
|
|
||||||
"auto": {domain.LangAuto, []string{"received:done", "audio:done", "engine_b:done", "save:done"}},
|
|
||||||
"swedish": {domain.LangSwedish, []string{"received:done", "audio:done", "engine_a:done", "engine_b:done", "save:done"}},
|
|
||||||
"no language (queued before the choice existed)": {"", []string{"received:done", "audio:done", "engine_a:done", "engine_b:done", "save:done"}},
|
|
||||||
} {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
r := newWRig(t, time.Minute)
|
|
||||||
w := newProgressWorker(t, r, langPlannedTr{r.tr})
|
|
||||||
job := r.spool(t, "a")
|
|
||||||
job.Language = tt.lang
|
|
||||||
|
|
||||||
w.Process(context.Background(), job)
|
|
||||||
|
|
||||||
assert.Equal(t, tt.want, keysAndStates(r.st.lastSteps()))
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,108 +0,0 @@
|
|||||||
package uploads_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"strconv"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/uploads"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The queue was append-only: 100,000 submits with the worker idle all succeeded, and each
|
|
||||||
// queued job holds up to the upload cap on disk until its turn (adversarial review F6).
|
|
||||||
|
|
||||||
func queueWorker(t *testing.T, total, perUser int) *uploads.Worker {
|
|
||||||
t.Helper()
|
|
||||||
r := newWRig(t, time.Minute)
|
|
||||||
return uploads.NewWorker(r.st, r.conv, r.tr, r.tmp, time.Minute, uploads.WithQueueLimits(total, perUser))
|
|
||||||
}
|
|
||||||
|
|
||||||
func submit(w *uploads.Worker, user string, n int) error {
|
|
||||||
return w.Submit(uploads.Job{UserID: user, UploadID: user + "-" + strconv.Itoa(n), Path: "/nonexistent"})
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWorkerSubmit_RefusesBeyondTheTotalQueueCap(t *testing.T) {
|
|
||||||
w := queueWorker(t, 3, 10)
|
|
||||||
|
|
||||||
for i := 0; i < 3; i++ {
|
|
||||||
require.NoError(t, submit(w, "u"+strconv.Itoa(i), i))
|
|
||||||
}
|
|
||||||
|
|
||||||
err := submit(w, "u9", 9)
|
|
||||||
assert.ErrorIs(t, err, uploads.ErrQueueFull)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWorkerSubmit_OneUserCannotFillTheQueueForEveryoneElse(t *testing.T) {
|
|
||||||
w := queueWorker(t, 100, 2)
|
|
||||||
|
|
||||||
require.NoError(t, submit(w, "alice", 1))
|
|
||||||
require.NoError(t, submit(w, "alice", 2))
|
|
||||||
errAlice := submit(w, "alice", 3)
|
|
||||||
errBob := submit(w, "bob", 1)
|
|
||||||
|
|
||||||
assert.True(t, errors.Is(errAlice, uploads.ErrQueueFull), "a third waiting upload from one user is refused")
|
|
||||||
assert.NoError(t, errBob, "another user is not affected by alice's queue")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWorkerSubmit_ThereIsADefaultCapSoANeglectedWorkerCannotBeFilledForever(t *testing.T) {
|
|
||||||
r := newWRig(t, time.Minute)
|
|
||||||
w := uploads.NewWorker(r.st, r.conv, r.tr, r.tmp, time.Minute)
|
|
||||||
|
|
||||||
refused := 0
|
|
||||||
for i := 0; i < 1000; i++ {
|
|
||||||
if err := submit(w, "u"+strconv.Itoa(i), i); errors.Is(err, uploads.ErrQueueFull) {
|
|
||||||
refused++
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
assert.Positive(t, refused, "with the defaults, 1,000 submits to an idle worker must not all be accepted")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWorkerSubmit_AJobThatHasLeftTheQueueFreesItsPlace(t *testing.T) {
|
|
||||||
r := newWRig(t, time.Minute)
|
|
||||||
w := uploads.NewWorker(r.st, r.conv, r.tr, r.tmp, time.Minute, uploads.WithQueueLimits(1, 1))
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
done := make(chan struct{})
|
|
||||||
go func() { defer close(done); w.Run(ctx) }()
|
|
||||||
defer func() { cancel(); <-done }()
|
|
||||||
|
|
||||||
job := r.spool(t, "first")
|
|
||||||
require.NoError(t, w.Submit(job))
|
|
||||||
require.Eventually(t, func() bool { c, _ := r.st.snapshot(); _, ok := c["first"]; return ok },
|
|
||||||
5*time.Second, 5*time.Millisecond, "the first job never completed")
|
|
||||||
|
|
||||||
assert.NoError(t, w.Submit(r.spool(t, "second")), "the cap counts jobs waiting, not jobs ever submitted")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWorkerHasRoom_SaysWhetherASubmitWouldBeAccepted(t *testing.T) {
|
|
||||||
w := queueWorker(t, 3, 2)
|
|
||||||
assert.True(t, w.HasRoom("alice"))
|
|
||||||
require.NoError(t, submit(w, "alice", 1))
|
|
||||||
require.NoError(t, submit(w, "alice", 2))
|
|
||||||
|
|
||||||
assert.False(t, w.HasRoom("alice"), "alice is at her share")
|
|
||||||
assert.True(t, w.HasRoom("bob"), "bob is not")
|
|
||||||
require.NoError(t, submit(w, "bob", 1))
|
|
||||||
assert.False(t, w.HasRoom("carol"), "the queue as a whole is at its cap")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestWorkerHasRoom_AgreesWithSubmitAtEveryStepAndIsFalseOnceStopped(t *testing.T) {
|
|
||||||
w := queueWorker(t, 2, 1)
|
|
||||||
for i, user := range []string{"a", "a", "b", "c"} {
|
|
||||||
room := w.HasRoom(user)
|
|
||||||
err := submit(w, user, i)
|
|
||||||
assert.Equal(t, room, err == nil, "step %d (%s): HasRoom=%v but Submit returned %v", i, user, room, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
r := newWRig(t, time.Minute)
|
|
||||||
stopped := uploads.NewWorker(r.st, r.conv, r.tr, r.tmp, time.Minute)
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
cancel()
|
|
||||||
stopped.Run(ctx)
|
|
||||||
assert.False(t, stopped.HasRoom("a"), "a worker that has stopped has no room")
|
|
||||||
}
|
|
||||||
@@ -23,8 +23,6 @@ type wStore struct {
|
|||||||
completed map[string]domain.Transcription
|
completed map[string]domain.Transcription
|
||||||
failed map[string]domain.UploadErrorKind
|
failed map[string]domain.UploadErrorKind
|
||||||
startErr error
|
startErr error
|
||||||
// completeErr makes CompleteUpload fail, as a database outage would.
|
|
||||||
completeErr error
|
|
||||||
saves [][]domain.ProgressStep
|
saves [][]domain.ProgressStep
|
||||||
beats int
|
beats int
|
||||||
// beatDelay makes each heartbeat write slow, so one is still in flight when a job
|
// beatDelay makes each heartbeat write slow, so one is still in flight when a job
|
||||||
@@ -46,17 +44,11 @@ func (s *wStore) StartTranscribing(_ context.Context, _, id string) error {
|
|||||||
func (s *wStore) CompleteUpload(_ context.Context, _, id string, t domain.Transcription) error {
|
func (s *wStore) CompleteUpload(_ context.Context, _, id string, t domain.Transcription) error {
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
defer s.mu.Unlock()
|
defer s.mu.Unlock()
|
||||||
if s.completeErr != nil {
|
|
||||||
return s.completeErr
|
|
||||||
}
|
|
||||||
s.completed[id] = t
|
s.completed[id] = t
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *wStore) FailUpload(ctx context.Context, _, id string, k domain.UploadErrorKind) error {
|
func (s *wStore) FailUpload(_ context.Context, _, id string, k domain.UploadErrorKind) error {
|
||||||
if err := ctx.Err(); err != nil { // like the real store: a cancelled context cannot write
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
defer s.mu.Unlock()
|
defer s.mu.Unlock()
|
||||||
s.failed[id] = k
|
s.failed[id] = k
|
||||||
@@ -70,7 +62,7 @@ func (s *wStore) SaveProgress(_ context.Context, _, _ string, steps []domain.Pro
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *wStore) Heartbeat(context.Context, string, string, time.Time) error {
|
func (s *wStore) Heartbeat(context.Context, string, string) error {
|
||||||
time.Sleep(s.beatDelay)
|
time.Sleep(s.beatDelay)
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
defer s.mu.Unlock()
|
defer s.mu.Unlock()
|
||||||
@@ -126,7 +118,6 @@ type fakeTranscriber struct {
|
|||||||
result domain.Transcription
|
result domain.Transcription
|
||||||
err error
|
err error
|
||||||
paths []string
|
paths []string
|
||||||
langs []domain.Language // what each call's context carried
|
|
||||||
// block, when non-nil, holds every call until it is closed or ctx ends.
|
// block, when non-nil, holds every call until it is closed or ctx ends.
|
||||||
block chan struct{}
|
block chan struct{}
|
||||||
running atomic.Int32
|
running atomic.Int32
|
||||||
@@ -135,7 +126,6 @@ type fakeTranscriber struct {
|
|||||||
|
|
||||||
func (f *fakeTranscriber) Transcribe(ctx context.Context, wav string) (domain.Transcription, error) {
|
func (f *fakeTranscriber) Transcribe(ctx context.Context, wav string) (domain.Transcription, error) {
|
||||||
f.paths = append(f.paths, wav)
|
f.paths = append(f.paths, wav)
|
||||||
f.langs = append(f.langs, domain.LanguageFrom(ctx))
|
|
||||||
n := f.running.Add(1)
|
n := f.running.Add(1)
|
||||||
defer f.running.Add(-1)
|
defer f.running.Add(-1)
|
||||||
for {
|
for {
|
||||||
@@ -218,10 +208,6 @@ func TestWorkerProcess_FailuresAreTypedAndStillDeleteTheAudio(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{"conversion fails", func(r *wrig) { r.conv.err = errors.New("moov atom not found") }},
|
{"conversion fails", func(r *wrig) { r.conv.err = errors.New("moov atom not found") }},
|
||||||
{"every transcription endpoint fails", func(r *wrig) { r.tr.err = errors.New("iguana down; berget down") }},
|
{"every transcription endpoint fails", func(r *wrig) { r.tr.err = errors.New("iguana down; berget down") }},
|
||||||
{"storing the finished transcript fails", func(r *wrig) { r.st.completeErr = errors.New("db down") }},
|
|
||||||
{"the temporary WAV cannot be created", func(r *wrig) {
|
|
||||||
r.w = uploads.NewWorker(r.st, r.conv, r.tr, filepath.Join(r.tmp, "missing"), time.Minute)
|
|
||||||
}},
|
|
||||||
}
|
}
|
||||||
for _, tt := range tests {
|
for _, tt := range tests {
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
@@ -247,9 +233,7 @@ func TestWorkerProcess_FailuresAreTypedAndStillDeleteTheAudio(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestWorkerProcess_ShutdownMidJobRecordsTheRowInterruptedAtOnce(t *testing.T) {
|
func TestWorkerProcess_ShutdownMidJobLeavesTheRowForTheStartupSweep(t *testing.T) {
|
||||||
// SIGTERM on a deploy used to leave the row "transcribing" until the NEXT pod's
|
|
||||||
// start-up sweep; the process may exit before any sweep runs (#50).
|
|
||||||
r := newWRig(t, time.Minute)
|
r := newWRig(t, time.Minute)
|
||||||
r.tr.block = make(chan struct{})
|
r.tr.block = make(chan struct{})
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
@@ -262,8 +246,8 @@ func TestWorkerProcess_ShutdownMidJobRecordsTheRowInterruptedAtOnce(t *testing.T
|
|||||||
<-done
|
<-done
|
||||||
|
|
||||||
_, f := r.st.snapshot()
|
_, f := r.st.snapshot()
|
||||||
assert.Equal(t, domain.UploadErrInterrupted, f["a"], "the row is recorded interrupted by the dying process, not left for a sweep")
|
assert.Empty(t, f, "a shutdown is not a transcription failure: the row stays for FailInterruptedUploads to mark interrupted")
|
||||||
assert.Empty(t, r.leftovers(t), "and the audio is still deleted")
|
assert.Empty(t, r.leftovers(t), "but the audio is still deleted")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestWorkerProcess_AJobThatOutlivesItsTimeoutIsFailedNotStuck(t *testing.T) {
|
func TestWorkerProcess_AJobThatOutlivesItsTimeoutIsFailedNotStuck(t *testing.T) {
|
||||||
@@ -331,32 +315,3 @@ func TestWorkerRun_StoppingDeletesQueuedAudioAndRefusesNewJobs(t *testing.T) {
|
|||||||
assert.Empty(t, r.leftovers(t), "audio of queued jobs is deleted on stop: the rows are swept to interrupted at next start")
|
assert.Empty(t, r.leftovers(t), "audio of queued jobs is deleted on stop: the rows are swept to interrupted at next start")
|
||||||
assert.ErrorIs(t, r.w.Submit(r.spool(t, "c")), uploads.ErrStopped)
|
assert.ErrorIs(t, r.w.Submit(r.spool(t, "c")), uploads.ErrStopped)
|
||||||
}
|
}
|
||||||
|
|
||||||
// dbErr is an error shaped like pgx's: a message that quotes a value, and an SQLSTATE.
|
|
||||||
type dbErr struct{}
|
|
||||||
|
|
||||||
func (dbErr) Error() string {
|
|
||||||
return `ERROR: invalid input syntax for type json: "zebrakaka secret words"`
|
|
||||||
}
|
|
||||||
func (dbErr) SQLState() string { return "22P02" }
|
|
||||||
|
|
||||||
func TestWorker_TheTranscriberSeesTheLanguageOfTheJobOnItsContext(t *testing.T) {
|
|
||||||
for name, tt := range map[string]struct {
|
|
||||||
job domain.Language
|
|
||||||
want domain.Language
|
|
||||||
}{
|
|
||||||
"auto": {domain.LangAuto, domain.LangAuto},
|
|
||||||
"swedish": {domain.LangSwedish, domain.LangSwedish},
|
|
||||||
"a job with no language": {"", domain.LangSwedish}, // queued before the field existed
|
|
||||||
} {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
r := newWRig(t, time.Minute)
|
|
||||||
job := r.spool(t, "a")
|
|
||||||
job.Language = tt.job
|
|
||||||
|
|
||||||
r.w.Process(context.Background(), job)
|
|
||||||
|
|
||||||
assert.Equal(t, []domain.Language{tt.want}, r.tr.langs)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,151 +0,0 @@
|
|||||||
package web_test
|
|
||||||
|
|
||||||
// Observes the cross-origin gate around Router (#48): SameSite=Lax does not stop a POST
|
|
||||||
// from a sibling *.d-ma.be origin, so the router itself must refuse a non-safe request
|
|
||||||
// whose Origin or Sec-Fetch-Site says it did not come from this site's own pages.
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"net/url"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
)
|
|
||||||
|
|
||||||
// foreign is a header set a browser sends for a form post made by another page.
|
|
||||||
var foreign = map[string]map[string]string{
|
|
||||||
"sibling subdomain, Sec-Fetch-Site": {"Origin": "https://evil.d-ma.be", "Sec-Fetch-Site": "same-site"},
|
|
||||||
"other site, Sec-Fetch-Site": {"Origin": "https://evil.example", "Sec-Fetch-Site": "cross-site"},
|
|
||||||
"sibling subdomain, Origin only": {"Origin": "https://evil.d-ma.be"},
|
|
||||||
"opaque origin": {"Origin": "null"},
|
|
||||||
}
|
|
||||||
|
|
||||||
// own is a header set for the app's own pages (or a non-browser client): must pass.
|
|
||||||
var own = map[string]map[string]string{
|
|
||||||
"same-origin Sec-Fetch-Site": {"Sec-Fetch-Site": "same-origin", "Origin": "http://example.com"},
|
|
||||||
"matching Origin only": {"Origin": "http://example.com"},
|
|
||||||
"no browser headers": {},
|
|
||||||
}
|
|
||||||
|
|
||||||
func withHeaders(req *http.Request, h map[string]string) *http.Request {
|
|
||||||
for k, v := range h {
|
|
||||||
req.Header.Set(k, v)
|
|
||||||
}
|
|
||||||
return req
|
|
||||||
}
|
|
||||||
|
|
||||||
func userCount(t *testing.T) int {
|
|
||||||
t.Helper()
|
|
||||||
var n int
|
|
||||||
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT count(*) FROM users WHERE id = $1`, userID).Scan(&n))
|
|
||||||
return n
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCrossOrigin_ARefusedDeleteAccountPostLeavesTheAccountInPlace(t *testing.T) {
|
|
||||||
for name, h := range foreign {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
app, _ := newAccountApp(t)
|
|
||||||
resetDB(t, rawPool(t))
|
|
||||||
seedConnection(t, app, "youtube", "ada@channel", "ref")
|
|
||||||
|
|
||||||
rec := do(t, app, withHeaders(httptest.NewRequest(http.MethodPost, "/account/delete", nil), h))
|
|
||||||
|
|
||||||
assert.Equal(t, http.StatusForbidden, rec.Code)
|
|
||||||
assert.Equal(t, 1, userCount(t), "the account must survive a refused post")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCrossOrigin_ASameOriginOrHeaderlessDeleteAccountPostIsServed(t *testing.T) {
|
|
||||||
for name, h := range own {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
app, _ := newAccountApp(t)
|
|
||||||
resetDB(t, rawPool(t))
|
|
||||||
seedConnection(t, app, "youtube", "ada@channel", "ref")
|
|
||||||
|
|
||||||
rec := do(t, app, withHeaders(httptest.NewRequest(http.MethodPost, "/account/delete", nil), h))
|
|
||||||
|
|
||||||
assert.Equal(t, http.StatusSeeOther, rec.Code)
|
|
||||||
assert.Equal(t, "/auth/logout", rec.Header().Get("Location"))
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCrossOrigin_ARefusedUploadPostSubmitsNoJob(t *testing.T) {
|
|
||||||
for name, h := range foreign {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
app, sub := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
|
|
||||||
rec := do(t, app, withHeaders(uploadReq(t, "m", "m.wav", []byte("RIFF....WAVE")), h))
|
|
||||||
|
|
||||||
assert.Equal(t, http.StatusForbidden, rec.Code)
|
|
||||||
assert.Empty(t, sub.jobs, "a refused post must not enqueue transcription")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCrossOrigin_ARefusedUploadChatPostCallsNoModel(t *testing.T) {
|
|
||||||
for name, h := range foreign {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
app, fc := newUploadChatApp(t)
|
|
||||||
id := seedUpload(t, userID, "Roadmap sync", "done", meetingText)
|
|
||||||
|
|
||||||
req := chatPost("/uploads/"+id+"/chat", url.Values{"message": {"hej"}})
|
|
||||||
rec := do(t, app, withHeaders(req, h))
|
|
||||||
|
|
||||||
assert.Equal(t, http.StatusForbidden, rec.Code)
|
|
||||||
assert.Zero(t, fc.callCount)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCrossOrigin_AForeignPostIsRefusedOnEachListedStateChangingPath(t *testing.T) {
|
|
||||||
routes := []string{
|
|
||||||
"/account/disconnect/youtube",
|
|
||||||
"/account/summarize-mode",
|
|
||||||
"/register",
|
|
||||||
"/paste",
|
|
||||||
"/v/abc/action",
|
|
||||||
"/v/abc/summarize",
|
|
||||||
"/v/abc/retry-now",
|
|
||||||
"/v/abc/chat",
|
|
||||||
}
|
|
||||||
for _, path := range routes {
|
|
||||||
t.Run(path, func(t *testing.T) {
|
|
||||||
app := newApp(t)
|
|
||||||
req := httptest.NewRequest(http.MethodPost, path, strings.NewReader(""))
|
|
||||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
|
||||||
rec := do(t, app, withHeaders(req, foreign["sibling subdomain, Sec-Fetch-Site"]))
|
|
||||||
assert.Equal(t, http.StatusForbidden, rec.Code)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCrossOrigin_ARefusedRecordingDeletePostLeavesTheRecordingInPlace(t *testing.T) {
|
|
||||||
for name, h := range foreign {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
|
||||||
|
|
||||||
rec := do(t, app, withHeaders(httptest.NewRequest(http.MethodPost, "/uploads/"+id+"/delete", nil), h))
|
|
||||||
|
|
||||||
assert.Equal(t, http.StatusForbidden, rec.Code)
|
|
||||||
assert.True(t, uploadExists(t, id), "a refused post must not delete the recording")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// A login redirect comes back to /auth/callback as a cross-site GET navigation, and
|
|
||||||
// health probes carry no headers: safe methods must never be refused on origin grounds.
|
|
||||||
func TestCrossOrigin_ACrossSiteGETIsNotRefused(t *testing.T) {
|
|
||||||
app := newApp(t)
|
|
||||||
for _, path := range []string{"/healthz", "/welcome"} {
|
|
||||||
req := httptest.NewRequest(http.MethodGet, path, nil)
|
|
||||||
req.Header.Set("Sec-Fetch-Site", "cross-site")
|
|
||||||
assert.Equal(t, http.StatusOK, do(t, app, req).Code, path)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -59,9 +59,6 @@ type Store interface {
|
|||||||
// non-owner's id is ErrUploadNotFound.
|
// non-owner's id is ErrUploadNotFound.
|
||||||
GetUpload(ctx context.Context, userID, id string) (*domain.UploadedTranscript, error)
|
GetUpload(ctx context.Context, userID, id string) (*domain.UploadedTranscript, error)
|
||||||
ListUploads(ctx context.Context, userID string) ([]domain.UploadedTranscript, error)
|
ListUploads(ctx context.Context, userID string) ([]domain.UploadedTranscript, error)
|
||||||
// DeleteUpload removes a finished upload; ErrUploadInProgress while it is still being
|
|
||||||
// processed, ErrUploadNotFound for a non-owner.
|
|
||||||
DeleteUpload(ctx context.Context, userID, id string) error
|
|
||||||
|
|
||||||
// SetTranscriptStatus clears or updates a video's transcript backoff state.
|
// SetTranscriptStatus clears or updates a video's transcript backoff state.
|
||||||
// Used by handleRetryNow to clear rate_limited_at before immediate processing.
|
// Used by handleRetryNow to clear rate_limited_at before immediate processing.
|
||||||
@@ -125,11 +122,6 @@ type App struct {
|
|||||||
// summarizer's (koala/phi4-mini, 8,192-token context) and cannot hold an hour of
|
// summarizer's (koala/phi4-mini, 8,192-token context) and cannot hold an hour of
|
||||||
// speech.
|
// speech.
|
||||||
UploadChatModel string
|
UploadChatModel string
|
||||||
// UploadChatModels is the set of models chat over an uploaded recording offers and
|
|
||||||
// accepts. The whole private transcript goes to the one chosen, so it must hold
|
|
||||||
// LOCAL models only, and the caller decides that. Nil or empty fails closed: no
|
|
||||||
// model is offered and none is called (ADR-035 review).
|
|
||||||
UploadChatModels []string
|
|
||||||
// Processing tracks in-flight immediate summarizations so the status endpoint
|
// Processing tracks in-flight immediate summarizations so the status endpoint
|
||||||
// shows the animation until the summary lands. The zero value is ready to use.
|
// shows the animation until the summary lands. The zero value is ready to use.
|
||||||
Processing ProcessingSet
|
Processing ProcessingSet
|
||||||
@@ -203,7 +195,6 @@ func (a *App) Router() http.Handler {
|
|||||||
app.HandleFunc("POST /uploads", a.handleUploadPost)
|
app.HandleFunc("POST /uploads", a.handleUploadPost)
|
||||||
app.HandleFunc("GET /uploads/{id}", a.handleUploadDetail)
|
app.HandleFunc("GET /uploads/{id}", a.handleUploadDetail)
|
||||||
app.HandleFunc("GET /uploads/{id}/status", a.handleUploadStatus)
|
app.HandleFunc("GET /uploads/{id}/status", a.handleUploadStatus)
|
||||||
app.HandleFunc("POST /uploads/{id}/delete", a.handleUploadDelete)
|
|
||||||
// Chat over the uploaded transcript reuses the ADR-027 Chatter; mounted only
|
// Chat over the uploaded transcript reuses the ADR-027 Chatter; mounted only
|
||||||
// when one is wired.
|
// when one is wired.
|
||||||
if a.Chat != nil {
|
if a.Chat != nil {
|
||||||
@@ -233,12 +224,7 @@ func (a *App) Router() http.Handler {
|
|||||||
// resolved user_id. /register lives inside the auth guard but is exempt from
|
// resolved user_id. /register lives inside the auth guard but is exempt from
|
||||||
// the registration gate (you must be able to reach it before you have a user).
|
// the registration gate (you must be able to reach it before you have a user).
|
||||||
root.Handle("/", a.Auth.Middleware(a.registrationGate(app)))
|
root.Handle("/", a.Auth.Middleware(a.registrationGate(app)))
|
||||||
|
return root
|
||||||
// The session cookie is SameSite=Lax, which still travels on a POST made from a
|
|
||||||
// sibling *.d-ma.be page (same site, different origin). Refuse any non-safe request
|
|
||||||
// whose Origin or Sec-Fetch-Site says it did not come from this origin (#48). It wraps
|
|
||||||
// everything, so a route added later is covered without remembering to ask.
|
|
||||||
return http.NewCrossOriginProtection().Handler(root)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// handleWelcome renders the public landing page (/welcome). It is mounted outside
|
// handleWelcome renders the public landing page (/welcome). It is mounted outside
|
||||||
@@ -313,7 +299,7 @@ func (a *App) handleList(w http.ResponseWriter, r *http.Request) {
|
|||||||
a.render(w, r, summaryList(buckets, hasConnected, autoSummarize))
|
a.render(w, r, summaryList(buckets, hasConnected, autoSummarize))
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
a.render(w, r, ListPage(buckets, f, stats, takeFlash(w, r), hasConnected, channels, autoSummarize, a.Uploads != nil))
|
a.render(w, r, ListPage(buckets, f, stats, takeFlash(w, r), hasConnected, channels, autoSummarize))
|
||||||
}
|
}
|
||||||
|
|
||||||
// handleDetail renders one summary in full (highlights, takeaways, action group).
|
// handleDetail renders one summary in full (highlights, takeaways, action group).
|
||||||
|
|||||||
@@ -1,61 +0,0 @@
|
|||||||
package oidc_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/web/oidc"
|
|
||||||
)
|
|
||||||
|
|
||||||
func testConfig(issuer string) oidc.Config {
|
|
||||||
return oidc.Config{
|
|
||||||
Issuer: issuer,
|
|
||||||
ClientID: testClientID,
|
|
||||||
ClientSecret: "test-client-secret",
|
|
||||||
RedirectURL: "http://tapir.test/auth/callback",
|
|
||||||
SessionSecret: "test-session-secret-please-change",
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// #59: a discovery 503 at start-up is retried instead of failing New.
|
|
||||||
func TestNewRetriesDiscoveryAfter503UntilIssuerAnswers(t *testing.T) {
|
|
||||||
f := newFakeIssuer(t)
|
|
||||||
f.unavailableFor = 2
|
|
||||||
|
|
||||||
auth, err := oidc.New(context.Background(), testConfig(f.server.URL),
|
|
||||||
oidc.WithDiscoveryRetry(5, time.Millisecond))
|
|
||||||
|
|
||||||
require.NoError(t, err)
|
|
||||||
require.NotNil(t, auth)
|
|
||||||
require.Equal(t, int32(3), f.discoveryHits.Load())
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestNewReturnsDiscoveryErrorAfterAttemptsExhausted(t *testing.T) {
|
|
||||||
f := newFakeIssuer(t)
|
|
||||||
f.unavailableFor = 1000
|
|
||||||
|
|
||||||
_, err := oidc.New(context.Background(), testConfig(f.server.URL),
|
|
||||||
oidc.WithDiscoveryRetry(3, time.Millisecond))
|
|
||||||
|
|
||||||
require.ErrorContains(t, err, "503")
|
|
||||||
require.Equal(t, int32(3), f.discoveryHits.Load())
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestNewStopsDiscoveryBackoffWhenContextCancelled(t *testing.T) {
|
|
||||||
f := newFakeIssuer(t)
|
|
||||||
f.unavailableFor = 1000
|
|
||||||
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
start := time.Now()
|
|
||||||
_, err := oidc.New(ctx, testConfig(f.server.URL),
|
|
||||||
oidc.WithDiscoveryRetry(5, time.Hour))
|
|
||||||
|
|
||||||
require.True(t, errors.Is(err, context.DeadlineExceeded), "err = %v", err)
|
|
||||||
require.Less(t, time.Since(start), 5*time.Second)
|
|
||||||
require.Equal(t, int32(1), f.discoveryHits.Load())
|
|
||||||
}
|
|
||||||
+16
-66
@@ -57,10 +57,6 @@ const (
|
|||||||
pendingTTL = 10 * time.Minute
|
pendingTTL = 10 * time.Minute
|
||||||
sessionCookie = "tapir_session"
|
sessionCookie = "tapir_session"
|
||||||
loginPath = "/auth/login"
|
loginPath = "/auth/login"
|
||||||
|
|
||||||
// Discovery retry (#59): 5 tries, 2+4+8+16 s back-off ≈ 30 s, then fail.
|
|
||||||
defaultDiscoveryAttempts = 5
|
|
||||||
defaultDiscoveryDelay = 2 * time.Second
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// DexAuth implements web.Auth against Dex.
|
// DexAuth implements web.Auth against Dex.
|
||||||
@@ -74,9 +70,6 @@ type DexAuth struct {
|
|||||||
sessionTTL time.Duration
|
sessionTTL time.Duration
|
||||||
now func() time.Time
|
now func() time.Time
|
||||||
insecure bool // tests only: drop the Secure attribute so cookies flow over http
|
insecure bool // tests only: drop the Secure attribute so cookies flow over http
|
||||||
|
|
||||||
discoveryAttempts int
|
|
||||||
discoveryDelay time.Duration
|
|
||||||
}
|
}
|
||||||
|
|
||||||
var _ web.Auth = (*DexAuth)(nil)
|
var _ web.Auth = (*DexAuth)(nil)
|
||||||
@@ -103,55 +96,14 @@ func WithSessionTTL(ttl time.Duration) Option {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// WithDiscoveryRetry overrides the discovery attempt count and the initial
|
|
||||||
// back-off delay (doubled after each failed attempt).
|
|
||||||
func WithDiscoveryRetry(attempts int, delay time.Duration) Option {
|
|
||||||
return func(d *DexAuth) {
|
|
||||||
if attempts > 0 {
|
|
||||||
d.discoveryAttempts = attempts
|
|
||||||
}
|
|
||||||
if delay > 0 {
|
|
||||||
d.discoveryDelay = delay
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// WithInsecureCookies drops the Secure cookie attribute. TEST ONLY — it lets
|
// WithInsecureCookies drops the Secure cookie attribute. TEST ONLY — it lets
|
||||||
// the session cookie travel over plain http; never use it in production.
|
// the session cookie travel over plain http; never use it in production.
|
||||||
func WithInsecureCookies() Option {
|
func WithInsecureCookies() Option {
|
||||||
return func(d *DexAuth) { d.insecure = true }
|
return func(d *DexAuth) { d.insecure = true }
|
||||||
}
|
}
|
||||||
|
|
||||||
// discover runs OIDC discovery up to attempts times, doubling delay between
|
// New discovers the issuer and returns a configured DexAuth. ctx bounds the
|
||||||
// tries (#59: an issuer briefly unavailable behind its proxy answered 503 and
|
// discovery request only.
|
||||||
// serve exited). A cancelled ctx ends the back-off immediately with ctx's error.
|
|
||||||
func discover(ctx context.Context, issuer string, attempts int, delay time.Duration) (*oidc.Provider, error) {
|
|
||||||
var lastErr error
|
|
||||||
for i := range attempts {
|
|
||||||
if i > 0 {
|
|
||||||
t := time.NewTimer(delay)
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
t.Stop()
|
|
||||||
return nil, fmt.Errorf("after %d attempts (last: %v): %w", i, lastErr, ctx.Err())
|
|
||||||
case <-t.C:
|
|
||||||
}
|
|
||||||
delay *= 2
|
|
||||||
}
|
|
||||||
p, err := oidc.NewProvider(ctx, issuer)
|
|
||||||
if err == nil {
|
|
||||||
return p, nil
|
|
||||||
}
|
|
||||||
lastErr = err
|
|
||||||
if ctx.Err() != nil {
|
|
||||||
return nil, fmt.Errorf("after %d attempts: %w", i+1, ctx.Err())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return nil, fmt.Errorf("after %d attempts: %w", attempts, lastErr)
|
|
||||||
}
|
|
||||||
|
|
||||||
// New discovers the issuer and returns a configured DexAuth. ctx bounds discovery,
|
|
||||||
// including its retry back-off.
|
|
||||||
func New(ctx context.Context, cfg Config, opts ...Option) (*DexAuth, error) {
|
func New(ctx context.Context, cfg Config, opts ...Option) (*DexAuth, error) {
|
||||||
for name, val := range map[string]string{
|
for name, val := range map[string]string{
|
||||||
"issuer": cfg.Issuer,
|
"issuer": cfg.Issuer,
|
||||||
@@ -165,30 +117,28 @@ func New(ctx context.Context, cfg Config, opts ...Option) (*DexAuth, error) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
d := &DexAuth{
|
provider, err := oidc.NewProvider(ctx, cfg.Issuer)
|
||||||
cfg: cfg,
|
|
||||||
pending: newPendingStore(),
|
|
||||||
secret: []byte(cfg.SessionSecret),
|
|
||||||
sessionTTL: defaultSessionTTL,
|
|
||||||
now: time.Now,
|
|
||||||
discoveryAttempts: defaultDiscoveryAttempts,
|
|
||||||
discoveryDelay: defaultDiscoveryDelay,
|
|
||||||
}
|
|
||||||
for _, o := range opts {
|
|
||||||
o(d)
|
|
||||||
}
|
|
||||||
|
|
||||||
provider, err := discover(ctx, cfg.Issuer, d.discoveryAttempts, d.discoveryDelay)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("oidc: discover issuer: %w", err)
|
return nil, fmt.Errorf("oidc: discover issuer: %w", err)
|
||||||
}
|
}
|
||||||
d.verifier = provider.Verifier(&oidc.Config{ClientID: cfg.ClientID})
|
|
||||||
d.oauth = &oauth2.Config{
|
d := &DexAuth{
|
||||||
|
cfg: cfg,
|
||||||
|
verifier: provider.Verifier(&oidc.Config{ClientID: cfg.ClientID}),
|
||||||
|
oauth: &oauth2.Config{
|
||||||
ClientID: cfg.ClientID,
|
ClientID: cfg.ClientID,
|
||||||
ClientSecret: cfg.ClientSecret,
|
ClientSecret: cfg.ClientSecret,
|
||||||
Endpoint: provider.Endpoint(),
|
Endpoint: provider.Endpoint(),
|
||||||
RedirectURL: cfg.RedirectURL,
|
RedirectURL: cfg.RedirectURL,
|
||||||
Scopes: []string{oidc.ScopeOpenID, "profile", "email"},
|
Scopes: []string{oidc.ScopeOpenID, "profile", "email"},
|
||||||
|
},
|
||||||
|
pending: newPendingStore(),
|
||||||
|
secret: []byte(cfg.SessionSecret),
|
||||||
|
sessionTTL: defaultSessionTTL,
|
||||||
|
now: time.Now,
|
||||||
|
}
|
||||||
|
for _, o := range opts {
|
||||||
|
o(d)
|
||||||
}
|
}
|
||||||
return d, nil
|
return d, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -8,7 +8,6 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"net/url"
|
"net/url"
|
||||||
"sync/atomic"
|
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -35,11 +34,6 @@ type fakeIssuer struct {
|
|||||||
sub string
|
sub string
|
||||||
email string
|
email string
|
||||||
nonce string
|
nonce string
|
||||||
|
|
||||||
// discoveryHits counts requests to the discovery document; the first
|
|
||||||
// unavailableFor of them answer 503 (issuer briefly down behind its proxy).
|
|
||||||
discoveryHits atomic.Int32
|
|
||||||
unavailableFor int32
|
|
||||||
}
|
}
|
||||||
|
|
||||||
const testKID = "test-key"
|
const testKID = "test-key"
|
||||||
@@ -53,10 +47,6 @@ func newFakeIssuer(t *testing.T) *fakeIssuer {
|
|||||||
|
|
||||||
mux := http.NewServeMux()
|
mux := http.NewServeMux()
|
||||||
mux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, _ *http.Request) {
|
mux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, _ *http.Request) {
|
||||||
if f.discoveryHits.Add(1) <= f.unavailableFor {
|
|
||||||
http.Error(w, "no available server", http.StatusServiceUnavailable)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
writeJSON(t, w, map[string]any{
|
writeJSON(t, w, map[string]any{
|
||||||
"issuer": f.server.URL,
|
"issuer": f.server.URL,
|
||||||
"authorization_endpoint": f.server.URL + "/authorize",
|
"authorization_endpoint": f.server.URL + "/authorize",
|
||||||
|
|||||||
@@ -66,7 +66,7 @@ func TestParseYouTubeVideoID(t *testing.T) {
|
|||||||
func TestListPageShowsPasteFormOnlyWhenConnected(t *testing.T) {
|
func TestListPageShowsPasteFormOnlyWhenConnected(t *testing.T) {
|
||||||
render := func(connected bool) string {
|
render := func(connected bool) string {
|
||||||
var buf bytes.Buffer
|
var buf bytes.Buffer
|
||||||
if err := ListPage(listBuckets{}, Filter{}, PipelineStats{}, "", connected, nil, true, false).Render(context.Background(), &buf); err != nil {
|
if err := ListPage(listBuckets{}, Filter{}, PipelineStats{}, "", connected, nil, true).Render(context.Background(), &buf); err != nil {
|
||||||
t.Fatalf("render: %v", err)
|
t.Fatalf("render: %v", err)
|
||||||
}
|
}
|
||||||
return buf.String()
|
return buf.String()
|
||||||
|
|||||||
+13
-228
@@ -8,19 +8,15 @@ import (
|
|||||||
"math"
|
"math"
|
||||||
"net/http"
|
"net/http"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"regexp"
|
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
"unicode"
|
|
||||||
"unicode/utf8"
|
"unicode/utf8"
|
||||||
|
|
||||||
"github.com/a-h/templ"
|
"github.com/a-h/templ"
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/adapters/chat"
|
"git.d-ma.be/mathias/tapir/internal/adapters/chat"
|
||||||
"git.d-ma.be/mathias/tapir/internal/adapters/store"
|
"git.d-ma.be/mathias/tapir/internal/adapters/store"
|
||||||
"git.d-ma.be/mathias/tapir/internal/config"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
"git.d-ma.be/mathias/tapir/internal/errkind"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/uploads"
|
"git.d-ma.be/mathias/tapir/internal/uploads"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -28,7 +24,7 @@ import (
|
|||||||
// *uploads.Intake satisfies it. It is the whole of what the handlers know about
|
// *uploads.Intake satisfies it. It is the whole of what the handlers know about
|
||||||
// validation, spooling and queueing.
|
// validation, spooling and queueing.
|
||||||
type Uploader interface {
|
type Uploader interface {
|
||||||
Accept(ctx context.Context, userID, title string, lang domain.Language, src io.Reader) (id string, err error)
|
Accept(ctx context.Context, userID, title string, src io.Reader) (id string, err error)
|
||||||
}
|
}
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -36,8 +32,6 @@ const (
|
|||||||
// and the title field; the cap itself is enforced on the file part by Intake.
|
// and the title field; the cap itself is enforced on the file part by Intake.
|
||||||
uploadOverhead = 1 << 20
|
uploadOverhead = 1 << 20
|
||||||
maxTitleRunes = 200
|
maxTitleRunes = 200
|
||||||
// maxLanguageBytes bounds what is read of the language field; a valid value is at most four bytes, so a longer one fails the parse.
|
|
||||||
maxLanguageBytes = 16
|
|
||||||
// uploadPollEvery is how often an unfinished upload re-asks for its status.
|
// uploadPollEvery is how often an unfinished upload re-asks for its status.
|
||||||
uploadPollEvery = "2s"
|
uploadPollEvery = "2s"
|
||||||
// serviceSilentAfter is how long a job may go without any signal (heartbeat or
|
// serviceSilentAfter is how long a job may go without any signal (heartbeat or
|
||||||
@@ -91,62 +85,24 @@ func (a *App) handleUploadPost(w http.ResponseWriter, r *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
title := ""
|
title := ""
|
||||||
lang := domain.LangSwedish
|
|
||||||
for {
|
for {
|
||||||
part, err := mr.NextPart()
|
part, err := mr.NextPart()
|
||||||
if errors.Is(err, io.EOF) {
|
if errors.Is(err, io.EOF) {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
if err != nil {
|
if err != nil {
|
||||||
var tooBig *http.MaxBytesError
|
|
||||||
if a.abandoned(w, r) {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if !errors.As(err, &tooBig) {
|
|
||||||
// A malformed multipart body (a header with a control byte, a truncated
|
|
||||||
// part) is the sender's to fix: a 400, not a server error, and the error
|
|
||||||
// text is never logged: it quotes the headers, which carry the file name.
|
|
||||||
a.logger().Info("upload rejected", "reason", "malformed")
|
|
||||||
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "That upload could not be read. Please try again.")
|
|
||||||
return
|
|
||||||
}
|
|
||||||
a.uploadFailure(w, r, userID, err)
|
a.uploadFailure(w, r, userID, err)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
switch part.FormName() {
|
switch part.FormName() {
|
||||||
case "title":
|
case "title":
|
||||||
b, err := io.ReadAll(io.LimitReader(part, maxTitleRunes*utf8.UTFMax))
|
b, _ := io.ReadAll(io.LimitReader(part, maxTitleRunes*utf8.UTFMax))
|
||||||
if err != nil {
|
title = clipRunes(strings.TrimSpace(string(b)), maxTitleRunes)
|
||||||
if a.abandoned(w, r) {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "That upload could not be read. Please try again.")
|
|
||||||
return
|
|
||||||
}
|
|
||||||
title = cleanTitle(string(b))
|
|
||||||
case "language":
|
|
||||||
// A closed set read before the file: anything else is refused before the file
|
|
||||||
// is spooled, so a crafted value never reaches an engine (tapir#61).
|
|
||||||
b, err := io.ReadAll(io.LimitReader(part, maxLanguageBytes+1))
|
|
||||||
if err != nil {
|
|
||||||
if a.abandoned(w, r) {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "That upload could not be read. Please try again.")
|
|
||||||
return
|
|
||||||
}
|
|
||||||
l, ok := domain.ParseLanguage(string(b))
|
|
||||||
if !ok {
|
|
||||||
a.logger().Info("upload rejected", "reason", "language")
|
|
||||||
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "Choose a language from the list.")
|
|
||||||
return
|
|
||||||
}
|
|
||||||
lang = l
|
|
||||||
case "file":
|
case "file":
|
||||||
if title == "" {
|
if title == "" {
|
||||||
title = titleFromFilename(part.FileName())
|
title = titleFromFilename(part.FileName())
|
||||||
}
|
}
|
||||||
id, err := a.Uploads.Accept(r.Context(), userID, title, lang, part)
|
id, err := a.Uploads.Accept(r.Context(), userID, title, part)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
a.uploadFailure(w, r, userID, err)
|
a.uploadFailure(w, r, userID, err)
|
||||||
return
|
return
|
||||||
@@ -158,27 +114,10 @@ func (a *App) handleUploadPost(w http.ResponseWriter, r *http.Request) {
|
|||||||
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "Choose an audio file to upload.")
|
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "Choose an audio file to upload.")
|
||||||
}
|
}
|
||||||
|
|
||||||
// abandoned reports, logs at INFO and answers 499, because the client has already gone: its request
|
|
||||||
// context is cancelled, so there is nobody to answer and rendering a page would run its
|
|
||||||
// database read on a dead context and log a false ERROR. A hang-up mid-upload is routine.
|
|
||||||
func (a *App) abandoned(w http.ResponseWriter, r *http.Request) bool {
|
|
||||||
if r.Context().Err() == nil {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
a.logger().Info("upload abandoned", "reason", "client_gone")
|
|
||||||
// 499 (the "client closed request" convention): a client that half-closed after sending
|
|
||||||
// can still read, and an unwritten status would reach it as a false, empty 200.
|
|
||||||
w.WriteHeader(499)
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|
||||||
// uploadFailure maps an Accept (or multipart) error to a response: a validation
|
// uploadFailure maps an Accept (or multipart) error to a response: a validation
|
||||||
// rejection is the user's to fix (422, or 413 for size) and carries its own
|
// rejection is the user's to fix (422, or 413 for size) and carries its own
|
||||||
// message; anything else is ours.
|
// message; anything else is ours.
|
||||||
func (a *App) uploadFailure(w http.ResponseWriter, r *http.Request, userID string, err error) {
|
func (a *App) uploadFailure(w http.ResponseWriter, r *http.Request, userID string, err error) {
|
||||||
if a.abandoned(w, r) {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
var rej *uploads.RejectError
|
var rej *uploads.RejectError
|
||||||
var tooBig *http.MaxBytesError
|
var tooBig *http.MaxBytesError
|
||||||
switch {
|
switch {
|
||||||
@@ -186,26 +125,16 @@ func (a *App) uploadFailure(w http.ResponseWriter, r *http.Request, userID strin
|
|||||||
// Logged by reason only: the filename and title are private content.
|
// Logged by reason only: the filename and title are private content.
|
||||||
a.logger().Info("upload rejected", "reason", string(rej.Reason))
|
a.logger().Info("upload rejected", "reason", string(rej.Reason))
|
||||||
status := http.StatusUnprocessableEntity
|
status := http.StatusUnprocessableEntity
|
||||||
switch rej.Reason {
|
if rej.Reason == uploads.ReasonTooLarge {
|
||||||
case uploads.ReasonTooLarge:
|
|
||||||
status = http.StatusRequestEntityTooLarge
|
status = http.StatusRequestEntityTooLarge
|
||||||
case uploads.ReasonBusy:
|
|
||||||
status = http.StatusTooManyRequests
|
|
||||||
}
|
}
|
||||||
a.renderUploadsPage(w, r, userID, status, rej.Message)
|
a.renderUploadsPage(w, r, userID, status, rej.Message)
|
||||||
case errors.As(err, &tooBig):
|
case errors.As(err, &tooBig):
|
||||||
a.logger().Info("upload rejected", "reason", string(uploads.ReasonTooLarge))
|
a.logger().Info("upload rejected", "reason", string(uploads.ReasonTooLarge))
|
||||||
a.renderUploadsPage(w, r, userID, http.StatusRequestEntityTooLarge,
|
a.renderUploadsPage(w, r, userID, http.StatusRequestEntityTooLarge,
|
||||||
"This file is larger than the upload limit.")
|
"This file is larger than the upload limit.")
|
||||||
case errors.Is(err, uploads.ErrClientRead):
|
|
||||||
// An error reading the CLIENT's body: theirs to retry, not a server fault. Only that
|
|
||||||
// marker counts: io.ErrUnexpectedEOF alone is also what a database drop looks like.
|
|
||||||
a.logger().Info("upload rejected", "reason", "truncated")
|
|
||||||
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "The upload was cut off before it finished. Please try again.")
|
|
||||||
default:
|
default:
|
||||||
// Not err: it can carry the file's name, a spool path or a database value.
|
a.serverError(w, r, "upload", err)
|
||||||
a.logger().Error("upload failed", "kind", errkind.Of(err))
|
|
||||||
http.Error(w, "internal error", http.StatusInternalServerError)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -231,45 +160,12 @@ func (a *App) handleUploadStatus(w http.ResponseWriter, r *http.Request) {
|
|||||||
a.render(w, r, uploadStatus(*u, a.Chat != nil, a.now()))
|
a.render(w, r, uploadStatus(*u, a.Chat != nil, a.now()))
|
||||||
}
|
}
|
||||||
|
|
||||||
// handleUploadDelete removes one of the caller's finished recordings and returns to the
|
|
||||||
// list. Another user's id is a 404 like every other upload route; a recording still being
|
|
||||||
// transcribed is refused (409) rather than deleted under the worker (#37).
|
|
||||||
func (a *App) handleUploadDelete(w http.ResponseWriter, r *http.Request) {
|
|
||||||
userID, ok := a.currentUserID(w, r)
|
|
||||||
if !ok {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
id := r.PathValue("id")
|
|
||||||
if !uuidShape.MatchString(id) {
|
|
||||||
http.NotFound(w, r)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
switch err := a.Store.DeleteUpload(r.Context(), userID, id); {
|
|
||||||
case err == nil:
|
|
||||||
http.Redirect(w, r, "/uploads", http.StatusSeeOther)
|
|
||||||
case errors.Is(err, store.ErrUploadNotFound):
|
|
||||||
http.NotFound(w, r)
|
|
||||||
case errors.Is(err, store.ErrUploadInProgress):
|
|
||||||
a.renderUploadsPage(w, r, userID, http.StatusConflict,
|
|
||||||
"This recording is still being transcribed. You can delete it once it has finished.")
|
|
||||||
default:
|
|
||||||
a.serverError(w, r, "delete upload", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (a *App) loadOwnedUpload(w http.ResponseWriter, r *http.Request) (*domain.UploadedTranscript, bool) {
|
func (a *App) loadOwnedUpload(w http.ResponseWriter, r *http.Request) (*domain.UploadedTranscript, bool) {
|
||||||
userID, ok := a.currentUserID(w, r)
|
userID, ok := a.currentUserID(w, r)
|
||||||
if !ok {
|
if !ok {
|
||||||
return nil, false
|
return nil, false
|
||||||
}
|
}
|
||||||
id := r.PathValue("id")
|
u, err := a.Store.GetUpload(r.Context(), userID, r.PathValue("id"))
|
||||||
if !uuidShape.MatchString(id) {
|
|
||||||
// Not an id at all: the same 404 as an unknown or foreign one, never a database
|
|
||||||
// error that surfaces as a 500 (and no oracle about which ids exist).
|
|
||||||
http.NotFound(w, r)
|
|
||||||
return nil, false
|
|
||||||
}
|
|
||||||
u, err := a.Store.GetUpload(r.Context(), userID, id)
|
|
||||||
if errors.Is(err, store.ErrUploadNotFound) {
|
if errors.Is(err, store.ErrUploadNotFound) {
|
||||||
http.NotFound(w, r)
|
http.NotFound(w, r)
|
||||||
return nil, false
|
return nil, false
|
||||||
@@ -284,87 +180,11 @@ func (a *App) loadOwnedUpload(w http.ResponseWriter, r *http.Request) (*domain.U
|
|||||||
// titleFromFilename is the default title: the file's name without its extension.
|
// titleFromFilename is the default title: the file's name without its extension.
|
||||||
func titleFromFilename(name string) string {
|
func titleFromFilename(name string) string {
|
||||||
base := filepath.Base(name)
|
base := filepath.Base(name)
|
||||||
t := cleanTitle(strings.TrimSuffix(base, filepath.Ext(base)))
|
t := strings.TrimSuffix(base, filepath.Ext(base))
|
||||||
if t == "" || t == "." {
|
if t == "" || t == "." {
|
||||||
return "Untitled recording"
|
return "Untitled recording"
|
||||||
}
|
}
|
||||||
return t
|
return clipRunes(t, maxTitleRunes)
|
||||||
}
|
|
||||||
|
|
||||||
// cloudProvider names the cloud service that produced a transcript, or false when it was
|
|
||||||
// served locally (or by an unknown endpoint). "Local" is config.IsLocalModel: the same
|
|
||||||
// definition the start-up validation uses, not a second string match here (#52).
|
|
||||||
//
|
|
||||||
// The endpoint is one alias, except for the two-engine path (ADR-034), which writes
|
|
||||||
// "<base> + <alt>, reconciled by <model>" (ensemble.go). That shape is taken apart exactly
|
|
||||||
// and each alias is judged whole, so a composite of local models is local, and anything
|
|
||||||
// else (a cloud alias, one with no provider prefix, one with a space or a connector word in
|
|
||||||
// it) is not. A string that is not that shape is one alias.
|
|
||||||
func cloudProvider(endpoint string) (string, bool) {
|
|
||||||
endpoint = strings.TrimSpace(endpoint)
|
|
||||||
if endpoint == "" {
|
|
||||||
return "", false
|
|
||||||
}
|
|
||||||
aliases := []string{endpoint}
|
|
||||||
if engines, reconciler, ok := strings.Cut(endpoint, ", reconciled by "); ok {
|
|
||||||
aliases = append(strings.Split(engines, " + "), reconciler)
|
|
||||||
}
|
|
||||||
for _, a := range aliases {
|
|
||||||
if a = strings.TrimSpace(a); !config.IsLocalModel(a) {
|
|
||||||
provider, _, _ := strings.Cut(a, "/")
|
|
||||||
if provider = strings.TrimSpace(provider); provider == "" {
|
|
||||||
provider = a
|
|
||||||
}
|
|
||||||
return provider, true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return "", false
|
|
||||||
}
|
|
||||||
|
|
||||||
// languageLabel is how the page names the language a recording was transcribed as.
|
|
||||||
func languageLabel(l domain.Language) string {
|
|
||||||
if l == domain.LangAuto {
|
|
||||||
return "Other (detected by the engine)"
|
|
||||||
}
|
|
||||||
return "Swedish"
|
|
||||||
}
|
|
||||||
|
|
||||||
// hasNote reports whether any phase carries a note. A note is the only warning for a
|
|
||||||
// recording that is shorter than its file claimed, so a record that has one is shown open.
|
|
||||||
func hasNote(steps []domain.ProgressStep) bool {
|
|
||||||
for _, s := range steps {
|
|
||||||
if s.Note != "" {
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
// uuidShape is the form of an upload id. Anything else cannot be one, so it is a 404.
|
|
||||||
var uuidShape = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
|
|
||||||
|
|
||||||
// isBidiControl reports the invisible characters that change text direction. Emoji joiners
|
|
||||||
// (ZWJ, ZWNJ) and variation selectors are not among them and are kept.
|
|
||||||
func isBidiControl(r rune) bool {
|
|
||||||
return r >= 0x202A && r <= 0x202E || r >= 0x2066 && r <= 0x2069 || r == 0x200E || r == 0x200F || r == 0x061C
|
|
||||||
}
|
|
||||||
|
|
||||||
// cleanTitle makes a title safe to store and show: invalid UTF-8 dropped, control
|
|
||||||
// characters (NUL, newlines, escapes) turned into spaces, runs of space collapsed, trimmed
|
|
||||||
// and clipped. A title is cosmetic; bad bytes in it must never fail an upload, and Postgres
|
|
||||||
// refuses NUL in text.
|
|
||||||
func cleanTitle(s string) string {
|
|
||||||
s = strings.ToValidUTF8(s, "")
|
|
||||||
s = strings.Map(func(r rune) rune {
|
|
||||||
switch {
|
|
||||||
case isBidiControl(r):
|
|
||||||
return -1 // dropped: U+202E would reverse the rest of the title on its owner's page
|
|
||||||
case unicode.IsControl(r):
|
|
||||||
return ' '
|
|
||||||
}
|
|
||||||
return r
|
|
||||||
}, s)
|
|
||||||
return clipRunes(strings.Join(strings.Fields(s), " "), maxTitleRunes)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func clipRunes(s string, n int) string {
|
func clipRunes(s string, n int) string {
|
||||||
@@ -380,10 +200,6 @@ func uploadFailureMessage(k domain.UploadErrorKind) string {
|
|||||||
switch k {
|
switch k {
|
||||||
case domain.UploadErrInterrupted:
|
case domain.UploadErrInterrupted:
|
||||||
return "The server restarted before this recording was transcribed. Please upload it again."
|
return "The server restarted before this recording was transcribed. Please upload it again."
|
||||||
case domain.UploadErrBusy:
|
|
||||||
return "Too many recordings were waiting to be transcribed when this one was uploaded. Please upload it again in a few minutes."
|
|
||||||
case domain.UploadErrTooLong:
|
|
||||||
return "This recording is longer than the limit once its audio is read (the file's header said otherwise). Please upload a shorter recording."
|
|
||||||
case domain.UploadErrTranscriptionFailed:
|
case domain.UploadErrTranscriptionFailed:
|
||||||
return "Transcription failed on every available service. Please upload it again later."
|
return "Transcription failed on every available service. Please upload it again later."
|
||||||
default:
|
default:
|
||||||
@@ -404,8 +220,7 @@ func (a *App) handleUploadChat(w http.ResponseWriter, r *http.Request) {
|
|||||||
http.Redirect(w, r, "/uploads/"+u.ID, http.StatusSeeOther)
|
http.Redirect(w, r, "/uploads/"+u.ID, http.StatusSeeOther)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
model, _ := a.resolveUploadModel("")
|
a.render(w, r, UploadChatPage(*u, a.uploadChatView(*u, a.Chat.DefaultModel(a.UploadChatModel), nil, false, "")))
|
||||||
a.render(w, r, UploadChatPage(*u, a.uploadChatView(*u, model, nil, false, "")))
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// handleUploadChatMessage answers one question against the owner's uploaded
|
// handleUploadChatMessage answers one question against the owner's uploaded
|
||||||
@@ -425,13 +240,7 @@ func (a *App) handleUploadChatMessage(w http.ResponseWriter, r *http.Request) {
|
|||||||
http.Error(w, "bad form", http.StatusBadRequest)
|
http.Error(w, "bad form", http.StatusBadRequest)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
model, ok := a.resolveUploadModel(r.FormValue("model"))
|
model := a.resolveModel(r.FormValue("model"), a.UploadChatModel)
|
||||||
if !ok {
|
|
||||||
a.logger().Error("upload chat: no local model is configured; nothing was sent", "upload", u.ID)
|
|
||||||
view := a.uploadChatView(*u, "", nil, false, "No local model is available for chat about a recording.")
|
|
||||||
a.render(w, r, UploadChatPage(*u, view))
|
|
||||||
return
|
|
||||||
}
|
|
||||||
history := parseHistory(r.Form["hq"], r.Form["ha"])
|
history := parseHistory(r.Form["hq"], r.Form["ha"])
|
||||||
question := strings.TrimSpace(r.FormValue("question"))
|
question := strings.TrimSpace(r.FormValue("question"))
|
||||||
|
|
||||||
@@ -445,7 +254,7 @@ func (a *App) handleUploadChatMessage(w http.ResponseWriter, r *http.Request) {
|
|||||||
MaxChars: a.UploadChatMaxChars,
|
MaxChars: a.UploadChatMaxChars,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
a.logger().Error("upload chat answer", "upload", u.ID, "model", model, "kind", errkind.Of(err))
|
a.logger().Error("upload chat answer", "upload", u.ID, "model", model, "err", err)
|
||||||
view.Error = "That model couldn't answer just now. Try again, or switch models."
|
view.Error = "That model couldn't answer just now. Try again, or switch models."
|
||||||
} else {
|
} else {
|
||||||
view.History = appendTurn(history, chat.Turn{Question: question, Answer: reply.Answer})
|
view.History = appendTurn(history, chat.Turn{Question: question, Answer: reply.Answer})
|
||||||
@@ -464,7 +273,7 @@ func (a *App) uploadChatView(u domain.UploadedTranscript, model string, history
|
|||||||
VideoID: u.ID,
|
VideoID: u.ID,
|
||||||
Title: u.Title,
|
Title: u.Title,
|
||||||
Available: true,
|
Available: true,
|
||||||
Models: a.UploadChatModels,
|
Models: a.Chat.Models(),
|
||||||
Selected: model,
|
Selected: model,
|
||||||
History: history,
|
History: history,
|
||||||
Truncated: truncated,
|
Truncated: truncated,
|
||||||
@@ -592,27 +401,3 @@ func minutesAgo(d time.Duration) string {
|
|||||||
}
|
}
|
||||||
return "under a minute"
|
return "under a minute"
|
||||||
}
|
}
|
||||||
|
|
||||||
// resolveUploadModel picks the model for chat over an uploaded recording from the
|
|
||||||
// offered LOCAL set only: the posted one when it is in the set, else the configured
|
|
||||||
// default when it is, else the first. ok is false when no model is offered, in which
|
|
||||||
// case nothing may be sent anywhere (the whole private transcript is the payload).
|
|
||||||
func (a *App) resolveUploadModel(posted string) (model string, ok bool) {
|
|
||||||
offered := func(m string) bool {
|
|
||||||
for _, o := range a.UploadChatModels {
|
|
||||||
if o == m {
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
switch {
|
|
||||||
case posted != "" && offered(posted):
|
|
||||||
return posted, true
|
|
||||||
case a.UploadChatModel != "" && offered(a.UploadChatModel):
|
|
||||||
return a.UploadChatModel, true
|
|
||||||
case len(a.UploadChatModels) > 0:
|
|
||||||
return a.UploadChatModels[0], true
|
|
||||||
}
|
|
||||||
return "", false
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -24,14 +24,6 @@ templ UploadsPage(v uploadsView) {
|
|||||||
}
|
}
|
||||||
<form method="post" action="/uploads" enctype="multipart/form-data">
|
<form method="post" action="/uploads" enctype="multipart/form-data">
|
||||||
<p><label>Title (optional) <input type="text" name="title" maxlength="200"/></label></p>
|
<p><label>Title (optional) <input type="text" name="title" maxlength="200"/></label></p>
|
||||||
<p>
|
|
||||||
<label>Language
|
|
||||||
<select name="language">
|
|
||||||
<option value="sv" selected>Swedish</option>
|
|
||||||
<option value="auto">Other (detected by the engine)</option>
|
|
||||||
</select>
|
|
||||||
</label>
|
|
||||||
</p>
|
|
||||||
<p><input type="file" name="file" accept="audio/*,.m4a,.mp3,.wav,.opus,.ogg,.mp4" required/></p>
|
<p><input type="file" name="file" accept="audio/*,.m4a,.mp3,.wav,.opus,.ogg,.mp4" required/></p>
|
||||||
<p><button type="submit">Upload and transcribe</button></p>
|
<p><button type="submit">Upload and transcribe</button></p>
|
||||||
</form>
|
</form>
|
||||||
@@ -57,7 +49,6 @@ templ UploadPage(u domain.UploadedTranscript, chatEnabled bool, now time.Time) {
|
|||||||
@Layout("Tapir — " + u.Title) {
|
@Layout("Tapir — " + u.Title) {
|
||||||
<article class="account">
|
<article class="account">
|
||||||
<h1>{ u.Title }</h1>
|
<h1>{ u.Title }</h1>
|
||||||
<p class="muted">Language: { languageLabel(u.Language) }</p>
|
|
||||||
@uploadStatus(u, chatEnabled, now)
|
@uploadStatus(u, chatEnabled, now)
|
||||||
<p><a href="/uploads">All recordings</a></p>
|
<p><a href="/uploads">All recordings</a></p>
|
||||||
</article>
|
</article>
|
||||||
@@ -75,15 +66,9 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
|
|||||||
Transcribed { uploads.FormatDuration(time.Duration(u.AudioSeconds * float64(time.Second))) }
|
Transcribed { uploads.FormatDuration(time.Duration(u.AudioSeconds * float64(time.Second))) }
|
||||||
of audio by <span class="chip">{ u.Endpoint }</span>.
|
of audio by <span class="chip">{ u.Endpoint }</span>.
|
||||||
</p>
|
</p>
|
||||||
if provider, cloud := cloudProvider(u.Endpoint); cloud {
|
|
||||||
<p class="cloud-note">
|
|
||||||
This recording's audio left the homelab: it was transcribed by a cloud service ({ provider })
|
|
||||||
because the local engine could not.
|
|
||||||
</p>
|
|
||||||
}
|
|
||||||
@reconciliationNote(u.Reconciliation)
|
@reconciliationNote(u.Reconciliation)
|
||||||
if len(u.Progress) > 0 {
|
if len(u.Progress) > 0 {
|
||||||
<details class="what-was-done" if hasNote(u.Progress) { open }>
|
<details class="what-was-done">
|
||||||
<summary>What was done</summary>
|
<summary>What was done</summary>
|
||||||
@progressSteps(u.Progress, now, false)
|
@progressSteps(u.Progress, now, false)
|
||||||
</details>
|
</details>
|
||||||
@@ -95,7 +80,6 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
|
|||||||
<summary>Transcript</summary>
|
<summary>Transcript</summary>
|
||||||
<pre class="transcript">{ u.Content }</pre>
|
<pre class="transcript">{ u.Content }</pre>
|
||||||
</details>
|
</details>
|
||||||
@deleteRecording(u.ID)
|
|
||||||
</div>
|
</div>
|
||||||
case domain.UploadFailed:
|
case domain.UploadFailed:
|
||||||
<div class="upload-status upload-failed flash flash-error" role="alert">
|
<div class="upload-status upload-failed flash flash-error" role="alert">
|
||||||
@@ -107,7 +91,6 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
|
|||||||
@progressSteps(u.Progress, now, false)
|
@progressSteps(u.Progress, now, false)
|
||||||
</details>
|
</details>
|
||||||
}
|
}
|
||||||
@deleteRecording(u.ID)
|
|
||||||
</div>
|
</div>
|
||||||
default:
|
default:
|
||||||
<div
|
<div
|
||||||
@@ -133,20 +116,6 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// deleteRecording is the confirm-then-delete control, shown only once a recording has
|
|
||||||
// finished (done or failed): deleting one that is still being processed is refused.
|
|
||||||
templ deleteRecording(id string) {
|
|
||||||
<details class="confirm-delete">
|
|
||||||
<summary class="btn-danger">Delete this recording…</summary>
|
|
||||||
<div class="confirm-body">
|
|
||||||
<p>This permanently deletes the recording and its transcript. Are you sure?</p>
|
|
||||||
<form method="post" action={ templ.SafeURL("/uploads/" + id + "/delete") }>
|
|
||||||
<button type="submit" class="btn-danger">Yes, permanently delete it</button>
|
|
||||||
</form>
|
|
||||||
</div>
|
|
||||||
</details>
|
|
||||||
}
|
|
||||||
|
|
||||||
// UploadChatPage is the no-JS full page for chatting with an uploaded transcript:
|
// UploadChatPage is the no-JS full page for chatting with an uploaded transcript:
|
||||||
// the same chat panel as a video's (ADR-027), scoped to this recording.
|
// the same chat panel as a video's (ADR-027), scoped to this recording.
|
||||||
templ UploadChatPage(u domain.UploadedTranscript, v chatView) {
|
templ UploadChatPage(u domain.UploadedTranscript, v chatView) {
|
||||||
@@ -249,9 +218,6 @@ templ progressSteps(steps []domain.ProgressStep, now time.Time, live bool) {
|
|||||||
if t := stepTime(s, now); t != "" {
|
if t := stepTime(s, now); t != "" {
|
||||||
<span class="step-time">{ t }</span>
|
<span class="step-time">{ t }</span>
|
||||||
}
|
}
|
||||||
if s.Note != "" {
|
|
||||||
<span class="step-note">{ s.Note }</span>
|
|
||||||
}
|
|
||||||
if s.State == domain.StepRunning && s.Total > 0 {
|
if s.State == domain.StepRunning && s.Total > 0 {
|
||||||
<progress class="step-bar" value={ strconv.Itoa(s.Done) } max={ strconv.Itoa(s.Total) }></progress>
|
<progress class="step-bar" value={ strconv.Itoa(s.Done) } max={ strconv.Itoa(s.Total) }></progress>
|
||||||
<span class="step-count">{ strconv.Itoa(s.Done) } of { strconv.Itoa(s.Total) }</span>
|
<span class="step-count">{ strconv.Itoa(s.Done) } of { strconv.Itoa(s.Total) }</span>
|
||||||
|
|||||||
@@ -1,93 +0,0 @@
|
|||||||
package web_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"fmt"
|
|
||||||
"log/slog"
|
|
||||||
"net/http"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Input that cannot be right is the caller's to fix: a 4xx, never a 500 (adversarial
|
|
||||||
// review F7). No cross-tenant oracle either way: unknown and foreign ids look the same.
|
|
||||||
|
|
||||||
func TestUploadRoutes_ANonUUIDIdIs404NotA500(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
for _, path := range []string{"/uploads/not-a-uuid", "/uploads/not-a-uuid/status", "/uploads/not-a-uuid/chat", "/uploads/%00/status", "/uploads/1/status"} {
|
|
||||||
t.Run(path, func(t *testing.T) {
|
|
||||||
rec := do(t, app, get(path))
|
|
||||||
|
|
||||||
assert.Equal(t, http.StatusNotFound, rec.Code)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadPost_ATitleWithControlCharactersOrInvalidUTF8IsCleanedNotA500(t *testing.T) {
|
|
||||||
tests := []struct{ name, title, filename string }{
|
|
||||||
{"a NUL in the title", "Sprint\x00review", "a.m4a"},
|
|
||||||
{"invalid UTF-8 in the title", "Sprint \xff\xfe review", "a.m4a"},
|
|
||||||
{"a newline in the title", "Sprint\nreview", "a.m4a"},
|
|
||||||
}
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
app, sub := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
|
|
||||||
rec := do(t, app, uploadReq(t, tt.title, tt.filename, []byte("synthetic audio bytes")))
|
|
||||||
|
|
||||||
require.Equal(t, http.StatusSeeOther, rec.Code, "bad bytes in a cosmetic title must not fail the upload: %s", body(t, rec))
|
|
||||||
require.Len(t, sub.jobs, 1)
|
|
||||||
var title string
|
|
||||||
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT title FROM uploaded_transcripts WHERE id = $1`, sub.jobs[0].UploadID).Scan(&title))
|
|
||||||
assert.NotContains(t, title, "\x00")
|
|
||||||
assert.NotContains(t, title, "\n")
|
|
||||||
assert.True(t, len(title) > 0, "a title is still shown")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadPost_AMalformedMultipartHeaderIsA400AndItsTextIsNotLogged(t *testing.T) {
|
|
||||||
// A control byte in the part's file name makes the multipart body malformed. It is the
|
|
||||||
// sender's to fix, and the parser's error quotes the header, which carries the name.
|
|
||||||
app, sub := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
buf := &lockedBuf{}
|
|
||||||
app.Log = slog.New(slog.NewTextHandler(buf, nil))
|
|
||||||
|
|
||||||
rec := do(t, app, uploadReq(t, "", "zebrakaka\x00weekly.m4a", []byte("x")))
|
|
||||||
|
|
||||||
assert.Equal(t, http.StatusBadRequest, rec.Code)
|
|
||||||
assert.Empty(t, sub.jobs)
|
|
||||||
assert.NotContains(t, buf.String(), "zebrakaka", "a file name is private content")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadPost_EveryBidirectionalControlCharacterIsStrippedFromATitle(t *testing.T) {
|
|
||||||
// U+202E reverses the text that follows it, so a title could render reversed on its
|
|
||||||
// owner's own page (second adversarial review). Every one of the twelve is pinned.
|
|
||||||
controls := []rune{0x202A, 0x202B, 0x202C, 0x202D, 0x202E, 0x2066, 0x2067, 0x2068, 0x2069, 0x200E, 0x200F, 0x061C}
|
|
||||||
for _, c := range controls {
|
|
||||||
t.Run(fmt.Sprintf("U+%04X", c), func(t *testing.T) {
|
|
||||||
app, sub := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
|
|
||||||
rec := do(t, app, uploadReq(t, "Sprint"+string(c)+"review", "a.m4a", []byte("x")))
|
|
||||||
|
|
||||||
require.Equal(t, http.StatusSeeOther, rec.Code)
|
|
||||||
var title string
|
|
||||||
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT title FROM uploaded_transcripts WHERE id = $1`, sub.jobs[0].UploadID).Scan(&title))
|
|
||||||
assert.Equal(t, "Sprintreview", title, "dropped, not replaced by a space: a space would leave a visible gap")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadPost_SwedishLettersAndEmojiSurviveTheTitleCleaning(t *testing.T) {
|
|
||||||
app, sub := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
|
|
||||||
// Includes a ZWJ sequence (woman + ZWJ + laptop) and a ZWNJ: joiners are not direction controls.
|
|
||||||
rec := do(t, app, uploadReq(t, "Möte åäö 🎙️ 👩\u200d💻 a\u200cb", "a.m4a", []byte("x")))
|
|
||||||
|
|
||||||
require.Equal(t, http.StatusSeeOther, rec.Code)
|
|
||||||
var title string
|
|
||||||
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT title FROM uploaded_transcripts WHERE id = $1`, sub.jobs[0].UploadID).Scan(&title))
|
|
||||||
assert.Equal(t, "Möte åäö 🎙️ 👩\u200d💻 a\u200cb", title, "joiners and variation selectors belong to the text")
|
|
||||||
}
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
package web_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"net/http"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/uploads"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestUploadPost_ABusyRejectionIs429WithItsMessage(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
app.Uploads = failingUploader{err: &uploads.RejectError{Reason: uploads.ReasonBusy, Message: "Too many recordings are waiting. Try again in a few minutes."}}
|
|
||||||
|
|
||||||
rec := do(t, app, uploadReq(t, "t", "a.m4a", []byte("x")))
|
|
||||||
|
|
||||||
require.Equal(t, http.StatusTooManyRequests, rec.Code, "the user's to retry: not a 500, not a 422")
|
|
||||||
assert.Contains(t, body(t, rec), "Too many recordings are waiting")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadStatus_ABusyFailureIsExplainedInWords(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
id := seedUpload(t, userID, "m", "failed", "queue_full")
|
|
||||||
|
|
||||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
|
||||||
|
|
||||||
assert.Contains(t, b, "waiting")
|
|
||||||
assert.NotContains(t, b, "This upload failed. Please upload it again.")
|
|
||||||
}
|
|
||||||
@@ -31,7 +31,6 @@ func newUploadChatApp(t *testing.T) (*web.App, *fakeChatter) {
|
|||||||
}
|
}
|
||||||
app.Chat = fc
|
app.Chat = fc
|
||||||
app.UploadChatMaxChars = 120000
|
app.UploadChatMaxChars = 120000
|
||||||
app.UploadChatModels = fc.models
|
|
||||||
return app, fc
|
return app, fc
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,73 +0,0 @@
|
|||||||
package web_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"net/http"
|
|
||||||
"net/url"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/adapters/chat"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/web"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The whole private transcript goes to the model chosen in the form, so the upload chat
|
|
||||||
// offers and accepts local models only, even when the shared chat service knows a cloud
|
|
||||||
// one (ADR-035 review). The cloud model here is only a name; no network is involved.
|
|
||||||
|
|
||||||
func newUploadChatAppWithCloud(t *testing.T) (*web.App, *fakeChatter) {
|
|
||||||
t.Helper()
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
fc := &fakeChatter{
|
|
||||||
models: []string{"koala/phi4-mini", "iguana/gemma4-26b", "berget/mistral-small"},
|
|
||||||
reply: chat.Reply{Answer: "Anders owns the timber order."},
|
|
||||||
}
|
|
||||||
app.Chat = fc
|
|
||||||
app.UploadChatMaxChars = 120000
|
|
||||||
app.UploadChatModels = []string{"koala/phi4-mini", "iguana/gemma4-26b"}
|
|
||||||
return app, fc
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadChatPage_DoesNotOfferACloudModelEvenThoughTheServiceKnowsOne(t *testing.T) {
|
|
||||||
app, _ := newUploadChatAppWithCloud(t)
|
|
||||||
id := seedUpload(t, userID, "Roadmap sync", "done", meetingText)
|
|
||||||
|
|
||||||
b := body(t, do(t, app, get("/uploads/"+id+"/chat")))
|
|
||||||
|
|
||||||
assert.Contains(t, b, "iguana/gemma4-26b")
|
|
||||||
assert.NotContains(t, b, "berget/mistral-small", "the switcher must not list a model the transcript must not be sent to")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadChatPost_AForgedCloudModelIsReplacedByALocalOneBeforeTheTranscriptIsSent(t *testing.T) {
|
|
||||||
app, fc := newUploadChatAppWithCloud(t)
|
|
||||||
id := seedUpload(t, userID, "Roadmap sync", "done", meetingText)
|
|
||||||
|
|
||||||
rec := do(t, app, chatPost("/uploads/"+id+"/chat", url.Values{"question": {"Who owns timber?"}, "model": {"berget/mistral-small"}}))
|
|
||||||
|
|
||||||
require.Equal(t, http.StatusOK, rec.Code)
|
|
||||||
require.Equal(t, 1, fc.callCount)
|
|
||||||
assert.NotEqual(t, "berget/mistral-small", fc.gotReq.Model, "a cloud model reached the chat service with the private transcript")
|
|
||||||
assert.Contains(t, []string{"koala/phi4-mini", "iguana/gemma4-26b"}, fc.gotReq.Model)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadChatPost_ALocalModelThatIsOfferedIsHonoured(t *testing.T) {
|
|
||||||
app, fc := newUploadChatAppWithCloud(t)
|
|
||||||
id := seedUpload(t, userID, "Roadmap sync", "done", meetingText)
|
|
||||||
|
|
||||||
rec := do(t, app, chatPost("/uploads/"+id+"/chat", url.Values{"question": {"q"}, "model": {"koala/phi4-mini"}}))
|
|
||||||
|
|
||||||
require.Equal(t, http.StatusOK, rec.Code)
|
|
||||||
assert.Equal(t, "koala/phi4-mini", fc.gotReq.Model)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadChat_WithNoLocalModelConfiguredNothingIsSentAnywhere(t *testing.T) {
|
|
||||||
app, fc := newUploadChatAppWithCloud(t)
|
|
||||||
app.UploadChatModels = nil
|
|
||||||
id := seedUpload(t, userID, "Roadmap sync", "done", meetingText)
|
|
||||||
|
|
||||||
rec := do(t, app, chatPost("/uploads/"+id+"/chat", url.Values{"question": {"q"}, "model": {"berget/mistral-small"}}))
|
|
||||||
|
|
||||||
require.Equal(t, http.StatusOK, rec.Code)
|
|
||||||
assert.Zero(t, fc.callCount, "failing closed: no local model offered means no model is called")
|
|
||||||
}
|
|
||||||
@@ -1,140 +0,0 @@
|
|||||||
package web_test
|
|
||||||
|
|
||||||
// A transcript produced by the cloud fallback means this recording's audio left the
|
|
||||||
// homelab, which the reader is told in plain words (#52). "Local" is config.IsLocalModel:
|
|
||||||
// one definition, not a second string match in the view.
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
)
|
|
||||||
|
|
||||||
func setEndpoint(t *testing.T, id, endpoint string) {
|
|
||||||
t.Helper()
|
|
||||||
_, err := rawPool(t).Exec(context.Background(), `UPDATE uploaded_transcripts SET endpoint = $2 WHERE id = $1`, id, endpoint)
|
|
||||||
require.NoError(t, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadStatus_ACloudServedTranscriptSaysTheAudioLeftTheHomelab(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
|
||||||
setEndpoint(t, id, "berget/kb-whisper")
|
|
||||||
|
|
||||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
|
||||||
|
|
||||||
assert.Contains(t, b, "left the homelab")
|
|
||||||
assert.Contains(t, b, "cloud service (berget)")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadStatus_ALocallyServedTranscriptCarriesNoCloudNote(t *testing.T) {
|
|
||||||
for _, endpoint := range []string{"iguana/kb-whisper", "koala/whisper", ""} {
|
|
||||||
t.Run(endpoint, func(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
|
||||||
setEndpoint(t, id, endpoint)
|
|
||||||
|
|
||||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
|
||||||
|
|
||||||
assert.NotContains(t, b, "left the homelab")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadStatus_AnEndpointThatOnlyLooksLocalIsStillNamedAsCloud(t *testing.T) {
|
|
||||||
// The prefix alone must not decide it: IsLocalModel refuses path tricks.
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
|
||||||
setEndpoint(t, id, "iguana/../berget/kb-whisper")
|
|
||||||
|
|
||||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
|
||||||
|
|
||||||
assert.Contains(t, b, "left the homelab")
|
|
||||||
}
|
|
||||||
|
|
||||||
// The two-engine path (ADR-034) writes one composite endpoint string. Every name in it is
|
|
||||||
// a local model, so the audio never left the homelab and the note must not say it did.
|
|
||||||
func TestUploadStatus_AReconciledAllLocalEndpointCarriesNoCloudNote(t *testing.T) {
|
|
||||||
for _, endpoint := range []string{
|
|
||||||
"iguana/kb-whisper + koala/whisper, reconciled by iguana/gemma4-26b",
|
|
||||||
"iguana/kb-whisper + koala/whisper, reconciled by iguana/gemma4-26b\n",
|
|
||||||
} {
|
|
||||||
t.Run(endpoint, func(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
|
||||||
setEndpoint(t, id, endpoint)
|
|
||||||
|
|
||||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
|
||||||
|
|
||||||
assert.NotContains(t, b, "left the homelab")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadStatus_AReconciledEndpointNamingACloudEngineStillNamesTheCloudProvider(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
|
||||||
setEndpoint(t, id, "iguana/kb-whisper + berget/kb-whisper, reconciled by iguana/gemma4-26b")
|
|
||||||
|
|
||||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
|
||||||
|
|
||||||
assert.Contains(t, b, "cloud service (berget)")
|
|
||||||
}
|
|
||||||
|
|
||||||
// An alias without a provider prefix is not a local model, wherever it sits in the
|
|
||||||
// composite: it must never be dropped from the judgement (PR #58 re-review).
|
|
||||||
func TestUploadStatus_AReconciledEndpointWithAnUnprefixedAliasIsNotCalledLocal(t *testing.T) {
|
|
||||||
for _, endpoint := range []string{
|
|
||||||
"iguana/kb-whisper + whisper-1, reconciled by iguana/gemma4-26b",
|
|
||||||
"iguana/kb-whisper + whisper 1, reconciled by iguana/gemma4-26b",
|
|
||||||
"iguana/kb-whisper + berget / x, reconciled by iguana/gemma4-26b",
|
|
||||||
} {
|
|
||||||
t.Run(endpoint, func(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
|
||||||
setEndpoint(t, id, endpoint)
|
|
||||||
|
|
||||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
|
||||||
|
|
||||||
assert.Contains(t, b, "left the homelab")
|
|
||||||
assert.NotContains(t, b, "cloud service ()", "the note names a provider")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// "by", "reconciled" and "+" are connectors only inside the composite format. As a whole
|
|
||||||
// endpoint, or as an alias within the composite, they are model names and are judged.
|
|
||||||
func TestUploadStatus_AConnectorWordUsedAsAModelNameIsStillJudged(t *testing.T) {
|
|
||||||
for _, endpoint := range []string{
|
|
||||||
"by",
|
|
||||||
"reconciled",
|
|
||||||
"+",
|
|
||||||
"iguana/kb-whisper + by, reconciled by iguana/gemma4-26b",
|
|
||||||
"iguana/kb-whisper + reconciled, reconciled by iguana/gemma4-26b",
|
|
||||||
"iguana/kb-whisper + +, reconciled by iguana/gemma4-26b",
|
|
||||||
"/x", // no provider at all: the note must still name something
|
|
||||||
} {
|
|
||||||
t.Run(endpoint, func(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
|
||||||
setEndpoint(t, id, endpoint)
|
|
||||||
|
|
||||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
|
||||||
|
|
||||||
assert.Contains(t, b, "left the homelab")
|
|
||||||
assert.NotContains(t, b, "cloud service ()")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadStatus_AReconcilerThatIsNotLocalNamesTheCloudProvider(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
|
||||||
setEndpoint(t, id, "iguana/kb-whisper + koala/whisper, reconciled by berget/mistral-large")
|
|
||||||
|
|
||||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
|
||||||
|
|
||||||
assert.Contains(t, b, "cloud service (berget)")
|
|
||||||
}
|
|
||||||
@@ -1,85 +0,0 @@
|
|||||||
package web_test
|
|
||||||
|
|
||||||
// Deleting one recording (#37). Owner-scoped like every other upload route: another user's
|
|
||||||
// id is a 404 and the row survives; an upload still being processed is refused.
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
)
|
|
||||||
|
|
||||||
func deleteReq(id string) *http.Request {
|
|
||||||
return httptest.NewRequest(http.MethodPost, "/uploads/"+id+"/delete", nil)
|
|
||||||
}
|
|
||||||
|
|
||||||
func uploadExists(t *testing.T, id string) bool {
|
|
||||||
t.Helper()
|
|
||||||
var n int
|
|
||||||
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT count(*) FROM uploaded_transcripts WHERE id = $1`, id).Scan(&n))
|
|
||||||
return n == 1
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadDelete_AnOwnerRemovesAFinishedOrFailedRecordingAndLandsOnTheList(t *testing.T) {
|
|
||||||
for _, status := range []string{"done", "failed"} {
|
|
||||||
t.Run(status, func(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
id := seedUpload(t, userID, "Roadmap sync", status, "transcription_failed")
|
|
||||||
|
|
||||||
rec := do(t, app, deleteReq(id))
|
|
||||||
|
|
||||||
assert.Equal(t, http.StatusSeeOther, rec.Code)
|
|
||||||
assert.Equal(t, "/uploads", rec.Header().Get("Location"))
|
|
||||||
assert.False(t, uploadExists(t, id))
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadDelete_AnotherUsersRecordingIs404AndSurvives(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
theirs := seedUpload(t, otherUserID, "Their private meeting", "done", meetingText)
|
|
||||||
|
|
||||||
rec := do(t, app, deleteReq(theirs))
|
|
||||||
|
|
||||||
assert.Equal(t, http.StatusNotFound, rec.Code)
|
|
||||||
assert.True(t, uploadExists(t, theirs), "a non-owner's delete must not remove the row")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadDelete_ARecordingStillBeingTranscribedIsRefusedWith409AndKept(t *testing.T) {
|
|
||||||
for _, status := range []string{"queued", "transcribing"} {
|
|
||||||
t.Run(status, func(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
id := seedUpload(t, userID, "m", status, "")
|
|
||||||
|
|
||||||
rec := do(t, app, deleteReq(id))
|
|
||||||
|
|
||||||
assert.Equal(t, http.StatusConflict, rec.Code)
|
|
||||||
assert.Contains(t, body(t, rec), "still being transcribed")
|
|
||||||
assert.True(t, uploadExists(t, id))
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadDelete_ANonIDIsA404NotAServerError(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
|
|
||||||
assert.Equal(t, http.StatusNotFound, do(t, app, deleteReq("not-a-uuid")).Code)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadPage_OffersDeleteOnlyForAFinishedOrFailedRecording(t *testing.T) {
|
|
||||||
for status, want := range map[string]bool{"done": true, "failed": true, "queued": false, "transcribing": false} {
|
|
||||||
t.Run(status, func(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
id := seedUpload(t, userID, "m", status, "transcription_failed")
|
|
||||||
|
|
||||||
b := body(t, do(t, app, get("/uploads/"+id)))
|
|
||||||
|
|
||||||
assert.Equal(t, want, strings.Contains(b, `action="/uploads/`+id+`/delete"`))
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,126 +0,0 @@
|
|||||||
package web_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"log/slog"
|
|
||||||
"net"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
|
||||||
)
|
|
||||||
|
|
||||||
// A client that hangs up in the middle of an upload is routine, not a server fault. The
|
|
||||||
// in-process recorder cannot see it (its context never cancels), so this talks to a real
|
|
||||||
// listener (third adversarial review: over a socket every such case logged an ERROR and
|
|
||||||
// half-close returned a 500, because the error page was rendered on a cancelled context).
|
|
||||||
|
|
||||||
// body positions at which the client can hang up: inside the title field, inside the file
|
|
||||||
// part's own headers, and mid-file. Every one must be handled alike (fourth adversarial
|
|
||||||
// review: the title branch had no guard and still logged an ERROR).
|
|
||||||
const (
|
|
||||||
inTitle = "in the title"
|
|
||||||
inHeaders = "in the file part's headers"
|
|
||||||
midFile = "mid-file"
|
|
||||||
)
|
|
||||||
|
|
||||||
func partialUpload(t *testing.T, addr, where string, hangUp func(*net.TCPConn)) {
|
|
||||||
t.Helper()
|
|
||||||
c, err := net.Dial("tcp", addr)
|
|
||||||
require.NoError(t, err)
|
|
||||||
tc := c.(*net.TCPConn)
|
|
||||||
const boundary = "xBOUNDARYx"
|
|
||||||
var body string
|
|
||||||
switch where {
|
|
||||||
case inTitle:
|
|
||||||
body = "--" + boundary + "\r\nContent-Disposition: form-data; name=\"title\"\r\n\r\nhalf a tit"
|
|
||||||
case inHeaders:
|
|
||||||
body = "--" + boundary + "\r\nContent-Disposition: form-data; name=\"file\"; filena"
|
|
||||||
default:
|
|
||||||
body = "--" + boundary + "\r\nContent-Disposition: form-data; name=\"file\"; filename=\"a.m4a\"\r\nContent-Type: audio/mp4\r\n\r\n" + strings.Repeat("a", 4000)
|
|
||||||
}
|
|
||||||
_, err = fmt.Fprintf(tc, "POST /uploads HTTP/1.1\r\nHost: x\r\nContent-Type: multipart/form-data; boundary=%s\r\nContent-Length: 200000\r\n\r\n%s", boundary, body)
|
|
||||||
require.NoError(t, err)
|
|
||||||
time.Sleep(50 * time.Millisecond) // let the server start reading
|
|
||||||
hangUp(tc)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadPost_AClientThatHangsUpAnywhereInTheBodyIsLoggedAtInfoAndNeverAsAnError(t *testing.T) {
|
|
||||||
hangUps := map[string]func(*net.TCPConn){
|
|
||||||
"a half close": func(c *net.TCPConn) { _ = c.CloseWrite(); time.Sleep(100 * time.Millisecond); _ = c.Close() },
|
|
||||||
"a full close": func(c *net.TCPConn) { _ = c.Close() },
|
|
||||||
"a reset (RST)": func(c *net.TCPConn) { _ = c.SetLinger(0); _ = c.Close() },
|
|
||||||
}
|
|
||||||
for _, where := range []string{inTitle, inHeaders, midFile} {
|
|
||||||
for name, hangUp := range hangUps {
|
|
||||||
t.Run(where+", "+name, func(t *testing.T) {
|
|
||||||
app, sub := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
buf := &lockedBuf{}
|
|
||||||
app.Log = slog.New(slog.NewTextHandler(buf, nil))
|
|
||||||
srv := httptest.NewServer(app.Router())
|
|
||||||
defer srv.Close()
|
|
||||||
|
|
||||||
partialUpload(t, strings.TrimPrefix(srv.URL, "http://"), where, hangUp)
|
|
||||||
|
|
||||||
require.Eventually(t, func() bool {
|
|
||||||
out := buf.String()
|
|
||||||
return strings.Contains(out, "upload abandoned") || strings.Contains(out, "upload rejected") || strings.Contains(out, "level=ERROR")
|
|
||||||
}, 3*time.Second, 20*time.Millisecond, "the server never finished handling the aborted upload")
|
|
||||||
time.Sleep(100 * time.Millisecond)
|
|
||||||
out := buf.String()
|
|
||||||
assert.NotContains(t, out, "level=ERROR", "a client hanging up is not a server fault:\n%s", out)
|
|
||||||
assert.Empty(t, sub.jobs)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// goneMidAccept is an Uploader during whose Accept the client disappears: the request's
|
|
||||||
// context is cancelled, then Accept fails the way a read from a dead connection does.
|
|
||||||
type goneMidAccept struct{ cancel context.CancelFunc }
|
|
||||||
|
|
||||||
func (g goneMidAccept) Accept(context.Context, string, string, domain.Language, io.Reader) (string, error) {
|
|
||||||
g.cancel()
|
|
||||||
return "", fmt.Errorf("receive: %w", io.ErrClosedPipe)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadPost_AnAbandonedRequestGetsA499NotAnImplicitEmpty200(t *testing.T) {
|
|
||||||
// If the context is cancelled but the connection can still be written to (a client that
|
|
||||||
// half-closed after sending), a missing status becomes 200 with an empty body: a false
|
|
||||||
// success (fourth adversarial review).
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
buf := &lockedBuf{}
|
|
||||||
app.Log = slog.New(slog.NewTextHandler(buf, nil))
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
defer cancel()
|
|
||||||
app.Uploads = goneMidAccept{cancel: cancel}
|
|
||||||
|
|
||||||
rec := do(t, app, uploadReq(t, "t", "a.m4a", []byte("x")).WithContext(ctx))
|
|
||||||
|
|
||||||
assert.Equal(t, 499, rec.Code, "never an implicit 200 for an upload that was not accepted")
|
|
||||||
assert.Contains(t, buf.String(), "upload abandoned")
|
|
||||||
assert.NotContains(t, buf.String(), "level=ERROR")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadPost_ADatabaseDropThatLooksLikeAnEOFIsStillAServerError(t *testing.T) {
|
|
||||||
// pgx turns a connection that drops mid-message into io.ErrUnexpectedEOF. Mapping that
|
|
||||||
// sentinel to "your upload was cut off" (a 400) would hide a real fault: only an error
|
|
||||||
// from reading the CLIENT's body is theirs (third adversarial review).
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
buf := &lockedBuf{}
|
|
||||||
app.Log = slog.New(slog.NewTextHandler(buf, nil))
|
|
||||||
app.Uploads = failingUploader{err: fmt.Errorf("failed to receive message: %w", io.ErrUnexpectedEOF)}
|
|
||||||
|
|
||||||
rec := do(t, app, uploadReq(t, "t", "a.m4a", []byte("x")))
|
|
||||||
|
|
||||||
assert.Equal(t, http.StatusInternalServerError, rec.Code)
|
|
||||||
assert.Contains(t, buf.String(), "level=ERROR")
|
|
||||||
}
|
|
||||||
@@ -16,7 +16,6 @@ import (
|
|||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/uploads"
|
"git.d-ma.be/mathias/tapir/internal/uploads"
|
||||||
"git.d-ma.be/mathias/tapir/internal/web"
|
"git.d-ma.be/mathias/tapir/internal/web"
|
||||||
)
|
)
|
||||||
@@ -133,7 +132,7 @@ func TestUploadPost_AcceptedAudioQueuesARowAndRedirectsToIt(t *testing.T) {
|
|||||||
assert.Equal(t, "Sprint review", title)
|
assert.Equal(t, "Sprint review", title)
|
||||||
assert.Equal(t, "/uploads/"+id, rec.Header().Get("Location"))
|
assert.Equal(t, "/uploads/"+id, rec.Header().Get("Location"))
|
||||||
require.Len(t, sub.jobs, 1)
|
require.Len(t, sub.jobs, 1)
|
||||||
assert.Equal(t, uploads.Job{UserID: userID, UploadID: id, Path: sub.jobs[0].Path, Language: domain.LangSwedish, Duration: audioOK.Duration}, sub.jobs[0])
|
assert.Equal(t, uploads.Job{UserID: userID, UploadID: id, Path: sub.jobs[0].Path}, sub.jobs[0])
|
||||||
spooled, err := os.ReadFile(sub.jobs[0].Path)
|
spooled, err := os.ReadFile(sub.jobs[0].Path)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Equal(t, "synthetic audio bytes", string(spooled))
|
assert.Equal(t, "synthetic audio bytes", string(spooled))
|
||||||
@@ -367,8 +366,8 @@ func setReconciliation(t *testing.T, id, json string) {
|
|||||||
const reconciledJSON = `{"status":"reconciled","base":"iguana/pianissimo","alt":"iguana/kb-whisper","reconciler":"iguana/gemma4-26b",
|
const reconciledJSON = `{"status":"reconciled","base":"iguana/pianissimo","alt":"iguana/kb-whisper","reconciler":"iguana/gemma4-26b",
|
||||||
"disputes":380,"auto_resolved":90,"adjudicated":285,"changed":120,"unsure":5,"errors":0,
|
"disputes":380,"auto_resolved":90,"adjudicated":285,"changed":120,"unsure":5,"errors":0,
|
||||||
"contested":[
|
"contested":[
|
||||||
{"seconds":2210.5,"base":"Axel","alt":"Alex","used":"alt","unsure":false},
|
{"seconds":2210.5,"base":"SEB","alt":"SCB","used":"alt","unsure":false},
|
||||||
{"seconds":3400,"base":"A+2","alt":"en A","used":"base","unsure":true}]}`
|
{"seconds":3400,"base":"T+1","alt":"en T","used":"base","unsure":true}]}`
|
||||||
|
|
||||||
func TestUploadStatus_ReconciledSaysWhatWasDoneInPlainWords(t *testing.T) {
|
func TestUploadStatus_ReconciledSaysWhatWasDoneInPlainWords(t *testing.T) {
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
@@ -395,11 +394,11 @@ func TestUploadStatus_ListsTheContestedSpotsWithAMinuteMarkBothReadingsAndWhichW
|
|||||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||||
|
|
||||||
assert.Contains(t, b, "~37 min", "2210.5 s")
|
assert.Contains(t, b, "~37 min", "2210.5 s")
|
||||||
assert.Contains(t, b, "Axel")
|
assert.Contains(t, b, "SEB")
|
||||||
assert.Contains(t, b, "Alex")
|
assert.Contains(t, b, "SCB")
|
||||||
assert.Contains(t, b, "~57 min", "3400 s")
|
assert.Contains(t, b, "~57 min", "3400 s")
|
||||||
assert.Contains(t, b, "A+2")
|
assert.Contains(t, b, "T+1")
|
||||||
assert.Contains(t, b, "en A")
|
assert.Contains(t, b, "en T")
|
||||||
assert.Contains(t, b, "uncertain", "an unsure spot is marked so the reader knows to check it")
|
assert.Contains(t, b, "uncertain", "an unsure spot is marked so the reader knows to check it")
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -461,20 +460,3 @@ func TestUploadStatus_AVeryLongListOfContestedSpotsIsShortenedWithItsTrueCount(t
|
|||||||
assert.NotContains(t, b, ">b119<", "the page does not render every one")
|
assert.NotContains(t, b, ">b119<", "the page does not render every one")
|
||||||
assert.Contains(t, b, "70 more", "120 stored, 50 shown")
|
assert.Contains(t, b, "70 more", "120 stored, 50 shown")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestUploadStatus_ALengthFailureIsExplainedInWordsAndNotAsAGenericError(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
tests := []struct{ kind, want string }{
|
|
||||||
{"audio_too_long", "longer than the limit"},
|
|
||||||
}
|
|
||||||
for _, tt := range tests {
|
|
||||||
t.Run(tt.kind, func(t *testing.T) {
|
|
||||||
id := seedUpload(t, userID, "m-"+tt.kind, "failed", tt.kind)
|
|
||||||
|
|
||||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
|
||||||
|
|
||||||
assert.Contains(t, b, tt.want)
|
|
||||||
assert.NotContains(t, b, "This upload failed. Please upload it again.")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,151 +0,0 @@
|
|||||||
package web_test
|
|
||||||
|
|
||||||
// The language of a recording is chosen on the upload form (tapir#61). These drive the real
|
|
||||||
// Router, Intake and Postgres store; the submitter is the process boundary and records the
|
|
||||||
// job the worker would receive.
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"context"
|
|
||||||
"mime/multipart"
|
|
||||||
"net/http"
|
|
||||||
"net/http/httptest"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
|
||||||
)
|
|
||||||
|
|
||||||
// langReq builds an upload whose parts come in the order given: "title", "language" and
|
|
||||||
// "file" (the file part always carries the same synthetic bytes).
|
|
||||||
func langReq(t *testing.T, order []string, language string) *http.Request {
|
|
||||||
t.Helper()
|
|
||||||
var buf bytes.Buffer
|
|
||||||
mw := multipart.NewWriter(&buf)
|
|
||||||
for _, part := range order {
|
|
||||||
switch part {
|
|
||||||
case "title":
|
|
||||||
require.NoError(t, mw.WriteField("title", "m"))
|
|
||||||
case "language":
|
|
||||||
require.NoError(t, mw.WriteField("language", language))
|
|
||||||
case "file":
|
|
||||||
fw, err := mw.CreateFormFile("file", "m.wav")
|
|
||||||
require.NoError(t, err)
|
|
||||||
_, err = fw.Write([]byte("RIFF....WAVE"))
|
|
||||||
require.NoError(t, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
require.NoError(t, mw.Close())
|
|
||||||
req := httptest.NewRequest(http.MethodPost, "/uploads", &buf)
|
|
||||||
req.Header.Set("Content-Type", mw.FormDataContentType())
|
|
||||||
return req
|
|
||||||
}
|
|
||||||
|
|
||||||
func rowLanguage(t *testing.T, location string) domain.Language {
|
|
||||||
t.Helper()
|
|
||||||
id := strings.TrimPrefix(location, "/uploads/")
|
|
||||||
got, err := newStore(t).GetUpload(context.Background(), userID, id)
|
|
||||||
require.NoError(t, err)
|
|
||||||
return got.Language
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadPost_TheLanguageFieldReachesTheJobAndTheRow(t *testing.T) {
|
|
||||||
for _, tt := range []struct {
|
|
||||||
name string
|
|
||||||
language string
|
|
||||||
want domain.Language
|
|
||||||
}{
|
|
||||||
{"other", "auto", domain.LangAuto},
|
|
||||||
{"swedish", "sv", domain.LangSwedish},
|
|
||||||
} {
|
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
|
||||||
app, sub := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
|
|
||||||
rec := do(t, app, langReq(t, []string{"title", "language", "file"}, tt.language))
|
|
||||||
|
|
||||||
require.Equal(t, http.StatusSeeOther, rec.Code)
|
|
||||||
require.Len(t, sub.jobs, 1)
|
|
||||||
assert.Equal(t, tt.want, sub.jobs[0].Language)
|
|
||||||
assert.Equal(t, tt.want, rowLanguage(t, rec.Header().Get("Location")))
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadPost_NoLanguageFieldIsSwedishAsBefore(t *testing.T) {
|
|
||||||
app, sub := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
|
|
||||||
rec := do(t, app, langReq(t, []string{"title", "file"}, ""))
|
|
||||||
|
|
||||||
require.Equal(t, http.StatusSeeOther, rec.Code)
|
|
||||||
require.Len(t, sub.jobs, 1)
|
|
||||||
assert.Equal(t, domain.LangSwedish, sub.jobs[0].Language)
|
|
||||||
assert.Equal(t, domain.LangSwedish, rowLanguage(t, rec.Header().Get("Location")))
|
|
||||||
}
|
|
||||||
|
|
||||||
// A part after the file is never read (the file part ends the stream), so a client that
|
|
||||||
// orders its fields wrongly gets Swedish, the old behaviour, not a guess.
|
|
||||||
func TestUploadPost_ALanguageFieldAfterTheFileIsNotReadAndTheUploadIsSwedish(t *testing.T) {
|
|
||||||
app, sub := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
|
|
||||||
rec := do(t, app, langReq(t, []string{"title", "file", "language"}, "auto"))
|
|
||||||
|
|
||||||
require.Equal(t, http.StatusSeeOther, rec.Code)
|
|
||||||
require.Len(t, sub.jobs, 1)
|
|
||||||
assert.Equal(t, domain.LangSwedish, sub.jobs[0].Language)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadPost_ALanguageOutsideTheClosedSetIsRefusedBeforeAnythingIsSpooledOrStored(t *testing.T) {
|
|
||||||
for _, bad := range []string{"en", "SV", " sv", "sv,auto", "../auto", strings.Repeat("a", 5000)} {
|
|
||||||
t.Run(bad[:min(len(bad), 12)], func(t *testing.T) {
|
|
||||||
app, sub := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
|
|
||||||
rec := do(t, app, langReq(t, []string{"title", "language", "file"}, bad))
|
|
||||||
|
|
||||||
assert.Equal(t, http.StatusBadRequest, rec.Code)
|
|
||||||
assert.Empty(t, sub.jobs, "a refused post must not enqueue transcription")
|
|
||||||
rows, err := newStore(t).ListUploads(context.Background(), userID)
|
|
||||||
require.NoError(t, err)
|
|
||||||
assert.Empty(t, rows, "and must leave no row")
|
|
||||||
assert.Contains(t, body(t, rec), "Choose a language from the list.")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadPage_TheFormOffersSwedishPreselectedAndOtherBeforeTheFilePart(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
|
|
||||||
b := body(t, do(t, app, get("/uploads")))
|
|
||||||
|
|
||||||
sel := strings.Index(b, `name="language"`)
|
|
||||||
file := strings.Index(b, `name="file"`)
|
|
||||||
require.True(t, sel >= 0, "the form has a language field")
|
|
||||||
assert.True(t, sel < file, "the field must come before the file part: the stream ends at the file")
|
|
||||||
assert.Regexp(t, `<option value="sv" selected`, b, "Swedish is preselected: today's behaviour")
|
|
||||||
assert.Contains(t, b, `<option value="auto"`)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadPage_NamesTheLanguageInEveryState(t *testing.T) {
|
|
||||||
for _, status := range []string{"queued", "transcribing", "done", "failed"} {
|
|
||||||
for _, tt := range []struct {
|
|
||||||
language domain.Language
|
|
||||||
want string
|
|
||||||
}{
|
|
||||||
{domain.LangSwedish, "Swedish"},
|
|
||||||
{domain.LangAuto, "Other (detected by the engine)"},
|
|
||||||
} {
|
|
||||||
t.Run(status+"/"+string(tt.language), func(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
id := seedUpload(t, userID, "m", status, "Synthetic transcript text.")
|
|
||||||
_, err := rawPool(t).Exec(context.Background(), `UPDATE uploaded_transcripts SET language = $2 WHERE id = $1`, id, string(tt.language))
|
|
||||||
require.NoError(t, err)
|
|
||||||
|
|
||||||
b := body(t, do(t, app, get("/uploads/"+id)))
|
|
||||||
|
|
||||||
assert.Contains(t, b, "Language: "+tt.want)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,74 +0,0 @@
|
|||||||
package web_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"io"
|
|
||||||
"log/slog"
|
|
||||||
"net/http"
|
|
||||||
"net/url"
|
|
||||||
"sync"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Error paths of the upload routes log a kind, never the error's text, which can carry
|
|
||||||
// a file name, a path, a database value or a provider's output (adversarial review F5).
|
|
||||||
|
|
||||||
type lockedBuf struct {
|
|
||||||
mu sync.Mutex
|
|
||||||
b bytes.Buffer
|
|
||||||
}
|
|
||||||
|
|
||||||
func (l *lockedBuf) Write(p []byte) (int, error) {
|
|
||||||
l.mu.Lock()
|
|
||||||
defer l.mu.Unlock()
|
|
||||||
return l.b.Write(p)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (l *lockedBuf) String() string {
|
|
||||||
l.mu.Lock()
|
|
||||||
defer l.mu.Unlock()
|
|
||||||
return l.b.String()
|
|
||||||
}
|
|
||||||
|
|
||||||
type failingUploader struct{ err error }
|
|
||||||
|
|
||||||
func (f failingUploader) Accept(context.Context, string, string, domain.Language, io.Reader) (string, error) {
|
|
||||||
return "", f.err
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadPost_AnUnexpectedFailureLogsAKindAndNotItsText(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
buf := &lockedBuf{}
|
|
||||||
app.Log = slog.New(slog.NewTextHandler(buf, nil))
|
|
||||||
app.Uploads = failingUploader{err: errors.New(`ffprobe: Error opening input file /spool/upload-555 for "zebrakaka-meeting.m4a"`)}
|
|
||||||
|
|
||||||
rec := do(t, app, uploadReq(t, "t", "zebrakaka-meeting.m4a", []byte("x")))
|
|
||||||
|
|
||||||
assert.Equal(t, http.StatusInternalServerError, rec.Code)
|
|
||||||
out := buf.String()
|
|
||||||
assert.NotContains(t, out, "zebrakaka", "a file name is private content")
|
|
||||||
assert.NotContains(t, out, "/spool/", "a spool path carries the upload's identity")
|
|
||||||
assert.Contains(t, out, "kind=error")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadChatPost_AModelFailureLogsAKindAndNotItsText(t *testing.T) {
|
|
||||||
app, fc := newUploadChatApp(t)
|
|
||||||
buf := &lockedBuf{}
|
|
||||||
app.Log = slog.New(slog.NewTextHandler(buf, nil))
|
|
||||||
fc.err = errors.New("LLM returned 400: the prompt was zebrakaka secret words")
|
|
||||||
id := seedUpload(t, userID, "m", "done", meetingText)
|
|
||||||
|
|
||||||
rec := do(t, app, chatPost("/uploads/"+id+"/chat", url.Values{"question": {"q"}}))
|
|
||||||
|
|
||||||
require.Equal(t, http.StatusOK, rec.Code)
|
|
||||||
out := buf.String()
|
|
||||||
assert.NotContains(t, out, "zebrakaka")
|
|
||||||
assert.Contains(t, out, "kind=error")
|
|
||||||
}
|
|
||||||
@@ -1,19 +0,0 @@
|
|||||||
package web_test
|
|
||||||
|
|
||||||
// /uploads was mounted but nothing linked to it (#37). The shared Layout has no access to
|
|
||||||
// the App, so the entry point is the home list, shown only when uploads are mounted.
|
|
||||||
|
|
||||||
import (
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestHomeList_LinksToRecordingsOnlyWhenUploadsAreMounted(t *testing.T) {
|
|
||||||
with, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
without := newApp(t)
|
|
||||||
|
|
||||||
assert.Contains(t, body(t, do(t, with, get("/"))), `href="/uploads"`)
|
|
||||||
assert.NotContains(t, body(t, do(t, without, get("/"))), `href="/uploads"`,
|
|
||||||
"a link to a route that is not mounted would be a dead end")
|
|
||||||
}
|
|
||||||
@@ -229,59 +229,16 @@ func TestUploadProgress_ADoneJobKeepsItsRecordOfWhatWasDoneAndHowLongEachTook(t
|
|||||||
assert.NotContains(t, b, "spinner")
|
assert.NotContains(t, b, "spinner")
|
||||||
}
|
}
|
||||||
|
|
||||||
// stepsBlock is the progress list of a page: from its opening tag to its closing one.
|
func TestUploadProgress_TheFullPageRendersTheSameDisplayAsTheFragment(t *testing.T) {
|
||||||
func stepsBlock(t *testing.T, html string) string {
|
|
||||||
t.Helper()
|
|
||||||
start := strings.Index(html, `<ol class="steps"`)
|
|
||||||
require.GreaterOrEqual(t, start, 0, "no progress list in:\n%s", html)
|
|
||||||
end := strings.Index(html[start:], "</ol>")
|
|
||||||
require.GreaterOrEqual(t, end, 0)
|
|
||||||
return html[start : start+end+len("</ol>")]
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadProgress_TheFullPageCarriesTheVerySameProgressListAsTheFragment(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
fixedClock(app)
|
fixedClock(app)
|
||||||
id := seedUpload(t, userID, "m", "transcribing", "")
|
id := seedUpload(t, userID, "m", "transcribing", "")
|
||||||
seedProgress(t, id, twoEngineRun(), progressNow.Add(-10*time.Second), progressNow.Add(-4*time.Second))
|
seedProgress(t, id, twoEngineRun(), progressNow.Add(-10*time.Second), progressNow.Add(-4*time.Second))
|
||||||
|
|
||||||
full := body(t, do(t, app, get("/uploads/"+id)))
|
rec := do(t, app, get("/uploads/"+id))
|
||||||
fragment := statusBody(t, app, id)
|
|
||||||
|
|
||||||
assert.Equal(t, stepsBlock(t, fragment), stepsBlock(t, full),
|
require.Equal(t, 200, rec.Code)
|
||||||
"the page a user opens and the fragment HTMX swaps in must show the same list, or the display jumps on the first poll")
|
b := body(t, rec)
|
||||||
}
|
assert.Contains(t, b, "Transcribing with iguana/kb-whisper", "the page a user opens, not only the polled fragment, shows the phases")
|
||||||
|
assert.Contains(t, b, "3m 12s")
|
||||||
func TestUploadProgress_AStepNoteIsShownDirectlyUnderItsStepAndEscaped(t *testing.T) {
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
fixedClock(app)
|
|
||||||
id := seedUpload(t, userID, "m", "transcribing", "")
|
|
||||||
steps := twoEngineRun()
|
|
||||||
steps[1].Note = "The audio is 20 min long; the file claimed 1 h 19 min. <b>cut off</b>?"
|
|
||||||
seedProgress(t, id, steps, progressNow.Add(-10*time.Second), progressNow.Add(-4*time.Second))
|
|
||||||
|
|
||||||
b := statusBody(t, app, id)
|
|
||||||
|
|
||||||
audio := strings.Index(b, "Preparing the audio")
|
|
||||||
note := strings.Index(b, "The audio is 20 min long")
|
|
||||||
next := strings.Index(b, "Transcribing with iguana/pianissimo")
|
|
||||||
require.True(t, audio >= 0 && note > audio && next > note, "the note must sit between its own step and the next one:\n%s", b)
|
|
||||||
assert.Contains(t, b, "step-note")
|
|
||||||
assert.NotContains(t, b, "<b>cut off</b>", "a note is words, never markup")
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUploadProgress_ADonePageShowsItsRecordOpenWhenAStepHasANote(t *testing.T) {
|
|
||||||
// A truncated recording must not look like a clean one: the note that replaced the
|
|
||||||
// refusal is the only warning, so it cannot sit inside a collapsed list (third review).
|
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
|
||||||
fixedClock(app)
|
|
||||||
noted := seedUpload(t, userID, "noted", "done", "Synthetic transcript text.")
|
|
||||||
steps := []domain.ProgressStep{step("audio", "Preparing the audio", domain.StepDone, 20*time.Minute, 40*time.Second, 0, 0)}
|
|
||||||
steps[0].Note = "The audio is 20 min long; the file claimed 1 h 19 min."
|
|
||||||
seedProgress(t, noted, steps, progressNow.Add(-time.Minute), progressNow.Add(-time.Minute))
|
|
||||||
clean := seedUpload(t, userID, "clean", "done", "Synthetic transcript text.")
|
|
||||||
seedProgress(t, clean, []domain.ProgressStep{step("audio", "Preparing the audio", domain.StepDone, 20*time.Minute, 40*time.Second, 0, 0)}, progressNow.Add(-time.Minute), progressNow.Add(-time.Minute))
|
|
||||||
|
|
||||||
assert.Contains(t, statusBody(t, app, noted), `<details class="what-was-done" open>`)
|
|
||||||
assert.NotContains(t, statusBody(t, app, clean), `<details class="what-was-done" open>`, "a clean record stays collapsed")
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -47,9 +47,6 @@ func TestRenderUploadProgressStatesForReview(t *testing.T) {
|
|||||||
step("save", "Saving the transcript", domain.StepDone, time.Minute, time.Second, 0, 0),
|
step("save", "Saving the transcript", domain.StepDone, time.Minute, time.Second, 0, 0),
|
||||||
}
|
}
|
||||||
|
|
||||||
noted := twoEngineRun()
|
|
||||||
noted[1].Note = "The audio is 20 min long; the file claimed 1 h 19 min. If the recording was cut off, upload the complete one."
|
|
||||||
|
|
||||||
cases := []struct {
|
cases := []struct {
|
||||||
name, status, content string
|
name, status, content string
|
||||||
steps []domain.ProgressStep
|
steps []domain.ProgressStep
|
||||||
@@ -61,7 +58,6 @@ func TestRenderUploadProgressStatesForReview(t *testing.T) {
|
|||||||
{"lost", "transcribing", "", twoEngineRun(), 3 * time.Minute, 3 * time.Minute},
|
{"lost", "transcribing", "", twoEngineRun(), 3 * time.Minute, 3 * time.Minute},
|
||||||
{"failed", "failed", "transcription_failed", failed, 2 * time.Minute, 2 * time.Minute},
|
{"failed", "failed", "transcription_failed", failed, 2 * time.Minute, 2 * time.Minute},
|
||||||
{"done", "done", "Synthetic transcript text for the review.", done, time.Minute, time.Minute},
|
{"done", "done", "Synthetic transcript text for the review.", done, time.Minute, time.Minute},
|
||||||
{"noted", "transcribing", "", noted, 10 * time.Second, 4 * time.Second},
|
|
||||||
{"queued", "queued", "", nil, 0, 0},
|
{"queued", "queued", "", nil, 0, 0},
|
||||||
}
|
}
|
||||||
for _, c := range cases {
|
for _, c := range cases {
|
||||||
|
|||||||
@@ -62,11 +62,11 @@ func TestUploadStatus_ASpotShowsTheWordsJustBeforeIt(t *testing.T) {
|
|||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript.")
|
id := seedUpload(t, userID, "m", "done", "Synthetic transcript.")
|
||||||
setReconciliation(t, id, `{"status":"reconciled","base":"a","alt":"b","reconciler":"c","disputes":1,"changed":1,
|
setReconciliation(t, id, `{"status":"reconciled","base":"a","alt":"b","reconciler":"c","disputes":1,"changed":1,
|
||||||
"contested":[{"seconds":2640,"base":"vanligt","alt":"KORT","used":"alt","unsure":false,"before":"grannen målar ett"}]}`)
|
"contested":[{"seconds":2640,"base":"special","alt":"CAMT","used":"alt","unsure":false,"before":"vi skickar in ett"}]}`)
|
||||||
|
|
||||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||||
|
|
||||||
assert.Contains(t, b, "grannen målar ett", "the words before the spot are how a reader finds it in the audio")
|
assert.Contains(t, b, "vi skickar in ett", "the words before the spot are how a reader finds it in the audio")
|
||||||
assert.Contains(t, b, "vanligt")
|
assert.Contains(t, b, "special")
|
||||||
assert.Contains(t, b, "KORT")
|
assert.Contains(t, b, "CAMT")
|
||||||
}
|
}
|
||||||
|
|||||||
+301
-418
File diff suppressed because it is too large
Load Diff
@@ -780,7 +780,6 @@ a.btn, a.btn:visited { color: var(--accent-fg); }
|
|||||||
.step-skipped { opacity: .7; }
|
.step-skipped { opacity: .7; }
|
||||||
.step-pending { opacity: .75; }
|
.step-pending { opacity: .75; }
|
||||||
.step-bar { flex: none; width: calc(100% - 1.6rem); margin: 0 0 0 1.6rem; height: .5rem; accent-color: var(--accent); }
|
.step-bar { flex: none; width: calc(100% - 1.6rem); margin: 0 0 0 1.6rem; height: .5rem; accent-color: var(--accent); }
|
||||||
.step-note { flex: none; width: calc(100% - 1.6rem); margin-left: 1.6rem; font-weight: 400; color: var(--muted); font-family: system-ui, -apple-system, sans-serif; font-size: .82rem; }
|
|
||||||
.step-count { flex: none; width: calc(100% - 1.6rem); margin-left: 1.6rem; font-weight: 400; color: var(--muted); }
|
.step-count { flex: none; width: calc(100% - 1.6rem); margin-left: 1.6rem; font-weight: 400; color: var(--muted); }
|
||||||
.step-running .step-mark-warn { color: var(--err-fg); }
|
.step-running .step-mark-warn { color: var(--err-fg); }
|
||||||
.spinner { display: inline-block; vertical-align: -.1em; width: .8rem; height: .8rem; box-sizing: border-box; border: 2px solid var(--line); border-top-color: var(--accent); border-radius: 50%; animation: tapir-spin 1s linear infinite; }
|
.spinner { display: inline-block; vertical-align: -.1em; width: .8rem; height: .8rem; box-sizing: border-box; border: 2px solid var(--line); border-top-color: var(--accent); border-radius: 50%; animation: tapir-spin 1s linear infinite; }
|
||||||
|
|||||||
@@ -107,12 +107,9 @@ templ flashBanner(code string) {
|
|||||||
// #summary-list region; a non-HTMX request renders the whole page. flash carries
|
// #summary-list region; a non-HTMX request renders the whole page. flash carries
|
||||||
// a one-shot notification (e.g. "connected", "registered") surfaced on arrival
|
// a one-shot notification (e.g. "connected", "registered") surfaced on arrival
|
||||||
// after a POST→redirect.
|
// after a POST→redirect.
|
||||||
templ ListPage(b listBuckets, f Filter, stats PipelineStats, flash string, hasConnected bool, channels []string, autoSummarize bool, uploadsEnabled bool) {
|
templ ListPage(b listBuckets, f Filter, stats PipelineStats, flash string, hasConnected bool, channels []string, autoSummarize bool) {
|
||||||
@Layout("Tapir — Summaries") {
|
@Layout("Tapir — Summaries") {
|
||||||
@flashBanner(flash)
|
@flashBanner(flash)
|
||||||
if uploadsEnabled {
|
|
||||||
<p class="muted"><a href="/uploads">Recordings</a>: upload audio and get a private transcript.</p>
|
|
||||||
}
|
|
||||||
if hasConnected {
|
if hasConnected {
|
||||||
@pasteForm()
|
@pasteForm()
|
||||||
}
|
}
|
||||||
|
|||||||
+403
-413
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user