Compare commits
177
Commits
@@ -121,6 +121,17 @@ The repo is **green and shipping** — last tag `v0.15.0`. `task check` passes (
|
|||||||
Two-engine reconciliation (ADR-034, migration 019, off unless `TAPIR_RECONCILE_BASE_MODEL`):
|
Two-engine reconciliation (ADR-034, migration 019, off unless `TAPIR_RECONCILE_BASE_MODEL`):
|
||||||
`internal/reconcile` (word diff, disputes, a local-model adjudicator) and
|
`internal/reconcile` (word diff, disputes, a local-model adjudicator) and
|
||||||
`transcribe.Ensemble`; every model must be a local alias, enforced at start-up.
|
`transcribe.Ensemble`; every model must be a local alias, enforced at start-up.
|
||||||
|
Progress and logging (ADR-035, migration 020): `internal/progress.Tracker` is the ONE writer of a
|
||||||
|
job's phases (saved to the row for the page, logged by the same call); engines and the reconciler
|
||||||
|
report through `progress.From(ctx)`, so `uploads.Transcriber` is unchanged, and a transcriber may
|
||||||
|
implement `uploads.Planner` to declare its own phases. The page tells running / quiet / lost from
|
||||||
|
two timestamps (`alive_at` heartbeat vs `progress_at`) and never shows a percentage that does not
|
||||||
|
come from real counts. **Nothing logged may carry transcript text, words from it, or the spool path.**
|
||||||
|
Errors reach a log, a row or a page as `errkind.Of(err)` (a kind such as `timeout` or `http_502`),
|
||||||
|
never `err.Error()`: error text can hold a provider's response body, the spool path or transcript
|
||||||
|
words (ADR-035 decision 8). Local-only for anything that carries transcript text: `config.IsLocalModel`.
|
||||||
|
`TAPIR_SHOT_DIR=… go test -run TestRenderUploadProgressStatesForReview ./internal/web` renders every
|
||||||
|
state for a screenshot review.
|
||||||
**Lint caveat:** `task check` does not fail on lint (#41: the Taskfile `|| echo` swallows
|
**Lint caveat:** `task check` does not fail on lint (#41: the Taskfile `|| echo` swallows
|
||||||
findings) — run `golangci-lint run ./...` yourself until that is fixed.
|
findings) — run `golangci-lint run ./...` yourself until that is fixed.
|
||||||
- `cmd/tapir` subcommands: `list`, `show`, `auth` (interactive host-side OAuth), `run` (batch
|
- `cmd/tapir` subcommands: `list`, `show`, `auth` (interactive host-side OAuth), `run` (batch
|
||||||
|
|||||||
+143
-5
@@ -1461,8 +1461,9 @@ an hour of Swedish *was* an unknown here; it is now measured, see decision 8.)
|
|||||||
- **The opening model.** The offered list leads with the summarizer default `koala/phi4-mini`,
|
- **The opening model.** The offered list leads with the summarizer default `koala/phi4-mini`,
|
||||||
whose 8,192-token context returns HTTP 400 on an hour of speech (~13,000 tokens). Chat over
|
whose 8,192-token context returns HTTP 400 on an hour of speech (~13,000 tokens). Chat over
|
||||||
an upload opens on `TAPIR_UPLOAD_CHAT_MODEL` (default `iguana/gemma4-26b`); the switcher
|
an upload opens on `TAPIR_UPLOAD_CHAT_MODEL` (default `iguana/gemma4-26b`); the switcher
|
||||||
still lists every model, and picking `phi4-mini` on a long recording fails loudly and
|
lists **local models only** (amended by ADR-035 after the adversarial review found it listed
|
||||||
inline. A per-model budget is not built.
|
the cloud model too: the whole private transcript goes to the one chosen), and picking
|
||||||
|
`phi4-mini` on a long recording fails loudly and inline. A per-model budget is not built.
|
||||||
Past ~90,000 chars (≈100 minutes) no offered model can answer by stuffing; see triggers.
|
Past ~90,000 chars (≈100 minutes) no offered model can answer by stuffing; see triggers.
|
||||||
|
|
||||||
9. **Not decided here.** Rich `AVSNITT`/`ALTERNATIV` analysis (gated on #30 — `analyze_srt.py`
|
9. **Not decided here.** Rich `AVSNITT`/`ALTERNATIV` analysis (gated on #30 — `analyze_srt.py`
|
||||||
@@ -1513,13 +1514,20 @@ deferred. The scope cut — chat first, no rich analysis, no video — is what k
|
|||||||
ADR-016 window, it was the interesting thing, not the useful one — remove it.
|
ADR-016 window, it was the interesting thing, not the useful one — remove it.
|
||||||
|
|
||||||
**Open items this ADR creates.**
|
**Open items this ADR creates.**
|
||||||
- **DECISION NEEDED — ffmpeg in the runtime image.** *Options:* (A) static ffmpeg/ffprobe copied
|
- **DECIDED (Mathias, 2026-10-02): option A.** Static ffmpeg and ffprobe 7.0.2 are copied into the
|
||||||
|
distroless image, pinned by version and sha256 (Dockerfile in this PR; #34). Measured at that time:
|
||||||
|
**about 160 MB** added, not the 70 MB first estimated (80 MB per binary); the build is **GPLv3**;
|
||||||
|
the checksum proves the bytes reviewed, not the publisher; upstream's `releases/` URL moves on, so
|
||||||
|
vendoring the verified tarball is the durable fix (not done). Uploads stay off until the deployment
|
||||||
|
sets `TAPIR_UPLOADS_ENABLED`. *The question as it was put:* *Options:* (A) static ffmpeg/ffprobe copied
|
||||||
into distroless **(recommended — the only option that satisfies "accept m4a/mp3/opus/mp4,
|
into distroless **(recommended — the only option that satisfies "accept m4a/mp3/opus/mp4,
|
||||||
validate duration, reject video"; pure-Go decoders cover wav/mp3 only)**; (B) accept WAV
|
validate duration, reject video"; pure-Go decoders cover wav/mp3 only)**; (B) accept WAV
|
||||||
only, convert client-side (fails "I just want to upload it"); (C) a transcode sidecar
|
only, convert client-side (fails "I just want to upload it"); (C) a transcode sidecar
|
||||||
(second deployable, ADR-018's reasoning against). *Blocks:* the Dockerfile slice, not the
|
(second deployable, ADR-018's reasoning against). *Blocks:* the Dockerfile slice, not the
|
||||||
tests, which use the port.
|
tests, which use the port.
|
||||||
- **DECISION NEEDED — berget fallback for private audio.** *Options:* (A) enabled by default as
|
- **DECIDED (Mathias, 2026-10-02): option A, enabled by default (#35).** Scope: the audio
|
||||||
|
transcription fallback only. Reconciliation text and chat over a recording never go to a cloud
|
||||||
|
model (ADR-034, ADR-035). *The question as it was put:* *Options:* (A) enabled by default as
|
||||||
#33 specifies, with the serving endpoint shown **(recommended, matches the issue)**; (B)
|
#33 specifies, with the serving endpoint shown **(recommended, matches the issue)**; (B)
|
||||||
disabled by default, a failed iguana job fails honestly and the user retries. *Blocks:*
|
disabled by default, a failed iguana job fails honestly and the user retries. *Blocks:*
|
||||||
the production default only; the adapter ships either way.
|
the production default only; the adapter ships either way.
|
||||||
@@ -1596,7 +1604,8 @@ on the slow numbers and is reopened: 31B scored 97% against 92%.
|
|||||||
timeout at a time. Both engines fail → ADR-033's fallback path.
|
timeout at a time. Both engines fail → ADR-033's fallback path.
|
||||||
|
|
||||||
6. **What is stored and shown.** `content` is the merged text. `segments` stay the base engine's timed
|
6. **What is stored and shown.** `content` is the merged text. `segments` stay the base engine's timed
|
||||||
segments, untouched: `content` is authoritative for chat, `segments` only for time. `endpoint`
|
segments, untouched (**amended by ADR-035: they now carry the merged words, timing stays the
|
||||||
|
base's**): `content` is authoritative for chat, `segments` only for time. `endpoint`
|
||||||
lists every contributor and the reconciler. A new `reconciliation` column (migration 019, same
|
lists every contributor and the reconciler. A new `reconciliation` column (migration 019, same
|
||||||
RLS-scoped table) records the status, counts, and the **uncertain and changed spots with a minute
|
RLS-scoped table) records the status, counts, and the **uncertain and changed spots with a minute
|
||||||
mark and both readings**, so the user can check them against the audio. The transcript page says in
|
mark and both readings**, so the user can check them against the audio. The transcript page says in
|
||||||
@@ -1641,6 +1650,19 @@ recording.
|
|||||||
~30 disputes where the reconciled text differs from pianissimo alone. Keep the feature only if the
|
~30 disputes where the reconciled text differs from pianissimo alone. Keep the feature only if the
|
||||||
reconciled reading is right at least 85% of the time **and** no worse than the better single engine.
|
reconciled reading is right at least 85% of the time **and** no worse than the better single engine.
|
||||||
If not, switch it off (config) and keep the single-engine path.
|
If not, switch it off (config) and keep the single-engine path.
|
||||||
|
**Decision (Mathias, 2026-10-02), on the first result below: keep it as an opt-in feature**, reading
|
||||||
|
the bar as 95% of decided items, with the second condition to be answered on the next real run.
|
||||||
|
For deployment the reconciler is `iguana/gemma4-31b` (97% against the speaker's answers, about 22
|
||||||
|
minutes for the 63.6-minute recording with Low Power Mode off). A cloud reconciler was raised and is
|
||||||
|
**not adopted**: it would send private transcript text to a cloud API on every upload; recorded as
|
||||||
|
an evaluation question in #45.
|
||||||
|
**First result, 2026-10-02, before merge** (`docs/research/reconcile-blind-check-2026-10-02.md`): 30
|
||||||
|
random replacement spots on the real 63.6-minute recording, judged blind by Opus from text (not by the
|
||||||
|
speaker, who delegated it): 21 right, 1 worse than pianissimo, 3 neither reading right, 5 undecidable.
|
||||||
|
That is 95% of decided items but 70% of all 30; the bar did not say how ties count, so it is **met on
|
||||||
|
one reading and missed on the other**, and the speaker has not ruled. The second condition, no worse
|
||||||
|
than the better single engine, is **not answered**: kept-base spots are not stored, so kb-whisper alone
|
||||||
|
cannot be compared. Left `Proposed`; the feature stays off by default.
|
||||||
- Reconcile time above 25 minutes per hour of audio: drop to a smaller adjudicator or skip more
|
- Reconcile time above 25 minutes per hour of audio: drop to a smaller adjudicator or skip more
|
||||||
disputes automatically.
|
disputes automatically.
|
||||||
- A different recording or speaker in which the engines agree far less (or far more): re-measure before
|
- A different recording or speaker in which the engines agree far less (or far more): re-measure before
|
||||||
@@ -1652,6 +1674,122 @@ unchanged.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## ADR-035 — Show an upload's progress as real phases, log the same transitions, and keep every judged spot
|
||||||
|
|
||||||
|
**Status:** Proposed (2026-10-02). Decider: Mathias (pending). Amends ADR-034 decision 6 (segments).
|
||||||
|
Closes #42, #43 and #44.
|
||||||
|
|
||||||
|
**Context (measured, not assumed).** The first real run (a 63.6-minute recording, 21 min 49 s) exposed
|
||||||
|
three gaps. (1) The page said only `queued`/`transcribing`/`done`; during 22 minutes nothing told the
|
||||||
|
user whether the job was working, hung or dead, and after a crash the row would say `transcribing`
|
||||||
|
until the next start. (2) The server log had no line about the job between start-up and the final
|
||||||
|
line, so an operator saw it only from outside (spool files, iguana CPU). (3) Evaluating the result
|
||||||
|
showed that spots the model *kept* as the base are counted but not stored (so ADR-034's "no worse than
|
||||||
|
the better single engine" cannot be measured after the fact, and the audio is not retained), insertions
|
||||||
|
and deletions cannot be placed without context, and the saved `segments` still carry the base engine's
|
||||||
|
wording while only `content` is merged.
|
||||||
|
|
||||||
|
**Decision.**
|
||||||
|
|
||||||
|
1. **Phases are data on the row.** A new `progress` JSONB column (migration 020, same RLS-scoped table)
|
||||||
|
holds an ordered list of steps `{key, label, state, started_at, finished_at, done, total}` with state
|
||||||
|
`pending|running|done|failed|skipped`. The plan is declared up front so the page shows what is coming:
|
||||||
|
received, preparing audio, then the transcriber's own steps (two-engine: engine A, engine B,
|
||||||
|
comparing, deciding n of N; single: transcribing), then saving.
|
||||||
|
2. **One source for the page and the log.** An `internal/progress` Tracker is the only writer. Every
|
||||||
|
transition is saved to the row and logged by the same call, so the UI and the log cannot disagree.
|
||||||
|
Engines and the reconciler report through a context-carried `Reporter` (no-op when absent), so the
|
||||||
|
`Transcriber` interface does not change.
|
||||||
|
3. **Alive is not the same as progressing, and the page says which it knows.** A heartbeat goroutine,
|
||||||
|
independent of any engine call, stamps `alive_at` every 10 s; `progress_at` moves only on a real
|
||||||
|
transition or advance. The page derives: *running*; *quiet* (the deciding step has not advanced for
|
||||||
|
5 minutes, so the model is probably waiting or slow); *lost* (`alive_at` older than 45 s, so the
|
||||||
|
service stopped reporting and may have restarted or crashed). The two engine calls are single blocking
|
||||||
|
requests, so inside them the page shows elapsed time only: **no percentage is ever shown that does not
|
||||||
|
come from real counts.** Only the deciding step has a bar.
|
||||||
|
4. **The page.** A server-rendered step list in tapir's existing tokens and mono font, swapped in place
|
||||||
|
by the existing HTMX poll (every 2 s while unfinished, none once done or failed). Finished steps
|
||||||
|
collapse to one line with their duration; the running step has a CSS-only spinner that respects
|
||||||
|
`prefers-reduced-motion`; a failed step and a lost service are visibly different from running.
|
||||||
|
No new JavaScript, framework or stylesheet.
|
||||||
|
5. **Logs.** One INFO line per phase transition and engine call (upload id, endpoint, elapsed), the
|
||||||
|
reconcile counts, `adjudicated n/N` at most once a minute, a WARN when an engine soft-fails or the
|
||||||
|
breaker trips, and a "still in phase X" line every 5 minutes. **Ids, counts, durations, endpoints and
|
||||||
|
error kinds only: never transcript text, disputed words or the filename**, asserted by a test.
|
||||||
|
6. **Every judged spot is stored.** `reconciliation.contested` keeps each model-decided spot with its
|
||||||
|
verdict (changed, kept base, uncertain) and a few words of left context, bounded at 2,000. The page
|
||||||
|
still lists only changed and uncertain spots. This makes ADR-034's second condition measurable on the
|
||||||
|
next real run.
|
||||||
|
7. **Segments carry the merged words.** The reconciled words replace the base's in the saved `segments`;
|
||||||
|
timing stays the base's. If the segments cannot be aligned to the merged text (a word wrapped across
|
||||||
|
a segment edge), they stay the base's and the result says so in the log. This replaces ADR-034
|
||||||
|
decision 6's "segments stay untouched".
|
||||||
|
|
||||||
|
8. **Hardening from the adversarial review** (`skills:pr-gatekeeper`, head `7029dd1`: BLOCK; recorded on
|
||||||
|
PR #39). Each finding was reproduced by a failing test first.
|
||||||
|
- **Chat over a recording offers and accepts local models only** (it listed `berget/mistral-small`);
|
||||||
|
`TAPIR_UPLOAD_CHAT_MODEL` must be local at start-up; with no local model, nothing is sent.
|
||||||
|
- **A model alias counts as local only if it is one plain name after `koala/` or `iguana/`**
|
||||||
|
(`koala/../berget/x`, `koala//x`, `koala/x?host=…` were accepted by a bare prefix check).
|
||||||
|
- **An error reaches a log line, the row or the page as a kind** (`timeout`, `canceled`, `http_502`,
|
||||||
|
`network`, `error`), never its text, which can hold a provider's response body, the spool path or
|
||||||
|
a word from the transcript (`internal/errkind`). The stored reconcile reason and an engine's
|
||||||
|
failure reason are kinds; the page no longer shows provider text.
|
||||||
|
- **Every local engine call has its own deadline** (`TAPIR_TRANSCRIBE_ENGINE_TIMEOUT`, default 20
|
||||||
|
minutes: two wedged engines in sequence then leave 20 of the job's 60 minutes for a fallback),
|
||||||
|
so a wedged engine fails while the job lives and the fallback can run; the job's own
|
||||||
|
deadline or a cancel still never reaches the cloud.
|
||||||
|
- **The duration limit is checked against what is converted**, not the file's header: ffmpeg is
|
||||||
|
capped at the limit plus a minute, a longer result fails the job (`audio_too_long`), and a result
|
||||||
|
under 80% of the claimed length and more than two minutes short is **transcribed and noted** on
|
||||||
|
the audio step ("the audio is 20 min long; the file claimed 1 h 19 min"), not refused. The second
|
||||||
|
adversarial review showed a complete variable-bitrate MP3 whose header over-estimated its length
|
||||||
|
by 4x was wrongly refused by the first version of this check; a note keeps a truncated file
|
||||||
|
visible without turning away real recordings.
|
||||||
|
- **The queue is bounded**, 20 waiting in total and 3 per user, and its room is checked before the
|
||||||
|
upload is spooled (not after 200 MB and an ffprobe run); a full queue is a 429 with a typed reason,
|
||||||
|
never a stuck row or a server error.
|
||||||
|
- **The configured upload chat model is callable.** It is added to the shared chat service's model
|
||||||
|
list when uploads are on and it is local, because the service re-maps a model it does not offer
|
||||||
|
(a valid `iguana/gemma4-31b` was being silently swapped for `koala/phi4-mini`). The video chat's own switcher
|
||||||
|
lists that local model too while uploads are on.
|
||||||
|
- **A client that hangs up mid-upload is not an error**: the handler logs INFO and writes nothing
|
||||||
|
(rendering the error page ran a database read on a cancelled context and logged a false ERROR). The
|
||||||
|
400 "cut off" answer is keyed on a marker for errors from *reading the client's body*
|
||||||
|
(`uploads.ErrClientRead`), not on `io.ErrUnexpectedEOF`, which is also what a database drop looks like.
|
||||||
|
- **A note is not hidden**: a finished upload's record opens by default when any step carries a note.
|
||||||
|
- **Bad input is a 4xx**: a non-UUID id is a 404, a malformed or cut-off multipart body is a 400
|
||||||
|
whose text is not logged, and a title with control bytes, bidirectional controls or invalid UTF-8 is
|
||||||
|
cleaned (emoji joiners, Swedish letters and variation selectors are kept), not a 500.
|
||||||
|
- **The image was built and inspected** (buildah on koala, 2026-10-02): the first build FAILED
|
||||||
|
(`tar: xz: Cannot exec`, the golang image has no xz); fixed, rebuilt at 180 MB, ffmpeg and ffprobe
|
||||||
|
run as `nonroot` in distroless. CI does not build the image on a pull request
|
||||||
|
(`if: github.event_name != 'pull_request'`), so the first CI build of it is the merge to main.
|
||||||
|
Not done, filed as follow-ups: a vendored ffmpeg tarball, CSRF cookie review, ffprobe CPU and
|
||||||
|
stream-count limits, mov external references, `alive_at` clock skew, a hung database blocking the
|
||||||
|
tracker, an unbounded gateway response body, a user-facing notice for a recording that is partial but
|
||||||
|
inside the tolerance.
|
||||||
|
|
||||||
|
**Alternatives rejected.** Server-sent events or a websocket: a long-lived connection through the
|
||||||
|
ingress, with buffering to get wrong, for a page that changes every few seconds at best; polling is what
|
||||||
|
the page already does. A percentage from the engine calls: there is no data behind it. A separate
|
||||||
|
progress table: progress belongs to the row and inherits its RLS. Logging only through the UI: an
|
||||||
|
operator has no page.
|
||||||
|
|
||||||
|
**Consequences.** Positive: running, quiet, lost and failed are distinguishable by the user and the
|
||||||
|
operator; the next blind check can answer ADR-034's second condition. Negative / watch: two more writes
|
||||||
|
per transition and one per 10 s per running job (one job at a time, so negligible); a heartbeat proves
|
||||||
|
the process is alive, not that an engine is making progress inside one request, which only the job
|
||||||
|
timeout (60 min) bounds. The stored spot list grows to a few hundred small objects per hour of audio.
|
||||||
|
|
||||||
|
**Validation.** A fake engine held open past the thresholds must change the page (quiet, lost); a log
|
||||||
|
capture on a synthetic transcript must contain none of its words; a screenshot review of the running,
|
||||||
|
quiet, lost, failed and done states against the existing pages (the visual claims tests cannot assert).
|
||||||
|
Revisit if the heartbeat write ever shows in the database's load, or if a longer recording cap makes
|
||||||
|
five-minute quiet detection too coarse.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Rejected alternatives
|
## Rejected alternatives
|
||||||
|
|
||||||
Approaches considered during the 2026-06-02 planning + grill session and **deliberately not
|
Approaches considered during the 2026-06-02 planning + grill session and **deliberately not
|
||||||
|
|||||||
+40
@@ -16,12 +16,52 @@ COPY . .
|
|||||||
# no libc. Trim symbols/DWARF to shrink the layer.
|
# no libc. Trim symbols/DWARF to shrink the layer.
|
||||||
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags='-s -w' -o /out/tapir ./cmd/tapir
|
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags='-s -w' -o /out/tapir ./cmd/tapir
|
||||||
|
|
||||||
|
# ── ffmpeg ──────────────────────────────────────────────────────────────────
|
||||||
|
# Audio upload probes files with ffprobe and converts them with ffmpeg (ADR-033,
|
||||||
|
# #34 option A). Static builds, so they run in the distroless image below.
|
||||||
|
#
|
||||||
|
# Pinned by version AND sha256: a changed upstream file fails the build instead of
|
||||||
|
# shipping. The checksum proves these are the bytes that were reviewed (2026-10-02;
|
||||||
|
# the publisher's md5 matched too). It does NOT prove authenticity: the publisher
|
||||||
|
# signs nothing we can check. amd64 only (the cluster is one amd64 node).
|
||||||
|
#
|
||||||
|
# These builds are GPLv3 (--enable-gpl --enable-version3). The service is private and
|
||||||
|
# the image is not distributed; the licence text is copied into the image anyway.
|
||||||
|
#
|
||||||
|
# The release URL moves on when upstream releases a newer version, so the pinned
|
||||||
|
# version is tried in releases/ and then old-releases/. If neither has it the build
|
||||||
|
# fails; vendoring the verified tarball is the durable fix.
|
||||||
|
FROM golang:1.26 AS ffmpeg
|
||||||
|
|
||||||
|
ARG FFMPEG_VERSION=7.0.2
|
||||||
|
ARG FFMPEG_SHA256=abda8d77ce8309141f83ab8edf0596834087c52467f6badf376a6a2a4c87cf67
|
||||||
|
|
||||||
|
# xz is not in the golang image (found by building it: `tar: xz: Cannot exec`).
|
||||||
|
RUN set -eu; \
|
||||||
|
apt-get update; \
|
||||||
|
apt-get install -y --no-install-recommends xz-utils; \
|
||||||
|
rm -rf /var/lib/apt/lists/*; \
|
||||||
|
mkdir -p /out; \
|
||||||
|
for dir in releases old-releases; do \
|
||||||
|
curl -fsSL -o /out/ffmpeg.tar.xz "https://johnvansickle.com/ffmpeg/${dir}/ffmpeg-${FFMPEG_VERSION}-amd64-static.tar.xz" && break; \
|
||||||
|
done; \
|
||||||
|
echo "${FFMPEG_SHA256} /out/ffmpeg.tar.xz" | sha256sum -c -; \
|
||||||
|
tar -xJf /out/ffmpeg.tar.xz -C /out --strip-components=1 --wildcards '*/ffmpeg' '*/ffprobe' '*/GPLv3.txt'; \
|
||||||
|
rm /out/ffmpeg.tar.xz
|
||||||
|
|
||||||
# ── runtime ─────────────────────────────────────────────────────────────────
|
# ── runtime ─────────────────────────────────────────────────────────────────
|
||||||
# distroless static + nonroot: no shell, no package manager, runs as uid 65532.
|
# distroless static + nonroot: no shell, no package manager, runs as uid 65532.
|
||||||
# ca-certificates are bundled, which the OIDC/HTTPS clients need.
|
# ca-certificates are bundled, which the OIDC/HTTPS clients need.
|
||||||
FROM gcr.io/distroless/static-debian12:nonroot
|
FROM gcr.io/distroless/static-debian12:nonroot
|
||||||
|
|
||||||
COPY --from=build /out/tapir /tapir
|
COPY --from=build /out/tapir /tapir
|
||||||
|
COPY --from=ffmpeg /out/ffmpeg /out/ffprobe /usr/local/bin/
|
||||||
|
COPY --from=ffmpeg /out/GPLv3.txt /usr/share/doc/ffmpeg/GPLv3.txt
|
||||||
|
|
||||||
|
# Where the service finds them. Uploads stay OFF until TAPIR_UPLOADS_ENABLED is set
|
||||||
|
# (a deployment decision, not an image one).
|
||||||
|
ENV TAPIR_FFMPEG_BIN=/usr/local/bin/ffmpeg \
|
||||||
|
TAPIR_FFPROBE_BIN=/usr/local/bin/ffprobe
|
||||||
|
|
||||||
EXPOSE 8080
|
EXPOSE 8080
|
||||||
USER nonroot:nonroot
|
USER nonroot:nonroot
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package main
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"mime/multipart"
|
"mime/multipart"
|
||||||
@@ -21,6 +22,7 @@ import (
|
|||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/adapters/store"
|
"git.d-ma.be/mathias/tapir/internal/adapters/store"
|
||||||
"git.d-ma.be/mathias/tapir/internal/config"
|
"git.d-ma.be/mathias/tapir/internal/config"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
"git.d-ma.be/mathias/tapir/internal/web"
|
"git.d-ma.be/mathias/tapir/internal/web"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -156,6 +158,23 @@ func TestE2E_UploadTranscribeAndChat(t *testing.T) {
|
|||||||
assert.Equal(t, "done", status)
|
assert.Equal(t, "done", status)
|
||||||
assert.NotEmpty(t, endpoint)
|
assert.NotEmpty(t, endpoint)
|
||||||
assert.NotEmpty(t, content)
|
assert.NotEmpty(t, content)
|
||||||
|
|
||||||
|
// ADR-035: a finished job's phases are all closed, and it was reporting while it ran.
|
||||||
|
var progJSON string
|
||||||
|
var progressAt, aliveAt *time.Time
|
||||||
|
require.NoError(t, raw.QueryRow(ctx,
|
||||||
|
`SELECT COALESCE(progress::text, '[]'), progress_at, alive_at FROM uploaded_transcripts WHERE user_id = $1`, uid).
|
||||||
|
Scan(&progJSON, &progressAt, &aliveAt))
|
||||||
|
var steps []domain.ProgressStep
|
||||||
|
require.NoError(t, json.Unmarshal([]byte(progJSON), &steps))
|
||||||
|
require.NotEmpty(t, steps, "the job recorded no phases")
|
||||||
|
for _, st := range steps {
|
||||||
|
assert.Contains(t, []domain.StepState{domain.StepDone, domain.StepSkipped}, st.State,
|
||||||
|
"phase %q was left %q after the job finished", st.Key, st.State)
|
||||||
|
}
|
||||||
|
assert.Equal(t, "save", steps[len(steps)-1].Key)
|
||||||
|
assert.NotNil(t, progressAt, "progress_at must have been stamped")
|
||||||
|
t.Logf("phases: %d, heartbeat stamped: %v", len(steps), aliveAt != nil)
|
||||||
entries, err := os.ReadDir(cfg.UploadTmpDir)
|
entries, err := os.ReadDir(cfg.UploadTmpDir)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Empty(t, entries, "the audio must be deleted once transcribed")
|
assert.Empty(t, entries, "the audio must be deleted once transcribed")
|
||||||
|
|||||||
@@ -134,9 +134,30 @@ func chatModels(cfg config.Config) []string {
|
|||||||
add(cfg.SummarizerModel)
|
add(cfg.SummarizerModel)
|
||||||
add(cfg.FallbackModel)
|
add(cfg.FallbackModel)
|
||||||
add(cfg.CloudFallbackModel)
|
add(cfg.CloudFallbackModel)
|
||||||
|
// The model chat over an uploaded recording opens on must be one the shared service can
|
||||||
|
// call: it re-maps a model it does not offer, which silently swapped a configured 31B for
|
||||||
|
// phi4-mini, whose context cannot hold an hour of speech. Local only, and only when
|
||||||
|
// uploads are on, so the video chat is unchanged otherwise (second adversarial review).
|
||||||
|
if cfg.UploadsEnabled && config.IsLocalModel(cfg.UploadChatModel) {
|
||||||
|
add(cfg.UploadChatModel)
|
||||||
|
}
|
||||||
return models
|
return models
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// uploadChatModels is the chat switcher for a PRIVATE uploaded recording: the same
|
||||||
|
// ordered set as chatModels, local models only. The whole transcript (up to 90,000
|
||||||
|
// characters) is sent to the one chosen, so a cloud model the video chat offers must not
|
||||||
|
// be offered here; with none local, none is offered (ADR-035 review).
|
||||||
|
func uploadChatModels(cfg config.Config) []string {
|
||||||
|
var local []string
|
||||||
|
for _, m := range chatModels(cfg) {
|
||||||
|
if config.IsLocalModel(m) {
|
||||||
|
local = append(local, m)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return local
|
||||||
|
}
|
||||||
|
|
||||||
// buildChat wires the per-video chat service (ADR-027): a Completer factory over
|
// buildChat wires the per-video chat service (ADR-027): a Completer factory over
|
||||||
// the SAME LiteLLM gateway the summarizer uses (a different alias per model, not a
|
// the SAME LiteLLM gateway the summarizer uses (a different alias per model, not a
|
||||||
// second client config) and the same transcript-truncation budget. It returns nil
|
// second client config) and the same transcript-truncation budget. It returns nil
|
||||||
|
|||||||
+16
-4
@@ -55,8 +55,8 @@ func buildUploads(ctx context.Context, cfg config.Config, st uploadsStore, log *
|
|||||||
log.Warn("uploads: marked interrupted by a restart (audio is not retained; they must be re-uploaded)", "count", n)
|
log.Warn("uploads: marked interrupted by a restart (audio is not retained; they must be re-uploaded)", "count", n)
|
||||||
}
|
}
|
||||||
|
|
||||||
worker := uploads.NewWorker(st, &audio.FFmpeg{Bin: cfg.FFmpegBin}, buildTranscriber(cfg),
|
worker := uploads.NewWorker(st, newConverter(cfg), buildTranscriber(cfg),
|
||||||
cfg.UploadTmpDir, cfg.TranscribeTimeout)
|
cfg.UploadTmpDir, cfg.TranscribeTimeout, uploads.WithMaxDuration(cfg.UploadMaxDuration))
|
||||||
intake := uploads.NewIntake(st, &audio.FFprobe{Bin: cfg.FFprobeBin}, worker,
|
intake := uploads.NewIntake(st, &audio.FFprobe{Bin: cfg.FFprobeBin}, worker,
|
||||||
uploadLimits(cfg), cfg.UploadTmpDir)
|
uploadLimits(cfg), cfg.UploadTmpDir)
|
||||||
return intake, worker, nil
|
return intake, worker, nil
|
||||||
@@ -91,6 +91,13 @@ func wireUploads(app *web.App, cfg config.Config, in *uploads.Intake) {
|
|||||||
app.UploadLimits = uploadLimits(cfg)
|
app.UploadLimits = uploadLimits(cfg)
|
||||||
app.UploadChatMaxChars = cfg.MaxUploadedTranscriptChars
|
app.UploadChatMaxChars = cfg.MaxUploadedTranscriptChars
|
||||||
app.UploadChatModel = cfg.UploadChatModel
|
app.UploadChatModel = cfg.UploadChatModel
|
||||||
|
app.UploadChatModels = uploadChatModels(cfg)
|
||||||
|
}
|
||||||
|
|
||||||
|
// newConverter is the ffmpeg adapter capped at the configured maximum duration, so a file
|
||||||
|
// whose header lies about its length cannot become hours of WAV (ADR-035 review).
|
||||||
|
func newConverter(cfg config.Config) *audio.FFmpeg {
|
||||||
|
return &audio.FFmpeg{Bin: cfg.FFmpegBin, MaxSeconds: int(cfg.UploadMaxDuration.Seconds())}
|
||||||
}
|
}
|
||||||
|
|
||||||
// adjudicatorTimeout bounds one adjudication call. A call takes ~2 s on gemma4-26b and ~10 s
|
// adjudicatorTimeout bounds one adjudication call. A call takes ~2 s on gemma4-26b and ~10 s
|
||||||
@@ -107,7 +114,12 @@ func buildTranscriber(cfg config.Config) uploads.Transcriber {
|
|||||||
// No overall client timeout: a 90 minute recording takes minutes. The deadline is
|
// No overall client timeout: a 90 minute recording takes minutes. The deadline is
|
||||||
// the worker's per-job context.
|
// the worker's per-job context.
|
||||||
client := &http.Client{}
|
client := &http.Client{}
|
||||||
primary := transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, cfg.TranscribeModel, client)
|
// Each LOCAL engine call gets its own deadline: a wedged engine does not answer, and
|
||||||
|
// without one it used the whole job before the fallback could run.
|
||||||
|
local := func(model string) transcribe.Backend {
|
||||||
|
return transcribe.WithTimeout(transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, model, client), cfg.TranscribeEngineTimeout)
|
||||||
|
}
|
||||||
|
primary := local(cfg.TranscribeModel)
|
||||||
var fallback transcribe.Backend
|
var fallback transcribe.Backend
|
||||||
if cfg.TranscribeFallbackModel != "" && cfg.TranscribeFallbackModel != cfg.TranscribeModel {
|
if cfg.TranscribeFallbackModel != "" && cfg.TranscribeFallbackModel != cfg.TranscribeModel {
|
||||||
fallback = transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, cfg.TranscribeFallbackModel, client)
|
fallback = transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, cfg.TranscribeFallbackModel, client)
|
||||||
@@ -116,7 +128,7 @@ func buildTranscriber(cfg config.Config) uploads.Transcriber {
|
|||||||
return transcribe.NewChain(primary, fallback)
|
return transcribe.NewChain(primary, fallback)
|
||||||
}
|
}
|
||||||
|
|
||||||
base := transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, cfg.ReconcileBaseModel, client)
|
base := local(cfg.ReconcileBaseModel)
|
||||||
// Temperature 0 and a short reply: the adjudicator's accuracy was measured that way.
|
// Temperature 0 and a short reply: the adjudicator's accuracy was measured that way.
|
||||||
chat := llm.New(cfg.GatewayURL, cfg.GatewayKey, cfg.ReconcileModel, adjudicatorTimeout,
|
chat := llm.New(cfg.GatewayURL, cfg.GatewayKey, cfg.ReconcileModel, adjudicatorTimeout,
|
||||||
llm.WithTemperature(0), llm.WithMaxTokens(200))
|
llm.WithTemperature(0), llm.WithMaxTokens(200))
|
||||||
|
|||||||
@@ -0,0 +1,85 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"reflect"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The chat over an uploaded recording sends the whole private transcript to the model
|
||||||
|
// the user picks, so the switcher for it may offer local models only (ADR-035 review).
|
||||||
|
|
||||||
|
func TestUploadChatModels_NeverOffersACloudModelButKeepsTheLocalOrder(t *testing.T) {
|
||||||
|
var cfg config.Config
|
||||||
|
cfg.SummarizerModel = "koala/phi4-mini"
|
||||||
|
cfg.FallbackModel = "iguana/gemma4-26b"
|
||||||
|
cfg.CloudFallbackModel = "berget/mistral-small"
|
||||||
|
|
||||||
|
got := uploadChatModels(cfg)
|
||||||
|
|
||||||
|
want := []string{"koala/phi4-mini", "iguana/gemma4-26b"}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Errorf("uploadChatModels = %v, want %v: the cloud model that the video chat offers must not be offered for a private recording", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadChatModels_ACloudPrimaryIsDroppedToo(t *testing.T) {
|
||||||
|
var cfg config.Config
|
||||||
|
cfg.SummarizerModel = "berget/mistral-small"
|
||||||
|
cfg.FallbackModel = "iguana/gemma4-26b"
|
||||||
|
|
||||||
|
got := uploadChatModels(cfg)
|
||||||
|
|
||||||
|
if !reflect.DeepEqual(got, []string{"iguana/gemma4-26b"}) {
|
||||||
|
t.Errorf("uploadChatModels = %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadChatModels_NoLocalModelMeansNoneOfferedNotACloudDefault(t *testing.T) {
|
||||||
|
var cfg config.Config
|
||||||
|
cfg.SummarizerModel = "berget/mistral-small"
|
||||||
|
|
||||||
|
if got := uploadChatModels(cfg); len(got) != 0 {
|
||||||
|
t.Errorf("uploadChatModels = %v, want none", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A valid local TAPIR_UPLOAD_CHAT_MODEL that is not in the video chat's chain used to be
|
||||||
|
// accepted by config and then silently ignored: the shared chat service re-maps a model it
|
||||||
|
// does not offer, so the user got koala/phi4-mini, whose 8,192-token context cannot hold an
|
||||||
|
// hour of speech (second adversarial review).
|
||||||
|
|
||||||
|
func TestChatModels_AnUploadChatModelOutsideTheChainIsOfferedWhenUploadsAreOn(t *testing.T) {
|
||||||
|
var cfg config.Config
|
||||||
|
cfg.SummarizerModel = "koala/phi4-mini"
|
||||||
|
cfg.FallbackModel = "iguana/gemma4-26b"
|
||||||
|
cfg.UploadsEnabled = true
|
||||||
|
cfg.UploadChatModel = "iguana/gemma4-31b"
|
||||||
|
|
||||||
|
service := chatModels(cfg)
|
||||||
|
upload := uploadChatModels(cfg)
|
||||||
|
|
||||||
|
if !reflect.DeepEqual(service, []string{"koala/phi4-mini", "iguana/gemma4-26b", "iguana/gemma4-31b"}) {
|
||||||
|
t.Errorf("chatModels = %v: the shared service must be able to call the configured upload model", service)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(upload, service) {
|
||||||
|
t.Errorf("uploadChatModels = %v, want %v", upload, service)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestChatModels_NothingIsAddedWhenUploadsAreOffOrTheModelIsNotLocal(t *testing.T) {
|
||||||
|
var cfg config.Config
|
||||||
|
cfg.SummarizerModel = "koala/phi4-mini"
|
||||||
|
cfg.UploadChatModel = "iguana/gemma4-31b"
|
||||||
|
|
||||||
|
if got := chatModels(cfg); !reflect.DeepEqual(got, []string{"koala/phi4-mini"}) {
|
||||||
|
t.Errorf("uploads off: chatModels = %v, want the video chain unchanged", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg.UploadsEnabled = true
|
||||||
|
cfg.UploadChatModel = "berget/mistral-small"
|
||||||
|
if got := chatModels(cfg); !reflect.DeepEqual(got, []string{"koala/phi4-mini"}) {
|
||||||
|
t.Errorf("a cloud upload chat model was added to the service: %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The converter is told the configured maximum, so a file whose header lies about its
|
||||||
|
// length cannot become hours of WAV (adversarial review F2).
|
||||||
|
|
||||||
|
func TestNewConverter_IsCappedAtTheConfiguredMaximumDuration(t *testing.T) {
|
||||||
|
var cfg config.Config
|
||||||
|
cfg.FFmpegBin = "/usr/local/bin/ffmpeg"
|
||||||
|
cfg.UploadMaxDuration = 90 * time.Minute
|
||||||
|
|
||||||
|
c := newConverter(cfg)
|
||||||
|
|
||||||
|
if c.Bin != "/usr/local/bin/ffmpeg" {
|
||||||
|
t.Errorf("Bin = %q", c.Bin)
|
||||||
|
}
|
||||||
|
if c.MaxSeconds != 5400 {
|
||||||
|
t.Errorf("MaxSeconds = %v, want 5400", c.MaxSeconds)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -6,6 +6,7 @@ import (
|
|||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
@@ -19,15 +20,17 @@ import (
|
|||||||
// fakeUploadsStore satisfies the store surface buildUploads needs, and counts the
|
// fakeUploadsStore satisfies the store surface buildUploads needs, and counts the
|
||||||
// start-up sweep.
|
// start-up sweep.
|
||||||
type fakeUploadsStore struct {
|
type fakeUploadsStore struct {
|
||||||
sweeps int
|
sweeps int
|
||||||
sweepErr error
|
sweepErr error
|
||||||
|
failedKinds []domain.UploadErrorKind
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *fakeUploadsStore) CreateUpload(context.Context, string, string, bool) (string, error) {
|
func (f *fakeUploadsStore) CreateUpload(context.Context, string, string, bool) (string, error) {
|
||||||
return "id", nil
|
return "id", nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *fakeUploadsStore) FailUpload(context.Context, string, string, domain.UploadErrorKind) error {
|
func (f *fakeUploadsStore) FailUpload(_ context.Context, _, _ string, k domain.UploadErrorKind) error {
|
||||||
|
f.failedKinds = append(f.failedKinds, k)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -37,6 +40,12 @@ func (f *fakeUploadsStore) CompleteUpload(context.Context, string, string, domai
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (f *fakeUploadsStore) SaveProgress(context.Context, string, string, []domain.ProgressStep, time.Time) error {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeUploadsStore) Heartbeat(context.Context, string, string) error { return nil }
|
||||||
|
|
||||||
func (f *fakeUploadsStore) FailInterruptedUploads(context.Context) (int, error) {
|
func (f *fakeUploadsStore) FailInterruptedUploads(context.Context) (int, error) {
|
||||||
f.sweeps++
|
f.sweeps++
|
||||||
return 2, f.sweepErr
|
return 2, f.sweepErr
|
||||||
|
|||||||
@@ -0,0 +1,78 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The wiring of the per-engine deadline: a gateway whose primary model never answers
|
||||||
|
// must still reach the fallback model inside the job's own deadline (adversarial review
|
||||||
|
// F4). The fake gateway answers per model from the multipart `model` field.
|
||||||
|
|
||||||
|
func TestBuildTranscriber_AWedgedPrimaryReachesTheFallbackBeforeTheJobDeadline(t *testing.T) {
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
_ = r.ParseMultipartForm(1 << 20)
|
||||||
|
if r.FormValue("model") == "iguana/kb-whisper" {
|
||||||
|
<-r.Context().Done() // wedged: never answers
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_, _ = w.Write([]byte(`{"text":"hej","duration":1,"segments":[{"start":0,"end":1,"text":"hej"}]}`))
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
cfg := uploadCfg(t, true)
|
||||||
|
cfg.GatewayURL = srv.URL
|
||||||
|
cfg.TranscribeModel = "iguana/kb-whisper"
|
||||||
|
cfg.TranscribeFallbackModel = "berget/kb-whisper"
|
||||||
|
cfg.TranscribeEngineTimeout = 150 * time.Millisecond
|
||||||
|
wav := filepath.Join(t.TempDir(), "a.wav")
|
||||||
|
require.NoError(t, os.WriteFile(wav, []byte("RIFF....WAVEfmt "), 0o600))
|
||||||
|
job, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
got, err := buildTranscriber(cfg).Transcribe(job, wav)
|
||||||
|
|
||||||
|
require.NoError(t, err, "the wedged primary used the whole job and the fallback never ran")
|
||||||
|
assert.Equal(t, "berget/kb-whisper", got.Endpoint)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildTranscriber_AWedgedBaseEngineIsASoftFailureInTheEnsemble(t *testing.T) {
|
||||||
|
// The same deadline must guard the ensemble's BASE engine, not only the chain's primary
|
||||||
|
// (second adversarial review: the wiring was tested for the chain alone).
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
_ = r.ParseMultipartForm(1 << 20)
|
||||||
|
if r.FormValue("model") == "iguana/pianissimo" {
|
||||||
|
<-r.Context().Done()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_, _ = w.Write([]byte(`{"text":"hej","duration":1,"segments":[{"start":0,"end":1,"text":"hej"}]}`))
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
cfg := uploadCfg(t, true)
|
||||||
|
cfg.GatewayURL = srv.URL
|
||||||
|
cfg.TranscribeModel = "iguana/kb-whisper"
|
||||||
|
cfg.ReconcileBaseModel = "iguana/pianissimo"
|
||||||
|
cfg.ReconcileModel = "iguana/gemma4-31b"
|
||||||
|
cfg.TranscribeEngineTimeout = 150 * time.Millisecond
|
||||||
|
wav := filepath.Join(t.TempDir(), "a.wav")
|
||||||
|
require.NoError(t, os.WriteFile(wav, []byte("RIFF....WAVEfmt "), 0o600))
|
||||||
|
job, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
got, err := buildTranscriber(cfg).Transcribe(job, wav)
|
||||||
|
|
||||||
|
require.NoError(t, err, "a wedged base engine must not use up the job")
|
||||||
|
assert.Equal(t, "hej", got.Text)
|
||||||
|
require.NotNil(t, got.Reconciliation)
|
||||||
|
assert.Equal(t, "single_engine", string(got.Reconciliation.Status))
|
||||||
|
assert.Contains(t, got.Reconciliation.Reason, "timeout")
|
||||||
|
}
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/uploads"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The worker buildUploads returns enforces the configured maximum on what was really
|
||||||
|
// converted (adversarial review F2). A stub ffmpeg writes a sparse 100-minute WAV however
|
||||||
|
// short the file claimed to be; the gateway is never reached because the job fails first.
|
||||||
|
|
||||||
|
func TestBuildUploads_TheWorkerRefusesAudioLongerThanTheConfiguredMaximum(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
stub := filepath.Join(dir, "ffmpeg")
|
||||||
|
require.NoError(t, os.WriteFile(stub, []byte("#!/bin/sh\nfor a; do last=\"$a\"; done\ntruncate -s 192000044 \"$last\"\n"), 0o700))
|
||||||
|
cfg := uploadCfg(t, true)
|
||||||
|
cfg.FFmpegBin = stub
|
||||||
|
cfg.UploadMaxDuration = 90 * time.Minute
|
||||||
|
cfg.TranscribeTimeout = time.Minute
|
||||||
|
st := &fakeUploadsStore{}
|
||||||
|
_, w, err := buildUploads(context.Background(), cfg, st, quietLog())
|
||||||
|
require.NoError(t, err)
|
||||||
|
src := filepath.Join(cfg.UploadTmpDir, "upload-x")
|
||||||
|
require.NoError(t, os.WriteFile(src, []byte("audio"), 0o600))
|
||||||
|
|
||||||
|
w.Process(context.Background(), uploads.Job{UserID: "u", UploadID: "id", Path: src, Duration: 10 * time.Second})
|
||||||
|
|
||||||
|
assert.Equal(t, []domain.UploadErrorKind{domain.UploadErrTooLong}, st.failedKinds,
|
||||||
|
"a file that claimed 10 s but converted to 100 minutes must fail, not be sent to an engine")
|
||||||
|
}
|
||||||
+5
-2
@@ -153,7 +153,10 @@ erDiagram
|
|||||||
float audio_seconds
|
float audio_seconds
|
||||||
text error_kind "typed: interrupted | transcription_failed — never a provider string"
|
text error_kind "typed: interrupted | transcription_failed — never a provider string"
|
||||||
bool had_video "the file carried a video track; only its audio was used (migration 018)"
|
bool had_video "the file carried a video track; only its audio was used (migration 018)"
|
||||||
jsonb reconciliation "NULL for a single engine; else status, exact counts and a bounded list of contested spots (migration 019, ADR-034)"
|
jsonb reconciliation "NULL for a single engine; else status, exact counts and a bounded list of every judged spot (migration 019, ADR-034; all spots stored, ADR-035)"
|
||||||
|
jsonb progress "the job's ordered phases: key, label, state, times, done/total; NULL until the job starts (migration 020, ADR-035)"
|
||||||
|
timestamptz progress_at "when a phase last really changed or advanced (migration 020)"
|
||||||
|
timestamptz alive_at "heartbeat from the running job, every ~10 s; not progress (migration 020)"
|
||||||
timestamptz created_at
|
timestamptz created_at
|
||||||
timestamptz finished_at
|
timestamptz finished_at
|
||||||
}
|
}
|
||||||
@@ -225,7 +228,7 @@ mechanism.
|
|||||||
grounds that it holds only public caption content, and a recorded meeting is the opposite.
|
grounds that it holds only public caption content, and a recorded meeting is the opposite.
|
||||||
`user_id NOT NULL ... ON DELETE CASCADE`, FORCE RLS'd. `status` runs `queued → transcribing →
|
`user_id NOT NULL ... ON DELETE CASCADE`, FORCE RLS'd. `status` runs `queued → transcribing →
|
||||||
done | failed`; `done` requires `content`, `failed` requires a typed `error_kind` (CHECK
|
done | failed`; `done` requires `content`, `failed` requires a typed `error_kind` (CHECK
|
||||||
constraints). `reconciliation` (migration 019, ADR-034) is NULL when one engine produced the text, so the page can never claim a second opinion that was not there; otherwise it holds the status (`reconciled` / `single_engine` / `skipped`), exact counts (differences, settled automatically, decided by the model, changed, uncertain, model errors) and up to 200 contested spots (each a position in seconds, both readings and which was used). The spots hold fragments of the private transcript, which is why they live in this RLS-scoped row. `had_video` (migration 018) records that the file carried a video track — a Teams meeting with the camera off is still a video file — so the page can say only its audio was used and the video was discarded. `endpoint` records which backend served it — `berget/...` means the audio left the
|
constraints). `reconciliation` (migration 019, ADR-034) is NULL when one engine produced the text, so the page can never claim a second opinion that was not there; otherwise it holds the status (`reconciled` / `single_engine` / `skipped`), exact counts (differences, settled automatically, decided by the model, changed, uncertain, model errors) and up to 2,000 judged spots (ADR-035: every one the model decided, kept or changed or unsure, each a position in seconds, both readings, which was used, and the few words before it; the page lists only the changed and uncertain ones). `progress`, `progress_at` and `alive_at` (migration 020, ADR-035) let the page show the job's phases and tell a running job from a quiet one from a lost service: `alive_at` is a heartbeat from a goroutine independent of any engine call, `progress_at` moves only when work really moved; the saved `segments` carry the reconciled words with the base engine's timing. The spots hold fragments of the private transcript, which is why they live in this RLS-scoped row. `had_video` (migration 018) records that the file carried a video track — a Teams meeting with the camera off is still a video file — so the page can say only its audio was used and the video was discarded. `endpoint` records which backend served it — `berget/...` means the audio left the
|
||||||
homelab. The audio itself is never stored. `segments` keeps the timestamps (re-transcribing 90
|
homelab. The audio itself is never stored. `segments` keeps the timestamps (re-transcribing 90
|
||||||
minutes costs ~7). Unlike `summary_actions`/`login_events` it **does** cascade on user deletion.
|
minutes costs ~7). Unlike `summary_actions`/`login_events` it **does** cascade on user deletion.
|
||||||
- **LOGIN_EVENTS** (migration 010) — throttled one-row-per-(user, date) login stamp. Used by the
|
- **LOGIN_EVENTS** (migration 010) — throttled one-row-per-(user, date) login stamp. Used by the
|
||||||
|
|||||||
@@ -23,15 +23,13 @@ it** — endpoints and aliases drift, and this file is a snapshot (2026-06-06),
|
|||||||
`{"error":"failed to process audio"}` string — it has had at least five causes. Tapir's knobs:
|
`{"error":"failed to process audio"}` string — it has had at least five causes. Tapir's knobs:
|
||||||
`TAPIR_UPLOADS_ENABLED` (off by default), `TAPIR_UPLOAD_MAX_MB` (200), `TAPIR_UPLOAD_MAX_MINUTES`
|
`TAPIR_UPLOADS_ENABLED` (off by default), `TAPIR_UPLOAD_MAX_MB` (200), `TAPIR_UPLOAD_MAX_MINUTES`
|
||||||
(90), `TAPIR_TRANSCRIBE_MODEL`, `TAPIR_TRANSCRIBE_FALLBACK_MODEL` (empty disables the cloud
|
(90), `TAPIR_TRANSCRIBE_MODEL`, `TAPIR_TRANSCRIBE_FALLBACK_MODEL` (empty disables the cloud
|
||||||
fallback), `TAPIR_TRANSCRIBE_TIMEOUT` (30m), `TAPIR_UPLOAD_TMP_DIR`, `TAPIR_FFMPEG_BIN`,
|
fallback), `TAPIR_TRANSCRIBE_TIMEOUT` (60m, the whole job), `TAPIR_TRANSCRIBE_ENGINE_TIMEOUT` (20m, one engine call: a wedged engine then fails while the job lives and the fallback can run), `TAPIR_UPLOAD_TMP_DIR`, `TAPIR_FFMPEG_BIN`,
|
||||||
`TAPIR_FFPROBE_BIN`, `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS` (90000 — the measured ceiling of
|
`TAPIR_FFPROBE_BIN`, `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS` (90000 — the measured ceiling of
|
||||||
`iguana/gemma4-26b`, which silently drops the head above it), `TAPIR_UPLOAD_CHAT_MODEL`
|
`iguana/gemma4-26b`, which silently drops the head above it), `TAPIR_UPLOAD_CHAT_MODEL`
|
||||||
(`iguana/gemma4-26b`; `koala/phi4-mini`'s 8,192-token context 400s on an hour of speech). Needs `ffmpeg` + `ffprobe`
|
(`iguana/gemma4-26b`; must be a local `koala/` or `iguana/` alias, since the whole private transcript is sent to it; `koala/phi4-mini`'s 8,192-token context 400s on an hour of speech). `ffmpeg` + `ffprobe` are in the runtime image (static, pinned by sha256; #34 decided A on 2026-10-02); Tapir's own ingress body limit for a 200 MB upload is unverified (infra).
|
||||||
in the runtime image; Tapir's own ingress body limit for a 200 MB upload is unverified (infra).
|
|
||||||
**Two-engine mode (ADR-034), off unless `TAPIR_RECONCILE_BASE_MODEL` is set:** it runs
|
**Two-engine mode (ADR-034), off unless `TAPIR_RECONCILE_BASE_MODEL` is set:** it runs
|
||||||
`iguana/pianissimo` (the base text) and `TAPIR_TRANSCRIBE_MODEL` (kb-whisper) on the same audio, then a
|
`iguana/pianissimo` (the base text) and `TAPIR_TRANSCRIBE_MODEL` (kb-whisper) on the same audio, then a
|
||||||
LOCAL chat model (`TAPIR_RECONCILE_MODEL`, e.g. `iguana/gemma4-26b`: 92% against a speaker's blind answers
|
LOCAL chat model (`TAPIR_RECONCILE_MODEL`) chooses between their disagreements. **Deployment choice (2026-10-02): `iguana/gemma4-31b`** (97% against a speaker's blind answers; 92% for `iguana/gemma4-26b`). With iguana's macOS **Low Power Mode off** the 31B decodes at 22 tok/s and reconciled a 63.6-minute recording in 21 min 49 s end to end; with it on it ran at 8 tok/s. Check `pmset -g | grep lowpowermode` before blaming a model or an engine for slowness.
|
||||||
at ~2 s a call; `iguana/gemma4-31b` 97% but ~5x slower) chooses between their disagreements.
|
|
||||||
`TAPIR_RECONCILE_CHECK_ORDER=true` asks twice per dispute with the order swapped (doubles the time).
|
`TAPIR_RECONCILE_CHECK_ORDER=true` asks twice per dispute with the order swapped (doubles the time).
|
||||||
Every model must start `koala/` or `iguana/`: a cloud alias is a start-up error. The default job timeout is
|
Every model must start `koala/` or `iguana/`: a cloud alias is a start-up error. The default job timeout is
|
||||||
60 minutes. `iguana/pianissimo` is KlangAI's Swedish ASR (a Parakeet/FastConformer on MLX, served by a small
|
60 minutes. `iguana/pianissimo` is KlangAI's Swedish ASR (a Parakeet/FastConformer on MLX, served by a small
|
||||||
|
|||||||
@@ -29,7 +29,7 @@ ingress configuration, which lives in `infra`. "About 10 MB" is an inference.
|
|||||||
## Method
|
## Method
|
||||||
|
|
||||||
- **Audio.** No known-good Swedish recording is available that is safe to use: the only real
|
- **Audio.** No known-good Swedish recording is available that is safe to use: the only real
|
||||||
one (`IMG_1233.wav`) is the private `/bygge` recording (#31 custody still open) and was
|
one is a private recording (#31 custody still open) and was
|
||||||
deliberately not touched. Instead: macOS `say -v Alva` (Swedish) on iguana, a 1,171-char
|
deliberately not touched. Instead: macOS `say -v Alva` (Swedish) on iguana, a 1,171-char
|
||||||
five-paragraph meeting script → 82.5 s of 16 kHz mono PCM WAV (`ffmpeg -ar 16000 -ac 1`).
|
five-paragraph meeting script → 82.5 s of 16 kHz mono PCM WAV (`ffmpeg -ar 16000 -ac 1`).
|
||||||
Concatenated ×4, ×8, ×16, ×32, ×44, ×66 with Python `wave` (the issue's own recipe).
|
Concatenated ×4, ×8, ×16, ×32, ×44, ×66 with Python `wave` (the issue's own recipe).
|
||||||
|
|||||||
@@ -0,0 +1,68 @@
|
|||||||
|
# Two-engine reconcile, first check on a real recording (tapir#33, ADR-034, 2026-10-02)
|
||||||
|
|
||||||
|
Aggregate numbers only. This repo is public and the recording is private (#31 custody is open), so no
|
||||||
|
transcript text, no disputed words and no per-item reading appears here.
|
||||||
|
|
||||||
|
## What was run
|
||||||
|
|
||||||
|
One real 63.6-minute Teams recording of one speaker dictating project notes (the video track, camera
|
||||||
|
off, was discarded). Build: PR #39 head `7f8e367`. Everything local: `iguana/pianissimo` as base,
|
||||||
|
`iguana/kb-whisper` as the second engine, `iguana/gemma4-31b` as reconciler (iguana Low Power Mode
|
||||||
|
off), cloud fallbacks off, a throwaway Postgres.
|
||||||
|
|
||||||
|
| Measure | Value |
|
||||||
|
|---|---|
|
||||||
|
| Job time, upload to saved | 21 min 49 s for 63.6 min of audio (20.6 min per hour of audio) |
|
||||||
|
| Differences between the engines | 362 |
|
||||||
|
| Settled automatically (compound splits, up to two filler words) | 80 |
|
||||||
|
| Decided by the model | 282 |
|
||||||
|
| Of those: alternative reading used / base kept / uncertain | 130 / 151 / 1 |
|
||||||
|
| Model call errors | 0 |
|
||||||
|
|
||||||
|
20.6 min per hour is inside the 25-minute resolution trigger in ADR-034, so no smaller adjudicator is
|
||||||
|
needed on speed grounds. The earlier estimate was about 10 min per hour on gemma4-26b; it was not
|
||||||
|
measured and this run used the 31B.
|
||||||
|
|
||||||
|
## Blind check
|
||||||
|
|
||||||
|
30 spots drawn at random (fixed seed) from the 89 replacement spots that could be placed in the base
|
||||||
|
text, across the whole recording (0:44 to 60:57). The two readings were shown as x and y in random
|
||||||
|
order, with the key kept apart from the judge. Judge: Claude Opus, text only, no audio, at the
|
||||||
|
speaker's request ("I trust Opus"). The speaker did not judge these items themselves.
|
||||||
|
|
||||||
|
| Verdict | Items |
|
||||||
|
|---|---|
|
||||||
|
| The reconciler's reading (the alternative) is right | 21 |
|
||||||
|
| The base reading (pianissimo) is right, so the change made it worse | 1 |
|
||||||
|
| Neither reading is right | 3 |
|
||||||
|
| Cannot tell from text (compound, hyphen and number-format differences) | 5 |
|
||||||
|
|
||||||
|
- Right of the items with a verdict: 21 of 22, 95% (95% interval 78% to 99%).
|
||||||
|
- Right of all 30, counting "neither" and "cannot tell" as not right: 21 of 30, 70% (52% to 83%).
|
||||||
|
- Right of the 25 that were not "cannot tell": 21 of 25, 84% (65% to 94%).
|
||||||
|
- Net effect against the base on these 30: 21 better, 1 worse, 8 unchanged in quality.
|
||||||
|
|
||||||
|
## Reading the result
|
||||||
|
|
||||||
|
ADR-034's bar was "right at least 85% of the time". It did not say how ties count, and that was a
|
||||||
|
flaw in the bar, not in the data. The literal reading (all 30) gives 70% and **does not meet** the bar;
|
||||||
|
counting only decided items gives 95% and meets it. Neither is a hidden pass or a hidden failure; both
|
||||||
|
are reported. What is not in doubt: on a random sample the reconciler improved the text far more often
|
||||||
|
than it harmed it (21 to 1), and the 8 unchanged cases are mostly spellings the text cannot settle.
|
||||||
|
|
||||||
|
## What this did not measure
|
||||||
|
|
||||||
|
- **Better single engine.** The second ADR-034 condition, "no worse than the better single engine", is
|
||||||
|
**not answered**. Only changed spots are stored, not the 151 where the model kept pianissimo, so the
|
||||||
|
reconciler cannot be compared with kb-whisper alone. The 21-to-1 result is against pianissimo.
|
||||||
|
- **The speaker as judge.** Opus judged from text. It cannot hear the audio, so it can be wrong where
|
||||||
|
the right word is audible but not inferable (items such as a mis-heard number or a name). Earlier,
|
||||||
|
Opus and the speaker agreed 30 of 30, but on items chosen as decidable; that does not transfer to a
|
||||||
|
random sample.
|
||||||
|
- **Insertions and deletions.** 42 of the 131 stored spots were not sampled: 39 are an added or
|
||||||
|
removed word, which cannot be placed in the base text, and 3 are the uncertain spot or could not be
|
||||||
|
located. The 39 are mostly small words ("och", "det"), so this sample says little about them.
|
||||||
|
- **One recording, one speaker, 30 items.** Wide intervals, and neighbouring spots (three pairs within
|
||||||
|
seconds of each other) are not independent.
|
||||||
|
- **Segments keep the base text.** The saved timed segments hold pianissimo's wording, not the merged
|
||||||
|
text; only the stored full text is merged. Found while building this sample; not yet an issue.
|
||||||
@@ -156,3 +156,103 @@ Feature: Upload an audio recording, get a private Swedish transcript, discuss it
|
|||||||
Scenario: A single-engine transcript claims no second opinion
|
Scenario: A single-engine transcript claims no second opinion
|
||||||
When only one engine was configured
|
When only one engine was configured
|
||||||
Then the page does not mention reconciliation
|
Then the page does not mention reconciliation
|
||||||
|
|
||||||
|
# ADR-035: progress, the log, and the judged spots
|
||||||
|
|
||||||
|
Scenario: I can see which phase my upload is in
|
||||||
|
Given a recording is being transcribed
|
||||||
|
When I open its page
|
||||||
|
Then it lists every phase in order, the finished ones with how long each took
|
||||||
|
And the running phase is marked, with how long it has run
|
||||||
|
And the page updates in place every two seconds until the job ends
|
||||||
|
|
||||||
|
Scenario: A bar appears only where the count is real
|
||||||
|
When the deciding phase has decided 142 of 282 differences
|
||||||
|
Then the page shows a bar with "142 of 282"
|
||||||
|
And an engine call, which is one request with nothing to count, never gets a bar or a percentage
|
||||||
|
|
||||||
|
Scenario: A slow engine call is not mistaken for a dead service
|
||||||
|
When an engine call has run for 25 minutes and the service is still reporting
|
||||||
|
Then the page shows the elapsed time and says nothing is wrong
|
||||||
|
|
||||||
|
Scenario: A service that stopped reporting is called lost
|
||||||
|
When nothing has been heard from the service for longer than the heartbeat allows
|
||||||
|
Then the page says the service stopped reporting and may have restarted or crashed
|
||||||
|
And the running phase carries a warning mark and no spinner
|
||||||
|
And the page keeps checking
|
||||||
|
|
||||||
|
Scenario: A model that stops advancing is called quiet
|
||||||
|
When the deciding phase has not advanced for five minutes and the service is still reporting
|
||||||
|
Then the page says nothing new has happened for that long and that the service is still running
|
||||||
|
And it does not say the service is lost
|
||||||
|
|
||||||
|
Scenario: A failed job shows where it stopped
|
||||||
|
When a job fails
|
||||||
|
Then the failed phase is marked and what never started stays pending
|
||||||
|
And the page stops checking
|
||||||
|
|
||||||
|
Scenario: A finished job keeps its record of what was done
|
||||||
|
When a job is done
|
||||||
|
Then the page keeps a collapsed list of the phases with how long each took
|
||||||
|
|
||||||
|
Scenario: The heartbeat runs on its own while an engine call is stuck
|
||||||
|
When an engine call does not return
|
||||||
|
Then the row is still stamped alive every few seconds
|
||||||
|
And no heartbeat outlives its job
|
||||||
|
|
||||||
|
Scenario: The log names every phase and never the transcript
|
||||||
|
When a job runs
|
||||||
|
Then the log has a line per phase in the order it ran, each naming the upload
|
||||||
|
And no line contains transcript text or the spool path
|
||||||
|
|
||||||
|
Scenario: Every judged spot is stored but only the ones to check are listed
|
||||||
|
When the model has changed, kept and been unsure about spots
|
||||||
|
Then every one is stored with its verdict and the words before it
|
||||||
|
And the page lists only the changed and uncertain ones, with the words before each
|
||||||
|
|
||||||
|
Scenario: The saved segments carry the merged words
|
||||||
|
When the model changes a word
|
||||||
|
Then the segment containing it carries the reconciled word
|
||||||
|
And the segment's timing is unchanged
|
||||||
|
|
||||||
|
# ADR-035 decision 8: hardening from the adversarial review
|
||||||
|
|
||||||
|
Scenario: Chat about a recording is offered with local models only
|
||||||
|
When I open the chat for my recording
|
||||||
|
Then the model list holds only models that run on the homelab
|
||||||
|
And a cloud model the shared chat service knows is not in it
|
||||||
|
|
||||||
|
Scenario: A forged cloud model is never sent the transcript
|
||||||
|
When a chat request names a cloud model
|
||||||
|
Then a local model answers instead, and nothing was sent to the cloud one
|
||||||
|
|
||||||
|
Scenario: A wedged engine does not use up the whole job
|
||||||
|
When the primary engine never answers
|
||||||
|
Then it fails at its own deadline while the job is still alive
|
||||||
|
And the fallback runs, unless the job itself was cancelled or timed out
|
||||||
|
|
||||||
|
Scenario: A file that lies about its length is refused
|
||||||
|
When a file claims to be short but converts to more than the limit
|
||||||
|
Then the job fails with a plain explanation and no engine receives the audio
|
||||||
|
|
||||||
|
Scenario: A recording much shorter than its file claimed is transcribed and noted
|
||||||
|
When a file converts to under 80 percent of the length it claimed and more than two minutes short
|
||||||
|
Then it is transcribed anyway and its audio step says how long the audio really is
|
||||||
|
And a complete recording whose header over-estimated its length is not refused
|
||||||
|
|
||||||
|
Scenario: A full queue is a retry, not an error
|
||||||
|
When too many recordings are already waiting
|
||||||
|
Then the upload is answered as busy, its row is failed honestly and its audio is deleted
|
||||||
|
|
||||||
|
Scenario: One user cannot fill the queue for everyone
|
||||||
|
When one user has several recordings waiting
|
||||||
|
Then further uploads from that user are refused and other users are not affected
|
||||||
|
|
||||||
|
Scenario: Errors are shown as kinds, never their text
|
||||||
|
When a model or an engine fails
|
||||||
|
Then the log, the stored reason and the page carry a kind such as timeout or http_502
|
||||||
|
And never the provider's response, a file path or a word from the transcript
|
||||||
|
|
||||||
|
Scenario: Input that cannot be right is a client error
|
||||||
|
When the id is not an id, or a header or title holds control bytes
|
||||||
|
Then the answer is a 4xx or the title is cleaned, never a server error
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"os/exec"
|
"os/exec"
|
||||||
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -17,8 +18,17 @@ import (
|
|||||||
type FFmpeg struct {
|
type FFmpeg struct {
|
||||||
// Bin is the ffmpeg executable ("ffmpeg" resolves via PATH).
|
// Bin is the ffmpeg executable ("ffmpeg" resolves via PATH).
|
||||||
Bin string
|
Bin string
|
||||||
|
// MaxSeconds caps the converted length when positive. The duration limit is a claim
|
||||||
|
// in the file's header and can be lied about (a 2 h file claiming 10 s converted to a
|
||||||
|
// 120-minute WAV); this bounds what a lie can cost. A minute of slack is added so a
|
||||||
|
// file at the limit is never cut, and the worker measures the result.
|
||||||
|
MaxSeconds int
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// lengthSlackSeconds is added to MaxSeconds so ffmpeg's rounding never clips a file that
|
||||||
|
// is really at the limit.
|
||||||
|
const lengthSlackSeconds = 60
|
||||||
|
|
||||||
// ToWAV writes src, converted, to dst. Video streams are dropped (-vn): by now the
|
// ToWAV writes src, converted, to dst. Video streams are dropped (-vn): by now the
|
||||||
// file has already been probed as audio-only, but a cover-art stream must not end
|
// file has already been probed as audio-only, but a cover-art stream must not end
|
||||||
// up in the WAV.
|
// up in the WAV.
|
||||||
@@ -29,7 +39,12 @@ func (f *FFmpeg) ToWAV(ctx context.Context, src, dst string) error {
|
|||||||
}
|
}
|
||||||
// Input restrictions go BEFORE -i: placed after it they would apply to the output.
|
// Input restrictions go BEFORE -i: placed after it they would apply to the output.
|
||||||
args := append([]string{"-y", "-v", "error"}, inputRestrictions...)
|
args := append([]string{"-y", "-v", "error"}, inputRestrictions...)
|
||||||
args = append(args, "-i", src, "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le", "-f", "wav", dst)
|
args = append(args, "-i", src, "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le")
|
||||||
|
if f.MaxSeconds > 0 {
|
||||||
|
// After -i: an OUTPUT cap. Before it, it would limit what is read, not written.
|
||||||
|
args = append(args, "-t", strconv.Itoa(f.MaxSeconds+lengthSlackSeconds))
|
||||||
|
}
|
||||||
|
args = append(args, "-f", "wav", dst)
|
||||||
cmd := exec.CommandContext(ctx, bin, args...)
|
cmd := exec.CommandContext(ctx, bin, args...)
|
||||||
cmd.WaitDelay = killGrace
|
cmd.WaitDelay = killGrace
|
||||||
if out, err := cmd.CombinedOutput(); err != nil {
|
if out, err := cmd.CombinedOutput(); err != nil {
|
||||||
|
|||||||
@@ -77,3 +77,35 @@ printf data > "$last"`)
|
|||||||
assert.Less(t, pw, in, "input options must come BEFORE -i or ffmpeg applies them to the output")
|
assert.Less(t, pw, in, "input options must come BEFORE -i or ffmpeg applies them to the output")
|
||||||
assert.Less(t, fw, in)
|
assert.Less(t, fw, in)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestFFmpegToWAV_PassesAnOutputLengthCapAfterTheInputWhenAMaximumIsSet(t *testing.T) {
|
||||||
|
// The duration limit is a CLAIM in the file's header; a file that lies about it can
|
||||||
|
// convert to hours of audio (adversarial review F2: a 2 h file claiming 10 s became a
|
||||||
|
// 120-minute WAV). The cap is an OUTPUT option, after -i, with a minute of slack.
|
||||||
|
rec := filepath.Join(t.TempDir(), "argv")
|
||||||
|
bin := stub(t, `printf '%s\n' "$@" > `+rec+`
|
||||||
|
for a; do last="$a"; done
|
||||||
|
printf data > "$last"`)
|
||||||
|
|
||||||
|
err := (&audio.FFmpeg{Bin: bin, MaxSeconds: 5400}).ToWAV(context.Background(), "/spool/in", filepath.Join(t.TempDir(), "out.wav"))
|
||||||
|
|
||||||
|
require.NoError(t, err)
|
||||||
|
raw, err := os.ReadFile(rec)
|
||||||
|
require.NoError(t, err)
|
||||||
|
joined := " " + strings.Join(strings.Split(strings.TrimSpace(string(raw)), "\n"), " ") + " "
|
||||||
|
assert.Contains(t, joined, " -t 5460 ", "limit plus a minute of slack")
|
||||||
|
assert.Less(t, strings.Index(joined, " -i /spool/in "), strings.Index(joined, " -t 5460 "), "-t after -i is an output cap; before it, an input one")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFFmpegToWAV_PassesNoLengthCapWhenNoMaximumIsSet(t *testing.T) {
|
||||||
|
rec := filepath.Join(t.TempDir(), "argv")
|
||||||
|
bin := stub(t, `printf '%s\n' "$@" > `+rec+`
|
||||||
|
for a; do last="$a"; done
|
||||||
|
printf data > "$last"`)
|
||||||
|
|
||||||
|
require.NoError(t, (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/spool/in", filepath.Join(t.TempDir(), "out.wav")))
|
||||||
|
|
||||||
|
raw, err := os.ReadFile(rec)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.NotContains(t, " "+strings.ReplaceAll(string(raw), "\n", " ")+" ", " -t ")
|
||||||
|
}
|
||||||
|
|||||||
@@ -55,6 +55,29 @@ func WithMaxTokens(n int) Option {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// maxErrorBody bounds how much of a failing response ends up in an error message.
|
||||||
|
const maxErrorBody = 512
|
||||||
|
|
||||||
|
// HTTPError is a non-200 answer from the gateway. Its text keeps the form callers that
|
||||||
|
// log it already rely on, with the body bounded; its Code is what code may branch on and
|
||||||
|
// what errkind turns into a kind. The body is provider text and can echo the prompt, so
|
||||||
|
// nothing private may persist or display Error() (ADR-035 review F3).
|
||||||
|
type HTTPError struct {
|
||||||
|
Code int
|
||||||
|
Body string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *HTTPError) Error() string {
|
||||||
|
body := e.Body
|
||||||
|
if len(body) > maxErrorBody {
|
||||||
|
body = body[:maxErrorBody] + "…"
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("LLM returned %d: %s", e.Code, body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// HTTPStatus implements errkind.HTTPStatuser.
|
||||||
|
func (e *HTTPError) HTTPStatus() int { return e.Code }
|
||||||
|
|
||||||
// WithTemperature overrides the sampling temperature. Zero IS a valid value and is sent
|
// WithTemperature overrides the sampling temperature. Zero IS a valid value and is sent
|
||||||
// as zero: the reconciler's accuracy was measured at temperature 0 and must run there.
|
// as zero: the reconciler's accuracy was measured at temperature 0 and must run there.
|
||||||
func WithTemperature(t float64) Option {
|
func WithTemperature(t float64) Option {
|
||||||
@@ -166,7 +189,7 @@ func (c *Client) Complete(ctx context.Context, system, user string) (string, err
|
|||||||
return "", fmt.Errorf("read response: %w", err)
|
return "", fmt.Errorf("read response: %w", err)
|
||||||
}
|
}
|
||||||
if resp.StatusCode != http.StatusOK {
|
if resp.StatusCode != http.StatusOK {
|
||||||
return "", fmt.Errorf("LLM returned %d: %s", resp.StatusCode, out)
|
return "", &HTTPError{Code: resp.StatusCode, Body: string(out)}
|
||||||
}
|
}
|
||||||
|
|
||||||
var cr chatResponse
|
var cr chatResponse
|
||||||
|
|||||||
@@ -0,0 +1,63 @@
|
|||||||
|
package llm
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/errkind"
|
||||||
|
)
|
||||||
|
|
||||||
|
// A non-200 from the gateway is an HTTP status error: its kind can be logged and stored
|
||||||
|
// without its body, which can echo the prompt (adversarial review F3).
|
||||||
|
|
||||||
|
func failingGateway(t *testing.T, status int, body string) *httptest.Server {
|
||||||
|
t.Helper()
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.WriteHeader(status)
|
||||||
|
_, _ = w.Write([]byte(body))
|
||||||
|
}))
|
||||||
|
t.Cleanup(srv.Close)
|
||||||
|
return srv
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestClient_ANon200IsAnHTTPStatusErrorWhoseKindCarriesNoBody(t *testing.T) {
|
||||||
|
srv := failingGateway(t, http.StatusBadRequest, `{"error":"prompt was: Anders ansvarar för virket"}`)
|
||||||
|
|
||||||
|
_, err := New(srv.URL, "", "m", 5*time.Second).Complete(context.Background(), "s", "u")
|
||||||
|
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected an error")
|
||||||
|
}
|
||||||
|
if got := errkind.Of(err); got != "http_400" {
|
||||||
|
t.Errorf("errkind.Of = %q, want http_400", got)
|
||||||
|
}
|
||||||
|
var he *HTTPError
|
||||||
|
if !errors.As(err, &he) || he.Code != http.StatusBadRequest {
|
||||||
|
t.Errorf("error is not an *HTTPError with the status code: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestClient_TheErrorTextIsUnchangedForCallersThatLogIt(t *testing.T) {
|
||||||
|
srv := failingGateway(t, http.StatusBadGateway, "upstream down")
|
||||||
|
|
||||||
|
_, err := New(srv.URL, "", "m", 5*time.Second).Complete(context.Background(), "s", "u")
|
||||||
|
|
||||||
|
if err == nil || err.Error() != "LLM returned 502: upstream down" {
|
||||||
|
t.Errorf("error text = %v: callers that log it already rely on this form", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestClient_AnEnormousErrorBodyIsBoundedInTheErrorText(t *testing.T) {
|
||||||
|
srv := failingGateway(t, http.StatusInternalServerError, strings.Repeat("x", 100_000))
|
||||||
|
|
||||||
|
_, err := New(srv.URL, "", "m", 5*time.Second).Complete(context.Background(), "s", "u")
|
||||||
|
|
||||||
|
if err == nil || len(err.Error()) > 1024 {
|
||||||
|
t.Errorf("error text is %d bytes: a gateway's whole body must not become an error message", len(err.Error()))
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
ALTER TABLE uploaded_transcripts
|
||||||
|
DROP COLUMN alive_at,
|
||||||
|
DROP COLUMN progress_at,
|
||||||
|
DROP COLUMN progress;
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
-- Migration 020: where an upload job is, and whether the service is still reporting (ADR-035).
|
||||||
|
--
|
||||||
|
-- progress the job's ordered phases as JSON (key, label, state, times, done/total).
|
||||||
|
-- progress_at when a phase last REALLY changed or advanced.
|
||||||
|
-- alive_at a heartbeat stamped every few seconds by a goroutine independent of any
|
||||||
|
-- engine call. The two differ on purpose: alive says the service is still
|
||||||
|
-- reporting, progress says work is moving, and a page that cannot tell a
|
||||||
|
-- crashed service from a slow model is the failure this exists to end.
|
||||||
|
--
|
||||||
|
-- All three are NULL until the job starts. They sit on the already-FORCE'd, RLS-scoped
|
||||||
|
-- uploaded_transcripts, so the exact-set isolation test is unaffected.
|
||||||
|
ALTER TABLE uploaded_transcripts
|
||||||
|
ADD COLUMN progress JSONB,
|
||||||
|
ADD COLUMN progress_at TIMESTAMPTZ,
|
||||||
|
ADD COLUMN alive_at TIMESTAMPTZ;
|
||||||
@@ -68,6 +68,26 @@ func (s *Store) CompleteUpload(ctx context.Context, userID, id string, t domain.
|
|||||||
t.Text, segs, t.Endpoint, t.AudioSeconds, rec)
|
t.Text, segs, t.Endpoint, t.AudioSeconds, rec)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SaveProgress records the job's phases and when something last really changed
|
||||||
|
// (ADR-035). It replaces the previous snapshot: the Tracker holds the whole plan.
|
||||||
|
func (s *Store) SaveProgress(ctx context.Context, userID, id string, steps []domain.ProgressStep, at time.Time) error {
|
||||||
|
b, err := json.Marshal(steps)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("store: save progress: marshal: %w", err)
|
||||||
|
}
|
||||||
|
return s.updateUpload(ctx, userID, id, "save progress",
|
||||||
|
`UPDATE uploaded_transcripts SET progress = $3, progress_at = $4
|
||||||
|
WHERE id = $1 AND user_id = $2`, b, at)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Heartbeat stamps the upload as alive now. It does not touch progress_at: alive says
|
||||||
|
// the service is still reporting, not that work is moving (ADR-035).
|
||||||
|
func (s *Store) Heartbeat(ctx context.Context, userID, id string) error {
|
||||||
|
return s.updateUpload(ctx, userID, id, "heartbeat",
|
||||||
|
`UPDATE uploaded_transcripts SET alive_at = NOW()
|
||||||
|
WHERE id = $1 AND user_id = $2`)
|
||||||
|
}
|
||||||
|
|
||||||
// FailUpload moves an upload to failed with a typed reason.
|
// FailUpload moves an upload to failed with a typed reason.
|
||||||
func (s *Store) FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error {
|
func (s *Store) FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error {
|
||||||
return s.updateUpload(ctx, userID, id, "fail upload",
|
return s.updateUpload(ctx, userID, id, "fail upload",
|
||||||
@@ -165,19 +185,21 @@ func (s *Store) FailInterruptedUploads(ctx context.Context) (int, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
const uploadSelect = `SELECT id, user_id, title, status, content, segments, endpoint,
|
const uploadSelect = `SELECT id, user_id, title, status, content, segments, endpoint,
|
||||||
audio_seconds, error_kind, had_video, reconciliation, created_at, finished_at FROM uploaded_transcripts`
|
audio_seconds, error_kind, had_video, reconciliation, progress, progress_at, alive_at,
|
||||||
|
created_at, finished_at FROM uploaded_transcripts`
|
||||||
|
|
||||||
// scanUpload scans one uploadSelect row. withText controls whether the transcript
|
// scanUpload scans one uploadSelect row. withText controls whether the transcript
|
||||||
// body and segments are kept (GetUpload) or dropped (ListUploads).
|
// body and segments are kept (GetUpload) or dropped (ListUploads).
|
||||||
func scanUpload(row pgx.Row, u *domain.UploadedTranscript, withText bool) error {
|
func scanUpload(row pgx.Row, u *domain.UploadedTranscript, withText bool) error {
|
||||||
var (
|
var (
|
||||||
status, errKind, endpoint, content *string
|
status, errKind, endpoint, content *string
|
||||||
segments, reconciliation []byte
|
segments, reconciliation, progress []byte
|
||||||
seconds *float64
|
seconds *float64
|
||||||
finished *time.Time
|
finished, progressAt, aliveAt *time.Time
|
||||||
)
|
)
|
||||||
if err := row.Scan(&u.ID, &u.UserID, &u.Title, &status, &content, &segments, &endpoint,
|
if err := row.Scan(&u.ID, &u.UserID, &u.Title, &status, &content, &segments, &endpoint,
|
||||||
&seconds, &errKind, &u.HadVideo, &reconciliation, &u.CreatedAt, &finished); err != nil {
|
&seconds, &errKind, &u.HadVideo, &reconciliation, &progress, &progressAt, &aliveAt,
|
||||||
|
&u.CreatedAt, &finished); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
u.Status = domain.UploadStatus(deref(status))
|
u.Status = domain.UploadStatus(deref(status))
|
||||||
@@ -192,6 +214,17 @@ func scanUpload(row pgx.Row, u *domain.UploadedTranscript, withText bool) error
|
|||||||
if !withText {
|
if !withText {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
if len(progress) > 0 {
|
||||||
|
if err := json.Unmarshal(progress, &u.Progress); err != nil {
|
||||||
|
return fmt.Errorf("unmarshal progress: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if progressAt != nil {
|
||||||
|
u.ProgressAt = *progressAt
|
||||||
|
}
|
||||||
|
if aliveAt != nil {
|
||||||
|
u.AliveAt = *aliveAt
|
||||||
|
}
|
||||||
if len(reconciliation) > 0 {
|
if len(reconciliation) > 0 {
|
||||||
var r domain.Reconciliation
|
var r domain.Reconciliation
|
||||||
if err := json.Unmarshal(reconciliation, &r); err != nil {
|
if err := json.Unmarshal(reconciliation, &r); err != nil {
|
||||||
|
|||||||
@@ -0,0 +1,137 @@
|
|||||||
|
package store_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/adapters/store"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Progress and heartbeat of an upload job (ADR-035, migration 020). Like the other
|
||||||
|
// upload store tests these run as the superuser, so the cross-user cases prove the
|
||||||
|
// statement's own user_id predicate; the database-enforced isolation is covered by the
|
||||||
|
// RLS tests, and migration 020 adds columns to the already-FORCE'd table, not a table.
|
||||||
|
|
||||||
|
func startedUpload(t *testing.T, s *store.Store) string {
|
||||||
|
t.Helper()
|
||||||
|
seedUsers(t, userA, userB)
|
||||||
|
id, err := s.CreateUpload(context.Background(), userA, "m", false)
|
||||||
|
require.NoError(t, err)
|
||||||
|
return id
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_AFreshUploadHasNoProgressAndNoHeartbeat(t *testing.T) {
|
||||||
|
s := newStore(t)
|
||||||
|
id := startedUpload(t, s)
|
||||||
|
|
||||||
|
got, err := s.GetUpload(context.Background(), userA, id)
|
||||||
|
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Empty(t, got.Progress)
|
||||||
|
assert.True(t, got.ProgressAt.IsZero())
|
||||||
|
assert.True(t, got.AliveAt.IsZero())
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_StepsAndTheirTimesRoundTripInOrder(t *testing.T) {
|
||||||
|
s := newStore(t)
|
||||||
|
id := startedUpload(t, s)
|
||||||
|
ctx := context.Background()
|
||||||
|
at := time.Date(2026, 10, 2, 8, 30, 0, 0, time.UTC)
|
||||||
|
steps := []domain.ProgressStep{
|
||||||
|
{Key: "audio", Label: "Preparing audio", State: domain.StepDone, StartedAt: at, FinishedAt: at.Add(40 * time.Second)},
|
||||||
|
{Key: "decide", Label: "Deciding", State: domain.StepRunning, StartedAt: at.Add(time.Minute), Done: 12, Total: 282},
|
||||||
|
{Key: "save", Label: "Saving", State: domain.StepPending},
|
||||||
|
}
|
||||||
|
|
||||||
|
require.NoError(t, s.SaveProgress(ctx, userA, id, steps, at.Add(2*time.Minute)))
|
||||||
|
|
||||||
|
got, err := s.GetUpload(ctx, userA, id)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, got.Progress, 3)
|
||||||
|
assert.Equal(t, []string{"audio", "decide", "save"}, []string{got.Progress[0].Key, got.Progress[1].Key, got.Progress[2].Key})
|
||||||
|
assert.Equal(t, domain.StepRunning, got.Progress[1].State)
|
||||||
|
assert.Equal(t, 12, got.Progress[1].Done)
|
||||||
|
assert.Equal(t, 282, got.Progress[1].Total)
|
||||||
|
assert.True(t, got.Progress[0].FinishedAt.Equal(at.Add(40*time.Second)), "a step's finish time must survive the round trip")
|
||||||
|
assert.True(t, got.Progress[2].StartedAt.IsZero(), "a pending step has no start time, not a zero date")
|
||||||
|
assert.True(t, got.ProgressAt.Equal(at.Add(2*time.Minute)))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_AHeartbeatMovesAliveAtAndNotProgressAt(t *testing.T) {
|
||||||
|
s := newStore(t)
|
||||||
|
id := startedUpload(t, s)
|
||||||
|
ctx := context.Background()
|
||||||
|
at := time.Date(2026, 10, 2, 8, 30, 0, 0, time.UTC)
|
||||||
|
require.NoError(t, s.SaveProgress(ctx, userA, id, []domain.ProgressStep{{Key: "audio", Label: "a", State: domain.StepRunning}}, at))
|
||||||
|
before := time.Now().Add(-time.Second)
|
||||||
|
|
||||||
|
require.NoError(t, s.Heartbeat(ctx, userA, id))
|
||||||
|
|
||||||
|
got, err := s.GetUpload(ctx, userA, id)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.True(t, got.AliveAt.After(before), "the heartbeat must stamp the row with now")
|
||||||
|
assert.True(t, got.ProgressAt.Equal(at), "alive is not progress: a heartbeat must not move progress_at")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_ASecondSaveReplacesTheFirst(t *testing.T) {
|
||||||
|
s := newStore(t)
|
||||||
|
id := startedUpload(t, s)
|
||||||
|
ctx := context.Background()
|
||||||
|
at := time.Date(2026, 10, 2, 8, 30, 0, 0, time.UTC)
|
||||||
|
require.NoError(t, s.SaveProgress(ctx, userA, id, []domain.ProgressStep{{Key: "a", Label: "A", State: domain.StepRunning}}, at))
|
||||||
|
|
||||||
|
require.NoError(t, s.SaveProgress(ctx, userA, id, []domain.ProgressStep{{Key: "a", Label: "A", State: domain.StepDone}, {Key: "b", Label: "B", State: domain.StepRunning}}, at.Add(time.Minute)))
|
||||||
|
|
||||||
|
got, err := s.GetUpload(ctx, userA, id)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, got.Progress, 2)
|
||||||
|
assert.Equal(t, domain.StepDone, got.Progress[0].State)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_ByAnotherUserChangesNothing(t *testing.T) {
|
||||||
|
s := newStore(t)
|
||||||
|
id := startedUpload(t, s)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
errSave := s.SaveProgress(ctx, userB, id, []domain.ProgressStep{{Key: "x", Label: "X", State: domain.StepDone}}, time.Now())
|
||||||
|
errBeat := s.Heartbeat(ctx, userB, id)
|
||||||
|
|
||||||
|
assert.ErrorIs(t, errSave, store.ErrUploadNotFound)
|
||||||
|
assert.ErrorIs(t, errBeat, store.ErrUploadNotFound)
|
||||||
|
got, err := s.GetUpload(ctx, userA, id)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Empty(t, got.Progress, "B's write must not have reached A's upload")
|
||||||
|
assert.True(t, got.AliveAt.IsZero())
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_ListDoesNotDragItAlong(t *testing.T) {
|
||||||
|
s := newStore(t)
|
||||||
|
id := startedUpload(t, s)
|
||||||
|
ctx := context.Background()
|
||||||
|
require.NoError(t, s.SaveProgress(ctx, userA, id, []domain.ProgressStep{{Key: "a", Label: "A", State: domain.StepRunning}}, time.Now()))
|
||||||
|
|
||||||
|
rows, err := s.ListUploads(ctx, userA)
|
||||||
|
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, rows, 1)
|
||||||
|
assert.Empty(t, rows[0].Progress, "the list view needs a status, not every row's phases")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_SurvivesCompletion(t *testing.T) {
|
||||||
|
s := newStore(t)
|
||||||
|
id := startedUpload(t, s)
|
||||||
|
ctx := context.Background()
|
||||||
|
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
||||||
|
require.NoError(t, s.SaveProgress(ctx, userA, id, []domain.ProgressStep{{Key: "a", Label: "A", State: domain.StepDone}}, time.Now()))
|
||||||
|
|
||||||
|
require.NoError(t, s.CompleteUpload(ctx, userA, id, domain.Transcription{Text: "x", Endpoint: "e"}))
|
||||||
|
|
||||||
|
got, err := s.GetUpload(ctx, userA, id)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, got.Progress, 1, "a finished upload keeps its record of what was done, and how long each step took")
|
||||||
|
}
|
||||||
@@ -242,8 +242,8 @@ func TestUploadLifecycle_ReconciliationRoundTripsWithItsContestedSpots(t *testin
|
|||||||
Status: domain.ReconcileDone, Base: "iguana/pianissimo", Alt: "iguana/kb-whisper", Reconciler: "iguana/gemma4-26b",
|
Status: domain.ReconcileDone, Base: "iguana/pianissimo", Alt: "iguana/kb-whisper", Reconciler: "iguana/gemma4-26b",
|
||||||
Disputes: 380, AutoResolved: 90, Adjudicated: 285, Changed: 120, Unsure: 5, Errors: 0,
|
Disputes: 380, AutoResolved: 90, Adjudicated: 285, Changed: 120, Unsure: 5, Errors: 0,
|
||||||
Contested: []domain.ContestedSpot{
|
Contested: []domain.ContestedSpot{
|
||||||
{Seconds: 2210.5, Base: "SEB", Alt: "SCB", Used: "alt"},
|
{Seconds: 2210.5, Base: "Axel", Alt: "Alex", Used: "alt"},
|
||||||
{Seconds: 3400, Base: "T+1", Alt: "en T", Used: "base", Unsure: true},
|
{Seconds: 3400, Base: "A+2", Alt: "en A", Used: "base", Unsure: true},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -6,6 +6,8 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/errkind"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/progress"
|
||||||
"git.d-ma.be/mathias/tapir/internal/reconcile"
|
"git.d-ma.be/mathias/tapir/internal/reconcile"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -30,13 +32,30 @@ func NewEnsemble(base, alt, fallback Backend, adj reconcile.Adjudicator, opts re
|
|||||||
return &Ensemble{base: base, alt: alt, fallback: fallback, adj: adj, opts: opts}
|
return &Ensemble{base: base, alt: alt, fallback: fallback, adj: adj, opts: opts}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Transcribe implements uploads.Transcriber.
|
// Plan implements uploads.Planner: the phases this ensemble reports, so the page can
|
||||||
|
// show them before they run. The fallback is a phase only when one is configured.
|
||||||
|
func (e *Ensemble) Plan() []progress.Planned {
|
||||||
|
plan := []progress.Planned{
|
||||||
|
{Key: "engine_a", Label: "Transcribing with " + e.base.Name()},
|
||||||
|
{Key: "engine_b", Label: "Transcribing with " + e.alt.Name()},
|
||||||
|
{Key: "compare", Label: "Comparing the two transcripts"},
|
||||||
|
{Key: "decide", Label: "Deciding the differences"},
|
||||||
|
}
|
||||||
|
if e.fallback != nil {
|
||||||
|
plan = append(plan, progress.Planned{Key: "fallback", Label: "Cloud fallback (" + e.fallback.Name() + ")"})
|
||||||
|
}
|
||||||
|
return plan
|
||||||
|
}
|
||||||
|
|
||||||
|
// Transcribe implements uploads.Transcriber. Each phase is reported through the
|
||||||
|
// progress.Reporter in ctx (a no-op when nobody listens).
|
||||||
func (e *Ensemble) Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error) {
|
func (e *Ensemble) Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error) {
|
||||||
b, berr := e.base.Transcribe(ctx, wavPath)
|
rep := progress.From(ctx)
|
||||||
|
b, berr := e.run(ctx, rep, "engine_a", e.base, wavPath)
|
||||||
if err := ctx.Err(); err != nil {
|
if err := ctx.Err(); err != nil {
|
||||||
return domain.Transcription{}, err
|
return domain.Transcription{}, err
|
||||||
}
|
}
|
||||||
a, aerr := e.alt.Transcribe(ctx, wavPath)
|
a, aerr := e.run(ctx, rep, "engine_b", e.alt, wavPath)
|
||||||
if err := ctx.Err(); err != nil {
|
if err := ctx.Err(); err != nil {
|
||||||
return domain.Transcription{}, err
|
return domain.Transcription{}, err
|
||||||
}
|
}
|
||||||
@@ -48,24 +67,42 @@ func (e *Ensemble) Transcribe(ctx context.Context, wavPath string) (domain.Trans
|
|||||||
return domain.Transcription{}, err
|
return domain.Transcription{}, err
|
||||||
}
|
}
|
||||||
out.Endpoint = fmt.Sprintf("%s + %s, reconciled by %s", e.base.Name(), e.alt.Name(), e.opts.Reconciler)
|
out.Endpoint = fmt.Sprintf("%s + %s, reconciled by %s", e.base.Name(), e.alt.Name(), e.opts.Reconciler)
|
||||||
|
if rc := out.Reconciliation; rc != nil {
|
||||||
|
rep.Note("reconciled", "status", string(rc.Status), "disputes", rc.Disputes, "auto_resolved", rc.AutoResolved,
|
||||||
|
"adjudicated", rc.Adjudicated, "changed", rc.Changed, "unsure", rc.Unsure, "errors", rc.Errors)
|
||||||
|
}
|
||||||
return out, nil
|
return out, nil
|
||||||
case berr == nil:
|
case berr == nil:
|
||||||
return e.single(b, e.base.Name(), fmt.Sprintf("%s failed: %v", e.alt.Name(), aerr)), nil
|
return e.single(b, e.base.Name(), fmt.Sprintf("%s failed (%s)", e.alt.Name(), errkind.Of(aerr))), nil
|
||||||
case aerr == nil:
|
case aerr == nil:
|
||||||
return e.single(a, e.alt.Name(), fmt.Sprintf("%s failed: %v", e.base.Name(), berr)), nil
|
return e.single(a, e.alt.Name(), fmt.Sprintf("%s failed (%s)", e.base.Name(), errkind.Of(berr))), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
both := errors.Join(fmt.Errorf("%s: %w", e.base.Name(), berr), fmt.Errorf("%s: %w", e.alt.Name(), aerr))
|
both := errors.Join(fmt.Errorf("%s: %w", e.base.Name(), berr), fmt.Errorf("%s: %w", e.alt.Name(), aerr))
|
||||||
if e.fallback == nil {
|
if e.fallback == nil {
|
||||||
return domain.Transcription{}, both
|
return domain.Transcription{}, both
|
||||||
}
|
}
|
||||||
f, ferr := e.fallback.Transcribe(ctx, wavPath)
|
f, ferr := e.run(ctx, rep, "fallback", e.fallback, wavPath)
|
||||||
if ferr != nil {
|
if ferr != nil {
|
||||||
return domain.Transcription{}, errors.Join(both, fmt.Errorf("%s: %w", e.fallback.Name(), ferr))
|
return domain.Transcription{}, errors.Join(both, fmt.Errorf("%s: %w", e.fallback.Name(), ferr))
|
||||||
}
|
}
|
||||||
return e.single(f, e.fallback.Name(), "both local engines failed"), nil
|
return e.single(f, e.fallback.Name(), "both local engines failed"), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// run is one engine call, reported as a phase. A job that was stopped is not a broken
|
||||||
|
// engine, so a cancelled context is never reported as a failed phase.
|
||||||
|
func (e *Ensemble) run(ctx context.Context, rep progress.Reporter, key string, be Backend, wav string) (domain.Transcription, error) {
|
||||||
|
rep.Begin(key, 0)
|
||||||
|
t, err := be.Transcribe(ctx, wav)
|
||||||
|
switch {
|
||||||
|
case err == nil:
|
||||||
|
rep.Done(key)
|
||||||
|
case ctx.Err() == nil:
|
||||||
|
rep.Fail(key, err)
|
||||||
|
}
|
||||||
|
return t, err
|
||||||
|
}
|
||||||
|
|
||||||
// single is a transcript from one engine only: nothing to reconcile, and the result
|
// single is a transcript from one engine only: nothing to reconcile, and the result
|
||||||
// says why, because a reader should know the second opinion was missing.
|
// says why, because a reader should know the second opinion was missing.
|
||||||
func (e *Ensemble) single(t domain.Transcription, engine, reason string) domain.Transcription {
|
func (e *Ensemble) single(t domain.Transcription, engine, reason string) domain.Transcription {
|
||||||
|
|||||||
@@ -0,0 +1,162 @@
|
|||||||
|
package transcribe_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/progress"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/reconcile"
|
||||||
|
)
|
||||||
|
|
||||||
|
// What the Ensemble reports to the page and the log (ADR-035). The reporter here is a
|
||||||
|
// recording double carried in the context, exactly as the worker's Tracker is.
|
||||||
|
|
||||||
|
type recReporter struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
events []string
|
||||||
|
notes []map[string]any
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *recReporter) add(e string) {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
r.events = append(r.events, e)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *recReporter) Begin(key string, _ int) { r.add("begin:" + key) }
|
||||||
|
func (r *recReporter) Advance(key string, _ int) { r.add("advance:" + key) }
|
||||||
|
func (r *recReporter) Done(key string) { r.add("done:" + key) }
|
||||||
|
func (r *recReporter) Fail(key string, _ error) { r.add("fail:" + key) }
|
||||||
|
func (r *recReporter) Note(msg string, kv ...any) {
|
||||||
|
m := map[string]any{"msg": msg}
|
||||||
|
for i := 0; i+1 < len(kv); i += 2 {
|
||||||
|
m[kv[i].(string)] = kv[i+1]
|
||||||
|
}
|
||||||
|
r.mu.Lock()
|
||||||
|
r.notes = append(r.notes, m)
|
||||||
|
r.mu.Unlock()
|
||||||
|
r.add("note:" + msg)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *recReporter) joined() string {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
return strings.Join(r.events, ",")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEnsemblePlan_DeclaresBothEnginesAndTheReconcileStepsByName(t *testing.T) {
|
||||||
|
r := newEnsemble("a", "a", nil, nil, false, &verdictAdj{})
|
||||||
|
|
||||||
|
plan := r.e.Plan()
|
||||||
|
|
||||||
|
require.Len(t, plan, 4)
|
||||||
|
assert.Equal(t, []string{"engine_a", "engine_b", "compare", "decide"}, []string{plan[0].Key, plan[1].Key, plan[2].Key, plan[3].Key})
|
||||||
|
assert.Contains(t, plan[0].Label, "iguana/pianissimo", "the page says which engine is running")
|
||||||
|
assert.Contains(t, plan[1].Label, "iguana/kb-whisper")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEnsemblePlan_TheCloudFallbackIsAStepOnlyWhenOneIsConfigured(t *testing.T) {
|
||||||
|
with := newEnsemble("a", "a", nil, nil, true, &verdictAdj{}).e.Plan()
|
||||||
|
without := newEnsemble("a", "a", nil, nil, false, &verdictAdj{}).e.Plan()
|
||||||
|
|
||||||
|
assert.Equal(t, "fallback", with[len(with)-1].Key)
|
||||||
|
assert.Len(t, without, 4, "no fallback configured: the page must not promise one")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEnsembleProgress_BothEnginesAreReportedInTheOrderTheyRun(t *testing.T) {
|
||||||
|
rep := &recReporter{}
|
||||||
|
r := newEnsemble("Totalt 1015 kronor idag.", "Totalt 1050 kronor idag.", nil, nil, false, &verdictAdj{v: reconcile.VerdictAlt})
|
||||||
|
|
||||||
|
_, err := r.e.Transcribe(progress.With(context.Background(), rep), "/a.wav")
|
||||||
|
|
||||||
|
require.NoError(t, err)
|
||||||
|
got := rep.joined()
|
||||||
|
a, aDone := strings.Index(got, "begin:engine_a"), strings.Index(got, "done:engine_a")
|
||||||
|
b, bDone := strings.Index(got, "begin:engine_b"), strings.Index(got, "done:engine_b")
|
||||||
|
require.True(t, a >= 0 && aDone > a && b > aDone && bDone > b, "events out of order: %s", got)
|
||||||
|
assert.NotContains(t, got, "fail:")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEnsembleProgress_AnEngineThatFailsIsShownFailedAndTheOtherStillRuns(t *testing.T) {
|
||||||
|
rep := &recReporter{}
|
||||||
|
r := newEnsemble("text", "text", errors.New("pianissimo down"), nil, false, &verdictAdj{})
|
||||||
|
|
||||||
|
got, err := r.e.Transcribe(progress.With(context.Background(), rep), "/a.wav")
|
||||||
|
|
||||||
|
require.NoError(t, err, "one engine down is a degraded result, not a failed job")
|
||||||
|
assert.Equal(t, domain.ReconcileSingle, got.Reconciliation.Status)
|
||||||
|
ev := rep.joined()
|
||||||
|
assert.Contains(t, ev, "fail:engine_a")
|
||||||
|
assert.Contains(t, ev, "done:engine_b")
|
||||||
|
assert.NotContains(t, ev, "begin:compare", "nothing to compare with one engine")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEnsembleProgress_TheFallbackIsReportedOnlyWhenBothEnginesFailed(t *testing.T) {
|
||||||
|
rep := &recReporter{}
|
||||||
|
r := newEnsemble("t", "t", errors.New("a down"), errors.New("b down"), true, &verdictAdj{})
|
||||||
|
|
||||||
|
_, err := r.e.Transcribe(progress.With(context.Background(), rep), "/a.wav")
|
||||||
|
|
||||||
|
require.NoError(t, err)
|
||||||
|
ev := rep.joined()
|
||||||
|
assert.Contains(t, ev, "fail:engine_a")
|
||||||
|
assert.Contains(t, ev, "fail:engine_b")
|
||||||
|
assert.Contains(t, ev, "begin:fallback")
|
||||||
|
assert.Contains(t, ev, "done:fallback")
|
||||||
|
|
||||||
|
rep2 := &recReporter{}
|
||||||
|
ok := newEnsemble("x", "x", nil, nil, true, &verdictAdj{})
|
||||||
|
_, err = ok.e.Transcribe(progress.With(context.Background(), rep2), "/a.wav")
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.NotContains(t, rep2.joined(), "fallback", "a healthy run never touches the cloud step")
|
||||||
|
}
|
||||||
|
|
||||||
|
// cancellingBackend ends the job in the middle of its own call, like a shutdown.
|
||||||
|
type cancellingBackend struct {
|
||||||
|
name string
|
||||||
|
cancel context.CancelFunc
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c cancellingBackend) Name() string { return c.name }
|
||||||
|
func (c cancellingBackend) Transcribe(ctx context.Context, _ string) (domain.Transcription, error) {
|
||||||
|
c.cancel()
|
||||||
|
return domain.Transcription{}, ctx.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEnsembleProgress_AShutdownMidEngineIsNotReportedAsAFailedEngine(t *testing.T) {
|
||||||
|
rep := &recReporter{}
|
||||||
|
ctx, cancel := context.WithCancel(progress.With(context.Background(), rep))
|
||||||
|
calls := &[]string{}
|
||||||
|
alt := &orderedBackend{name: "iguana/kb-whisper", text: "x", calls: calls}
|
||||||
|
e := transcribe.NewEnsemble(cancellingBackend{name: "iguana/pianissimo", cancel: cancel}, alt, nil, &verdictAdj{}, ensembleOpts)
|
||||||
|
|
||||||
|
_, err := e.Transcribe(ctx, "/a.wav")
|
||||||
|
|
||||||
|
require.ErrorIs(t, err, context.Canceled)
|
||||||
|
assert.NotContains(t, rep.joined(), "fail:", "a stopped job is not a broken engine: the page must not blame one")
|
||||||
|
assert.Empty(t, *calls, "and the second engine never starts")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEnsembleProgress_TheReconcileCountsAreNotedForTheOperator(t *testing.T) {
|
||||||
|
rep := &recReporter{}
|
||||||
|
r := newEnsemble("Totalt 1015 kronor idag.", "Totalt 1050 kronor idag.", nil, nil, false, &verdictAdj{v: reconcile.VerdictAlt})
|
||||||
|
|
||||||
|
_, err := r.e.Transcribe(progress.With(context.Background(), rep), "/a.wav")
|
||||||
|
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, rep.notes, 1)
|
||||||
|
n := rep.notes[0]
|
||||||
|
assert.Equal(t, "reconciled", n["msg"])
|
||||||
|
assert.Equal(t, 1, n["disputes"])
|
||||||
|
assert.Equal(t, 1, n["changed"])
|
||||||
|
assert.Equal(t, 0, n["unsure"])
|
||||||
|
assert.Equal(t, 0, n["errors"])
|
||||||
|
}
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
package transcribe_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/reconcile"
|
||||||
|
)
|
||||||
|
|
||||||
|
// When one engine is down the page says so and why; the "why" is a kind, not the
|
||||||
|
// engine's error text, which can carry provider output (adversarial review F3).
|
||||||
|
|
||||||
|
func TestEnsemble_AFailedEnginesReasonIsAKindNotItsErrorText(t *testing.T) {
|
||||||
|
r := newEnsemble("text", "text", nil, errors.New("kb-whisper said: zebrakaka secret words"), false, &verdictAdj{v: reconcile.VerdictAlt})
|
||||||
|
|
||||||
|
got, err := r.e.Transcribe(context.Background(), "/a.wav")
|
||||||
|
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.NotContains(t, got.Reconciliation.Reason, "zebrakaka")
|
||||||
|
assert.Contains(t, got.Reconciliation.Reason, "iguana/kb-whisper failed", "which engine failed is still said")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEnsemble_AnHTTPFailureShowsItsStatusCodeAsTheKind(t *testing.T) {
|
||||||
|
r := newEnsemble("text", "text", &transcribe.StatusError{Code: 502, Snippet: "zebrakaka"}, nil, false, &verdictAdj{})
|
||||||
|
|
||||||
|
got, err := r.e.Transcribe(context.Background(), "/a.wav")
|
||||||
|
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Contains(t, got.Reconciliation.Reason, "http_502")
|
||||||
|
assert.NotContains(t, got.Reconciliation.Reason, "zebrakaka")
|
||||||
|
}
|
||||||
@@ -31,6 +31,9 @@ type StatusError struct {
|
|||||||
Snippet string
|
Snippet string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// HTTPStatus implements errkind.HTTPStatuser: the code, never the snippet.
|
||||||
|
func (e *StatusError) HTTPStatus() int { return e.Code }
|
||||||
|
|
||||||
func (e *StatusError) Error() string {
|
func (e *StatusError) Error() string {
|
||||||
return fmt.Sprintf("transcribe: gateway returned HTTP %d: %s", e.Code, e.Snippet)
|
return fmt.Sprintf("transcribe: gateway returned HTTP %d: %s", e.Code, e.Snippet)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,11 +3,13 @@ package transcribe_test
|
|||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
@@ -15,6 +17,7 @@ import (
|
|||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
|
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/errkind"
|
||||||
)
|
)
|
||||||
|
|
||||||
// The httptest server stands in for the LiteLLM gateway: it is the process
|
// The httptest server stands in for the LiteLLM gateway: it is the process
|
||||||
@@ -230,3 +233,25 @@ func TestGateway_WithNoSegmentsTheTextFieldIsUsedWithWhitespaceCollapsed(t *test
|
|||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Equal(t, "Hej allihopa. Vi börjar.", got.Text)
|
assert.Equal(t, "Hej allihopa. Vi börjar.", got.Text)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestStatusError_ExposesItsCodeAsAKindAndNotItsSnippet(t *testing.T) {
|
||||||
|
err := fmt.Errorf("transcribe: %w", &transcribe.StatusError{Code: 502, Snippet: "echoed: Anders ansvarar"})
|
||||||
|
|
||||||
|
assert.Equal(t, "http_502", errkind.Of(err))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGateway_AnErrorBodyIsBoundedToTwoHundredBytesInTheStatusError(t *testing.T) {
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusInternalServerError)
|
||||||
|
_, _ = w.Write([]byte(strings.Repeat("x", 100_000)))
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
wav := filepath.Join(t.TempDir(), "a.wav")
|
||||||
|
require.NoError(t, os.WriteFile(wav, []byte("RIFF....WAVEfmt "), 0o600))
|
||||||
|
|
||||||
|
_, err := transcribe.NewGateway(srv.URL, "", "iguana/kb-whisper", srv.Client()).Transcribe(context.Background(), wav)
|
||||||
|
|
||||||
|
var se *transcribe.StatusError
|
||||||
|
require.ErrorAs(t, err, &se)
|
||||||
|
assert.LessOrEqual(t, len(se.Snippet), 200, "a gateway's whole body must not become part of an error")
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,33 @@
|
|||||||
|
package transcribe
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
)
|
||||||
|
|
||||||
|
// WithTimeout gives every call to b its own deadline. A wedged engine does not answer at
|
||||||
|
// all (iguana after a Metal OOM, infra#273), and with only the job's deadline it used up
|
||||||
|
// the whole job before a fallback could run -- and Chain then refuses the fallback because
|
||||||
|
// the job context has expired. With its own deadline the call fails while the job is still
|
||||||
|
// alive, which is exactly when the fallback, or the other engine, can take over.
|
||||||
|
//
|
||||||
|
// A zero or negative d returns b unchanged. The wrapper is invisible: Name is b's.
|
||||||
|
func WithTimeout(b Backend, d time.Duration) Backend {
|
||||||
|
if d <= 0 {
|
||||||
|
return b
|
||||||
|
}
|
||||||
|
return timeoutBackend{Backend: b, d: d}
|
||||||
|
}
|
||||||
|
|
||||||
|
type timeoutBackend struct {
|
||||||
|
Backend
|
||||||
|
d time.Duration
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t timeoutBackend) Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error) {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, t.d)
|
||||||
|
defer cancel()
|
||||||
|
return t.Backend.Transcribe(ctx, wavPath)
|
||||||
|
}
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
package transcribe_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/errkind"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/reconcile"
|
||||||
|
)
|
||||||
|
|
||||||
|
// A wedged engine (iguana after a Metal OOM, infra#273) does not answer at all. With only
|
||||||
|
// the job's deadline, the primary used up the whole job before any fallback could run, and
|
||||||
|
// the chain then refused the fallback because the job context had expired. Each engine call
|
||||||
|
// gets its own deadline (adversarial review F4).
|
||||||
|
|
||||||
|
type hangingBackend struct{ name string }
|
||||||
|
|
||||||
|
func (h hangingBackend) Name() string { return h.name }
|
||||||
|
func (hangingBackend) Transcribe(ctx context.Context, _ string) (domain.Transcription, error) {
|
||||||
|
<-ctx.Done()
|
||||||
|
return domain.Transcription{}, ctx.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWithTimeout_AHungBackendFailsAtItsOwnDeadlineWhileTheJobIsStillAlive(t *testing.T) {
|
||||||
|
job, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
b := transcribe.WithTimeout(hangingBackend{name: "iguana/kb-whisper"}, 40*time.Millisecond)
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
_, err := b.Transcribe(job, "/a.wav")
|
||||||
|
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.Equal(t, "timeout", errkind.Of(err))
|
||||||
|
assert.Less(t, time.Since(start), 2*time.Second, "it must give up at its own deadline, not wait for the job's")
|
||||||
|
assert.NoError(t, job.Err(), "the job context is untouched, which is what lets a fallback run")
|
||||||
|
assert.Equal(t, "iguana/kb-whisper", b.Name(), "the wrapper is invisible in the recorded endpoint")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWithTimeout_ASuccessPassesStraightThrough(t *testing.T) {
|
||||||
|
calls := &[]string{}
|
||||||
|
inner := &orderedBackend{name: "iguana/kb-whisper", text: "hej", calls: calls}
|
||||||
|
|
||||||
|
got, err := transcribe.WithTimeout(inner, time.Minute).Transcribe(context.Background(), "/a.wav")
|
||||||
|
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, "hej", got.Text)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWithTimeout_ZeroMeansNoDeadlineOfItsOwn(t *testing.T) {
|
||||||
|
inner := &orderedBackend{name: "x", text: "hej", calls: &[]string{}}
|
||||||
|
|
||||||
|
assert.Same(t, inner, transcribe.WithTimeout(inner, 0), "a zero timeout must not wrap or change anything")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestChain_AHungPrimaryReachesTheFallbackWhenItHasItsOwnDeadline(t *testing.T) {
|
||||||
|
calls := &[]string{}
|
||||||
|
fb := &orderedBackend{name: "berget/kb-whisper", text: "fallback text", calls: calls}
|
||||||
|
c := transcribe.NewChain(transcribe.WithTimeout(hangingBackend{name: "iguana/kb-whisper"}, 40*time.Millisecond), fb)
|
||||||
|
job, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
got, err := c.Transcribe(job, "/a.wav")
|
||||||
|
|
||||||
|
require.NoError(t, err, "a wedged primary is exactly what the fallback exists for")
|
||||||
|
assert.Equal(t, "berget/kb-whisper", got.Endpoint)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestChain_AJobThatEndedWhileThePrimaryHungStillNeverReachesTheFallback(t *testing.T) {
|
||||||
|
calls := &[]string{}
|
||||||
|
fb := &orderedBackend{name: "berget/kb-whisper", text: "fallback text", calls: calls}
|
||||||
|
c := transcribe.NewChain(transcribe.WithTimeout(hangingBackend{name: "iguana/kb-whisper"}, time.Minute), fb)
|
||||||
|
job, cancel := context.WithTimeout(context.Background(), 40*time.Millisecond)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
_, err := c.Transcribe(job, "/a.wav")
|
||||||
|
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.Empty(t, *calls, "the job's own deadline or a cancel must still stop the audio from reaching a cloud endpoint")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEnsemble_AHungEngineIsASoftFailureAndTheOtherEnginesTextIsUsed(t *testing.T) {
|
||||||
|
calls := &[]string{}
|
||||||
|
alt := &orderedBackend{name: "iguana/kb-whisper", text: "text from the other engine", calls: calls}
|
||||||
|
e := transcribe.NewEnsemble(
|
||||||
|
transcribe.WithTimeout(hangingBackend{name: "iguana/pianissimo"}, 40*time.Millisecond), alt, nil,
|
||||||
|
&verdictAdj{v: reconcile.VerdictAlt}, ensembleOpts)
|
||||||
|
job, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
got, err := e.Transcribe(job, "/a.wav")
|
||||||
|
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, "text from the other engine", got.Text)
|
||||||
|
assert.Equal(t, domain.ReconcileSingle, got.Reconciliation.Status)
|
||||||
|
assert.Contains(t, got.Reconciliation.Reason, "timeout")
|
||||||
|
assert.False(t, errors.Is(err, context.DeadlineExceeded))
|
||||||
|
}
|
||||||
+30
-12
@@ -3,6 +3,7 @@ package config
|
|||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
|
"regexp"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
@@ -27,6 +28,10 @@ type UploadConfig struct {
|
|||||||
// TranscribeTimeout bounds one job (conversion + transcription); a 90 minute
|
// TranscribeTimeout bounds one job (conversion + transcription); a 90 minute
|
||||||
// recording took ~7.5 minutes on iguana.
|
// recording took ~7.5 minutes on iguana.
|
||||||
TranscribeTimeout time.Duration
|
TranscribeTimeout time.Duration
|
||||||
|
// TranscribeEngineTimeout bounds ONE engine call, so a wedged engine fails while the
|
||||||
|
// job is still alive and the fallback (or the other engine) can take over. The job
|
||||||
|
// timeout alone cannot do that (ADR-035 review).
|
||||||
|
TranscribeEngineTimeout time.Duration
|
||||||
// UploadTmpDir is where uploads are spooled and converted. Audio is deleted
|
// UploadTmpDir is where uploads are spooled and converted. Audio is deleted
|
||||||
// after transcription, so this holds at most the in-flight files.
|
// after transcription, so this holds at most the in-flight files.
|
||||||
UploadTmpDir string
|
UploadTmpDir string
|
||||||
@@ -61,6 +66,7 @@ const (
|
|||||||
defaultTranscribeModel = "iguana/kb-whisper"
|
defaultTranscribeModel = "iguana/kb-whisper"
|
||||||
defaultTranscribeFallbackModel = "berget/kb-whisper"
|
defaultTranscribeFallbackModel = "berget/kb-whisper"
|
||||||
defaultTranscribeTimeout = 60 * time.Minute
|
defaultTranscribeTimeout = 60 * time.Minute
|
||||||
|
defaultTranscribeEngineTimeout = 20 * time.Minute
|
||||||
defaultMaxUploadedChars = 90000
|
defaultMaxUploadedChars = 90000
|
||||||
defaultUploadChatModel = "iguana/gemma4-26b"
|
defaultUploadChatModel = "iguana/gemma4-26b"
|
||||||
)
|
)
|
||||||
@@ -106,6 +112,15 @@ func loadUploadConfig() (UploadConfig, error) {
|
|||||||
}
|
}
|
||||||
u.TranscribeTimeout = timeout
|
u.TranscribeTimeout = timeout
|
||||||
|
|
||||||
|
engineTimeout, err := durationOr("TAPIR_TRANSCRIBE_ENGINE_TIMEOUT", defaultTranscribeEngineTimeout)
|
||||||
|
if err != nil {
|
||||||
|
return UploadConfig{}, err
|
||||||
|
}
|
||||||
|
if engineTimeout <= 0 {
|
||||||
|
return UploadConfig{}, fmt.Errorf("config: TAPIR_TRANSCRIBE_ENGINE_TIMEOUT must be positive, got %v", engineTimeout)
|
||||||
|
}
|
||||||
|
u.TranscribeEngineTimeout = engineTimeout
|
||||||
|
|
||||||
if v := os.Getenv("TAPIR_RECONCILE_CHECK_ORDER"); v != "" {
|
if v := os.Getenv("TAPIR_RECONCILE_CHECK_ORDER"); v != "" {
|
||||||
on, err := strconv.ParseBool(v)
|
on, err := strconv.ParseBool(v)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -116,6 +131,9 @@ func loadUploadConfig() (UploadConfig, error) {
|
|||||||
if err := u.validateReconcile(); err != nil {
|
if err := u.validateReconcile(); err != nil {
|
||||||
return UploadConfig{}, err
|
return UploadConfig{}, err
|
||||||
}
|
}
|
||||||
|
if !IsLocalModel(u.UploadChatModel) {
|
||||||
|
return UploadConfig{}, fmt.Errorf("config: TAPIR_UPLOAD_CHAT_MODEL=%q is not a local model (koala/ or iguana/): the whole private transcript is sent to it", u.UploadChatModel)
|
||||||
|
}
|
||||||
|
|
||||||
chars, err := intOr("TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS", defaultMaxUploadedChars)
|
chars, err := intOr("TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS", defaultMaxUploadedChars)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -141,18 +159,18 @@ func positiveInt(key string, fallback int) (int, error) {
|
|||||||
return n, nil
|
return n, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// localPrefixes are the host prefixes of the models that run on the homelab. Private
|
// localAlias is the shape of a model alias that runs on the homelab: a local host
|
||||||
// meeting content may be sent to these and to nothing else for reconciliation
|
// prefix, then one plain name. A bare prefix check let through aliases that are shaped
|
||||||
// (ADR-034): a cloud alias is refused at start-up rather than discovered later.
|
// like an escape (koala/../berget/x, koala//x, koala/x?host=berget): whether the
|
||||||
var localPrefixes = []string{"koala/", "iguana/"}
|
// gateway would route them elsewhere is unknown, so they are refused (ADR-035 review).
|
||||||
|
var localAlias = regexp.MustCompile(`^(koala|iguana)/[A-Za-z0-9][A-Za-z0-9._:-]*$`)
|
||||||
|
|
||||||
func isLocalModel(alias string) bool {
|
// IsLocalModel reports whether alias is a local model. Private meeting content may be
|
||||||
for _, p := range localPrefixes {
|
// sent to these and to nothing else: reconciliation (ADR-034) and chat over an uploaded
|
||||||
if strings.HasPrefix(alias, p) {
|
// recording (ADR-035) are both held to it, and a cloud alias is refused at start-up
|
||||||
return true
|
// rather than discovered later.
|
||||||
}
|
func IsLocalModel(alias string) bool {
|
||||||
}
|
return localAlias.MatchString(alias) && !strings.Contains(alias, "..")
|
||||||
return false
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// validateReconcile checks the two-engine settings. They are inert while no base
|
// validateReconcile checks the two-engine settings. They are inert while no base
|
||||||
@@ -168,7 +186,7 @@ func (u UploadConfig) validateReconcile() error {
|
|||||||
{"TAPIR_RECONCILE_BASE_MODEL", u.ReconcileBaseModel},
|
{"TAPIR_RECONCILE_BASE_MODEL", u.ReconcileBaseModel},
|
||||||
{"TAPIR_RECONCILE_MODEL", u.ReconcileModel},
|
{"TAPIR_RECONCILE_MODEL", u.ReconcileModel},
|
||||||
} {
|
} {
|
||||||
if !isLocalModel(m.alias) {
|
if !IsLocalModel(m.alias) {
|
||||||
return fmt.Errorf("config: %s=%q is not a local model (koala/ or iguana/): private meeting content must not leave the homelab for reconciliation", m.name, m.alias)
|
return fmt.Errorf("config: %s=%q is not a local model (koala/ or iguana/): private meeting content must not leave the homelab for reconciliation", m.name, m.alias)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,101 @@
|
|||||||
|
package config
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// What counts as a LOCAL model alias, and where that is demanded (ADR-034, ADR-035 review).
|
||||||
|
// A private transcript may be sent to a local model and to nothing else; the check is a
|
||||||
|
// guard against a misconfiguration or a crafted alias, so it must be strict about shape,
|
||||||
|
// not just about a prefix.
|
||||||
|
|
||||||
|
func TestIsLocalModel_AcceptsRealLocalAliasesAndNothingShapedLikeAnEscape(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
alias string
|
||||||
|
want bool
|
||||||
|
}{
|
||||||
|
{"koala/phi4-mini", true},
|
||||||
|
{"iguana/gemma4-26b", true},
|
||||||
|
{"iguana/gemma4-31b-think", true},
|
||||||
|
{"koala/qwen36-35b-a3b", true},
|
||||||
|
{"iguana/kb-whisper", true},
|
||||||
|
{"iguana/hf.co-model_v1:q4", true},
|
||||||
|
{"", false},
|
||||||
|
{"koala/", false},
|
||||||
|
{"iguana/", false},
|
||||||
|
{"gemma4-26b", false},
|
||||||
|
{"berget/mistral-small", false},
|
||||||
|
{"berget/koala/x", false},
|
||||||
|
{"xkoala/x", false},
|
||||||
|
{"Koala/x", false},
|
||||||
|
{" koala/x", false},
|
||||||
|
{"koala/x ", false},
|
||||||
|
{"koala/x y", false},
|
||||||
|
{"koala/x\n", false},
|
||||||
|
{"koala/../berget/kb", false},
|
||||||
|
{"koala//berget/mistral-small", false},
|
||||||
|
{"koala/x/y", false},
|
||||||
|
{"koala/x?host=berget", false},
|
||||||
|
{"koala/x#frag", false},
|
||||||
|
{"koala/x@evil", false},
|
||||||
|
{"koalа/x", false}, // Cyrillic а
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.alias, func(t *testing.T) {
|
||||||
|
if got := IsLocalModel(tt.alias); got != tt.want {
|
||||||
|
t.Errorf("IsLocalModel(%q) = %v, want %v", tt.alias, got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoad_UploadChatModelMustBeLocalBecauseTheWholeTranscriptIsSentToIt(t *testing.T) {
|
||||||
|
for _, alias := range []string{"berget/mistral-small", "gemini/gemini-2.5-pro", "gemma4-26b", "koala/../berget/kb"} {
|
||||||
|
t.Run(alias, func(t *testing.T) {
|
||||||
|
clearUploadEnv(t)
|
||||||
|
setEnv(t, map[string]string{"TAPIR_UPLOAD_CHAT_MODEL": alias})
|
||||||
|
|
||||||
|
if _, err := Load(); err == nil {
|
||||||
|
t.Errorf("Load accepted TAPIR_UPLOAD_CHAT_MODEL=%q: a private transcript of up to 90,000 characters would be sent to it", alias)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoad_ALocalUploadChatModelIsAccepted(t *testing.T) {
|
||||||
|
clearUploadEnv(t)
|
||||||
|
setEnv(t, map[string]string{"TAPIR_UPLOAD_CHAT_MODEL": "iguana/gemma4-31b"})
|
||||||
|
|
||||||
|
cfg, err := Load()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load: %v", err)
|
||||||
|
}
|
||||||
|
if cfg.UploadChatModel != "iguana/gemma4-31b" {
|
||||||
|
t.Errorf("UploadChatModel = %q", cfg.UploadChatModel)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoad_TheEngineTimeoutDefaultsToTwentyMinutesAndMustBePositive(t *testing.T) {
|
||||||
|
clearUploadEnv(t)
|
||||||
|
cfg, err := Load()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load: %v", err)
|
||||||
|
}
|
||||||
|
if cfg.TranscribeEngineTimeout != 20*time.Minute {
|
||||||
|
t.Errorf("TranscribeEngineTimeout = %v, want 20m: a 90-minute recording takes about 8 minutes on kb-whisper, and two wedged engines in sequence must leave the job time to reach a fallback", cfg.TranscribeEngineTimeout)
|
||||||
|
}
|
||||||
|
|
||||||
|
setEnv(t, map[string]string{"TAPIR_TRANSCRIBE_ENGINE_TIMEOUT": "7m"})
|
||||||
|
cfg, err = Load()
|
||||||
|
if err != nil || cfg.TranscribeEngineTimeout != 7*time.Minute {
|
||||||
|
t.Errorf("override: %v, %v", cfg.TranscribeEngineTimeout, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, bad := range []string{"0", "-1m", "soon"} {
|
||||||
|
setEnv(t, map[string]string{"TAPIR_TRANSCRIBE_ENGINE_TIMEOUT": bad})
|
||||||
|
if _, err := Load(); err == nil {
|
||||||
|
t.Errorf("Load accepted TAPIR_TRANSCRIBE_ENGINE_TIMEOUT=%q", bad)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -12,7 +12,7 @@ func clearUploadEnv(t *testing.T) {
|
|||||||
t.Helper()
|
t.Helper()
|
||||||
for _, k := range []string{
|
for _, k := range []string{
|
||||||
"TAPIR_UPLOADS_ENABLED", "TAPIR_UPLOAD_MAX_MB", "TAPIR_UPLOAD_MAX_MINUTES",
|
"TAPIR_UPLOADS_ENABLED", "TAPIR_UPLOAD_MAX_MB", "TAPIR_UPLOAD_MAX_MINUTES",
|
||||||
"TAPIR_TRANSCRIBE_MODEL", "TAPIR_TRANSCRIBE_FALLBACK_MODEL", "TAPIR_TRANSCRIBE_TIMEOUT",
|
"TAPIR_TRANSCRIBE_MODEL", "TAPIR_TRANSCRIBE_FALLBACK_MODEL", "TAPIR_TRANSCRIBE_TIMEOUT", "TAPIR_TRANSCRIBE_ENGINE_TIMEOUT",
|
||||||
"TAPIR_UPLOAD_TMP_DIR", "TAPIR_FFMPEG_BIN", "TAPIR_FFPROBE_BIN",
|
"TAPIR_UPLOAD_TMP_DIR", "TAPIR_FFMPEG_BIN", "TAPIR_FFPROBE_BIN",
|
||||||
"TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS", "TAPIR_UPLOAD_CHAT_MODEL",
|
"TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS", "TAPIR_UPLOAD_CHAT_MODEL",
|
||||||
"TAPIR_RECONCILE_BASE_MODEL", "TAPIR_RECONCILE_MODEL", "TAPIR_RECONCILE_CHECK_ORDER",
|
"TAPIR_RECONCILE_BASE_MODEL", "TAPIR_RECONCILE_MODEL", "TAPIR_RECONCILE_CHECK_ORDER",
|
||||||
|
|||||||
@@ -147,14 +147,18 @@ const (
|
|||||||
ReconcileSkipped ReconcileStatus = "skipped"
|
ReconcileSkipped ReconcileStatus = "skipped"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ContestedSpot is a place the engines disagreed that a reader may want to check
|
// ContestedSpot is a place the engines disagreed that the model judged: it changed it,
|
||||||
// against the audio: one the model changed, or one it was unsure about.
|
// kept the base's reading, or could not decide. Every judged spot is stored, not only
|
||||||
|
// the ones a reader should check, because the audio is not retained and the kept ones
|
||||||
|
// are what a later evaluation needs (ADR-035). The page lists only the changed and
|
||||||
|
// uncertain ones.
|
||||||
type ContestedSpot struct {
|
type ContestedSpot struct {
|
||||||
Seconds float64 `json:"seconds"` // roughly where in the recording
|
Seconds float64 `json:"seconds"` // roughly where in the recording
|
||||||
Base string `json:"base"` // what the base engine heard
|
Base string `json:"base"` // what the base engine heard
|
||||||
Alt string `json:"alt"` // what the other engine heard
|
Alt string `json:"alt"` // what the other engine heard
|
||||||
Used string `json:"used"` // "base" or "alt": the reading now in the text
|
Used string `json:"used"` // "base" or "alt": the reading now in the text
|
||||||
Unsure bool `json:"unsure"` // the model could not decide; the base was kept
|
Unsure bool `json:"unsure"` // the model could not decide; the base was kept
|
||||||
|
Before string `json:"before,omitempty"` // the few words just before the spot, to find it by
|
||||||
}
|
}
|
||||||
|
|
||||||
// Reconciliation records how a merged transcript came about, so the page can say so
|
// Reconciliation records how a merged transcript came about, so the page can say so
|
||||||
@@ -175,6 +179,36 @@ type Reconciliation struct {
|
|||||||
Contested []ContestedSpot `json:"contested,omitempty"`
|
Contested []ContestedSpot `json:"contested,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// StepState is where one phase of an upload job stands (ADR-035).
|
||||||
|
type StepState string
|
||||||
|
|
||||||
|
const (
|
||||||
|
StepPending StepState = "pending"
|
||||||
|
StepRunning StepState = "running"
|
||||||
|
StepDone StepState = "done"
|
||||||
|
StepFailed StepState = "failed"
|
||||||
|
// StepSkipped is a step that never ran because the job finished without it, for
|
||||||
|
// example the second engine's step when the first engine's text was used alone.
|
||||||
|
StepSkipped StepState = "skipped"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ProgressStep is one named phase of an upload job. Total is zero for a phase with no
|
||||||
|
// count; only a phase whose work is really countable carries one (ADR-035: no
|
||||||
|
// percentage that does not come from real counts).
|
||||||
|
type ProgressStep struct {
|
||||||
|
Key string `json:"key"`
|
||||||
|
Label string `json:"label"`
|
||||||
|
State StepState `json:"state"`
|
||||||
|
StartedAt time.Time `json:"started_at,omitzero"`
|
||||||
|
FinishedAt time.Time `json:"finished_at,omitzero"`
|
||||||
|
Done int `json:"done,omitempty"`
|
||||||
|
Total int `json:"total,omitempty"`
|
||||||
|
// Note is a short plain remark about the phase that the reader should see even though
|
||||||
|
// it did not fail, for example that the audio is much shorter than the file claimed.
|
||||||
|
// Durations and counts only: never transcript text, a name or a path.
|
||||||
|
Note string `json:"note,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
// UploadStatus is the state of an uploaded-audio transcription job.
|
// UploadStatus is the state of an uploaded-audio transcription job.
|
||||||
type UploadStatus string
|
type UploadStatus string
|
||||||
|
|
||||||
@@ -196,6 +230,13 @@ const (
|
|||||||
UploadErrInterrupted UploadErrorKind = "interrupted"
|
UploadErrInterrupted UploadErrorKind = "interrupted"
|
||||||
// UploadErrTranscriptionFailed: every configured transcription endpoint failed.
|
// UploadErrTranscriptionFailed: every configured transcription endpoint failed.
|
||||||
UploadErrTranscriptionFailed UploadErrorKind = "transcription_failed"
|
UploadErrTranscriptionFailed UploadErrorKind = "transcription_failed"
|
||||||
|
// UploadErrTooLong: the audio, once converted, is longer than the limit. The limit is
|
||||||
|
// first checked against the file's header, which a hostile or damaged file can lie
|
||||||
|
// about (ADR-035 review).
|
||||||
|
UploadErrTooLong UploadErrorKind = "audio_too_long"
|
||||||
|
// UploadErrBusy: too many recordings were already waiting, so this one was not
|
||||||
|
// queued. The user's to retry later (ADR-035 review).
|
||||||
|
UploadErrBusy UploadErrorKind = "queue_full"
|
||||||
)
|
)
|
||||||
|
|
||||||
// UploadedTranscript is a user's PRIVATE transcript of audio they uploaded. It is
|
// UploadedTranscript is a user's PRIVATE transcript of audio they uploaded. It is
|
||||||
@@ -218,6 +259,13 @@ type UploadedTranscript struct {
|
|||||||
// Reconciliation is how a merged transcript came about, or nil for a single
|
// Reconciliation is how a merged transcript came about, or nil for a single
|
||||||
// engine (ADR-034). Set by Get, not by List.
|
// engine (ADR-034). Set by Get, not by List.
|
||||||
Reconciliation *Reconciliation
|
Reconciliation *Reconciliation
|
||||||
CreatedAt time.Time
|
// Progress is the job's phases, in order. Set by Get, not by List (ADR-035).
|
||||||
FinishedAt time.Time // zero until the job reaches done or failed
|
Progress []ProgressStep
|
||||||
|
// ProgressAt is when a phase last really changed or advanced; AliveAt is the
|
||||||
|
// heartbeat. They differ on purpose: alive says the service is still reporting,
|
||||||
|
// progress says work is moving (ADR-035). Both are zero until the job starts.
|
||||||
|
ProgressAt time.Time
|
||||||
|
AliveAt time.Time
|
||||||
|
CreatedAt time.Time
|
||||||
|
FinishedAt time.Time // zero until the job reaches done or failed
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,51 @@
|
|||||||
|
// Package errkind reduces an error to a short kind that is safe to log, store and show.
|
||||||
|
//
|
||||||
|
// An error's own text can hold a provider's response body (a gateway that echoes the
|
||||||
|
// prompt), a file path (ffmpeg names the spool file), or a word from a private
|
||||||
|
// transcript. None of that may reach a log line, a database column or a page about a
|
||||||
|
// private recording (ADR-033: "typed, never a provider's string"; ADR-035). Of returns
|
||||||
|
// only a fixed vocabulary, so it can never carry any.
|
||||||
|
package errkind
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"net"
|
||||||
|
)
|
||||||
|
|
||||||
|
// HTTPStatuser is implemented by errors that are a non-2xx answer from a service.
|
||||||
|
type HTTPStatuser interface{ HTTPStatus() int }
|
||||||
|
|
||||||
|
// SQLStater is implemented by database errors that carry a SQLSTATE (pgx's PgError does).
|
||||||
|
// The five-character code is a fixed vocabulary; the error's message is not: Postgres quotes
|
||||||
|
// the offending value, which for a transcript row is transcript text.
|
||||||
|
type SQLStater interface{ SQLState() string }
|
||||||
|
|
||||||
|
// Of is the kind of err: "timeout", "canceled", "http_<code>", "sql_<state>", "network",
|
||||||
|
// or "error".
|
||||||
|
// It is empty for nil.
|
||||||
|
func Of(err error) string {
|
||||||
|
if err == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
var hs HTTPStatuser
|
||||||
|
var ss SQLStater
|
||||||
|
var ne net.Error
|
||||||
|
switch {
|
||||||
|
case errors.Is(err, context.DeadlineExceeded):
|
||||||
|
return "timeout"
|
||||||
|
case errors.Is(err, context.Canceled):
|
||||||
|
return "canceled"
|
||||||
|
case errors.As(err, &hs):
|
||||||
|
return fmt.Sprintf("http_%d", hs.HTTPStatus())
|
||||||
|
case errors.As(err, &ss) && ss.SQLState() != "":
|
||||||
|
return "sql_" + ss.SQLState()
|
||||||
|
case errors.As(err, &ne):
|
||||||
|
if ne.Timeout() {
|
||||||
|
return "timeout"
|
||||||
|
}
|
||||||
|
return "network"
|
||||||
|
}
|
||||||
|
return "error"
|
||||||
|
}
|
||||||
@@ -0,0 +1,82 @@
|
|||||||
|
package errkind_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"net"
|
||||||
|
"net/url"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/errkind"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Of reduces an error to a short kind that is safe to log, store and show: it never
|
||||||
|
// carries the error's own text, because that text can hold a provider's response body, a
|
||||||
|
// file path or a word from a private transcript (ADR-035 review F3, F5).
|
||||||
|
|
||||||
|
type statusErr struct{ code int }
|
||||||
|
|
||||||
|
func (e statusErr) Error() string {
|
||||||
|
return fmt.Sprintf("gateway said: %d and echoed the transcript", e.code)
|
||||||
|
}
|
||||||
|
func (e statusErr) HTTPStatus() int { return e.code }
|
||||||
|
|
||||||
|
type timeoutNet struct{}
|
||||||
|
|
||||||
|
func (timeoutNet) Error() string { return "i/o timeout to 10.0.0.1" }
|
||||||
|
func (timeoutNet) Timeout() bool { return true }
|
||||||
|
func (timeoutNet) Temporary() bool { return true }
|
||||||
|
|
||||||
|
func TestOf_ReducesAnErrorToAKindAndNeverItsText(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
err error
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{"nothing", nil, ""},
|
||||||
|
{"a deadline", context.DeadlineExceeded, "timeout"},
|
||||||
|
{"a wrapped deadline", fmt.Errorf("transcribe: %w", context.DeadlineExceeded), "timeout"},
|
||||||
|
{"a cancellation", fmt.Errorf("job: %w", context.Canceled), "canceled"},
|
||||||
|
{"an HTTP status", statusErr{code: 502}, "http_502"},
|
||||||
|
{"a wrapped HTTP status", fmt.Errorf("model: %w", statusErr{code: 400}), "http_400"},
|
||||||
|
{"a network timeout", timeoutNet{}, "timeout"},
|
||||||
|
{"a network failure", &net.OpError{Op: "dial", Err: errors.New("connection refused")}, "network"},
|
||||||
|
{"a url error wrapping a refusal", &url.Error{Op: "Post", URL: "http://gateway/secret-path", Err: errors.New("refused")}, "network"},
|
||||||
|
{"anything else", errors.New("moov atom not found in /spool/upload-555"), "error"},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
assert.Equal(t, tt.want, errkind.Of(tt.err))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type sqlErr struct{}
|
||||||
|
|
||||||
|
func (sqlErr) Error() string {
|
||||||
|
return `ERROR: invalid input syntax for type json: "Anders ansvarar för virket"`
|
||||||
|
}
|
||||||
|
func (sqlErr) SQLState() string { return "22P02" }
|
||||||
|
|
||||||
|
func TestOf_ADatabaseErrorIsReducedToItsSQLStateNeverItsMessage(t *testing.T) {
|
||||||
|
// A Postgres error can quote the offending value, which for a transcript row is
|
||||||
|
// transcript text. The five-character SQLSTATE is a fixed vocabulary (fourth review).
|
||||||
|
err := fmt.Errorf("store: complete upload: %w", sqlErr{})
|
||||||
|
|
||||||
|
got := errkind.Of(err)
|
||||||
|
|
||||||
|
assert.Equal(t, "sql_22P02", got)
|
||||||
|
assert.NotContains(t, got, "Anders")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOf_AKindCarriesNoWordFromTheError(t *testing.T) {
|
||||||
|
err := fmt.Errorf("convert /spool/upload-555: %w", errors.New("Anders ansvarar för virket"))
|
||||||
|
|
||||||
|
got := errkind.Of(err)
|
||||||
|
|
||||||
|
assert.NotContains(t, got, "Anders")
|
||||||
|
assert.NotContains(t, got, "spool")
|
||||||
|
}
|
||||||
@@ -0,0 +1,284 @@
|
|||||||
|
// Package progress tracks the phases of one upload job (ADR-035). A Tracker is the
|
||||||
|
// only writer of a job's progress: every transition is saved to the job's row and
|
||||||
|
// logged by the same call, so what the page shows and what the log says cannot
|
||||||
|
// disagree. The engines and the reconciler report through a Reporter carried in the
|
||||||
|
// context, so the Transcriber interface stays what it was.
|
||||||
|
package progress
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"log/slog"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/errkind"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Planned is one phase of a job, declared up front so the page can show what is coming.
|
||||||
|
type Planned struct {
|
||||||
|
Key, Label string
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reporter is what a long-running component reports to. Begin starts a phase (total
|
||||||
|
// is the number of countable units, 0 when there is no honest count), Advance says how
|
||||||
|
// many are done, Done closes the phase. Fail closes a phase that did not work while
|
||||||
|
// the job goes on (an engine that was down, the other one's text used instead) and
|
||||||
|
// says why in the log. Note is a log line for the operator that costs no write.
|
||||||
|
type Reporter interface {
|
||||||
|
Begin(key string, total int)
|
||||||
|
Advance(key string, done int)
|
||||||
|
Done(key string)
|
||||||
|
Fail(key string, err error)
|
||||||
|
Note(msg string, kv ...any)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sink is where a job's progress is persisted. Both calls are best effort: progress
|
||||||
|
// is for people, so a failed write is logged and never stops the job.
|
||||||
|
type Sink interface {
|
||||||
|
SaveProgress(ctx context.Context, steps []domain.ProgressStep, at time.Time) error
|
||||||
|
Beat(ctx context.Context) error
|
||||||
|
}
|
||||||
|
|
||||||
|
const (
|
||||||
|
defaultMinSave = 2 * time.Second
|
||||||
|
logAdvanceEach = time.Minute
|
||||||
|
)
|
||||||
|
|
||||||
|
// Option tunes a Tracker.
|
||||||
|
type Option func(*Tracker)
|
||||||
|
|
||||||
|
// WithClock fixes the clock, for tests.
|
||||||
|
func WithClock(now func() time.Time) Option { return func(t *Tracker) { t.now = now } }
|
||||||
|
|
||||||
|
// WithMinSave sets the least time between two saves caused by Advance alone. Begin,
|
||||||
|
// Done, Fail and Finish always save.
|
||||||
|
func WithMinSave(d time.Duration) Option { return func(t *Tracker) { t.minSave = d } }
|
||||||
|
|
||||||
|
// Tracker implements Reporter for one job.
|
||||||
|
type Tracker struct {
|
||||||
|
ctx context.Context
|
||||||
|
sink Sink
|
||||||
|
log *slog.Logger
|
||||||
|
now func() time.Time
|
||||||
|
minSave time.Duration
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
steps []domain.ProgressStep
|
||||||
|
lastAdvanceSave time.Time
|
||||||
|
lastAdvanceLog time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewTracker plans a job's phases. ctx is used for the saves, so it should outlive the
|
||||||
|
// job's own timeout: a job that timed out still gets its failure recorded.
|
||||||
|
func NewTracker(ctx context.Context, sink Sink, log *slog.Logger, plan []Planned, opts ...Option) *Tracker {
|
||||||
|
t := &Tracker{ctx: ctx, sink: sink, log: log, now: time.Now, minSave: defaultMinSave}
|
||||||
|
for _, o := range opts {
|
||||||
|
o(t)
|
||||||
|
}
|
||||||
|
for _, p := range plan {
|
||||||
|
t.steps = append(t.steps, domain.ProgressStep{Key: p.Key, Label: p.Label, State: domain.StepPending})
|
||||||
|
}
|
||||||
|
return t
|
||||||
|
}
|
||||||
|
|
||||||
|
// Start saves the plan, every phase pending.
|
||||||
|
func (t *Tracker) Start() {
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
t.log.Info("job started", "steps", len(t.steps))
|
||||||
|
t.save()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Begin implements Reporter.
|
||||||
|
func (t *Tracker) Begin(key string, total int) {
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
s := t.find(key)
|
||||||
|
if s == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.State, s.StartedAt, s.FinishedAt, s.Done, s.Total = domain.StepRunning, t.now(), time.Time{}, 0, total
|
||||||
|
t.log.Info("step started", "step", key, "total", total)
|
||||||
|
t.save()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Advance implements Reporter. The count is always kept; it is saved at most once per
|
||||||
|
// minimum interval and logged at most once a minute, so a counted phase does not turn
|
||||||
|
// into a write and a log line per unit.
|
||||||
|
func (t *Tracker) Advance(key string, done int) {
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
s := t.find(key)
|
||||||
|
if s == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.Done = done
|
||||||
|
now := t.now()
|
||||||
|
if t.lastAdvanceLog.IsZero() || now.Sub(t.lastAdvanceLog) >= logAdvanceEach {
|
||||||
|
t.lastAdvanceLog = now
|
||||||
|
t.log.Info("step progress", "step", key, "done", done, "total", s.Total)
|
||||||
|
}
|
||||||
|
if t.lastAdvanceSave.IsZero() || now.Sub(t.lastAdvanceSave) >= t.minSave {
|
||||||
|
t.lastAdvanceSave = now
|
||||||
|
t.save()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Done implements Reporter.
|
||||||
|
func (t *Tracker) Done(key string) {
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
s := t.find(key)
|
||||||
|
if s == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
t.close(s, domain.StepDone)
|
||||||
|
t.log.Info("step finished", "step", key, "elapsed", s.FinishedAt.Sub(s.StartedAt))
|
||||||
|
t.save()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fail implements Reporter: one phase failed and the job carries on.
|
||||||
|
func (t *Tracker) Fail(key string, err error) {
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
s := t.find(key)
|
||||||
|
if s == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
t.close(s, domain.StepFailed)
|
||||||
|
t.log.Warn("step failed", "step", key, "elapsed", s.FinishedAt.Sub(s.StartedAt), "kind", errkind.Of(err))
|
||||||
|
t.save()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Note implements Reporter.
|
||||||
|
func (t *Tracker) Note(msg string, kv ...any) { t.log.Info(msg, kv...) }
|
||||||
|
|
||||||
|
// FailRunning marks the phase that was running as failed, because the whole job
|
||||||
|
// failed. What never started stays pending, which is what shows the user where the
|
||||||
|
// job stopped.
|
||||||
|
func (t *Tracker) FailRunning() {
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
for i := range t.steps {
|
||||||
|
if t.steps[i].State == domain.StepRunning {
|
||||||
|
t.close(&t.steps[i], domain.StepFailed)
|
||||||
|
t.log.Warn("step failed", "step", t.steps[i].Key, "elapsed", t.steps[i].FinishedAt.Sub(t.steps[i].StartedAt))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
t.save()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Finish closes the job: a phase still running is done, a phase that never ran is
|
||||||
|
// skipped.
|
||||||
|
func (t *Tracker) Finish() {
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
for i := range t.steps {
|
||||||
|
switch t.steps[i].State {
|
||||||
|
case domain.StepRunning:
|
||||||
|
t.close(&t.steps[i], domain.StepDone)
|
||||||
|
case domain.StepPending:
|
||||||
|
t.steps[i].State = domain.StepSkipped
|
||||||
|
}
|
||||||
|
}
|
||||||
|
t.log.Info("job finished")
|
||||||
|
t.save()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Annotate attaches a short plain note to a phase and saves it at once. It is for a
|
||||||
|
// remark the reader should see although nothing failed. Durations and counts only: a note
|
||||||
|
// is shown on a page about a private recording.
|
||||||
|
func (t *Tracker) Annotate(key, note string) {
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
s := t.find(key)
|
||||||
|
if s == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.Note = note
|
||||||
|
t.save()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Heartbeat stamps the job's row every beatEvery and logs which phase is still running
|
||||||
|
// every stillEvery, until ctx ends. It runs on its own, independent of any engine call:
|
||||||
|
// a heartbeat that stopped means the process is gone, not that a request is slow.
|
||||||
|
func (t *Tracker) Heartbeat(ctx context.Context, beatEvery, stillEvery time.Duration) {
|
||||||
|
beat, still := time.NewTicker(beatEvery), time.NewTicker(stillEvery)
|
||||||
|
defer beat.Stop()
|
||||||
|
defer still.Stop()
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-beat.C:
|
||||||
|
if err := t.sink.Beat(ctx); err != nil && ctx.Err() == nil {
|
||||||
|
t.log.Warn("progress: heartbeat not saved", "err", err)
|
||||||
|
}
|
||||||
|
case <-still.C:
|
||||||
|
t.logStill()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Tracker) logStill() {
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
for _, s := range t.steps {
|
||||||
|
if s.State == domain.StepRunning {
|
||||||
|
t.log.Info("still running", "step", s.Key, "elapsed", t.now().Sub(s.StartedAt), "done", s.Done, "total", s.Total)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// close ends a phase. Callers hold t.mu.
|
||||||
|
func (t *Tracker) close(s *domain.ProgressStep, state domain.StepState) {
|
||||||
|
s.State, s.FinishedAt = state, t.now()
|
||||||
|
if s.StartedAt.IsZero() {
|
||||||
|
s.StartedAt = s.FinishedAt
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// find returns the phase with this key, or nil. Callers hold t.mu.
|
||||||
|
func (t *Tracker) find(key string) *domain.ProgressStep {
|
||||||
|
for i := range t.steps {
|
||||||
|
if t.steps[i].Key == key {
|
||||||
|
return &t.steps[i]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// save persists a copy of the phases. Callers hold t.mu. A failed write is logged and
|
||||||
|
// dropped: progress is cosmetic and must never fail the job it describes.
|
||||||
|
func (t *Tracker) save() {
|
||||||
|
snapshot := append([]domain.ProgressStep(nil), t.steps...)
|
||||||
|
if err := t.sink.SaveProgress(t.ctx, snapshot, t.now()); err != nil && t.ctx.Err() == nil {
|
||||||
|
t.log.Warn("progress: not saved", "err", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type ctxKey struct{}
|
||||||
|
|
||||||
|
// With returns a context that carries r.
|
||||||
|
func With(ctx context.Context, r Reporter) context.Context {
|
||||||
|
return context.WithValue(ctx, ctxKey{}, r)
|
||||||
|
}
|
||||||
|
|
||||||
|
// From returns the Reporter carried by ctx, or one that does nothing, so a component
|
||||||
|
// can always report without checking whether anyone is listening.
|
||||||
|
func From(ctx context.Context) Reporter {
|
||||||
|
if r, ok := ctx.Value(ctxKey{}).(Reporter); ok {
|
||||||
|
return r
|
||||||
|
}
|
||||||
|
return noop{}
|
||||||
|
}
|
||||||
|
|
||||||
|
type noop struct{}
|
||||||
|
|
||||||
|
func (noop) Begin(string, int) {}
|
||||||
|
func (noop) Advance(string, int) {}
|
||||||
|
func (noop) Done(string) {}
|
||||||
|
func (noop) Fail(string, error) {}
|
||||||
|
func (noop) Note(string, ...any) {}
|
||||||
@@ -0,0 +1,402 @@
|
|||||||
|
package progress_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"log/slog"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/progress"
|
||||||
|
)
|
||||||
|
|
||||||
|
// fakeSink records every saved snapshot and every heartbeat.
|
||||||
|
type fakeSink struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
saves [][]domain.ProgressStep
|
||||||
|
at []time.Time
|
||||||
|
beats int
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeSink) SaveProgress(_ context.Context, steps []domain.ProgressStep, at time.Time) error {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
f.saves = append(f.saves, append([]domain.ProgressStep(nil), steps...))
|
||||||
|
f.at = append(f.at, at)
|
||||||
|
return f.err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeSink) Beat(context.Context) error {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
f.beats++
|
||||||
|
return f.err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeSink) last(t *testing.T) []domain.ProgressStep {
|
||||||
|
t.Helper()
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
require.NotEmpty(t, f.saves, "nothing was saved")
|
||||||
|
return f.saves[len(f.saves)-1]
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeSink) saveCount() int {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
return len(f.saves)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeSink) beatCount() int {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
return f.beats
|
||||||
|
}
|
||||||
|
|
||||||
|
// syncBuf is a log sink safe to read while the heartbeat goroutine is still writing.
|
||||||
|
type syncBuf struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
b strings.Builder
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *syncBuf) Write(p []byte) (int, error) {
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
return s.b.Write(p)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *syncBuf) String() string {
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
return s.b.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
// clock is a hand-wound clock so durations and throttling are exact.
|
||||||
|
type clock struct{ t time.Time }
|
||||||
|
|
||||||
|
func (c *clock) now() time.Time { return c.t }
|
||||||
|
func (c *clock) tick(d time.Duration) { c.t = c.t.Add(d) }
|
||||||
|
func newClock() *clock { return &clock{t: time.Date(2026, 10, 2, 8, 0, 0, 0, time.UTC)} }
|
||||||
|
|
||||||
|
func plan() []progress.Planned {
|
||||||
|
return []progress.Planned{{Key: "audio", Label: "Preparing audio"}, {Key: "decide", Label: "Deciding"}, {Key: "save", Label: "Saving"}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newTracker(t *testing.T) (*progress.Tracker, *fakeSink, *clock, *syncBuf) {
|
||||||
|
t.Helper()
|
||||||
|
sink, clk, buf := &fakeSink{}, newClock(), &syncBuf{}
|
||||||
|
log := slog.New(slog.NewTextHandler(buf, &slog.HandlerOptions{Level: slog.LevelDebug}))
|
||||||
|
return progress.NewTracker(context.Background(), sink, log, plan(), progress.WithClock(clk.now), progress.WithMinSave(2*time.Second)), sink, clk, buf
|
||||||
|
}
|
||||||
|
|
||||||
|
func stateOf(steps []domain.ProgressStep, key string) domain.StepState {
|
||||||
|
for _, s := range steps {
|
||||||
|
if s.Key == key {
|
||||||
|
return s.State
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTracker_PlanIsShownUpFrontInOrderAllPending(t *testing.T) {
|
||||||
|
tr, sink, _, _ := newTracker(t)
|
||||||
|
|
||||||
|
tr.Start()
|
||||||
|
|
||||||
|
steps := sink.last(t)
|
||||||
|
require.Len(t, steps, 3)
|
||||||
|
require.Equal(t, []string{"audio", "decide", "save"}, []string{steps[0].Key, steps[1].Key, steps[2].Key})
|
||||||
|
for _, s := range steps {
|
||||||
|
require.Equal(t, domain.StepPending, s.State, s.Key)
|
||||||
|
}
|
||||||
|
require.Equal(t, "Preparing audio", steps[0].Label)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTracker_BeginRunsAStepAndDoneRecordsItsDuration(t *testing.T) {
|
||||||
|
tr, sink, clk, _ := newTracker(t)
|
||||||
|
tr.Start()
|
||||||
|
|
||||||
|
tr.Begin("audio", 0)
|
||||||
|
require.Equal(t, domain.StepRunning, stateOf(sink.last(t), "audio"))
|
||||||
|
clk.tick(90 * time.Second)
|
||||||
|
tr.Done("audio")
|
||||||
|
|
||||||
|
s := sink.last(t)[0]
|
||||||
|
require.Equal(t, domain.StepDone, s.State)
|
||||||
|
require.Equal(t, 90*time.Second, s.FinishedAt.Sub(s.StartedAt), "the finished step must carry how long it took")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTracker_AdvanceCountsAreSavedAndThrottled(t *testing.T) {
|
||||||
|
tr, sink, clk, _ := newTracker(t)
|
||||||
|
tr.Start()
|
||||||
|
tr.Begin("decide", 282)
|
||||||
|
base := sink.saveCount()
|
||||||
|
|
||||||
|
tr.Advance("decide", 1)
|
||||||
|
clk.tick(500 * time.Millisecond)
|
||||||
|
tr.Advance("decide", 2)
|
||||||
|
clk.tick(500 * time.Millisecond)
|
||||||
|
tr.Advance("decide", 3)
|
||||||
|
|
||||||
|
require.Equal(t, base+1, sink.saveCount(), "advances inside the minimum interval must not each hit the database")
|
||||||
|
clk.tick(2 * time.Second)
|
||||||
|
tr.Advance("decide", 4)
|
||||||
|
require.Equal(t, base+2, sink.saveCount())
|
||||||
|
got := sink.last(t)[1]
|
||||||
|
require.Equal(t, 4, got.Done)
|
||||||
|
require.Equal(t, 282, got.Total)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTracker_DoneFlushesTheFinalCountEvenInsideTheThrottle(t *testing.T) {
|
||||||
|
tr, sink, _, _ := newTracker(t)
|
||||||
|
tr.Start()
|
||||||
|
tr.Begin("decide", 10)
|
||||||
|
tr.Advance("decide", 9)
|
||||||
|
|
||||||
|
tr.Advance("decide", 10)
|
||||||
|
tr.Done("decide")
|
||||||
|
|
||||||
|
got := sink.last(t)[1]
|
||||||
|
require.Equal(t, domain.StepDone, got.State)
|
||||||
|
require.Equal(t, 10, got.Done, "a finished step must show its full count, not the last throttled one")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTracker_FinishSkipsWhatNeverRanAndClosesWhatDid(t *testing.T) {
|
||||||
|
tr, sink, _, _ := newTracker(t)
|
||||||
|
tr.Start()
|
||||||
|
tr.Begin("audio", 0)
|
||||||
|
tr.Done("audio")
|
||||||
|
tr.Begin("decide", 5)
|
||||||
|
|
||||||
|
tr.Finish()
|
||||||
|
|
||||||
|
steps := sink.last(t)
|
||||||
|
require.Equal(t, domain.StepDone, stateOf(steps, "audio"))
|
||||||
|
require.Equal(t, domain.StepDone, stateOf(steps, "decide"), "a step still running when the job completes is closed")
|
||||||
|
require.Equal(t, domain.StepSkipped, stateOf(steps, "save"), "a step that never ran is skipped, not left pending")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTracker_FailRunningMarksTheRunningStepFailedAndLeavesTheRestPending(t *testing.T) {
|
||||||
|
tr, sink, _, _ := newTracker(t)
|
||||||
|
tr.Start()
|
||||||
|
tr.Begin("audio", 0)
|
||||||
|
tr.Done("audio")
|
||||||
|
tr.Begin("decide", 5)
|
||||||
|
|
||||||
|
tr.FailRunning()
|
||||||
|
|
||||||
|
steps := sink.last(t)
|
||||||
|
require.Equal(t, domain.StepDone, stateOf(steps, "audio"))
|
||||||
|
require.Equal(t, domain.StepFailed, stateOf(steps, "decide"))
|
||||||
|
require.Equal(t, domain.StepPending, stateOf(steps, "save"), "what never started stays pending: it shows where the job stopped")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTracker_AKeyedFailMarksOneStepFailedLogsItsKindAndTheJobGoesOn(t *testing.T) {
|
||||||
|
tr, sink, _, buf := newTracker(t)
|
||||||
|
tr.Start()
|
||||||
|
tr.Begin("audio", 0)
|
||||||
|
|
||||||
|
tr.Fail("audio", errors.New("gateway said no"))
|
||||||
|
tr.Begin("decide", 5)
|
||||||
|
|
||||||
|
steps := sink.last(t)
|
||||||
|
require.Equal(t, domain.StepFailed, stateOf(steps, "audio"))
|
||||||
|
require.Equal(t, domain.StepRunning, stateOf(steps, "decide"), "a soft failure must not stop the next step from running")
|
||||||
|
require.Contains(t, buf.String(), "level=WARN")
|
||||||
|
require.NotContains(t, buf.String(), "gateway said no", "an error's text can carry provider output or a path: the log gets its kind")
|
||||||
|
require.Contains(t, buf.String(), "kind=error")
|
||||||
|
require.Contains(t, buf.String(), "step=audio")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTracker_NoteIsALogLineAndNeverASave(t *testing.T) {
|
||||||
|
tr, sink, _, buf := newTracker(t)
|
||||||
|
tr.Start()
|
||||||
|
before := sink.saveCount()
|
||||||
|
|
||||||
|
tr.Note("reconciled", "disputes", 362, "auto_resolved", 80)
|
||||||
|
|
||||||
|
require.Equal(t, before, sink.saveCount(), "a note is for the operator; it must not cost a database write")
|
||||||
|
require.Contains(t, buf.String(), "reconciled")
|
||||||
|
require.Contains(t, buf.String(), "disputes=362")
|
||||||
|
require.Contains(t, buf.String(), "auto_resolved=80")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTracker_AnUnknownKeyIsIgnored(t *testing.T) {
|
||||||
|
tr, sink, _, _ := newTracker(t)
|
||||||
|
tr.Start()
|
||||||
|
before := sink.saveCount()
|
||||||
|
|
||||||
|
require.NotPanics(t, func() {
|
||||||
|
tr.Begin("no-such-step", 0)
|
||||||
|
tr.Advance("no-such-step", 1)
|
||||||
|
tr.Done("no-such-step")
|
||||||
|
tr.Fail("no-such-step", errors.New("x"))
|
||||||
|
})
|
||||||
|
require.Equal(t, before, sink.saveCount())
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTracker_ASinkFailureIsLoggedAndNeverStopsTheJob(t *testing.T) {
|
||||||
|
tr, sink, _, buf := newTracker(t)
|
||||||
|
sink.err = errors.New("db down")
|
||||||
|
|
||||||
|
require.NotPanics(t, func() {
|
||||||
|
tr.Start()
|
||||||
|
tr.Begin("audio", 0)
|
||||||
|
tr.Done("audio")
|
||||||
|
})
|
||||||
|
require.Contains(t, buf.String(), "level=WARN", "a lost progress write is worth a WARN")
|
||||||
|
require.Contains(t, buf.String(), "db down")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTracker_EveryTransitionIsLoggedByTheSameCall(t *testing.T) {
|
||||||
|
tr, _, clk, buf := newTracker(t)
|
||||||
|
tr.Start()
|
||||||
|
tr.Begin("audio", 0)
|
||||||
|
clk.tick(3 * time.Second)
|
||||||
|
tr.Done("audio")
|
||||||
|
tr.Begin("decide", 5)
|
||||||
|
clk.tick(2 * time.Second)
|
||||||
|
tr.FailRunning()
|
||||||
|
tr.Finish()
|
||||||
|
|
||||||
|
out := buf.String()
|
||||||
|
for _, want := range []string{
|
||||||
|
`msg="job started"`,
|
||||||
|
`msg="step started" step=audio`,
|
||||||
|
`msg="step finished" step=audio elapsed=3s`,
|
||||||
|
`msg="step started" step=decide total=5`,
|
||||||
|
`msg="step failed" step=decide elapsed=2s`,
|
||||||
|
`msg="job finished"`,
|
||||||
|
} {
|
||||||
|
require.Contains(t, out, want, "a transition with no log line is invisible to the operator")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTracker_AKeyedFailIsSavedAtOnceNotOnTheNextTransition(t *testing.T) {
|
||||||
|
tr, sink, _, _ := newTracker(t)
|
||||||
|
tr.Start()
|
||||||
|
tr.Begin("audio", 0)
|
||||||
|
before := sink.saveCount()
|
||||||
|
|
||||||
|
tr.Fail("audio", errors.New("down"))
|
||||||
|
|
||||||
|
require.Equal(t, before+1, sink.saveCount(), "a soft-failed phase must show as failed on the page now")
|
||||||
|
require.Equal(t, domain.StepFailed, stateOf(sink.last(t), "audio"))
|
||||||
|
}
|
||||||
|
|
||||||
|
// retainingSink keeps the very slice it is handed, like a sink that queues it for later.
|
||||||
|
type retainingSink struct{ got [][]domain.ProgressStep }
|
||||||
|
|
||||||
|
func (r *retainingSink) SaveProgress(_ context.Context, steps []domain.ProgressStep, _ time.Time) error {
|
||||||
|
r.got = append(r.got, steps)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
func (r *retainingSink) Beat(context.Context) error { return nil }
|
||||||
|
|
||||||
|
func TestTracker_ASnapshotHandedToTheSinkIsNotChangedByLaterTransitions(t *testing.T) {
|
||||||
|
sink := &retainingSink{}
|
||||||
|
tr := progress.NewTracker(context.Background(), sink, slog.New(slog.NewTextHandler(&syncBuf{}, nil)), plan(),
|
||||||
|
progress.WithClock(newClock().now))
|
||||||
|
tr.Start()
|
||||||
|
first := sink.got[0]
|
||||||
|
|
||||||
|
tr.Begin("audio", 0)
|
||||||
|
tr.Done("audio")
|
||||||
|
|
||||||
|
require.Equal(t, domain.StepPending, first[0].State, "a sink that keeps a snapshot must not see it change under it")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTracker_AdvanceIsLoggedAtMostOncePerMinute(t *testing.T) {
|
||||||
|
tr, _, clk, buf := newTracker(t)
|
||||||
|
tr.Start()
|
||||||
|
tr.Begin("decide", 100)
|
||||||
|
|
||||||
|
for i := 1; i <= 30; i++ {
|
||||||
|
clk.tick(1 * time.Second)
|
||||||
|
tr.Advance("decide", i)
|
||||||
|
}
|
||||||
|
require.LessOrEqual(t, strings.Count(buf.String(), "step progress"), 1,
|
||||||
|
"30 seconds of progress is at most one log line, not thirty")
|
||||||
|
clk.tick(61 * time.Second)
|
||||||
|
tr.Advance("decide", 31)
|
||||||
|
require.Equal(t, 2, strings.Count(buf.String(), "step progress"))
|
||||||
|
require.Contains(t, buf.String(), "done=31")
|
||||||
|
require.Contains(t, buf.String(), "total=100")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTracker_HeartbeatBeatsOnItsOwnUntilStopped(t *testing.T) {
|
||||||
|
tr, sink, _, _ := newTracker(t)
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
done := make(chan struct{})
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
defer close(done)
|
||||||
|
tr.Heartbeat(ctx, 5*time.Millisecond, time.Hour)
|
||||||
|
}()
|
||||||
|
|
||||||
|
require.Eventually(t, func() bool { return sink.beatCount() >= 3 }, 2*time.Second, 5*time.Millisecond,
|
||||||
|
"a heartbeat that does not depend on any engine call is the point")
|
||||||
|
cancel()
|
||||||
|
select {
|
||||||
|
case <-done:
|
||||||
|
case <-time.After(2 * time.Second):
|
||||||
|
t.Fatal("Heartbeat did not stop when its context ended")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTracker_HeartbeatSaysWhichStepIsStillRunningAndForHowLong(t *testing.T) {
|
||||||
|
tr, _, clk, buf := newTracker(t)
|
||||||
|
tr.Start()
|
||||||
|
tr.Begin("decide", 100)
|
||||||
|
clk.tick(7 * time.Minute)
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
go tr.Heartbeat(ctx, time.Hour, 5*time.Millisecond)
|
||||||
|
|
||||||
|
require.Eventually(t, func() bool { return strings.Contains(buf.String(), "still running") }, 2*time.Second, 5*time.Millisecond)
|
||||||
|
require.Contains(t, buf.String(), "step=decide")
|
||||||
|
require.Contains(t, buf.String(), "elapsed=7m0s")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFrom_WithoutAReporterReturnsOneThatDoesNothing(t *testing.T) {
|
||||||
|
r := progress.From(context.Background())
|
||||||
|
|
||||||
|
require.NotNil(t, r)
|
||||||
|
require.NotPanics(t, func() {
|
||||||
|
r.Begin("x", 3)
|
||||||
|
r.Advance("x", 1)
|
||||||
|
r.Done("x")
|
||||||
|
r.Fail("x", errors.New("e"))
|
||||||
|
r.Note("n", "k", 1)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWithAndFrom_ACarriedReporterIsTheOneReturned(t *testing.T) {
|
||||||
|
tr, sink, _, _ := newTracker(t)
|
||||||
|
tr.Start()
|
||||||
|
ctx := progress.With(context.Background(), tr)
|
||||||
|
|
||||||
|
progress.From(ctx).Begin("audio", 0)
|
||||||
|
|
||||||
|
require.Equal(t, domain.StepRunning, stateOf(sink.last(t), "audio"))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTracker_AnnotateAttachesANoteToAStepAndSavesItAtOnce(t *testing.T) {
|
||||||
|
tr, sink, _, _ := newTracker(t)
|
||||||
|
tr.Start()
|
||||||
|
tr.Begin("audio", 0)
|
||||||
|
before := sink.saveCount()
|
||||||
|
|
||||||
|
tr.Annotate("audio", "20 min of the 79 the file claimed")
|
||||||
|
|
||||||
|
require.Equal(t, before+1, sink.saveCount())
|
||||||
|
require.Equal(t, "20 min of the 79 the file claimed", sink.last(t)[0].Note)
|
||||||
|
require.NotPanics(t, func() { tr.Annotate("no-such-step", "x") })
|
||||||
|
}
|
||||||
@@ -32,7 +32,7 @@ func (f *fakeCompleter) Complete(_ context.Context, system, user string) (string
|
|||||||
return f.replies[i], nil
|
return f.replies[i], nil
|
||||||
}
|
}
|
||||||
|
|
||||||
var req = reconcile.Request{Before: "kunderna ska då", After: "De ska kunna betala", Base: "onboardade", Alt: "omvårdade"}
|
var req = reconcile.Request{Before: "grannarna ska då", After: "De ska kunna måla", Base: "målade", Alt: "mölade"}
|
||||||
|
|
||||||
func TestLLMAdjudicator_SendsTheMeasuredPromptWithTheReadingsInTheRequestedOrder(t *testing.T) {
|
func TestLLMAdjudicator_SendsTheMeasuredPromptWithTheReadingsInTheRequestedOrder(t *testing.T) {
|
||||||
fc := &fakeCompleter{replies: []string{"ANSWER: 1"}}
|
fc := &fakeCompleter{replies: []string{"ANSWER: 1"}}
|
||||||
@@ -45,10 +45,10 @@ func TestLLMAdjudicator_SendsTheMeasuredPromptWithTheReadingsInTheRequestedOrder
|
|||||||
assert.Contains(t, fc.systems[0], "two automatic transcriptions of the same Swedish speech")
|
assert.Contains(t, fc.systems[0], "two automatic transcriptions of the same Swedish speech")
|
||||||
assert.Contains(t, fc.systems[0], "ANSWER: 1")
|
assert.Contains(t, fc.systems[0], "ANSWER: 1")
|
||||||
u := fc.users[0]
|
u := fc.users[0]
|
||||||
assert.Contains(t, u, "Text before the spot:\nkunderna ska då")
|
assert.Contains(t, u, "Text before the spot:\ngrannarna ska då")
|
||||||
assert.Contains(t, u, "[[ THE SPOT ]]")
|
assert.Contains(t, u, "[[ THE SPOT ]]")
|
||||||
assert.Contains(t, u, "Text after the spot:\nDe ska kunna betala")
|
assert.Contains(t, u, "Text after the spot:\nDe ska kunna måla")
|
||||||
assert.Contains(t, u, "Reading 1: onboardade\nReading 2: omvårdade", "base first when AltFirst is false")
|
assert.Contains(t, u, "Reading 1: målade\nReading 2: mölade", "base first when AltFirst is false")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestLLMAdjudicator_AltFirstSwapsWhichReadingIsNumberOne(t *testing.T) {
|
func TestLLMAdjudicator_AltFirstSwapsWhichReadingIsNumberOne(t *testing.T) {
|
||||||
@@ -59,7 +59,7 @@ func TestLLMAdjudicator_AltFirstSwapsWhichReadingIsNumberOne(t *testing.T) {
|
|||||||
_, err := reconcile.NewLLMAdjudicator(fc).Adjudicate(context.Background(), r)
|
_, err := reconcile.NewLLMAdjudicator(fc).Adjudicate(context.Background(), r)
|
||||||
|
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Contains(t, fc.users[0], "Reading 1: omvårdade\nReading 2: onboardade")
|
assert.Contains(t, fc.users[0], "Reading 1: mölade\nReading 2: målade")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestLLMAdjudicator_MapsTheAnswerBackToBaseOrAltWhicheverOrderWasShown(t *testing.T) {
|
func TestLLMAdjudicator_MapsTheAnswerBackToBaseOrAltWhicheverOrderWasShown(t *testing.T) {
|
||||||
@@ -134,8 +134,8 @@ func TestOrderChecked_AsksTwiceWithTheOrderSwappedAndKeepsOnlyAgreement(t *testi
|
|||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Equal(t, tt.want, got)
|
assert.Equal(t, tt.want, got)
|
||||||
require.Len(t, fc.users, 2)
|
require.Len(t, fc.users, 2)
|
||||||
assert.True(t, strings.Contains(fc.users[0], "Reading 1: onboardade"), "first call shows the base first")
|
assert.True(t, strings.Contains(fc.users[0], "Reading 1: målade"), "first call shows the base first")
|
||||||
assert.True(t, strings.Contains(fc.users[1], "Reading 1: omvårdade"), "second call shows the alternative first")
|
assert.True(t, strings.Contains(fc.users[1], "Reading 1: mölade"), "second call shows the alternative first")
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -19,12 +19,12 @@ func TestFindDisputes_NoneWhenTheTextsAgreeWordForWord(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFindDisputes_OneReplacedWordIsOneDisputeCarryingBothReadings(t *testing.T) {
|
func TestFindDisputes_OneReplacedWordIsOneDisputeCarryingBothReadings(t *testing.T) {
|
||||||
got := disputes("Vi ska onboardade kunderna.", "Vi ska omvårdade kunderna.")
|
got := disputes("Vi ska målade grannarna.", "Vi ska mölade grannarna.")
|
||||||
|
|
||||||
require.Len(t, got, 1)
|
require.Len(t, got, 1)
|
||||||
d := got[0]
|
d := got[0]
|
||||||
assert.Equal(t, "onboardade", d.Base)
|
assert.Equal(t, "målade", d.Base)
|
||||||
assert.Equal(t, "omvårdade", d.Alt)
|
assert.Equal(t, "mölade", d.Alt)
|
||||||
assert.Equal(t, 2, d.BaseStart)
|
assert.Equal(t, 2, d.BaseStart)
|
||||||
assert.Equal(t, 3, d.BaseEnd)
|
assert.Equal(t, 3, d.BaseEnd)
|
||||||
assert.Equal(t, 2, d.AltStart)
|
assert.Equal(t, 2, d.AltStart)
|
||||||
@@ -32,11 +32,11 @@ func TestFindDisputes_OneReplacedWordIsOneDisputeCarryingBothReadings(t *testing
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestFindDisputes_ReadingsKeepTheirOwnSpellingAndPunctuation(t *testing.T) {
|
func TestFindDisputes_ReadingsKeepTheirOwnSpellingAndPunctuation(t *testing.T) {
|
||||||
got := disputes("Det blir T+1, alltså.", "Det blir T 1, alltså.")
|
got := disputes("Det blir A+2, alltså.", "Det blir A 2, alltså.")
|
||||||
|
|
||||||
require.Len(t, got, 1)
|
require.Len(t, got, 1)
|
||||||
assert.Equal(t, "T+1,", got[0].Base)
|
assert.Equal(t, "A+2,", got[0].Base)
|
||||||
assert.Equal(t, "T 1,", got[0].Alt, "the alternative's words replace the span as the engine wrote them")
|
assert.Equal(t, "A 2,", got[0].Alt, "the alternative's words replace the span as the engine wrote them")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestFindDisputes_AWordOnlyOneEngineHeardHasAnEmptyOtherSide(t *testing.T) {
|
func TestFindDisputes_AWordOnlyOneEngineHeardHasAnEmptyOtherSide(t *testing.T) {
|
||||||
@@ -90,12 +90,12 @@ func TestAutoResolve(t *testing.T) {
|
|||||||
base, alt string
|
base, alt string
|
||||||
resolved bool
|
resolved bool
|
||||||
}{
|
}{
|
||||||
{"a compound written as two words or one", "i crossborder delen", "i crossborderdelen", true},
|
{"a compound written as two words or one", "i takrenns delen", "i takrennsdelen", true},
|
||||||
{"a compound with a hyphen", "crossborder-delen", "crossborder delen", true},
|
{"a compound with a hyphen", "takrenns-delen", "takrenns delen", true},
|
||||||
{"a filler only the alternative heard", "vi ska ju betala", "vi ska betala", true},
|
{"a filler only the alternative heard", "vi ska ju betala", "vi ska betala", true},
|
||||||
{"a filler only the base heard", "vi ska betala", "vi ska ju betala", true},
|
{"a filler only the base heard", "vi ska betala", "vi ska ju betala", true},
|
||||||
{"two fillers in a row", "vi ska ja ju betala", "vi ska betala", true},
|
{"two fillers in a row", "vi ska ja ju betala", "vi ska betala", true},
|
||||||
{"a different word is for the model", "vi ska onboardade", "vi ska omvårdade", false},
|
{"a different word is for the model", "vi ska målade", "vi ska mölade", false},
|
||||||
{"a different number is for the model", "totalt 1050 kronor", "totalt 2050 kronor", false},
|
{"a different number is for the model", "totalt 1050 kronor", "totalt 2050 kronor", false},
|
||||||
{"a content word only one side heard is for the model", "ett förslag", "ett coolt förslag", false},
|
{"a content word only one side heard is for the model", "ett förslag", "ett coolt förslag", false},
|
||||||
{"three filler words are too many to wave through", "vi ska ju då liksom betala", "vi ska betala", false},
|
{"three filler words are too many to wave through", "vi ska ju då liksom betala", "vi ska betala", false},
|
||||||
|
|||||||
+118
-29
@@ -6,11 +6,18 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/errkind"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/progress"
|
||||||
)
|
)
|
||||||
|
|
||||||
// MaxContested bounds the stored list of contested spots. The counts on the
|
// MaxContested is the default bound on the stored list of judged spots. A 90-minute
|
||||||
// Reconciliation are always exact; only this list is capped.
|
// recording has a few hundred, so this is a guard against a runaway, not a budget. The
|
||||||
const MaxContested = 200
|
// counts on the Reconciliation are always exact; only the list is capped.
|
||||||
|
const MaxContested = 2000
|
||||||
|
|
||||||
|
// beforeWords is how many words of left context a stored spot carries: enough to find
|
||||||
|
// the place again, small enough to stay out of the way.
|
||||||
|
const beforeWords = 8
|
||||||
|
|
||||||
// Options configure Reconcile. Zero Window and BreakAfter take their defaults.
|
// Options configure Reconcile. Zero Window and BreakAfter take their defaults.
|
||||||
type Options struct {
|
type Options struct {
|
||||||
@@ -20,6 +27,8 @@ type Options struct {
|
|||||||
// BreakAfter is how many consecutive model errors stop the reconciliation, so a
|
// BreakAfter is how many consecutive model errors stop the reconciliation, so a
|
||||||
// model that is down costs three timeouts, not one per dispute.
|
// model that is down costs three timeouts, not one per dispute.
|
||||||
BreakAfter int
|
BreakAfter int
|
||||||
|
// MaxContested caps the stored judged spots; zero takes the package default.
|
||||||
|
MaxContested int
|
||||||
// BaseEngine, AltEngine and Reconciler are recorded on the result.
|
// BaseEngine, AltEngine and Reconciler are recorded on the result.
|
||||||
BaseEngine, AltEngine, Reconciler string
|
BaseEngine, AltEngine, Reconciler string
|
||||||
}
|
}
|
||||||
@@ -31,22 +40,50 @@ type Options struct {
|
|||||||
// and the result says why. The only error it returns is a cancelled context, which
|
// and the result says why. The only error it returns is a cancelled context, which
|
||||||
// is the job stopping, not a verdict on the model.
|
// is the job stopping, not a verdict on the model.
|
||||||
func Reconcile(ctx context.Context, base, alt domain.Transcription, adj Adjudicator, o Options) (domain.Transcription, error) {
|
func Reconcile(ctx context.Context, base, alt domain.Transcription, adj Adjudicator, o Options) (domain.Transcription, error) {
|
||||||
window, breakAfter := o.Window, o.BreakAfter
|
window, breakAfter, maxSpots := o.Window, o.BreakAfter, o.MaxContested
|
||||||
if window <= 0 {
|
if window <= 0 {
|
||||||
window = 100
|
window = 100
|
||||||
}
|
}
|
||||||
if breakAfter <= 0 {
|
if breakAfter <= 0 {
|
||||||
breakAfter = 3
|
breakAfter = 3
|
||||||
}
|
}
|
||||||
|
if maxSpots <= 0 {
|
||||||
|
maxSpots = MaxContested
|
||||||
|
}
|
||||||
|
rep := progress.From(ctx)
|
||||||
|
|
||||||
|
rep.Begin("compare", 0)
|
||||||
bt, at := Tokenize(base.Text), Tokenize(alt.Text)
|
bt, at := Tokenize(base.Text), Tokenize(alt.Text)
|
||||||
ds := FindDisputes(bt, at)
|
ds := FindDisputes(bt, at)
|
||||||
|
rep.Done("compare")
|
||||||
|
|
||||||
|
// The deciding phase counts what the MODEL decides: the trivial disputes are settled
|
||||||
|
// without it, so counting them would leave the bar short of its end.
|
||||||
|
deciding := 0
|
||||||
|
for _, d := range ds {
|
||||||
|
if !AutoResolve(d) {
|
||||||
|
deciding++
|
||||||
|
}
|
||||||
|
}
|
||||||
rec := &domain.Reconciliation{
|
rec := &domain.Reconciliation{
|
||||||
Status: domain.ReconcileDone, Base: o.BaseEngine, Alt: o.AltEngine, Reconciler: o.Reconciler, Disputes: len(ds),
|
Status: domain.ReconcileDone, Base: o.BaseEngine, Alt: o.AltEngine, Reconciler: o.Reconciler, Disputes: len(ds),
|
||||||
}
|
}
|
||||||
seconds := positioner(base, len(bt))
|
seconds := positioner(base, len(bt))
|
||||||
|
spot := func(d Dispute, used string, unsure bool) {
|
||||||
|
before, _ := ContextAround(bt, d, window)
|
||||||
|
if w := strings.Fields(before); len(w) > beforeWords {
|
||||||
|
before = strings.Join(w[len(w)-beforeWords:], " ")
|
||||||
|
}
|
||||||
|
if len(rec.Contested) < maxSpots {
|
||||||
|
rec.Contested = append(rec.Contested, domain.ContestedSpot{
|
||||||
|
Seconds: seconds(d.BaseStart), Base: d.Base, Alt: d.Alt, Used: used, Unsure: unsure, Before: before,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
rep.Begin("decide", deciding)
|
||||||
verdicts := make([]Verdict, len(ds))
|
verdicts := make([]Verdict, len(ds))
|
||||||
consecutive := 0
|
consecutive, handled := 0, 0
|
||||||
for i, d := range ds {
|
for i, d := range ds {
|
||||||
if AutoResolve(d) {
|
if AutoResolve(d) {
|
||||||
rec.AutoResolved++
|
rec.AutoResolved++
|
||||||
@@ -65,7 +102,8 @@ func Reconcile(ctx context.Context, base, alt domain.Transcription, adj Adjudica
|
|||||||
rec.Errors++
|
rec.Errors++
|
||||||
consecutive++
|
consecutive++
|
||||||
if consecutive >= breakAfter {
|
if consecutive >= breakAfter {
|
||||||
return skipped(base, rec, fmt.Sprintf("the model failed %d times in a row: %v", consecutive, err)), nil
|
rep.Fail("decide", err)
|
||||||
|
return skipped(base, rec, fmt.Sprintf("the model failed %d times in a row (%s)", consecutive, errkind.Of(err))), nil
|
||||||
}
|
}
|
||||||
v = VerdictUnsure
|
v = VerdictUnsure
|
||||||
} else {
|
} else {
|
||||||
@@ -76,30 +114,94 @@ func Reconcile(ctx context.Context, base, alt domain.Transcription, adj Adjudica
|
|||||||
switch v {
|
switch v {
|
||||||
case VerdictAlt:
|
case VerdictAlt:
|
||||||
rec.Changed++
|
rec.Changed++
|
||||||
addContested(rec, domain.ContestedSpot{Seconds: seconds(d.BaseStart), Base: d.Base, Alt: d.Alt, Used: "alt"})
|
spot(d, "alt", false)
|
||||||
case VerdictUnsure:
|
case VerdictUnsure:
|
||||||
rec.Unsure++
|
rec.Unsure++
|
||||||
addContested(rec, domain.ContestedSpot{Seconds: seconds(d.BaseStart), Base: d.Base, Alt: d.Alt, Used: "base", Unsure: true})
|
spot(d, "base", true)
|
||||||
|
default:
|
||||||
|
spot(d, "base", false)
|
||||||
}
|
}
|
||||||
|
handled++
|
||||||
|
rep.Advance("decide", handled)
|
||||||
|
}
|
||||||
|
rep.Done("decide")
|
||||||
|
|
||||||
|
out := base
|
||||||
|
out.Text, out.Segments = merge(base, bt, at, ds, verdicts)
|
||||||
|
out.Reconciliation = rec
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// merge applies the verdicts: the base's words stand except where the alternative's
|
||||||
|
// reading was chosen. It returns the merged text and the base's segments carrying the
|
||||||
|
// merged words (timing is the base's). Segments are rewritten only when their words line
|
||||||
|
// up with the text; a word wrapped across a segment edge makes them differ, and guessing
|
||||||
|
// would put words in the wrong place, so then the base's segments are returned as they
|
||||||
|
// were (ADR-035). The base's own segments are never edited in place.
|
||||||
|
func merge(base domain.Transcription, bt, at []Token, ds []Dispute, verdicts []Verdict) (string, []domain.Segment) {
|
||||||
|
segOf, aligned := segmentIndex(base.Segments, len(bt))
|
||||||
|
var perSeg [][]string
|
||||||
|
var changed map[int]bool
|
||||||
|
if aligned {
|
||||||
|
perSeg, changed = make([][]string, len(base.Segments)), map[int]bool{}
|
||||||
}
|
}
|
||||||
|
|
||||||
var parts []string
|
var parts []string
|
||||||
|
keep := func(from, to int) {
|
||||||
|
for i := from; i < to; i++ {
|
||||||
|
parts = append(parts, bt[i].Text)
|
||||||
|
if aligned {
|
||||||
|
perSeg[segOf[i]] = append(perSeg[segOf[i]], bt[i].Text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
pos := 0
|
pos := 0
|
||||||
for i, d := range ds {
|
for i, d := range ds {
|
||||||
parts = appendTexts(parts, bt[pos:d.BaseStart])
|
keep(pos, d.BaseStart)
|
||||||
if verdicts[i] == VerdictAlt {
|
if verdicts[i] == VerdictAlt {
|
||||||
parts = appendTexts(parts, at[d.AltStart:d.AltEnd])
|
for _, t := range at[d.AltStart:d.AltEnd] {
|
||||||
|
parts = append(parts, t.Text)
|
||||||
|
}
|
||||||
|
if aligned {
|
||||||
|
anchor := min(d.BaseStart, len(bt)-1)
|
||||||
|
for _, t := range at[d.AltStart:d.AltEnd] {
|
||||||
|
perSeg[segOf[anchor]] = append(perSeg[segOf[anchor]], t.Text)
|
||||||
|
}
|
||||||
|
changed[segOf[anchor]] = true
|
||||||
|
for k := d.BaseStart; k < d.BaseEnd; k++ {
|
||||||
|
changed[segOf[k]] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
} else {
|
} else {
|
||||||
parts = appendTexts(parts, bt[d.BaseStart:d.BaseEnd])
|
keep(d.BaseStart, d.BaseEnd)
|
||||||
}
|
}
|
||||||
pos = d.BaseEnd
|
pos = d.BaseEnd
|
||||||
}
|
}
|
||||||
parts = appendTexts(parts, bt[pos:])
|
keep(pos, len(bt))
|
||||||
|
|
||||||
out := base
|
if !aligned {
|
||||||
out.Text = strings.Join(parts, " ")
|
return strings.Join(parts, " "), base.Segments
|
||||||
out.Reconciliation = rec
|
}
|
||||||
return out, nil
|
segs := append([]domain.Segment(nil), base.Segments...)
|
||||||
|
for k := range changed {
|
||||||
|
segs[k].Text = strings.Join(perSeg[k], " ")
|
||||||
|
}
|
||||||
|
return strings.Join(parts, " "), segs
|
||||||
|
}
|
||||||
|
|
||||||
|
// segmentIndex maps each base token to the segment it came from. aligned is false when
|
||||||
|
// there are no segments or their word count is not the text's.
|
||||||
|
func segmentIndex(segs []domain.Segment, tokens int) (segOf []int, aligned bool) {
|
||||||
|
if len(segs) == 0 || tokens == 0 {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
segOf = make([]int, 0, tokens)
|
||||||
|
for k, s := range segs {
|
||||||
|
for range Tokenize(s.Text) {
|
||||||
|
segOf = append(segOf, k)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return segOf, len(segOf) == tokens
|
||||||
}
|
}
|
||||||
|
|
||||||
// skipped is the base text, none of the partial changes, and the reason.
|
// skipped is the base text, none of the partial changes, and the reason.
|
||||||
@@ -112,13 +214,6 @@ func skipped(base domain.Transcription, rec *domain.Reconciliation, reason strin
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func appendTexts(dst []string, ts []Token) []string {
|
|
||||||
for _, t := range ts {
|
|
||||||
dst = append(dst, t.Text)
|
|
||||||
}
|
|
||||||
return dst
|
|
||||||
}
|
|
||||||
|
|
||||||
// positioner maps a token index of the base to the second it starts at: through the
|
// positioner maps a token index of the base to the second it starts at: through the
|
||||||
// segments when there are any, proportionally otherwise.
|
// segments when there are any, proportionally otherwise.
|
||||||
func positioner(base domain.Transcription, tokens int) func(int) float64 {
|
func positioner(base domain.Transcription, tokens int) func(int) float64 {
|
||||||
@@ -144,9 +239,3 @@ func positioner(base domain.Transcription, tokens int) func(int) float64 {
|
|||||||
return base.Segments[k].Start
|
return base.Segments[k].Start
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func addContested(rec *domain.Reconciliation, c domain.ContestedSpot) {
|
|
||||||
if len(rec.Contested) < MaxContested {
|
|
||||||
rec.Contested = append(rec.Contested, c)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -0,0 +1,116 @@
|
|||||||
|
package reconcile_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/progress"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/reconcile"
|
||||||
|
)
|
||||||
|
|
||||||
|
// What Reconcile reports while it works (ADR-035): the comparison, then the deciding
|
||||||
|
// phase with the real number of model decisions as its total.
|
||||||
|
|
||||||
|
type repRec struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
events []string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *repRec) add(e string) {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
r.events = append(r.events, e)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *repRec) Begin(key string, total int) { r.add(fmt.Sprintf("begin:%s:%d", key, total)) }
|
||||||
|
func (r *repRec) Advance(key string, done int) { r.add(fmt.Sprintf("advance:%s:%d", key, done)) }
|
||||||
|
func (r *repRec) Done(key string) { r.add("done:" + key) }
|
||||||
|
func (r *repRec) Fail(key string, _ error) { r.add("fail:" + key) }
|
||||||
|
func (r *repRec) Note(string, ...any) {}
|
||||||
|
func (r *repRec) joined() string {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
return strings.Join(r.events, ",")
|
||||||
|
}
|
||||||
|
func withRep(r *repRec) context.Context { return progress.With(context.Background(), r) }
|
||||||
|
func reconcileWith(t *testing.T, ctx context.Context, base, alt string, adj reconcile.Adjudicator) error {
|
||||||
|
t.Helper()
|
||||||
|
_, err := reconcile.Reconcile(ctx, tr(base), tr(alt), adj, opts)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// two compound/filler disputes the model is never asked about, then two it is.
|
||||||
|
const (
|
||||||
|
progBase = "i takrenns delen ska vi ju betala totalt 1015 kronor till Axel idag"
|
||||||
|
progAlt = "i takrennsdelen ska vi betala totalt 1050 kronor till Alex idag"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestReconcileProgress_TheDecidingPhaseTotalIsTheModelDecisionsNotTheDisputes(t *testing.T) {
|
||||||
|
rep := &repRec{}
|
||||||
|
|
||||||
|
require.NoError(t, reconcileWith(t, withRep(rep), progBase, progAlt, always(reconcile.VerdictAlt)))
|
||||||
|
|
||||||
|
assert.Equal(t,
|
||||||
|
"begin:compare:0,done:compare,begin:decide:2,advance:decide:1,advance:decide:2,done:decide",
|
||||||
|
rep.joined(),
|
||||||
|
"4 disputes, 2 settled without the model: the bar counts the 2 the model decides, so it can reach its end")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReconcileProgress_TextsThatAgreeStillClosePhasesSoNothingIsLeftRunning(t *testing.T) {
|
||||||
|
rep := &repRec{}
|
||||||
|
|
||||||
|
require.NoError(t, reconcileWith(t, withRep(rep), "samma text hela vägen", "samma text hela vägen", always(reconcile.VerdictAlt)))
|
||||||
|
|
||||||
|
assert.Equal(t, "begin:compare:0,done:compare,begin:decide:0,done:decide", rep.joined())
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReconcileProgress_AModelErrorStillCountsAsADecisionTaken(t *testing.T) {
|
||||||
|
rep := &repRec{}
|
||||||
|
adj := &scripted{fn: func(i int, _ reconcile.Request) (reconcile.Verdict, error) {
|
||||||
|
if i == 0 {
|
||||||
|
return reconcile.VerdictBase, errors.New("one timeout")
|
||||||
|
}
|
||||||
|
return reconcile.VerdictAlt, nil
|
||||||
|
}}
|
||||||
|
|
||||||
|
require.NoError(t, reconcileWith(t, withRep(rep), progBase, progAlt, adj))
|
||||||
|
|
||||||
|
assert.Contains(t, rep.joined(), "advance:decide:1,advance:decide:2", "a spot that errored was still handled; the bar must move past it")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReconcileProgress_TheBreakerTrippingShowsTheDecidingPhaseFailedNotDone(t *testing.T) {
|
||||||
|
rep := &repRec{}
|
||||||
|
adj := &scripted{fn: func(int, reconcile.Request) (reconcile.Verdict, error) {
|
||||||
|
return reconcile.VerdictBase, errors.New("model down")
|
||||||
|
}}
|
||||||
|
small := opts
|
||||||
|
small.BreakAfter = 2
|
||||||
|
|
||||||
|
_, err := reconcile.Reconcile(withRep(rep), tr(progBase), tr(progAlt), adj, small)
|
||||||
|
|
||||||
|
require.NoError(t, err, "a model that is down degrades the result; it does not fail the job")
|
||||||
|
assert.Contains(t, rep.joined(), "fail:decide")
|
||||||
|
assert.NotContains(t, rep.joined(), "done:decide")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReconcileProgress_AShutdownIsNeitherDoneNorFailed(t *testing.T) {
|
||||||
|
rep := &repRec{}
|
||||||
|
ctx, cancel := context.WithCancel(withRep(rep))
|
||||||
|
adj := &scripted{fn: func(int, reconcile.Request) (reconcile.Verdict, error) {
|
||||||
|
cancel()
|
||||||
|
return reconcile.VerdictAlt, nil
|
||||||
|
}}
|
||||||
|
|
||||||
|
err := reconcileWith(t, ctx, progBase, progAlt, adj)
|
||||||
|
|
||||||
|
require.ErrorIs(t, err, context.Canceled)
|
||||||
|
assert.NotContains(t, rep.joined(), "fail:decide")
|
||||||
|
assert.NotContains(t, rep.joined(), "done:decide")
|
||||||
|
}
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
package reconcile_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/reconcile"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The reason a reconciliation was skipped is stored and shown on a page about a private
|
||||||
|
// recording, so it carries a kind, never the model's error text, which can echo the
|
||||||
|
// prompt (adversarial review F3).
|
||||||
|
|
||||||
|
func TestReconcile_TheBreakerReasonCarriesAKindNotTheModelsText(t *testing.T) {
|
||||||
|
adj := &scripted{fn: func(int, reconcile.Request) (reconcile.Verdict, error) {
|
||||||
|
return reconcile.VerdictBase, errors.New("gateway echoed the prompt: zebrakaka secret words")
|
||||||
|
}}
|
||||||
|
small := opts
|
||||||
|
small.BreakAfter = 2
|
||||||
|
|
||||||
|
out, err := reconcile.Reconcile(context.Background(), tr(progBase), tr(progAlt), adj, small)
|
||||||
|
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Equal(t, domain.ReconcileSkipped, out.Reconciliation.Status)
|
||||||
|
assert.NotContains(t, out.Reconciliation.Reason, "zebrakaka", "provider text must not be stored or shown")
|
||||||
|
assert.Contains(t, out.Reconciliation.Reason, "2 times in a row", "the reason still says what happened")
|
||||||
|
assert.Contains(t, out.Reconciliation.Reason, "error", "and carries a kind")
|
||||||
|
}
|
||||||
@@ -0,0 +1,130 @@
|
|||||||
|
package reconcile_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/reconcile"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Every judged spot is stored with a few words of left context, and the saved segments
|
||||||
|
// carry the merged words while keeping the base's timing (ADR-035 decisions 6 and 7).
|
||||||
|
|
||||||
|
func TestReconcileSpots_AKeptSpotAndAChangedSpotAreBothStoredWithTheirVerdict(t *testing.T) {
|
||||||
|
adj := &scripted{fn: func(i int, _ reconcile.Request) (reconcile.Verdict, error) {
|
||||||
|
if i == 0 {
|
||||||
|
return reconcile.VerdictBase, nil
|
||||||
|
}
|
||||||
|
return reconcile.VerdictAlt, nil
|
||||||
|
}}
|
||||||
|
|
||||||
|
out := merge(t, tr(progBase), tr(progAlt), adj)
|
||||||
|
|
||||||
|
require.Len(t, out.Reconciliation.Contested, 2)
|
||||||
|
assert.Equal(t, "base", out.Reconciliation.Contested[0].Used)
|
||||||
|
assert.Equal(t, "1015", out.Reconciliation.Contested[0].Base)
|
||||||
|
assert.Equal(t, "alt", out.Reconciliation.Contested[1].Used)
|
||||||
|
assert.Equal(t, 1, out.Reconciliation.Changed, "the exact counts are unchanged by storing more")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReconcileSpots_AutoResolvedDisputesAreNotStored(t *testing.T) {
|
||||||
|
out := merge(t, tr("i takrenns delen ska vi ju betala"), tr("i takrennsdelen ska vi betala"), always(reconcile.VerdictAlt))
|
||||||
|
|
||||||
|
assert.Empty(t, out.Reconciliation.Contested, "compounds and fillers were never judged; they are counted, not listed")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReconcileSpots_ASpotCarriesTheWordsJustBeforeIt(t *testing.T) {
|
||||||
|
out := merge(t, tr("grannen målar ett vanligt staket mot gatan"), tr("grannen målar ett KORT staket mot gatan"), always(reconcile.VerdictAlt))
|
||||||
|
|
||||||
|
require.Len(t, out.Reconciliation.Contested, 1)
|
||||||
|
assert.Equal(t, "grannen målar ett", out.Reconciliation.Contested[0].Before)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReconcileSpots_TheContextIsAFewWordsNotTheWholeMeeting(t *testing.T) {
|
||||||
|
long := "ett två tre fyra fem sex sju åtta nio tio elva tolv"
|
||||||
|
out := merge(t, tr(long+" X slut"), tr(long+" Y slut"), always(reconcile.VerdictAlt))
|
||||||
|
|
||||||
|
require.Len(t, out.Reconciliation.Contested, 1)
|
||||||
|
assert.Equal(t, "fem sex sju åtta nio tio elva tolv", out.Reconciliation.Contested[0].Before,
|
||||||
|
"eight words: enough to find the place, small enough to stay out of the way")
|
||||||
|
}
|
||||||
|
|
||||||
|
func segmented(text string, segs ...domain.Segment) domain.Transcription {
|
||||||
|
t := tr(text)
|
||||||
|
t.Segments = segs
|
||||||
|
return t
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReconcileSegments_AChangedWordReplacesTheBaseWordInItsOwnSegmentOnly(t *testing.T) {
|
||||||
|
base := segmented("ett två tre. fyra fem sex. sju åtta nio.",
|
||||||
|
domain.Segment{Start: 0, End: 10, Text: "ett två tre."},
|
||||||
|
domain.Segment{Start: 10, End: 20, Text: "fyra fem sex."},
|
||||||
|
domain.Segment{Start: 20, End: 30, Text: "sju åtta nio."})
|
||||||
|
|
||||||
|
out := merge(t, base, tr("ett två tre. fyra femm sex. sju åtta nio."), always(reconcile.VerdictAlt))
|
||||||
|
|
||||||
|
require.Len(t, out.Segments, 3)
|
||||||
|
assert.Equal(t, "ett två tre.", out.Segments[0].Text, "an untouched segment keeps its exact text")
|
||||||
|
assert.Equal(t, "fyra femm sex.", out.Segments[1].Text)
|
||||||
|
assert.Equal(t, "sju åtta nio.", out.Segments[2].Text)
|
||||||
|
assert.InDelta(t, 10.0, out.Segments[1].Start, 1e-9, "timing stays the base's")
|
||||||
|
assert.InDelta(t, 20.0, out.Segments[1].End, 1e-9)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReconcileSegments_AWordOnlyTheAlternativeHeardLandsInTheSegmentItBelongsTo(t *testing.T) {
|
||||||
|
base := segmented("a b c. d e f.", domain.Segment{Start: 0, End: 5, Text: "a b c."}, domain.Segment{Start: 5, End: 10, Text: "d e f."})
|
||||||
|
|
||||||
|
out := merge(t, base, tr("a b c. d NY e f."), always(reconcile.VerdictAlt))
|
||||||
|
|
||||||
|
assert.Equal(t, "a b c.", out.Segments[0].Text)
|
||||||
|
assert.Equal(t, "d NY e f.", out.Segments[1].Text)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReconcileSegments_AWordOnlyTheBaseHeardIsRemovedFromItsSegment(t *testing.T) {
|
||||||
|
base := segmented("a b c. d X e f.", domain.Segment{Start: 0, End: 5, Text: "a b c."}, domain.Segment{Start: 5, End: 10, Text: "d X e f."})
|
||||||
|
|
||||||
|
out := merge(t, base, tr("a b c. d e f."), always(reconcile.VerdictAlt))
|
||||||
|
|
||||||
|
assert.Equal(t, "d e f.", out.Segments[1].Text)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReconcileSegments_AKeptBaseLeavesEverySegmentAsItWas(t *testing.T) {
|
||||||
|
base := segmented("a x b", domain.Segment{Start: 0, End: 4, Text: "a x b"})
|
||||||
|
|
||||||
|
out := merge(t, base, tr("a y b"), always(reconcile.VerdictBase))
|
||||||
|
|
||||||
|
assert.Equal(t, base.Segments, out.Segments)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReconcileSegments_TheCallersSegmentsAreNeverChangedInPlace(t *testing.T) {
|
||||||
|
segs := []domain.Segment{{Start: 0, End: 4, Text: "a x b"}}
|
||||||
|
base := segmented("a x b", segs...)
|
||||||
|
|
||||||
|
_ = merge(t, base, tr("a y b"), always(reconcile.VerdictAlt))
|
||||||
|
|
||||||
|
assert.Equal(t, "a x b", segs[0].Text, "Reconcile builds new segments; it does not edit the base transcription's")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReconcileSegments_SegmentsThatDoNotLineUpWithTheTextAreLeftAsTheBases(t *testing.T) {
|
||||||
|
// A word wrapped across a segment edge makes the segments' word count differ from
|
||||||
|
// the joined text's. Guessing would put words in the wrong place, so leave them.
|
||||||
|
base := segmented("a x b c", domain.Segment{Start: 0, End: 4, Text: "a x"}, domain.Segment{Start: 4, End: 8, Text: "b c d"})
|
||||||
|
|
||||||
|
out := merge(t, base, tr("a y b c"), always(reconcile.VerdictAlt))
|
||||||
|
|
||||||
|
assert.Equal(t, base.Segments, out.Segments)
|
||||||
|
assert.Equal(t, "a y b c", out.Text, "the merged text is unaffected")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReconcileSegments_ASpanThatCrossesASegmentEdgeIsRemovedFromBothSegments(t *testing.T) {
|
||||||
|
base := segmented("a b c d", domain.Segment{Start: 0, End: 5, Text: "a b"}, domain.Segment{Start: 5, End: 10, Text: "c d"})
|
||||||
|
|
||||||
|
out := merge(t, base, tr("a X d"), always(reconcile.VerdictAlt))
|
||||||
|
|
||||||
|
require.Equal(t, "a X d", out.Text)
|
||||||
|
assert.Equal(t, "a X", out.Segments[0].Text)
|
||||||
|
assert.Equal(t, "d", out.Segments[1].Text, "'c' was replaced too: it must not linger in the second segment")
|
||||||
|
}
|
||||||
@@ -58,10 +58,10 @@ func TestReconcile_IdenticalTextsNeedNoModelAndComeBackUnchanged(t *testing.T) {
|
|||||||
func TestReconcile_TrivialDisputesAreResolvedWithoutAskingTheModel(t *testing.T) {
|
func TestReconcile_TrivialDisputesAreResolvedWithoutAskingTheModel(t *testing.T) {
|
||||||
adj := always(reconcile.VerdictAlt)
|
adj := always(reconcile.VerdictAlt)
|
||||||
|
|
||||||
out := merge(t, tr("i crossborder delen ska vi ju betala"), tr("i crossborderdelen ska vi betala"), adj)
|
out := merge(t, tr("i takrenns delen ska vi ju betala"), tr("i takrennsdelen ska vi betala"), adj)
|
||||||
|
|
||||||
assert.Empty(t, adj.calls, "a compound split and a filler particle are not worth a model call")
|
assert.Empty(t, adj.calls, "a compound split and a filler particle are not worth a model call")
|
||||||
assert.Equal(t, "i crossborder delen ska vi ju betala", out.Text, "the base reading is kept")
|
assert.Equal(t, "i takrenns delen ska vi ju betala", out.Text, "the base reading is kept")
|
||||||
assert.Equal(t, 2, out.Reconciliation.Disputes)
|
assert.Equal(t, 2, out.Reconciliation.Disputes)
|
||||||
assert.Equal(t, 2, out.Reconciliation.AutoResolved)
|
assert.Equal(t, 2, out.Reconciliation.AutoResolved)
|
||||||
assert.Zero(t, out.Reconciliation.Adjudicated)
|
assert.Zero(t, out.Reconciliation.Adjudicated)
|
||||||
@@ -76,16 +76,21 @@ func TestReconcile_AltVerdictReplacesTheSpanWithTheAlternativesOwnWords(t *testi
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestReconcile_BaseAndUnsureVerdictsKeepTheBaseButOnlyUnsureIsFlagged(t *testing.T) {
|
func TestReconcile_BaseAndUnsureVerdictsKeepTheBaseButOnlyUnsureIsFlagged(t *testing.T) {
|
||||||
base := tr("Totalt 1015 kronor till SEB.")
|
base := tr("Totalt 1015 kronor till Axel.")
|
||||||
alt := tr("Totalt 1050 kronor till SCB.")
|
alt := tr("Totalt 1050 kronor till Alex.")
|
||||||
|
|
||||||
keptBase := merge(t, base, alt, always(reconcile.VerdictBase))
|
keptBase := merge(t, base, alt, always(reconcile.VerdictBase))
|
||||||
unsure := merge(t, base, alt, always(reconcile.VerdictUnsure))
|
unsure := merge(t, base, alt, always(reconcile.VerdictUnsure))
|
||||||
|
|
||||||
assert.Equal(t, "Totalt 1015 kronor till SEB.", keptBase.Text)
|
assert.Equal(t, "Totalt 1015 kronor till Axel.", keptBase.Text)
|
||||||
assert.Zero(t, keptBase.Reconciliation.Unsure)
|
assert.Zero(t, keptBase.Reconciliation.Unsure)
|
||||||
assert.Empty(t, keptBase.Reconciliation.Contested, "a confident 'the base is right' is not a spot worth the reader's time")
|
require.Len(t, keptBase.Reconciliation.Contested, 2,
|
||||||
assert.Equal(t, "Totalt 1015 kronor till SEB.", unsure.Text, "unsure keeps the base")
|
"every judged spot is stored, kept ones too: without them the reconciler cannot be compared with either engine alone (ADR-035)")
|
||||||
|
for _, c := range keptBase.Reconciliation.Contested {
|
||||||
|
assert.Equal(t, "base", c.Used)
|
||||||
|
assert.False(t, c.Unsure, "a confident 'the base is right' is stored but is not an uncertain spot")
|
||||||
|
}
|
||||||
|
assert.Equal(t, "Totalt 1015 kronor till Axel.", unsure.Text, "unsure keeps the base")
|
||||||
assert.Equal(t, 2, unsure.Reconciliation.Unsure)
|
assert.Equal(t, 2, unsure.Reconciliation.Unsure)
|
||||||
require.Len(t, unsure.Reconciliation.Contested, 2)
|
require.Len(t, unsure.Reconciliation.Contested, 2)
|
||||||
assert.True(t, unsure.Reconciliation.Contested[0].Unsure)
|
assert.True(t, unsure.Reconciliation.Contested[0].Unsure)
|
||||||
@@ -93,12 +98,12 @@ func TestReconcile_BaseAndUnsureVerdictsKeepTheBaseButOnlyUnsureIsFlagged(t *tes
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestReconcile_ChangedSpotsAreRecordedWithBothReadings(t *testing.T) {
|
func TestReconcile_ChangedSpotsAreRecordedWithBothReadings(t *testing.T) {
|
||||||
out := merge(t, tr("det blir T+1 imorgon"), tr("det blir en T imorgon"), always(reconcile.VerdictAlt))
|
out := merge(t, tr("det blir A+2 imorgon"), tr("det blir en A imorgon"), always(reconcile.VerdictAlt))
|
||||||
|
|
||||||
require.Len(t, out.Reconciliation.Contested, 1)
|
require.Len(t, out.Reconciliation.Contested, 1)
|
||||||
c := out.Reconciliation.Contested[0]
|
c := out.Reconciliation.Contested[0]
|
||||||
assert.Equal(t, "T+1", c.Base)
|
assert.Equal(t, "A+2", c.Base)
|
||||||
assert.Equal(t, "en T", c.Alt)
|
assert.Equal(t, "en A", c.Alt)
|
||||||
assert.Equal(t, "alt", c.Used)
|
assert.Equal(t, "alt", c.Used)
|
||||||
assert.False(t, c.Unsure)
|
assert.False(t, c.Unsure)
|
||||||
}
|
}
|
||||||
@@ -117,16 +122,16 @@ func TestReconcile_AWordOnlyTheBaseHeardCanBeRemoved(t *testing.T) {
|
|||||||
|
|
||||||
func TestReconcile_SeveralDisputesAreAppliedIndependentlyAndInOrder(t *testing.T) {
|
func TestReconcile_SeveralDisputesAreAppliedIndependentlyAndInOrder(t *testing.T) {
|
||||||
adj := &scripted{fn: func(i int, r reconcile.Request) (reconcile.Verdict, error) {
|
adj := &scripted{fn: func(i int, r reconcile.Request) (reconcile.Verdict, error) {
|
||||||
if r.Alt == "omvårdade" || r.Alt == "SCB" {
|
if r.Alt == "mölade" || r.Alt == "Alex" {
|
||||||
return reconcile.VerdictAlt, nil
|
return reconcile.VerdictAlt, nil
|
||||||
}
|
}
|
||||||
return reconcile.VerdictBase, nil
|
return reconcile.VerdictBase, nil
|
||||||
}}
|
}}
|
||||||
|
|
||||||
out := merge(t, tr("kunderna onboardade nu och sen betalas till SEB via ombud med 1050 kronor"),
|
out := merge(t, tr("kunderna målade nu och sen betalas till Axel via ombud med 1050 kronor"),
|
||||||
tr("kunderna omvårdade nu och sen betalas till SCB via ombud med 2050 kronor"), adj)
|
tr("kunderna mölade nu och sen betalas till Alex via ombud med 2050 kronor"), adj)
|
||||||
|
|
||||||
assert.Equal(t, "kunderna omvårdade nu och sen betalas till SCB via ombud med 1050 kronor", out.Text)
|
assert.Equal(t, "kunderna mölade nu och sen betalas till Alex via ombud med 1050 kronor", out.Text)
|
||||||
assert.Equal(t, 3, out.Reconciliation.Disputes)
|
assert.Equal(t, 3, out.Reconciliation.Disputes)
|
||||||
assert.Equal(t, 2, out.Reconciliation.Changed)
|
assert.Equal(t, 2, out.Reconciliation.Changed)
|
||||||
}
|
}
|
||||||
@@ -188,7 +193,8 @@ func TestReconcile_ConsecutiveModelErrorsStopTheWholeReconciliationInsteadOfWait
|
|||||||
assert.Len(t, adj.calls, 3, "three consecutive failures and it stops asking")
|
assert.Len(t, adj.calls, 3, "three consecutive failures and it stops asking")
|
||||||
assert.Equal(t, strings.Join(a, " "), out.Text, "the base text, with none of the partial changes")
|
assert.Equal(t, strings.Join(a, " "), out.Text, "the base text, with none of the partial changes")
|
||||||
assert.Equal(t, domain.ReconcileSkipped, out.Reconciliation.Status)
|
assert.Equal(t, domain.ReconcileSkipped, out.Reconciliation.Status)
|
||||||
assert.Contains(t, out.Reconciliation.Reason, "gateway down")
|
assert.Contains(t, out.Reconciliation.Reason, "3 times in a row")
|
||||||
|
assert.NotContains(t, out.Reconciliation.Reason, "gateway down", "the reason is shown on the page: a kind, not the model's own text")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestReconcile_ASuccessBetweenFailuresResetsTheBreaker(t *testing.T) {
|
func TestReconcile_ASuccessBetweenFailuresResetsTheBreaker(t *testing.T) {
|
||||||
@@ -224,14 +230,16 @@ func TestReconcile_ACancelledJobIsAnErrorNotASkippedReconciliation(t *testing.T)
|
|||||||
assert.ErrorIs(t, err, context.Canceled, "a shutdown must not be recorded as 'the model was unavailable'")
|
assert.ErrorIs(t, err, context.Canceled, "a shutdown must not be recorded as 'the model was unavailable'")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestReconcile_KeepsTheBaseSegmentsDurationAndNamesTheEngines(t *testing.T) {
|
func TestReconcile_KeepsTheBaseTimingAndDurationAndNamesTheEngines(t *testing.T) {
|
||||||
base := tr("a x b")
|
base := tr("a x b")
|
||||||
base.Segments = []domain.Segment{{Start: 0, End: 4, Text: "a x b"}}
|
base.Segments = []domain.Segment{{Start: 0, End: 4, Text: "a x b"}}
|
||||||
base.AudioSeconds = 3821.5
|
base.AudioSeconds = 3821.5
|
||||||
|
|
||||||
out := merge(t, base, tr("a y b"), always(reconcile.VerdictAlt))
|
out := merge(t, base, tr("a y b"), always(reconcile.VerdictAlt))
|
||||||
|
|
||||||
assert.Equal(t, base.Segments, out.Segments, "the timed segments are the base engine's, untouched")
|
require.Len(t, out.Segments, 1)
|
||||||
|
assert.InDelta(t, 0.0, out.Segments[0].Start, 1e-9, "timing is the base engine's")
|
||||||
|
assert.InDelta(t, 4.0, out.Segments[0].End, 1e-9)
|
||||||
assert.InDelta(t, 3821.5, out.AudioSeconds, 1e-9)
|
assert.InDelta(t, 3821.5, out.AudioSeconds, 1e-9)
|
||||||
assert.Equal(t, "iguana/pianissimo", out.Reconciliation.Base)
|
assert.Equal(t, "iguana/pianissimo", out.Reconciliation.Base)
|
||||||
assert.Equal(t, "iguana/kb-whisper", out.Reconciliation.Alt)
|
assert.Equal(t, "iguana/kb-whisper", out.Reconciliation.Alt)
|
||||||
@@ -264,18 +272,25 @@ func TestReconcile_WithNoSegmentsASpotIsPlacedProportionally(t *testing.T) {
|
|||||||
assert.InDelta(t, 90.0, out.Reconciliation.Contested[0].Seconds, 1e-9, "token 9 of 10 in 100 s")
|
assert.InDelta(t, 90.0, out.Reconciliation.Contested[0].Seconds, 1e-9, "token 9 of 10 in 100 s")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestReconcile_ContestedSpotsAreCappedButTheCountsAreExact(t *testing.T) {
|
func TestReconcile_JudgedSpotsAreCappedButTheCountsAreExact(t *testing.T) {
|
||||||
a, b := make([]string, 0, 600), make([]string, 0, 600)
|
a, b := make([]string, 0, 600), make([]string, 0, 600)
|
||||||
for i := 0; i < 300; i++ {
|
for i := 0; i < 300; i++ {
|
||||||
a = append(a, fmt.Sprintf("w%d", i), "same", "same2")
|
a = append(a, fmt.Sprintf("w%d", i), "same", "same2")
|
||||||
b = append(b, fmt.Sprintf("v%d", i), "same", "same2")
|
b = append(b, fmt.Sprintf("v%d", i), "same", "same2")
|
||||||
}
|
}
|
||||||
|
capped := opts
|
||||||
|
capped.MaxContested = 50
|
||||||
|
|
||||||
out := merge(t, tr(strings.Join(a, " ")), tr(strings.Join(b, " ")), always(reconcile.VerdictUnsure))
|
out, err := reconcile.Reconcile(context.Background(), tr(strings.Join(a, " ")), tr(strings.Join(b, " ")), always(reconcile.VerdictUnsure), capped)
|
||||||
|
|
||||||
|
require.NoError(t, err)
|
||||||
assert.Equal(t, 300, out.Reconciliation.Unsure)
|
assert.Equal(t, 300, out.Reconciliation.Unsure)
|
||||||
assert.Equal(t, 300, out.Reconciliation.Disputes)
|
assert.Equal(t, 300, out.Reconciliation.Disputes)
|
||||||
assert.Len(t, out.Reconciliation.Contested, reconcile.MaxContested, "the stored list is bounded; the counts are not")
|
assert.Len(t, out.Reconciliation.Contested, 50, "the stored list is bounded; the counts are not")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReconcile_TheDefaultCapIsAFewThousandSoAWholeMeetingIsKept(t *testing.T) {
|
||||||
|
assert.Equal(t, 2000, reconcile.MaxContested, "a 90-minute recording has a few hundred judged spots; the cap is a guard, not a budget")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestReconcile_AVerdictReturnedAlongsideAnErrorIsIgnored(t *testing.T) {
|
func TestReconcile_AVerdictReturnedAlongsideAnErrorIsIgnored(t *testing.T) {
|
||||||
|
|||||||
@@ -34,9 +34,9 @@ func TestTokenize_KeepsTheWordAsWrittenAndKeysItWithoutCaseOrPunctuation(t *test
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestTokenize_KeysIgnoreHyphensAndPlusSoOnlyWordsDecideEquality(t *testing.T) {
|
func TestTokenize_KeysIgnoreHyphensAndPlusSoOnlyWordsDecideEquality(t *testing.T) {
|
||||||
got := reconcile.Tokenize("end-to-end T+1 5 000 kr")
|
got := reconcile.Tokenize("end-to-end A+2 5 000 kr")
|
||||||
|
|
||||||
assert.Equal(t, []string{"endtoend", "t1", "5", "000", "kr"}, keys(got))
|
assert.Equal(t, []string{"endtoend", "a2", "5", "000", "kr"}, keys(got))
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestTokenize_APunctuationOnlyWordJoinsThePreviousWordInsteadOfBecomingOne(t *testing.T) {
|
func TestTokenize_APunctuationOnlyWordJoinsThePreviousWordInsteadOfBecomingOne(t *testing.T) {
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
|
"time"
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
)
|
)
|
||||||
@@ -33,6 +34,9 @@ type Job struct {
|
|||||||
UserID string
|
UserID string
|
||||||
UploadID string
|
UploadID string
|
||||||
Path string
|
Path string
|
||||||
|
// Duration is what the file CLAIMED when probed. The worker compares it with what
|
||||||
|
// was really converted; zero means unknown (ADR-035 review).
|
||||||
|
Duration time.Duration
|
||||||
}
|
}
|
||||||
|
|
||||||
// Submitter hands accepted jobs to the in-process worker (ADR-033).
|
// Submitter hands accepted jobs to the in-process worker (ADR-033).
|
||||||
@@ -54,12 +58,42 @@ func NewIntake(s Store, p Prober, sub Submitter, l Limits, tmpDir string) *Intak
|
|||||||
return &Intake{store: s, prober: p, sub: sub, limits: l, tmpDir: tmpDir}
|
return &Intake{store: s, prober: p, sub: sub, limits: l, tmpDir: tmpDir}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ErrClientRead marks an error that came from READING the client's body: a connection that
|
||||||
|
// dropped, a body cut short. It is the sender's to retry, unlike a spool, probe or database
|
||||||
|
// fault, which is ours. It is a marker on the error, not a sentinel that others may wrap:
|
||||||
|
// io.ErrUnexpectedEOF alone is also what a database drop looks like.
|
||||||
|
var ErrClientRead = errors.New("uploads: client body read failed")
|
||||||
|
|
||||||
|
// clientReader remembers the last non-EOF error its source returned, so Accept can tell
|
||||||
|
// a failure reading the client from a failure writing the spool file.
|
||||||
|
type clientReader struct {
|
||||||
|
r io.Reader
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *clientReader) Read(p []byte) (int, error) {
|
||||||
|
n, err := c.r.Read(p)
|
||||||
|
if err != nil && !errors.Is(err, io.EOF) {
|
||||||
|
c.err = err
|
||||||
|
}
|
||||||
|
return n, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// busyMessage is what a user is told when the queue is full.
|
||||||
|
const busyMessage = "Too many recordings are waiting to be transcribed. Please try again in a few minutes."
|
||||||
|
|
||||||
// Accept spools src to disk (bounded by MaxBytes), probes it, validates it, records
|
// Accept spools src to disk (bounded by MaxBytes), probes it, validates it, records
|
||||||
// a queued upload and submits the job. It returns the upload id. A file that fails
|
// a queued upload and submits the job. It returns the upload id. A file that fails
|
||||||
// validation yields a *RejectError and creates NO row — rejection is synchronous
|
// validation yields a *RejectError and creates NO row — rejection is synchronous
|
||||||
// and leaves nothing behind. On success the temp file is handed to the job and the
|
// and leaves nothing behind. On success the temp file is handed to the job and the
|
||||||
// worker deletes it; on every failure path Accept deletes it itself.
|
// worker deletes it; on every failure path Accept deletes it itself.
|
||||||
func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reader) (string, error) {
|
func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reader) (string, error) {
|
||||||
|
// Refuse before spooling when the queue has no room, if the submitter can say so: a full
|
||||||
|
// queue otherwise cost up to the upload cap on disk, an ffprobe run and a failed row for
|
||||||
|
// every refused attempt.
|
||||||
|
if rc, ok := in.sub.(interface{ HasRoom(userID string) bool }); ok && !rc.HasRoom(userID) {
|
||||||
|
return "", &RejectError{Reason: ReasonBusy, Message: busyMessage}
|
||||||
|
}
|
||||||
f, err := os.CreateTemp(in.tmpDir, "upload-*")
|
f, err := os.CreateTemp(in.tmpDir, "upload-*")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", fmt.Errorf("uploads: spool: %w", err)
|
return "", fmt.Errorf("uploads: spool: %w", err)
|
||||||
@@ -73,11 +107,15 @@ func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reade
|
|||||||
}()
|
}()
|
||||||
|
|
||||||
// Read one byte past the cap so "exactly at the limit" and "over" are told apart.
|
// Read one byte past the cap so "exactly at the limit" and "over" are told apart.
|
||||||
n, err := io.Copy(f, io.LimitReader(src, in.limits.MaxBytes+1))
|
cr := &clientReader{r: io.LimitReader(src, in.limits.MaxBytes+1)}
|
||||||
|
n, err := io.Copy(f, cr)
|
||||||
if cerr := f.Close(); err == nil {
|
if cerr := f.Close(); err == nil {
|
||||||
err = cerr
|
err = cerr
|
||||||
}
|
}
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
if cr.err != nil && errors.Is(err, cr.err) {
|
||||||
|
return "", fmt.Errorf("%w: %w", ErrClientRead, err)
|
||||||
|
}
|
||||||
return "", fmt.Errorf("uploads: receive: %w", err)
|
return "", fmt.Errorf("uploads: receive: %w", err)
|
||||||
}
|
}
|
||||||
if n > in.limits.MaxBytes {
|
if n > in.limits.MaxBytes {
|
||||||
@@ -103,7 +141,15 @@ func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reade
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return "", fmt.Errorf("uploads: create: %w", err)
|
return "", fmt.Errorf("uploads: create: %w", err)
|
||||||
}
|
}
|
||||||
if err := in.sub.Submit(Job{UserID: userID, UploadID: id, Path: path}); err != nil {
|
if err := in.sub.Submit(Job{UserID: userID, UploadID: id, Path: path, Duration: probe.Duration}); err != nil {
|
||||||
|
if errors.Is(err, ErrQueueFull) {
|
||||||
|
// The user's to retry, not a server fault: refuse it as such, and do not leave
|
||||||
|
// the row queued -- that is the stuck spinner.
|
||||||
|
if ferr := in.store.FailUpload(ctx, userID, id, domain.UploadErrBusy); ferr != nil {
|
||||||
|
return "", fmt.Errorf("uploads: submit: %w (and failing the row: %v)", err, ferr)
|
||||||
|
}
|
||||||
|
return "", &RejectError{Reason: ReasonBusy, Message: busyMessage}
|
||||||
|
}
|
||||||
// A row that will never run must not sit queued: that is the stuck spinner.
|
// A row that will never run must not sit queued: that is the stuck spinner.
|
||||||
if ferr := in.store.FailUpload(ctx, userID, id, domain.UploadErrInterrupted); ferr != nil {
|
if ferr := in.store.FailUpload(ctx, userID, id, domain.UploadErrInterrupted); ferr != nil {
|
||||||
return "", fmt.Errorf("uploads: submit: %w (and failing the row: %v)", err, ferr)
|
return "", fmt.Errorf("uploads: submit: %w (and failing the row: %v)", err, ferr)
|
||||||
|
|||||||
@@ -0,0 +1,96 @@
|
|||||||
|
package uploads_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/uploads"
|
||||||
|
)
|
||||||
|
|
||||||
|
// A full queue is the user's to retry, not a server fault: a typed rejection, the row
|
||||||
|
// failed honestly, the audio removed (adversarial review F6).
|
||||||
|
|
||||||
|
func TestAccept_AFullQueueIsABusyRejectionNotAServerErrorAndLeavesNothingBehind(t *testing.T) {
|
||||||
|
r := newRig(t)
|
||||||
|
r.sub.err = uploads.ErrQueueFull
|
||||||
|
|
||||||
|
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||||
|
|
||||||
|
var rej *uploads.RejectError
|
||||||
|
require.True(t, errors.As(err, &rej), "a full queue must be a rejection the web layer can answer with 429: %v", err)
|
||||||
|
assert.Equal(t, uploads.ReasonBusy, rej.Reason)
|
||||||
|
assert.Contains(t, rej.Message, "waiting")
|
||||||
|
assert.Equal(t, domain.UploadErrBusy, r.st.failed["up-1"], "the row must not sit queued: that is the stuck spinner")
|
||||||
|
assert.Empty(t, r.tmpFiles(t), "the audio is deleted")
|
||||||
|
}
|
||||||
|
|
||||||
|
// roomSubmitter is a Submitter that can say, before anything is spooled, that there is no room.
|
||||||
|
type roomSubmitter struct {
|
||||||
|
fakeSubmitter
|
||||||
|
room bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *roomSubmitter) HasRoom(string) bool { return r.room }
|
||||||
|
|
||||||
|
func TestAccept_ANoRoomQueueIsRefusedBeforeAnythingIsSpooledProbedOrStored(t *testing.T) {
|
||||||
|
// Checking only at Submit meant 200 MB was spooled and ffprobe run for every refused
|
||||||
|
// attempt, and a failed row was left behind each time (second adversarial review).
|
||||||
|
r := newRig(t)
|
||||||
|
full := &roomSubmitter{room: false}
|
||||||
|
in := uploads.NewIntake(r.st, r.pr, full, uploads.Limits{MaxBytes: 1024, MaxDuration: 90 * time.Minute}, r.tmp)
|
||||||
|
|
||||||
|
_, err := in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||||
|
|
||||||
|
var rej *uploads.RejectError
|
||||||
|
require.True(t, errors.As(err, &rej))
|
||||||
|
assert.Equal(t, uploads.ReasonBusy, rej.Reason)
|
||||||
|
assert.Empty(t, r.st.created, "no row for an upload that was never accepted")
|
||||||
|
assert.Empty(t, r.pr.paths, "ffprobe was never run on it")
|
||||||
|
assert.Empty(t, r.tmpFiles(t), "nothing was spooled")
|
||||||
|
assert.Empty(t, full.jobs)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAccept_ASubmitterThatCannotSayStillWorksAsBefore(t *testing.T) {
|
||||||
|
r := newRig(t)
|
||||||
|
|
||||||
|
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||||
|
|
||||||
|
require.NoError(t, err, "a Submitter without HasRoom is accepted: the check at Submit remains")
|
||||||
|
}
|
||||||
|
|
||||||
|
// brokenBody is a client body that fails mid-read, like a connection that dropped.
|
||||||
|
type brokenBody struct{ n int }
|
||||||
|
|
||||||
|
func (b *brokenBody) Read(p []byte) (int, error) {
|
||||||
|
if b.n > 0 {
|
||||||
|
b.n = 0
|
||||||
|
return copy(p, "partial audio"), nil
|
||||||
|
}
|
||||||
|
return 0, io.ErrUnexpectedEOF
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAccept_AnErrorReadingTheClientsBodyIsMarkedAsTheClientsAndAWrappedSubmitErrorIsNot(t *testing.T) {
|
||||||
|
r := newRig(t)
|
||||||
|
|
||||||
|
_, err := r.in.Accept(context.Background(), "user-1", "t", &brokenBody{n: 1})
|
||||||
|
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.ErrorIs(t, err, uploads.ErrClientRead, "a dropped client body is the sender's to retry")
|
||||||
|
assert.Empty(t, r.tmpFiles(t))
|
||||||
|
|
||||||
|
// A fault that is ours must NOT carry the marker, even if it wraps the same sentinel.
|
||||||
|
r2 := newRig(t)
|
||||||
|
r2.sub.err = fmt.Errorf("db: %w", io.ErrUnexpectedEOF)
|
||||||
|
_, err = r2.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.NotErrorIs(t, err, uploads.ErrClientRead)
|
||||||
|
}
|
||||||
@@ -106,7 +106,8 @@ func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
|
|||||||
assert.Equal(t, audio, r.pr.seen, "the prober must see the complete uploaded bytes, on disk")
|
assert.Equal(t, audio, r.pr.seen, "the prober must see the complete uploaded bytes, on disk")
|
||||||
require.Len(t, r.sub.jobs, 1)
|
require.Len(t, r.sub.jobs, 1)
|
||||||
job := r.sub.jobs[0]
|
job := r.sub.jobs[0]
|
||||||
assert.Equal(t, uploads.Job{UserID: "user-1", UploadID: "up-1", Path: job.Path}, job)
|
assert.Equal(t, uploads.Job{UserID: "user-1", UploadID: "up-1", Path: job.Path, Duration: 10 * time.Minute}, job,
|
||||||
|
"the job carries what the file CLAIMED, so the worker can compare it with what was converted")
|
||||||
assert.Equal(t, filepath.Dir(job.Path), r.tmp, "the temp file lives in the configured directory")
|
assert.Equal(t, filepath.Dir(job.Path), r.tmp, "the temp file lives in the configured directory")
|
||||||
got, err := os.ReadFile(job.Path)
|
got, err := os.ReadFile(job.Path)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|||||||
@@ -34,6 +34,8 @@ const (
|
|||||||
ReasonTooLong Reason = "too_long"
|
ReasonTooLong Reason = "too_long"
|
||||||
ReasonTooLarge Reason = "too_large"
|
ReasonTooLarge Reason = "too_large"
|
||||||
ReasonUnreadable Reason = "unreadable"
|
ReasonUnreadable Reason = "unreadable"
|
||||||
|
// ReasonBusy: the queue is full. Not about the file: the user retries later.
|
||||||
|
ReasonBusy Reason = "busy"
|
||||||
)
|
)
|
||||||
|
|
||||||
// RejectError is a synchronous, user-presentable refusal. Message is written for
|
// RejectError is a synchronous, user-presentable refusal. Message is written for
|
||||||
|
|||||||
+222
-17
@@ -10,11 +10,18 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/errkind"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/progress"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ErrStopped is returned by Submit once the worker has stopped.
|
// ErrStopped is returned by Submit once the worker has stopped.
|
||||||
var ErrStopped = errors.New("uploads: worker stopped")
|
var ErrStopped = errors.New("uploads: worker stopped")
|
||||||
|
|
||||||
|
// ErrQueueFull is returned by Submit when the queue, or one user's share of it, is at
|
||||||
|
// its cap. Every queued job holds up to the upload cap on disk until its turn, so an
|
||||||
|
// unbounded queue is a disk-filling lever (ADR-035 review).
|
||||||
|
var ErrQueueFull = errors.New("uploads: queue full")
|
||||||
|
|
||||||
// Converter turns the uploaded file into the 16 kHz mono PCM WAV whisper wants.
|
// Converter turns the uploaded file into the 16 kHz mono PCM WAV whisper wants.
|
||||||
type Converter interface {
|
type Converter interface {
|
||||||
ToWAV(ctx context.Context, src, dst string) error
|
ToWAV(ctx context.Context, src, dst string) error
|
||||||
@@ -26,13 +33,66 @@ type Transcriber interface {
|
|||||||
Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error)
|
Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Planner is an optional extra on a Transcriber: it declares its own phases (for
|
||||||
|
// example one per engine) so the page can show them. A Transcriber that does not
|
||||||
|
// implement it gets a single "transcribing" phase from the worker. Whatever a Planner
|
||||||
|
// declares it reports itself, through progress.From(ctx) (ADR-035).
|
||||||
|
type Planner interface {
|
||||||
|
Plan() []progress.Planned
|
||||||
|
}
|
||||||
|
|
||||||
// WorkerStore is the slice of the private transcript store the worker drives.
|
// WorkerStore is the slice of the private transcript store the worker drives.
|
||||||
type WorkerStore interface {
|
type WorkerStore interface {
|
||||||
StartTranscribing(ctx context.Context, userID, id string) error
|
StartTranscribing(ctx context.Context, userID, id string) error
|
||||||
CompleteUpload(ctx context.Context, userID, id string, t domain.Transcription) error
|
CompleteUpload(ctx context.Context, userID, id string, t domain.Transcription) error
|
||||||
FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error
|
FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error
|
||||||
|
SaveProgress(ctx context.Context, userID, id string, steps []domain.ProgressStep, at time.Time) error
|
||||||
|
Heartbeat(ctx context.Context, userID, id string) error
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// WorkerOption tunes a Worker.
|
||||||
|
type WorkerOption func(*Worker)
|
||||||
|
|
||||||
|
// WithHeartbeat sets how often a running job stamps its row alive and how often it logs
|
||||||
|
// which phase is still running. The defaults are 10 s and 5 min (ADR-035).
|
||||||
|
func WithHeartbeat(beat, still time.Duration) WorkerOption {
|
||||||
|
return func(w *Worker) { w.beatEvery, w.stillEvery = beat, still }
|
||||||
|
}
|
||||||
|
|
||||||
|
// WithQueueLimits bounds how many jobs may WAIT: in total, and per user. A job being
|
||||||
|
// processed is not waiting. The defaults are 20 and 3 (one instance, one job at a time).
|
||||||
|
func WithQueueLimits(total, perUser int) WorkerOption {
|
||||||
|
return func(w *Worker) { w.maxQueued, w.maxPerUser = total, perUser }
|
||||||
|
}
|
||||||
|
|
||||||
|
// WithMaxDuration makes the worker measure the converted audio and fail a job whose real
|
||||||
|
// length exceeds the limit, or falls far short of what the file claimed. Zero (the
|
||||||
|
// default) measures nothing. The limit is first checked against the file's header, which
|
||||||
|
// can lie (ADR-035 review).
|
||||||
|
func WithMaxDuration(d time.Duration) WorkerOption { return func(w *Worker) { w.maxDuration = d } }
|
||||||
|
|
||||||
|
// WithLogger replaces slog.Default() as the worker's logger.
|
||||||
|
func WithLogger(l *slog.Logger) WorkerOption { return func(w *Worker) { w.log = l } }
|
||||||
|
|
||||||
|
const (
|
||||||
|
defaultMaxQueued = 20
|
||||||
|
defaultMaxPerUser = 3
|
||||||
|
defaultBeatEvery = 10 * time.Second
|
||||||
|
defaultStillEvery = 5 * time.Minute
|
||||||
|
)
|
||||||
|
|
||||||
|
// jobSink binds the store to one job so a progress.Tracker can persist through it.
|
||||||
|
type jobSink struct {
|
||||||
|
st WorkerStore
|
||||||
|
userID, id string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s jobSink) SaveProgress(ctx context.Context, steps []domain.ProgressStep, at time.Time) error {
|
||||||
|
return s.st.SaveProgress(ctx, s.userID, s.id, steps, at)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s jobSink) Beat(ctx context.Context) error { return s.st.Heartbeat(ctx, s.userID, s.id) }
|
||||||
|
|
||||||
// Worker transcribes accepted uploads in-process, ONE AT A TIME (ADR-033). It is
|
// Worker transcribes accepted uploads in-process, ONE AT A TIME (ADR-033). It is
|
||||||
// not a second deployable: it shares the web process's lifetime, which is why it
|
// not a second deployable: it shares the web process's lifetime, which is why it
|
||||||
// inherits ADR-018's single-replica assumption — the queue is process-local. The
|
// inherits ADR-018's single-replica assumption — the queue is process-local. The
|
||||||
@@ -45,6 +105,11 @@ type Worker struct {
|
|||||||
tmpDir string
|
tmpDir string
|
||||||
timeout time.Duration
|
timeout time.Duration
|
||||||
|
|
||||||
|
log *slog.Logger
|
||||||
|
beatEvery, stillEvery time.Duration
|
||||||
|
maxDuration time.Duration
|
||||||
|
maxQueued, maxPerUser int
|
||||||
|
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
queue []Job
|
queue []Job
|
||||||
stopped bool
|
stopped bool
|
||||||
@@ -53,8 +118,34 @@ type Worker struct {
|
|||||||
|
|
||||||
// NewWorker wires a Worker. timeout bounds each job (conversion + transcription);
|
// NewWorker wires a Worker. timeout bounds each job (conversion + transcription);
|
||||||
// a job that outlives it is failed rather than left spinning.
|
// a job that outlives it is failed rather than left spinning.
|
||||||
func NewWorker(s WorkerStore, c Converter, t Transcriber, tmpDir string, timeout time.Duration) *Worker {
|
func NewWorker(s WorkerStore, c Converter, t Transcriber, tmpDir string, timeout time.Duration, opts ...WorkerOption) *Worker {
|
||||||
return &Worker{store: s, conv: c, tr: t, tmpDir: tmpDir, timeout: timeout, wake: make(chan struct{}, 1)}
|
w := &Worker{
|
||||||
|
store: s, conv: c, tr: t, tmpDir: tmpDir, timeout: timeout, wake: make(chan struct{}, 1),
|
||||||
|
beatEvery: defaultBeatEvery, stillEvery: defaultStillEvery,
|
||||||
|
maxQueued: defaultMaxQueued, maxPerUser: defaultMaxPerUser,
|
||||||
|
}
|
||||||
|
for _, o := range opts {
|
||||||
|
o(w)
|
||||||
|
}
|
||||||
|
return w
|
||||||
|
}
|
||||||
|
|
||||||
|
// HasRoom reports whether a job from this user would be accepted right now. It lets the
|
||||||
|
// intake refuse BEFORE spooling up to the upload cap and running ffprobe; Submit still
|
||||||
|
// enforces the same limits atomically, so a race costs at worst one refused submit.
|
||||||
|
func (w *Worker) HasRoom(userID string) bool {
|
||||||
|
w.mu.Lock()
|
||||||
|
defer w.mu.Unlock()
|
||||||
|
if w.stopped || len(w.queue) >= w.maxQueued {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
mine := 0
|
||||||
|
for _, q := range w.queue {
|
||||||
|
if q.UserID == userID {
|
||||||
|
mine++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return mine < w.maxPerUser
|
||||||
}
|
}
|
||||||
|
|
||||||
// Submit queues a job. It never blocks. It satisfies Submitter.
|
// Submit queues a job. It never blocks. It satisfies Submitter.
|
||||||
@@ -64,6 +155,18 @@ func (w *Worker) Submit(j Job) error {
|
|||||||
if w.stopped {
|
if w.stopped {
|
||||||
return ErrStopped
|
return ErrStopped
|
||||||
}
|
}
|
||||||
|
if len(w.queue) >= w.maxQueued {
|
||||||
|
return ErrQueueFull
|
||||||
|
}
|
||||||
|
mine := 0
|
||||||
|
for _, q := range w.queue {
|
||||||
|
if q.UserID == j.UserID {
|
||||||
|
mine++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if mine >= w.maxPerUser {
|
||||||
|
return ErrQueueFull
|
||||||
|
}
|
||||||
w.queue = append(w.queue, j)
|
w.queue = append(w.queue, j)
|
||||||
select {
|
select {
|
||||||
case w.wake <- struct{}{}:
|
case w.wake <- struct{}{}:
|
||||||
@@ -125,55 +228,118 @@ func (w *Worker) stop() {
|
|||||||
// uploaded file and the converted WAV are deleted on every path — the audio is
|
// uploaded file and the converted WAV are deleted on every path — the audio is
|
||||||
// never retained (ADR-033). A failure is recorded with a TYPED kind; a shutdown
|
// never retained (ADR-033). A failure is recorded with a TYPED kind; a shutdown
|
||||||
// mid-job is not a failure and leaves the row for the start-up sweep.
|
// mid-job is not a failure and leaves the row for the start-up sweep.
|
||||||
|
//
|
||||||
|
// Every phase is reported through one progress.Tracker, which saves it to the row for
|
||||||
|
// the page and logs it, and a heartbeat stamps the row on its own while the job runs
|
||||||
|
// (ADR-035). Nothing logged or saved here carries transcript text or the file's name.
|
||||||
func (w *Worker) Process(ctx context.Context, job Job) {
|
func (w *Worker) Process(ctx context.Context, job Job) {
|
||||||
defer func() { _ = os.Remove(job.Path) }()
|
defer func() { _ = os.Remove(job.Path) }()
|
||||||
log := slog.Default().With("upload", job.UploadID)
|
base := w.log
|
||||||
|
if base == nil {
|
||||||
|
base = slog.Default()
|
||||||
|
}
|
||||||
|
log := base.With("upload", job.UploadID)
|
||||||
start := time.Now()
|
start := time.Now()
|
||||||
|
|
||||||
if err := w.store.StartTranscribing(ctx, job.UserID, job.UploadID); err != nil {
|
if err := w.store.StartTranscribing(ctx, job.UserID, job.UploadID); err != nil {
|
||||||
log.Error("upload: start transcribing", "err", err)
|
log.Error("upload: start transcribing", "kind", errkind.Of(err))
|
||||||
w.fail(ctx, job, log)
|
w.fail(ctx, job, log, domain.UploadErrTranscriptionFailed)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
plan := []progress.Planned{{Key: "received", Label: "Received"}, {Key: "audio", Label: "Preparing the audio"}}
|
||||||
|
planner, planned := w.tr.(Planner)
|
||||||
|
if planned {
|
||||||
|
plan = append(plan, planner.Plan()...)
|
||||||
|
} else {
|
||||||
|
plan = append(plan, progress.Planned{Key: "transcribe", Label: "Transcribing"})
|
||||||
|
}
|
||||||
|
plan = append(plan, progress.Planned{Key: "save", Label: "Saving the transcript"})
|
||||||
|
tr := progress.NewTracker(ctx, jobSink{st: w.store, userID: job.UserID, id: job.UploadID}, log, plan)
|
||||||
|
tr.Start()
|
||||||
|
tr.Begin("received", 0)
|
||||||
|
tr.Done("received")
|
||||||
|
|
||||||
|
hbCtx, stopBeat := context.WithCancel(ctx)
|
||||||
|
hbDone := make(chan struct{})
|
||||||
|
go func() {
|
||||||
|
defer close(hbDone)
|
||||||
|
tr.Heartbeat(hbCtx, w.beatEvery, w.stillEvery)
|
||||||
|
}()
|
||||||
|
defer func() { stopBeat(); <-hbDone }() // no heartbeat outlives its job
|
||||||
|
|
||||||
|
// abort records where the job stopped. A shutdown (the parent context ended) is not
|
||||||
|
// a failure: the row is left for the start-up sweep, so the step stays as it was.
|
||||||
|
abort := func(kind domain.UploadErrorKind) {
|
||||||
|
if ctx.Err() == nil {
|
||||||
|
tr.FailRunning()
|
||||||
|
}
|
||||||
|
w.fail(ctx, job, log, kind)
|
||||||
|
}
|
||||||
|
|
||||||
wav, err := w.tempWAV()
|
wav, err := w.tempWAV()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Error("upload: temp wav", "err", err)
|
log.Error("upload: temp wav", "kind", errkind.Of(err))
|
||||||
w.fail(ctx, job, log)
|
abort(domain.UploadErrTranscriptionFailed)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
defer func() { _ = os.Remove(wav) }()
|
defer func() { _ = os.Remove(wav) }()
|
||||||
|
|
||||||
jobCtx, cancel := context.WithTimeout(ctx, w.timeout)
|
jobCtx, cancel := context.WithTimeout(ctx, w.timeout)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
tr.Begin("audio", 0)
|
||||||
if err := w.conv.ToWAV(jobCtx, job.Path, wav); err != nil {
|
if err := w.conv.ToWAV(jobCtx, job.Path, wav); err != nil {
|
||||||
log.Error("upload: convert", "err", err)
|
log.Error("upload: convert", "kind", errkind.Of(err))
|
||||||
w.fail(ctx, job, log)
|
abort(domain.UploadErrTranscriptionFailed)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
t, err := w.tr.Transcribe(jobCtx, wav)
|
got, tooLong := w.audioLengthProblem(wav)
|
||||||
|
if tooLong {
|
||||||
|
abort(domain.UploadErrTooLong)
|
||||||
|
log.Error("upload: converted audio is over the limit", "audio_s", int(got.Seconds()), "limit_s", int(w.maxDuration.Seconds()))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if w.maxDuration > 0 && job.Duration > 0 {
|
||||||
|
if note := shortfallNote(got, job.Duration); note != "" {
|
||||||
|
log.Warn("upload: audio is much shorter than the file claimed", "audio_s", int(got.Seconds()), "claimed_s", int(job.Duration.Seconds()))
|
||||||
|
tr.Annotate("audio", note)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
tr.Done("audio")
|
||||||
|
|
||||||
|
if !planned {
|
||||||
|
tr.Begin("transcribe", 0)
|
||||||
|
}
|
||||||
|
t, err := w.tr.Transcribe(progress.With(jobCtx, tr), wav)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Error("upload: transcribe", "err", err, "elapsed", time.Since(start).Round(time.Second))
|
log.Error("upload: transcribe", "kind", errkind.Of(err), "elapsed", time.Since(start).Round(time.Second))
|
||||||
w.fail(ctx, job, log)
|
abort(domain.UploadErrTranscriptionFailed)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
if !planned {
|
||||||
|
tr.Done("transcribe")
|
||||||
|
}
|
||||||
|
|
||||||
|
tr.Begin("save", 0)
|
||||||
if err := w.store.CompleteUpload(ctx, job.UserID, job.UploadID, t); err != nil {
|
if err := w.store.CompleteUpload(ctx, job.UserID, job.UploadID, t); err != nil {
|
||||||
log.Error("upload: store transcript", "err", err)
|
log.Error("upload: store transcript", "kind", errkind.Of(err))
|
||||||
w.fail(ctx, job, log)
|
abort(domain.UploadErrTranscriptionFailed)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
tr.Done("save")
|
||||||
|
tr.Finish()
|
||||||
log.Info("upload transcribed", "endpoint", t.Endpoint, "audio_s", t.AudioSeconds,
|
log.Info("upload transcribed", "endpoint", t.Endpoint, "audio_s", t.AudioSeconds,
|
||||||
"segments", len(t.Segments), "elapsed", time.Since(start).Round(time.Second))
|
"segments", len(t.Segments), "elapsed", time.Since(start).Round(time.Second))
|
||||||
}
|
}
|
||||||
|
|
||||||
// fail records a transcription failure — unless the PARENT context ended, which
|
// fail records a transcription failure — unless the PARENT context ended, which
|
||||||
// is a shutdown, not a failure: the row is left for the start-up sweep.
|
// is a shutdown, not a failure: the row is left for the start-up sweep.
|
||||||
func (w *Worker) fail(ctx context.Context, job Job, log *slog.Logger) {
|
func (w *Worker) fail(ctx context.Context, job Job, log *slog.Logger, kind domain.UploadErrorKind) {
|
||||||
if ctx.Err() != nil {
|
if ctx.Err() != nil {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if err := w.store.FailUpload(ctx, job.UserID, job.UploadID, domain.UploadErrTranscriptionFailed); err != nil {
|
if err := w.store.FailUpload(ctx, job.UserID, job.UploadID, kind); err != nil {
|
||||||
log.Error("upload: record failure", "err", err)
|
log.Error("upload: record failure", "kind", errkind.Of(err))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -189,3 +355,42 @@ func (w *Worker) tempWAV() (string, error) {
|
|||||||
}
|
}
|
||||||
return name, nil
|
return name, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const (
|
||||||
|
// tooLongSlack is how far over the limit the converted audio may be before the claim
|
||||||
|
// in the file's header is called a lie: ffmpeg rounds.
|
||||||
|
tooLongSlack = 30 * time.Second
|
||||||
|
// A file counts as truncated when it converts to under 80% of the length it claimed
|
||||||
|
// AND more than two minutes short. Probe estimates for variable-bitrate files are
|
||||||
|
// imprecise; refusing real recordings would be worse than the problem.
|
||||||
|
incompleteBelow = 0.8
|
||||||
|
incompleteShortfall = 2 * time.Minute
|
||||||
|
)
|
||||||
|
|
||||||
|
// audioLengthProblem measures the converted WAV (16 kHz mono 16-bit PCM is 32,000 bytes a
|
||||||
|
// second after a 44-byte header) against the limit. It returns the real length and whether
|
||||||
|
// that is over the limit by more than the tolerance.
|
||||||
|
func (w *Worker) audioLengthProblem(wav string) (got time.Duration, tooLong bool) {
|
||||||
|
if w.maxDuration <= 0 {
|
||||||
|
return 0, false
|
||||||
|
}
|
||||||
|
info, err := os.Stat(wav)
|
||||||
|
if err != nil {
|
||||||
|
return 0, false
|
||||||
|
}
|
||||||
|
got = time.Duration(float64(max(info.Size()-44, 0)) / 32000 * float64(time.Second))
|
||||||
|
return got, got > w.maxDuration+tooLongSlack
|
||||||
|
}
|
||||||
|
|
||||||
|
// shortfallNote is a remark for the audio step when the converted audio is much shorter
|
||||||
|
// than the file claimed, or "" when it is not. It is a NOTE and never a failure: a
|
||||||
|
// container header can over-estimate a complete file (a variable-bitrate MP3 without a Xing
|
||||||
|
// header claimed 79 minutes for 20 real ones), so refusing would turn away real
|
||||||
|
// recordings, while saying nothing about a truncated one would hide a partial transcript.
|
||||||
|
func shortfallNote(got, claimed time.Duration) string {
|
||||||
|
if float64(got) >= incompleteBelow*float64(claimed) || claimed-got <= incompleteShortfall {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("The audio is %s long; the file claimed %s. If the recording was cut off, upload the complete one.",
|
||||||
|
FormatDuration(got), FormatDuration(claimed))
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,187 @@
|
|||||||
|
package uploads_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"os"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/uploads"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The duration limit is a claim in the file's header. The worker measures what was
|
||||||
|
// really converted and refuses both a recording that is longer than the limit (the
|
||||||
|
// header lied low) and one that ends far earlier than the file claimed (it was truncated
|
||||||
|
// and would otherwise yield a partial transcript with no warning) -- adversarial review F2.
|
||||||
|
|
||||||
|
// sizedConverter writes a (sparse) 16 kHz mono PCM WAV of the given length, without
|
||||||
|
// actually allocating it: 32,000 bytes per second plus a 44-byte header.
|
||||||
|
type sizedConverter struct{ seconds float64 }
|
||||||
|
|
||||||
|
func (c sizedConverter) ToWAV(_ context.Context, _, dst string) error {
|
||||||
|
f, err := os.Create(dst)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer func() { _ = f.Close() }()
|
||||||
|
return f.Truncate(44 + int64(c.seconds*32000))
|
||||||
|
}
|
||||||
|
|
||||||
|
func durationWorker(t *testing.T, r *wrig, wavSeconds float64, max time.Duration) *uploads.Worker {
|
||||||
|
t.Helper()
|
||||||
|
opts := []uploads.WorkerOption{}
|
||||||
|
if max > 0 {
|
||||||
|
opts = append(opts, uploads.WithMaxDuration(max))
|
||||||
|
}
|
||||||
|
return uploads.NewWorker(r.st, sizedConverter{seconds: wavSeconds}, r.tr, r.tmp, time.Minute, opts...)
|
||||||
|
}
|
||||||
|
|
||||||
|
func jobWith(r *wrig, t *testing.T, claimed time.Duration) uploads.Job {
|
||||||
|
j := r.spool(t, "a")
|
||||||
|
j.Duration = claimed
|
||||||
|
return j
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerDuration_AConvertedLengthOverTheLimitFailsTheJobBeforeAnyEngineSeesIt(t *testing.T) {
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
w := durationWorker(t, r, 100*60, 90*time.Minute)
|
||||||
|
|
||||||
|
w.Process(context.Background(), jobWith(r, t, 10*time.Second))
|
||||||
|
|
||||||
|
_, failed := r.st.snapshot()
|
||||||
|
assert.Equal(t, domain.UploadErrTooLong, failed["a"], "a header that claimed 10 s must not let 100 minutes through")
|
||||||
|
assert.Empty(t, r.tr.paths, "no engine may be sent audio that is over the limit")
|
||||||
|
assert.Empty(t, r.leftovers(t), "the audio is still deleted")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerDuration_JustOverTheLimitWithinTheSlackStillTranscribes(t *testing.T) {
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
w := durationWorker(t, r, 90*60+20, 90*time.Minute)
|
||||||
|
|
||||||
|
w.Process(context.Background(), jobWith(r, t, 90*time.Minute))
|
||||||
|
|
||||||
|
completed, failed := r.st.snapshot()
|
||||||
|
assert.Empty(t, failed)
|
||||||
|
assert.Contains(t, completed, "a", "ffmpeg rounds: 20 s over a 90-minute limit is not a lie")
|
||||||
|
}
|
||||||
|
|
||||||
|
func audioStepNote(r *wrig) string {
|
||||||
|
for _, st := range r.st.lastSteps() {
|
||||||
|
if st.Key == "audio" {
|
||||||
|
return st.Note
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerDuration_AFileThatEndsFarEarlierThanItClaimedIsStillTranscribedAndSaysSo(t *testing.T) {
|
||||||
|
// A container header can over-estimate a complete file (VBR MP3 without a Xing header
|
||||||
|
// claimed 79 minutes for 20 real ones). Refusing it was wrong; staying silent about a
|
||||||
|
// truncated one is wrong too. So it is transcribed, and the audio step carries a note.
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
w := durationWorker(t, r, 20*60, 90*time.Minute)
|
||||||
|
|
||||||
|
w.Process(context.Background(), jobWith(r, t, 79*time.Minute))
|
||||||
|
|
||||||
|
completed, failed := r.st.snapshot()
|
||||||
|
assert.Empty(t, failed, "a complete recording with a bad header estimate must not be refused")
|
||||||
|
assert.Contains(t, completed, "a")
|
||||||
|
note := audioStepNote(r)
|
||||||
|
assert.Contains(t, note, "20 min", "what was really transcribed")
|
||||||
|
assert.Contains(t, note, "1 h 19 min", "what the file claimed")
|
||||||
|
assert.Contains(t, note, "cut off", "and what to do if that is not a surprise")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerDuration_TheShortfallNoteIsPinnedAtEightyPercentAndTwoMinutes(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
claimed time.Duration
|
||||||
|
wav float64
|
||||||
|
noted bool
|
||||||
|
}{
|
||||||
|
{"79 percent of an hour", 60 * time.Minute, 0.79 * 3600, true},
|
||||||
|
{"exactly 80 percent of an hour", 60 * time.Minute, 2880, false},
|
||||||
|
{"one second under 80 percent of an hour", 60 * time.Minute, 2879, true},
|
||||||
|
{"81 percent of an hour", 60 * time.Minute, 0.81 * 3600, false},
|
||||||
|
{"exactly two minutes short and under 80 percent", 6 * time.Minute, 4 * 60, false},
|
||||||
|
{"two minutes and a second short and under 80 percent", 6 * time.Minute, 4*60 - 1, true},
|
||||||
|
{"unknown claimed length", 0, 20 * 60, false},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
w := durationWorker(t, r, tt.wav, 90*time.Minute)
|
||||||
|
|
||||||
|
w.Process(context.Background(), jobWith(r, t, tt.claimed))
|
||||||
|
|
||||||
|
assert.Equal(t, tt.noted, audioStepNote(r) != "")
|
||||||
|
completed, _ := r.st.snapshot()
|
||||||
|
assert.Contains(t, completed, "a", "a note never stops the job")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerDuration_TheTooLongToleranceIsExactlyThirtySeconds(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
wav float64
|
||||||
|
fail bool
|
||||||
|
}{
|
||||||
|
{"at the limit plus 30 s", 90*60 + 30, false},
|
||||||
|
{"at the limit plus 31 s", 90*60 + 31, true},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
w := durationWorker(t, r, tt.wav, 90*time.Minute)
|
||||||
|
|
||||||
|
w.Process(context.Background(), jobWith(r, t, 90*time.Minute))
|
||||||
|
|
||||||
|
_, failed := r.st.snapshot()
|
||||||
|
if tt.fail {
|
||||||
|
assert.Equal(t, domain.UploadErrTooLong, failed["a"])
|
||||||
|
} else {
|
||||||
|
assert.Empty(t, failed)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerDuration_AnEstimateThatIsMerelyALittleOffIsNotTruncation(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
claimed time.Duration
|
||||||
|
wav float64
|
||||||
|
}{
|
||||||
|
{"five percent short", 60 * time.Minute, 57 * 60},
|
||||||
|
{"under two minutes short of a long file", 60 * time.Minute, 59 * 60},
|
||||||
|
{"unknown claimed length", 0, 20 * 60},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
w := durationWorker(t, r, tt.wav, 90*time.Minute)
|
||||||
|
|
||||||
|
w.Process(context.Background(), jobWith(r, t, tt.claimed))
|
||||||
|
|
||||||
|
completed, failed := r.st.snapshot()
|
||||||
|
assert.Empty(t, failed)
|
||||||
|
assert.Contains(t, completed, "a")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerDuration_WithNoMaximumConfiguredNothingIsMeasured(t *testing.T) {
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
w := durationWorker(t, r, 100*60, 0)
|
||||||
|
|
||||||
|
w.Process(context.Background(), jobWith(r, t, 0))
|
||||||
|
|
||||||
|
completed, _ := r.st.snapshot()
|
||||||
|
assert.Contains(t, completed, "a")
|
||||||
|
require.NotNil(t, w)
|
||||||
|
}
|
||||||
@@ -0,0 +1,215 @@
|
|||||||
|
package uploads_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"log/slog"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/progress"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/uploads"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Progress, heartbeat and logging of the worker (ADR-035, #42, #43).
|
||||||
|
|
||||||
|
// plannedTr is a transcriber that declares its own phases and reports them, like the
|
||||||
|
// two-engine Ensemble does.
|
||||||
|
type plannedTr struct{ *fakeTranscriber }
|
||||||
|
|
||||||
|
func (plannedTr) Plan() []progress.Planned {
|
||||||
|
return []progress.Planned{{Key: "engine_a", Label: "Transcribing with A"}, {Key: "engine_b", Label: "Transcribing with B"}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p plannedTr) Transcribe(ctx context.Context, wav string) (domain.Transcription, error) {
|
||||||
|
r := progress.From(ctx)
|
||||||
|
r.Begin("engine_a", 0)
|
||||||
|
r.Done("engine_a")
|
||||||
|
r.Begin("engine_b", 0)
|
||||||
|
r.Done("engine_b")
|
||||||
|
return p.fakeTranscriber.Transcribe(ctx, wav)
|
||||||
|
}
|
||||||
|
|
||||||
|
func keysAndStates(steps []domain.ProgressStep) []string {
|
||||||
|
var out []string
|
||||||
|
for _, s := range steps {
|
||||||
|
out = append(out, s.Key+":"+string(s.State))
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// syncLog is a log sink safe to read while the worker's goroutines write.
|
||||||
|
type syncLog struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
b bytes.Buffer
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *syncLog) Write(p []byte) (int, error) {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
return l.b.Write(p)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *syncLog) String() string {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
return l.b.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
func newProgressWorker(t *testing.T, r *wrig, tr uploads.Transcriber, opts ...uploads.WorkerOption) *uploads.Worker {
|
||||||
|
t.Helper()
|
||||||
|
return uploads.NewWorker(r.st, r.conv, tr, r.tmp, time.Minute, opts...)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerProgress_ASingleEngineJobShowsEveryPhaseDone(t *testing.T) {
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
r.w.Process(context.Background(), r.spool(t, "a"))
|
||||||
|
|
||||||
|
assert.Equal(t, []string{"received:done", "audio:done", "transcribe:done", "save:done"}, keysAndStates(r.st.lastSteps()))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerProgress_ATranscriberThatPlansItsOwnPhasesGetsThemBetweenAudioAndSaving(t *testing.T) {
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
w := newProgressWorker(t, r, plannedTr{r.tr})
|
||||||
|
|
||||||
|
w.Process(context.Background(), r.spool(t, "a"))
|
||||||
|
|
||||||
|
assert.Equal(t, []string{"received:done", "audio:done", "engine_a:done", "engine_b:done", "save:done"}, keysAndStates(r.st.lastSteps()))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerProgress_AFailureMarksWhereItStoppedAndLeavesTheRestPending(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
setup func(*wrig)
|
||||||
|
want []string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
"conversion fails", func(r *wrig) { r.conv.err = errors.New("moov atom not found") },
|
||||||
|
[]string{"received:done", "audio:failed", "transcribe:pending", "save:pending"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"every engine fails", func(r *wrig) { r.tr.err = errors.New("iguana down") },
|
||||||
|
[]string{"received:done", "audio:done", "transcribe:failed", "save:pending"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
tt.setup(r)
|
||||||
|
|
||||||
|
r.w.Process(context.Background(), r.spool(t, "a"))
|
||||||
|
|
||||||
|
assert.Equal(t, tt.want, keysAndStates(r.st.lastSteps()))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerProgress_AShutdownIsNotRecordedAsAFailedStep(t *testing.T) {
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
r.tr.block = make(chan struct{})
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
done := make(chan struct{})
|
||||||
|
go func() { defer close(done); r.w.Process(ctx, r.spool(t, "a")) }()
|
||||||
|
require.Eventually(t, func() bool {
|
||||||
|
return strings.Contains(strings.Join(keysAndStates(r.st.lastSteps()), ","), "transcribe:running")
|
||||||
|
}, 2*time.Second, time.Millisecond)
|
||||||
|
|
||||||
|
cancel()
|
||||||
|
<-done
|
||||||
|
|
||||||
|
assert.Contains(t, keysAndStates(r.st.lastSteps()), "transcribe:running",
|
||||||
|
"a shutdown leaves the row for the start-up sweep; it must not pose as a failure of the step")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerProgress_TheHeartbeatBeatsWhileAnEngineCallIsStuckAndStopsWhenTheJobEnds(t *testing.T) {
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
r.tr.block = make(chan struct{})
|
||||||
|
r.st.beatDelay = 20 * time.Millisecond
|
||||||
|
w := newProgressWorker(t, r, r.tr, uploads.WithHeartbeat(5*time.Millisecond, time.Hour))
|
||||||
|
done := make(chan struct{})
|
||||||
|
go func() { defer close(done); w.Process(context.Background(), r.spool(t, "a")) }()
|
||||||
|
|
||||||
|
require.Eventually(t, func() bool { return r.st.beatCount() >= 3 }, 2*time.Second, 5*time.Millisecond,
|
||||||
|
"the page tells a slow engine from a dead service by this heartbeat, so it must not wait on the engine")
|
||||||
|
close(r.tr.block)
|
||||||
|
<-done
|
||||||
|
after := r.st.beatCount()
|
||||||
|
time.Sleep(40 * time.Millisecond)
|
||||||
|
|
||||||
|
assert.Equal(t, after, r.st.beatCount(), "no heartbeat may outlive its job")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerProgress_OnASuccessfulJobTheLogNamesEveryPhaseInOrderAndNeverTheTranscriptOrThePath(t *testing.T) {
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
r.tr.result = domain.Transcription{Text: "zebrakaka secret meeting words", Endpoint: "iguana/kb-whisper", AudioSeconds: 3}
|
||||||
|
buf := &syncLog{}
|
||||||
|
w := newProgressWorker(t, r, r.tr, uploads.WithLogger(slog.New(slog.NewTextHandler(buf, nil))))
|
||||||
|
|
||||||
|
w.Process(context.Background(), r.spool(t, "a"))
|
||||||
|
|
||||||
|
out := buf.String()
|
||||||
|
audio := strings.Index(out, "step=audio")
|
||||||
|
transcribe := strings.Index(out, "step=transcribe")
|
||||||
|
save := strings.Index(out, "step=save")
|
||||||
|
require.True(t, audio >= 0 && transcribe > audio && save > transcribe, "phases must be logged in the order they ran:\n%s", out)
|
||||||
|
assert.Contains(t, out, "upload=a", "every line says which upload it is about")
|
||||||
|
assert.NotContains(t, out, "zebrakaka", "no transcript text in the log")
|
||||||
|
assert.NotContains(t, out, r.tmp, "no spool path, which carries the upload's identity, in the log")
|
||||||
|
assert.Contains(t, out, "upload transcribed", "the job's own summary line")
|
||||||
|
assert.Contains(t, out, "endpoint=iguana/kb-whisper", "which endpoint served it is a quality fact")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerProgress_AFailureLogsAKindNeverTheErrorTextOrThePath(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
setup func(*wrig)
|
||||||
|
line string
|
||||||
|
}{
|
||||||
|
{"conversion fails and names the spool file", func(r *wrig) {
|
||||||
|
r.conv.err = errors.New("Error opening input file " + r.tmp + "/upload-a: Invalid data zebrakaka")
|
||||||
|
}, "upload: convert"},
|
||||||
|
{"every engine fails and the gateway echoed a word", func(r *wrig) {
|
||||||
|
r.tr.err = errors.New("gateway echoed: zebrakaka secret words")
|
||||||
|
}, "upload: transcribe"},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
tt.setup(r)
|
||||||
|
buf := &syncLog{}
|
||||||
|
w := newProgressWorker(t, r, r.tr, uploads.WithLogger(slog.New(slog.NewTextHandler(buf, nil))))
|
||||||
|
|
||||||
|
w.Process(context.Background(), r.spool(t, "a"))
|
||||||
|
|
||||||
|
out := buf.String()
|
||||||
|
assert.NotContains(t, out, "zebrakaka", "provider text or file contents in the log")
|
||||||
|
assert.NotContains(t, out, r.tmp, "the spool path in the log")
|
||||||
|
assert.Contains(t, out, tt.line, "the failure must be logged at all, not only scrubbed")
|
||||||
|
assert.Contains(t, out, "kind=error")
|
||||||
|
assert.Contains(t, out, "upload=a")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerProgress_AStoreFailureLogsItsSQLStateNeverTheValueItQuotes(t *testing.T) {
|
||||||
|
// A Postgres error can quote the offending value; for a transcript row that is the
|
||||||
|
// transcript (fourth adversarial review). The log gets the SQLSTATE.
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
r.st.completeErr = dbErr{}
|
||||||
|
buf := &syncLog{}
|
||||||
|
w := newProgressWorker(t, r, r.tr, uploads.WithLogger(slog.New(slog.NewTextHandler(buf, nil))))
|
||||||
|
|
||||||
|
w.Process(context.Background(), r.spool(t, "a"))
|
||||||
|
|
||||||
|
out := buf.String()
|
||||||
|
assert.NotContains(t, out, "zebrakaka")
|
||||||
|
assert.Contains(t, out, "upload: store transcript")
|
||||||
|
assert.Contains(t, out, "kind=sql_22P02")
|
||||||
|
}
|
||||||
@@ -0,0 +1,108 @@
|
|||||||
|
package uploads_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"strconv"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/uploads"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The queue was append-only: 100,000 submits with the worker idle all succeeded, and each
|
||||||
|
// queued job holds up to the upload cap on disk until its turn (adversarial review F6).
|
||||||
|
|
||||||
|
func queueWorker(t *testing.T, total, perUser int) *uploads.Worker {
|
||||||
|
t.Helper()
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
return uploads.NewWorker(r.st, r.conv, r.tr, r.tmp, time.Minute, uploads.WithQueueLimits(total, perUser))
|
||||||
|
}
|
||||||
|
|
||||||
|
func submit(w *uploads.Worker, user string, n int) error {
|
||||||
|
return w.Submit(uploads.Job{UserID: user, UploadID: user + "-" + strconv.Itoa(n), Path: "/nonexistent"})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerSubmit_RefusesBeyondTheTotalQueueCap(t *testing.T) {
|
||||||
|
w := queueWorker(t, 3, 10)
|
||||||
|
|
||||||
|
for i := 0; i < 3; i++ {
|
||||||
|
require.NoError(t, submit(w, "u"+strconv.Itoa(i), i))
|
||||||
|
}
|
||||||
|
|
||||||
|
err := submit(w, "u9", 9)
|
||||||
|
assert.ErrorIs(t, err, uploads.ErrQueueFull)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerSubmit_OneUserCannotFillTheQueueForEveryoneElse(t *testing.T) {
|
||||||
|
w := queueWorker(t, 100, 2)
|
||||||
|
|
||||||
|
require.NoError(t, submit(w, "alice", 1))
|
||||||
|
require.NoError(t, submit(w, "alice", 2))
|
||||||
|
errAlice := submit(w, "alice", 3)
|
||||||
|
errBob := submit(w, "bob", 1)
|
||||||
|
|
||||||
|
assert.True(t, errors.Is(errAlice, uploads.ErrQueueFull), "a third waiting upload from one user is refused")
|
||||||
|
assert.NoError(t, errBob, "another user is not affected by alice's queue")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerSubmit_ThereIsADefaultCapSoANeglectedWorkerCannotBeFilledForever(t *testing.T) {
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
w := uploads.NewWorker(r.st, r.conv, r.tr, r.tmp, time.Minute)
|
||||||
|
|
||||||
|
refused := 0
|
||||||
|
for i := 0; i < 1000; i++ {
|
||||||
|
if err := submit(w, "u"+strconv.Itoa(i), i); errors.Is(err, uploads.ErrQueueFull) {
|
||||||
|
refused++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
assert.Positive(t, refused, "with the defaults, 1,000 submits to an idle worker must not all be accepted")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerSubmit_AJobThatHasLeftTheQueueFreesItsPlace(t *testing.T) {
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
w := uploads.NewWorker(r.st, r.conv, r.tr, r.tmp, time.Minute, uploads.WithQueueLimits(1, 1))
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
done := make(chan struct{})
|
||||||
|
go func() { defer close(done); w.Run(ctx) }()
|
||||||
|
defer func() { cancel(); <-done }()
|
||||||
|
|
||||||
|
job := r.spool(t, "first")
|
||||||
|
require.NoError(t, w.Submit(job))
|
||||||
|
require.Eventually(t, func() bool { c, _ := r.st.snapshot(); _, ok := c["first"]; return ok },
|
||||||
|
5*time.Second, 5*time.Millisecond, "the first job never completed")
|
||||||
|
|
||||||
|
assert.NoError(t, w.Submit(r.spool(t, "second")), "the cap counts jobs waiting, not jobs ever submitted")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerHasRoom_SaysWhetherASubmitWouldBeAccepted(t *testing.T) {
|
||||||
|
w := queueWorker(t, 3, 2)
|
||||||
|
assert.True(t, w.HasRoom("alice"))
|
||||||
|
require.NoError(t, submit(w, "alice", 1))
|
||||||
|
require.NoError(t, submit(w, "alice", 2))
|
||||||
|
|
||||||
|
assert.False(t, w.HasRoom("alice"), "alice is at her share")
|
||||||
|
assert.True(t, w.HasRoom("bob"), "bob is not")
|
||||||
|
require.NoError(t, submit(w, "bob", 1))
|
||||||
|
assert.False(t, w.HasRoom("carol"), "the queue as a whole is at its cap")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerHasRoom_AgreesWithSubmitAtEveryStepAndIsFalseOnceStopped(t *testing.T) {
|
||||||
|
w := queueWorker(t, 2, 1)
|
||||||
|
for i, user := range []string{"a", "a", "b", "c"} {
|
||||||
|
room := w.HasRoom(user)
|
||||||
|
err := submit(w, user, i)
|
||||||
|
assert.Equal(t, room, err == nil, "step %d (%s): HasRoom=%v but Submit returned %v", i, user, room, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
stopped := uploads.NewWorker(r.st, r.conv, r.tr, r.tmp, time.Minute)
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
stopped.Run(ctx)
|
||||||
|
assert.False(t, stopped.HasRoom("a"), "a worker that has stopped has no room")
|
||||||
|
}
|
||||||
@@ -23,6 +23,13 @@ type wStore struct {
|
|||||||
completed map[string]domain.Transcription
|
completed map[string]domain.Transcription
|
||||||
failed map[string]domain.UploadErrorKind
|
failed map[string]domain.UploadErrorKind
|
||||||
startErr error
|
startErr error
|
||||||
|
// completeErr makes CompleteUpload fail, as a database outage would.
|
||||||
|
completeErr error
|
||||||
|
saves [][]domain.ProgressStep
|
||||||
|
beats int
|
||||||
|
// beatDelay makes each heartbeat write slow, so one is still in flight when a job
|
||||||
|
// ends -- which is what proves the worker waits for its heartbeat to finish.
|
||||||
|
beatDelay time.Duration
|
||||||
}
|
}
|
||||||
|
|
||||||
func newWStore() *wStore {
|
func newWStore() *wStore {
|
||||||
@@ -39,6 +46,9 @@ func (s *wStore) StartTranscribing(_ context.Context, _, id string) error {
|
|||||||
func (s *wStore) CompleteUpload(_ context.Context, _, id string, t domain.Transcription) error {
|
func (s *wStore) CompleteUpload(_ context.Context, _, id string, t domain.Transcription) error {
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
defer s.mu.Unlock()
|
defer s.mu.Unlock()
|
||||||
|
if s.completeErr != nil {
|
||||||
|
return s.completeErr
|
||||||
|
}
|
||||||
s.completed[id] = t
|
s.completed[id] = t
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -50,6 +60,36 @@ func (s *wStore) FailUpload(_ context.Context, _, id string, k domain.UploadErro
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (s *wStore) SaveProgress(_ context.Context, _, _ string, steps []domain.ProgressStep, _ time.Time) error {
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
s.saves = append(s.saves, append([]domain.ProgressStep(nil), steps...))
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *wStore) Heartbeat(context.Context, string, string) error {
|
||||||
|
time.Sleep(s.beatDelay)
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
s.beats++
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *wStore) lastSteps() []domain.ProgressStep {
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
if len(s.saves) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return s.saves[len(s.saves)-1]
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *wStore) beatCount() int {
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
return s.beats
|
||||||
|
}
|
||||||
|
|
||||||
func (s *wStore) snapshot() (map[string]domain.Transcription, map[string]domain.UploadErrorKind) {
|
func (s *wStore) snapshot() (map[string]domain.Transcription, map[string]domain.UploadErrorKind) {
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
defer s.mu.Unlock()
|
defer s.mu.Unlock()
|
||||||
@@ -173,6 +213,10 @@ func TestWorkerProcess_FailuresAreTypedAndStillDeleteTheAudio(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
{"conversion fails", func(r *wrig) { r.conv.err = errors.New("moov atom not found") }},
|
{"conversion fails", func(r *wrig) { r.conv.err = errors.New("moov atom not found") }},
|
||||||
{"every transcription endpoint fails", func(r *wrig) { r.tr.err = errors.New("iguana down; berget down") }},
|
{"every transcription endpoint fails", func(r *wrig) { r.tr.err = errors.New("iguana down; berget down") }},
|
||||||
|
{"storing the finished transcript fails", func(r *wrig) { r.st.completeErr = errors.New("db down") }},
|
||||||
|
{"the temporary WAV cannot be created", func(r *wrig) {
|
||||||
|
r.w = uploads.NewWorker(r.st, r.conv, r.tr, filepath.Join(r.tmp, "missing"), time.Minute)
|
||||||
|
}},
|
||||||
}
|
}
|
||||||
for _, tt := range tests {
|
for _, tt := range tests {
|
||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
@@ -280,3 +324,11 @@ func TestWorkerRun_StoppingDeletesQueuedAudioAndRefusesNewJobs(t *testing.T) {
|
|||||||
assert.Empty(t, r.leftovers(t), "audio of queued jobs is deleted on stop: the rows are swept to interrupted at next start")
|
assert.Empty(t, r.leftovers(t), "audio of queued jobs is deleted on stop: the rows are swept to interrupted at next start")
|
||||||
assert.ErrorIs(t, r.w.Submit(r.spool(t, "c")), uploads.ErrStopped)
|
assert.ErrorIs(t, r.w.Submit(r.spool(t, "c")), uploads.ErrStopped)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// dbErr is an error shaped like pgx's: a message that quotes a value, and an SQLSTATE.
|
||||||
|
type dbErr struct{}
|
||||||
|
|
||||||
|
func (dbErr) Error() string {
|
||||||
|
return `ERROR: invalid input syntax for type json: "zebrakaka secret words"`
|
||||||
|
}
|
||||||
|
func (dbErr) SQLState() string { return "22P02" }
|
||||||
|
|||||||
@@ -122,6 +122,11 @@ type App struct {
|
|||||||
// summarizer's (koala/phi4-mini, 8,192-token context) and cannot hold an hour of
|
// summarizer's (koala/phi4-mini, 8,192-token context) and cannot hold an hour of
|
||||||
// speech.
|
// speech.
|
||||||
UploadChatModel string
|
UploadChatModel string
|
||||||
|
// UploadChatModels is the set of models chat over an uploaded recording offers and
|
||||||
|
// accepts. The whole private transcript goes to the one chosen, so it must hold
|
||||||
|
// LOCAL models only, and the caller decides that. Nil or empty fails closed: no
|
||||||
|
// model is offered and none is called (ADR-035 review).
|
||||||
|
UploadChatModels []string
|
||||||
// Processing tracks in-flight immediate summarizations so the status endpoint
|
// Processing tracks in-flight immediate summarizations so the status endpoint
|
||||||
// shows the animation until the summary lands. The zero value is ready to use.
|
// shows the animation until the summary lands. The zero value is ready to use.
|
||||||
Processing ProcessingSet
|
Processing ProcessingSet
|
||||||
|
|||||||
+255
-21
@@ -8,7 +8,10 @@ import (
|
|||||||
"math"
|
"math"
|
||||||
"net/http"
|
"net/http"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
"strings"
|
"strings"
|
||||||
|
"time"
|
||||||
|
"unicode"
|
||||||
"unicode/utf8"
|
"unicode/utf8"
|
||||||
|
|
||||||
"github.com/a-h/templ"
|
"github.com/a-h/templ"
|
||||||
@@ -16,6 +19,7 @@ import (
|
|||||||
"git.d-ma.be/mathias/tapir/internal/adapters/chat"
|
"git.d-ma.be/mathias/tapir/internal/adapters/chat"
|
||||||
"git.d-ma.be/mathias/tapir/internal/adapters/store"
|
"git.d-ma.be/mathias/tapir/internal/adapters/store"
|
||||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/errkind"
|
||||||
"git.d-ma.be/mathias/tapir/internal/uploads"
|
"git.d-ma.be/mathias/tapir/internal/uploads"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -32,7 +36,15 @@ const (
|
|||||||
uploadOverhead = 1 << 20
|
uploadOverhead = 1 << 20
|
||||||
maxTitleRunes = 200
|
maxTitleRunes = 200
|
||||||
// uploadPollEvery is how often an unfinished upload re-asks for its status.
|
// uploadPollEvery is how often an unfinished upload re-asks for its status.
|
||||||
uploadPollEvery = "5s"
|
uploadPollEvery = "2s"
|
||||||
|
// serviceSilentAfter is how long a job may go without any signal (heartbeat or
|
||||||
|
// progress) before the page calls the service lost. The heartbeat is every 10 s, so
|
||||||
|
// 45 s is four missed beats (ADR-035).
|
||||||
|
serviceSilentAfter = 45 * time.Second
|
||||||
|
// countedQuietAfter is how long a COUNTED phase (the deciding one) may go without
|
||||||
|
// advancing before the page says it is quiet. Each decision has a two-minute timeout,
|
||||||
|
// so five minutes is more than two in a row. A phase with no count is never quiet.
|
||||||
|
countedQuietAfter = 5 * time.Minute
|
||||||
)
|
)
|
||||||
|
|
||||||
// uploadsView is what the /uploads page renders.
|
// uploadsView is what the /uploads page renders.
|
||||||
@@ -82,13 +94,32 @@ func (a *App) handleUploadPost(w http.ResponseWriter, r *http.Request) {
|
|||||||
break
|
break
|
||||||
}
|
}
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
var tooBig *http.MaxBytesError
|
||||||
|
if a.abandoned(w, r) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !errors.As(err, &tooBig) {
|
||||||
|
// A malformed multipart body (a header with a control byte, a truncated
|
||||||
|
// part) is the sender's to fix: a 400, not a server error, and the error
|
||||||
|
// text is never logged: it quotes the headers, which carry the file name.
|
||||||
|
a.logger().Info("upload rejected", "reason", "malformed")
|
||||||
|
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "That upload could not be read. Please try again.")
|
||||||
|
return
|
||||||
|
}
|
||||||
a.uploadFailure(w, r, userID, err)
|
a.uploadFailure(w, r, userID, err)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
switch part.FormName() {
|
switch part.FormName() {
|
||||||
case "title":
|
case "title":
|
||||||
b, _ := io.ReadAll(io.LimitReader(part, maxTitleRunes*utf8.UTFMax))
|
b, err := io.ReadAll(io.LimitReader(part, maxTitleRunes*utf8.UTFMax))
|
||||||
title = clipRunes(strings.TrimSpace(string(b)), maxTitleRunes)
|
if err != nil {
|
||||||
|
if a.abandoned(w, r) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "That upload could not be read. Please try again.")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
title = cleanTitle(string(b))
|
||||||
case "file":
|
case "file":
|
||||||
if title == "" {
|
if title == "" {
|
||||||
title = titleFromFilename(part.FileName())
|
title = titleFromFilename(part.FileName())
|
||||||
@@ -105,10 +136,27 @@ func (a *App) handleUploadPost(w http.ResponseWriter, r *http.Request) {
|
|||||||
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "Choose an audio file to upload.")
|
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "Choose an audio file to upload.")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// abandoned reports, logs at INFO and answers 499, because the client has already gone: its request
|
||||||
|
// context is cancelled, so there is nobody to answer and rendering a page would run its
|
||||||
|
// database read on a dead context and log a false ERROR. A hang-up mid-upload is routine.
|
||||||
|
func (a *App) abandoned(w http.ResponseWriter, r *http.Request) bool {
|
||||||
|
if r.Context().Err() == nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
a.logger().Info("upload abandoned", "reason", "client_gone")
|
||||||
|
// 499 (the "client closed request" convention): a client that half-closed after sending
|
||||||
|
// can still read, and an unwritten status would reach it as a false, empty 200.
|
||||||
|
w.WriteHeader(499)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
// uploadFailure maps an Accept (or multipart) error to a response: a validation
|
// uploadFailure maps an Accept (or multipart) error to a response: a validation
|
||||||
// rejection is the user's to fix (422, or 413 for size) and carries its own
|
// rejection is the user's to fix (422, or 413 for size) and carries its own
|
||||||
// message; anything else is ours.
|
// message; anything else is ours.
|
||||||
func (a *App) uploadFailure(w http.ResponseWriter, r *http.Request, userID string, err error) {
|
func (a *App) uploadFailure(w http.ResponseWriter, r *http.Request, userID string, err error) {
|
||||||
|
if a.abandoned(w, r) {
|
||||||
|
return
|
||||||
|
}
|
||||||
var rej *uploads.RejectError
|
var rej *uploads.RejectError
|
||||||
var tooBig *http.MaxBytesError
|
var tooBig *http.MaxBytesError
|
||||||
switch {
|
switch {
|
||||||
@@ -116,16 +164,26 @@ func (a *App) uploadFailure(w http.ResponseWriter, r *http.Request, userID strin
|
|||||||
// Logged by reason only: the filename and title are private content.
|
// Logged by reason only: the filename and title are private content.
|
||||||
a.logger().Info("upload rejected", "reason", string(rej.Reason))
|
a.logger().Info("upload rejected", "reason", string(rej.Reason))
|
||||||
status := http.StatusUnprocessableEntity
|
status := http.StatusUnprocessableEntity
|
||||||
if rej.Reason == uploads.ReasonTooLarge {
|
switch rej.Reason {
|
||||||
|
case uploads.ReasonTooLarge:
|
||||||
status = http.StatusRequestEntityTooLarge
|
status = http.StatusRequestEntityTooLarge
|
||||||
|
case uploads.ReasonBusy:
|
||||||
|
status = http.StatusTooManyRequests
|
||||||
}
|
}
|
||||||
a.renderUploadsPage(w, r, userID, status, rej.Message)
|
a.renderUploadsPage(w, r, userID, status, rej.Message)
|
||||||
case errors.As(err, &tooBig):
|
case errors.As(err, &tooBig):
|
||||||
a.logger().Info("upload rejected", "reason", string(uploads.ReasonTooLarge))
|
a.logger().Info("upload rejected", "reason", string(uploads.ReasonTooLarge))
|
||||||
a.renderUploadsPage(w, r, userID, http.StatusRequestEntityTooLarge,
|
a.renderUploadsPage(w, r, userID, http.StatusRequestEntityTooLarge,
|
||||||
"This file is larger than the upload limit.")
|
"This file is larger than the upload limit.")
|
||||||
|
case errors.Is(err, uploads.ErrClientRead):
|
||||||
|
// An error reading the CLIENT's body: theirs to retry, not a server fault. Only that
|
||||||
|
// marker counts: io.ErrUnexpectedEOF alone is also what a database drop looks like.
|
||||||
|
a.logger().Info("upload rejected", "reason", "truncated")
|
||||||
|
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "The upload was cut off before it finished. Please try again.")
|
||||||
default:
|
default:
|
||||||
a.serverError(w, r, "upload", err)
|
// Not err: it can carry the file's name, a spool path or a database value.
|
||||||
|
a.logger().Error("upload failed", "kind", errkind.Of(err))
|
||||||
|
http.Error(w, "internal error", http.StatusInternalServerError)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -137,7 +195,7 @@ func (a *App) handleUploadDetail(w http.ResponseWriter, r *http.Request) {
|
|||||||
if !ok {
|
if !ok {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
a.render(w, r, UploadPage(*u, a.Chat != nil))
|
a.render(w, r, UploadPage(*u, a.Chat != nil, a.now()))
|
||||||
}
|
}
|
||||||
|
|
||||||
// handleUploadStatus renders the status fragment HTMX polls. Unfinished states
|
// handleUploadStatus renders the status fragment HTMX polls. Unfinished states
|
||||||
@@ -148,7 +206,7 @@ func (a *App) handleUploadStatus(w http.ResponseWriter, r *http.Request) {
|
|||||||
if !ok {
|
if !ok {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
a.render(w, r, uploadStatus(*u, a.Chat != nil))
|
a.render(w, r, uploadStatus(*u, a.Chat != nil, a.now()))
|
||||||
}
|
}
|
||||||
|
|
||||||
func (a *App) loadOwnedUpload(w http.ResponseWriter, r *http.Request) (*domain.UploadedTranscript, bool) {
|
func (a *App) loadOwnedUpload(w http.ResponseWriter, r *http.Request) (*domain.UploadedTranscript, bool) {
|
||||||
@@ -156,7 +214,14 @@ func (a *App) loadOwnedUpload(w http.ResponseWriter, r *http.Request) (*domain.U
|
|||||||
if !ok {
|
if !ok {
|
||||||
return nil, false
|
return nil, false
|
||||||
}
|
}
|
||||||
u, err := a.Store.GetUpload(r.Context(), userID, r.PathValue("id"))
|
id := r.PathValue("id")
|
||||||
|
if !uuidShape.MatchString(id) {
|
||||||
|
// Not an id at all: the same 404 as an unknown or foreign one, never a database
|
||||||
|
// error that surfaces as a 500 (and no oracle about which ids exist).
|
||||||
|
http.NotFound(w, r)
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
u, err := a.Store.GetUpload(r.Context(), userID, id)
|
||||||
if errors.Is(err, store.ErrUploadNotFound) {
|
if errors.Is(err, store.ErrUploadNotFound) {
|
||||||
http.NotFound(w, r)
|
http.NotFound(w, r)
|
||||||
return nil, false
|
return nil, false
|
||||||
@@ -171,11 +236,49 @@ func (a *App) loadOwnedUpload(w http.ResponseWriter, r *http.Request) (*domain.U
|
|||||||
// titleFromFilename is the default title: the file's name without its extension.
|
// titleFromFilename is the default title: the file's name without its extension.
|
||||||
func titleFromFilename(name string) string {
|
func titleFromFilename(name string) string {
|
||||||
base := filepath.Base(name)
|
base := filepath.Base(name)
|
||||||
t := strings.TrimSuffix(base, filepath.Ext(base))
|
t := cleanTitle(strings.TrimSuffix(base, filepath.Ext(base)))
|
||||||
if t == "" || t == "." {
|
if t == "" || t == "." {
|
||||||
return "Untitled recording"
|
return "Untitled recording"
|
||||||
}
|
}
|
||||||
return clipRunes(t, maxTitleRunes)
|
return t
|
||||||
|
}
|
||||||
|
|
||||||
|
// hasNote reports whether any phase carries a note. A note is the only warning for a
|
||||||
|
// recording that is shorter than its file claimed, so a record that has one is shown open.
|
||||||
|
func hasNote(steps []domain.ProgressStep) bool {
|
||||||
|
for _, s := range steps {
|
||||||
|
if s.Note != "" {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// uuidShape is the form of an upload id. Anything else cannot be one, so it is a 404.
|
||||||
|
var uuidShape = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
|
||||||
|
|
||||||
|
// isBidiControl reports the invisible characters that change text direction. Emoji joiners
|
||||||
|
// (ZWJ, ZWNJ) and variation selectors are not among them and are kept.
|
||||||
|
func isBidiControl(r rune) bool {
|
||||||
|
return r >= 0x202A && r <= 0x202E || r >= 0x2066 && r <= 0x2069 || r == 0x200E || r == 0x200F || r == 0x061C
|
||||||
|
}
|
||||||
|
|
||||||
|
// cleanTitle makes a title safe to store and show: invalid UTF-8 dropped, control
|
||||||
|
// characters (NUL, newlines, escapes) turned into spaces, runs of space collapsed, trimmed
|
||||||
|
// and clipped. A title is cosmetic; bad bytes in it must never fail an upload, and Postgres
|
||||||
|
// refuses NUL in text.
|
||||||
|
func cleanTitle(s string) string {
|
||||||
|
s = strings.ToValidUTF8(s, "")
|
||||||
|
s = strings.Map(func(r rune) rune {
|
||||||
|
switch {
|
||||||
|
case isBidiControl(r):
|
||||||
|
return -1 // dropped: U+202E would reverse the rest of the title on its owner's page
|
||||||
|
case unicode.IsControl(r):
|
||||||
|
return ' '
|
||||||
|
}
|
||||||
|
return r
|
||||||
|
}, s)
|
||||||
|
return clipRunes(strings.Join(strings.Fields(s), " "), maxTitleRunes)
|
||||||
}
|
}
|
||||||
|
|
||||||
func clipRunes(s string, n int) string {
|
func clipRunes(s string, n int) string {
|
||||||
@@ -191,6 +294,10 @@ func uploadFailureMessage(k domain.UploadErrorKind) string {
|
|||||||
switch k {
|
switch k {
|
||||||
case domain.UploadErrInterrupted:
|
case domain.UploadErrInterrupted:
|
||||||
return "The server restarted before this recording was transcribed. Please upload it again."
|
return "The server restarted before this recording was transcribed. Please upload it again."
|
||||||
|
case domain.UploadErrBusy:
|
||||||
|
return "Too many recordings were waiting to be transcribed when this one was uploaded. Please upload it again in a few minutes."
|
||||||
|
case domain.UploadErrTooLong:
|
||||||
|
return "This recording is longer than the limit once its audio is read (the file's header said otherwise). Please upload a shorter recording."
|
||||||
case domain.UploadErrTranscriptionFailed:
|
case domain.UploadErrTranscriptionFailed:
|
||||||
return "Transcription failed on every available service. Please upload it again later."
|
return "Transcription failed on every available service. Please upload it again later."
|
||||||
default:
|
default:
|
||||||
@@ -211,7 +318,8 @@ func (a *App) handleUploadChat(w http.ResponseWriter, r *http.Request) {
|
|||||||
http.Redirect(w, r, "/uploads/"+u.ID, http.StatusSeeOther)
|
http.Redirect(w, r, "/uploads/"+u.ID, http.StatusSeeOther)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
a.render(w, r, UploadChatPage(*u, a.uploadChatView(*u, a.Chat.DefaultModel(a.UploadChatModel), nil, false, "")))
|
model, _ := a.resolveUploadModel("")
|
||||||
|
a.render(w, r, UploadChatPage(*u, a.uploadChatView(*u, model, nil, false, "")))
|
||||||
}
|
}
|
||||||
|
|
||||||
// handleUploadChatMessage answers one question against the owner's uploaded
|
// handleUploadChatMessage answers one question against the owner's uploaded
|
||||||
@@ -231,7 +339,13 @@ func (a *App) handleUploadChatMessage(w http.ResponseWriter, r *http.Request) {
|
|||||||
http.Error(w, "bad form", http.StatusBadRequest)
|
http.Error(w, "bad form", http.StatusBadRequest)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
model := a.resolveModel(r.FormValue("model"), a.UploadChatModel)
|
model, ok := a.resolveUploadModel(r.FormValue("model"))
|
||||||
|
if !ok {
|
||||||
|
a.logger().Error("upload chat: no local model is configured; nothing was sent", "upload", u.ID)
|
||||||
|
view := a.uploadChatView(*u, "", nil, false, "No local model is available for chat about a recording.")
|
||||||
|
a.render(w, r, UploadChatPage(*u, view))
|
||||||
|
return
|
||||||
|
}
|
||||||
history := parseHistory(r.Form["hq"], r.Form["ha"])
|
history := parseHistory(r.Form["hq"], r.Form["ha"])
|
||||||
question := strings.TrimSpace(r.FormValue("question"))
|
question := strings.TrimSpace(r.FormValue("question"))
|
||||||
|
|
||||||
@@ -245,7 +359,7 @@ func (a *App) handleUploadChatMessage(w http.ResponseWriter, r *http.Request) {
|
|||||||
MaxChars: a.UploadChatMaxChars,
|
MaxChars: a.UploadChatMaxChars,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
a.logger().Error("upload chat answer", "upload", u.ID, "model", model, "err", err)
|
a.logger().Error("upload chat answer", "upload", u.ID, "model", model, "kind", errkind.Of(err))
|
||||||
view.Error = "That model couldn't answer just now. Try again, or switch models."
|
view.Error = "That model couldn't answer just now. Try again, or switch models."
|
||||||
} else {
|
} else {
|
||||||
view.History = appendTurn(history, chat.Turn{Question: question, Answer: reply.Answer})
|
view.History = appendTurn(history, chat.Turn{Question: question, Answer: reply.Answer})
|
||||||
@@ -264,7 +378,7 @@ func (a *App) uploadChatView(u domain.UploadedTranscript, model string, history
|
|||||||
VideoID: u.ID,
|
VideoID: u.ID,
|
||||||
Title: u.Title,
|
Title: u.Title,
|
||||||
Available: true,
|
Available: true,
|
||||||
Models: a.Chat.Models(),
|
Models: a.UploadChatModels,
|
||||||
Selected: model,
|
Selected: model,
|
||||||
History: history,
|
History: history,
|
||||||
Truncated: truncated,
|
Truncated: truncated,
|
||||||
@@ -274,9 +388,9 @@ func (a *App) uploadChatView(u domain.UploadedTranscript, model string, history
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// maxSpotsShown is how many contested spots the page lists. The stored list holds up to
|
// maxSpotsShown is how many spots to check the page lists. The stored list holds every
|
||||||
// 200 and the counts are exact; a page of two hundred "check this" items is not a page
|
// judged spot and the counts are exact; a page of two hundred "check this" items is not a
|
||||||
// anyone reads, so the rest are counted, not listed.
|
// page anyone reads, so the rest are counted, not listed.
|
||||||
const maxSpotsShown = 50
|
const maxSpotsShown = 50
|
||||||
|
|
||||||
// minuteMark renders where in the recording a spot is, as the nearest minute.
|
// minuteMark renders where in the recording a spot is, as the nearest minute.
|
||||||
@@ -284,15 +398,135 @@ func minuteMark(seconds float64) string {
|
|||||||
return fmt.Sprintf("~%d min", int(math.Round(seconds/60)))
|
return fmt.Sprintf("~%d min", int(math.Round(seconds/60)))
|
||||||
}
|
}
|
||||||
|
|
||||||
// shownSpots is the part of the contested list the page renders.
|
// shownSpots is the part of the stored list the page renders: the spots a reader should
|
||||||
|
// check, changed or uncertain, in order, capped. A spot the model confidently kept is
|
||||||
|
// stored (ADR-035) but is not one to check, so it is not listed.
|
||||||
func shownSpots(r *domain.Reconciliation) []domain.ContestedSpot {
|
func shownSpots(r *domain.Reconciliation) []domain.ContestedSpot {
|
||||||
if len(r.Contested) > maxSpotsShown {
|
var out []domain.ContestedSpot
|
||||||
return r.Contested[:maxSpotsShown]
|
for _, c := range r.Contested {
|
||||||
|
if c.Used == "alt" || c.Unsure {
|
||||||
|
out = append(out, c)
|
||||||
|
if len(out) == maxSpotsShown {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return r.Contested
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
// moreSpots is how many spots to check are not listed, from the exact counts.
|
// moreSpots is how many spots to check are not listed, from the exact counts.
|
||||||
func moreSpots(r *domain.Reconciliation) int {
|
func moreSpots(r *domain.Reconciliation) int {
|
||||||
return max(0, r.Changed+r.Unsure-len(shownSpots(r)))
|
return max(0, r.Changed+r.Unsure-len(shownSpots(r)))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// jobHealth is what the page can honestly say about a job it is still polling.
|
||||||
|
type jobHealth string
|
||||||
|
|
||||||
|
const (
|
||||||
|
// healthRunning: the service is reporting and work is moving, or there is nothing yet
|
||||||
|
// to say otherwise.
|
||||||
|
healthRunning jobHealth = "running"
|
||||||
|
// healthQuiet: the service is alive but a counted phase has not advanced for a while.
|
||||||
|
healthQuiet jobHealth = "quiet"
|
||||||
|
// healthLost: nothing has been heard from the service for too long. It may have
|
||||||
|
// restarted or crashed; the start-up sweep fails the row once it is back.
|
||||||
|
healthLost jobHealth = "lost"
|
||||||
|
)
|
||||||
|
|
||||||
|
// uploadHealth tells running from quiet from lost for an upload being transcribed (the
|
||||||
|
// page calls it for no other state: a queued or finished job has no heartbeat to judge). The
|
||||||
|
// two timestamps mean different things (ADR-035): alive_at is the heartbeat, which says
|
||||||
|
// the service is still reporting; progress_at moves only when work really moved. A
|
||||||
|
// single blocking engine call has no sub-progress, so only a counted phase can be quiet.
|
||||||
|
// silent is how long since the signal that decided it.
|
||||||
|
func uploadHealth(u domain.UploadedTranscript, now time.Time) (health jobHealth, silent time.Duration) {
|
||||||
|
last := u.AliveAt
|
||||||
|
if u.ProgressAt.After(last) {
|
||||||
|
last = u.ProgressAt
|
||||||
|
}
|
||||||
|
if last.IsZero() {
|
||||||
|
return healthRunning, 0
|
||||||
|
}
|
||||||
|
if d := now.Sub(last); d > serviceSilentAfter {
|
||||||
|
return healthLost, d
|
||||||
|
}
|
||||||
|
for _, s := range u.Progress {
|
||||||
|
if s.State == domain.StepRunning && s.Total > 0 && !u.ProgressAt.IsZero() {
|
||||||
|
if d := now.Sub(u.ProgressAt); d > countedQuietAfter {
|
||||||
|
return healthQuiet, d
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return healthRunning, 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// stepDuration renders how long a phase took or has run: "<1s", "42s", "3m 12s", "1h 02m".
|
||||||
|
func stepDuration(d time.Duration) string {
|
||||||
|
switch {
|
||||||
|
case d < time.Second:
|
||||||
|
return "<1s"
|
||||||
|
case d < time.Minute:
|
||||||
|
return fmt.Sprintf("%ds", int(d.Seconds()))
|
||||||
|
case d < time.Hour:
|
||||||
|
return fmt.Sprintf("%dm %02ds", int(d.Minutes()), int(d.Seconds())%60)
|
||||||
|
default:
|
||||||
|
return fmt.Sprintf("%dh %02dm", int(d.Hours()), int(d.Minutes())%60)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// stepTime is the time shown beside a phase: how long it has run, or how long it took.
|
||||||
|
func stepTime(s domain.ProgressStep, now time.Time) string {
|
||||||
|
switch s.State {
|
||||||
|
case domain.StepRunning:
|
||||||
|
return stepDuration(now.Sub(s.StartedAt))
|
||||||
|
case domain.StepDone, domain.StepFailed:
|
||||||
|
return stepDuration(s.FinishedAt.Sub(s.StartedAt))
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// stepMark is the glyph at the head of a phase line.
|
||||||
|
func stepMark(s domain.ProgressStep) string {
|
||||||
|
switch s.State {
|
||||||
|
case domain.StepDone:
|
||||||
|
return "✓"
|
||||||
|
case domain.StepFailed:
|
||||||
|
return "✗"
|
||||||
|
case domain.StepSkipped:
|
||||||
|
return "–"
|
||||||
|
}
|
||||||
|
return "·"
|
||||||
|
}
|
||||||
|
|
||||||
|
// minutesAgo is a whole number of minutes for the health notes: "6 min", or "under a
|
||||||
|
// minute" when it is less.
|
||||||
|
func minutesAgo(d time.Duration) string {
|
||||||
|
if m := int(d.Minutes()); m >= 1 {
|
||||||
|
return fmt.Sprintf("%d min", m)
|
||||||
|
}
|
||||||
|
return "under a minute"
|
||||||
|
}
|
||||||
|
|
||||||
|
// resolveUploadModel picks the model for chat over an uploaded recording from the
|
||||||
|
// offered LOCAL set only: the posted one when it is in the set, else the configured
|
||||||
|
// default when it is, else the first. ok is false when no model is offered, in which
|
||||||
|
// case nothing may be sent anywhere (the whole private transcript is the payload).
|
||||||
|
func (a *App) resolveUploadModel(posted string) (model string, ok bool) {
|
||||||
|
offered := func(m string) bool {
|
||||||
|
for _, o := range a.UploadChatModels {
|
||||||
|
if o == m {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
switch {
|
||||||
|
case posted != "" && offered(posted):
|
||||||
|
return posted, true
|
||||||
|
case a.UploadChatModel != "" && offered(a.UploadChatModel):
|
||||||
|
return a.UploadChatModel, true
|
||||||
|
case len(a.UploadChatModels) > 0:
|
||||||
|
return a.UploadChatModels[0], true
|
||||||
|
}
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
|||||||
@@ -45,11 +45,11 @@ templ UploadsPage(v uploadsView) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// UploadPage is one upload's page: its title and the polling status region.
|
// UploadPage is one upload's page: its title and the polling status region.
|
||||||
templ UploadPage(u domain.UploadedTranscript, chatEnabled bool) {
|
templ UploadPage(u domain.UploadedTranscript, chatEnabled bool, now time.Time) {
|
||||||
@Layout("Tapir — " + u.Title) {
|
@Layout("Tapir — " + u.Title) {
|
||||||
<article class="account">
|
<article class="account">
|
||||||
<h1>{ u.Title }</h1>
|
<h1>{ u.Title }</h1>
|
||||||
@uploadStatus(u, chatEnabled)
|
@uploadStatus(u, chatEnabled, now)
|
||||||
<p><a href="/uploads">All recordings</a></p>
|
<p><a href="/uploads">All recordings</a></p>
|
||||||
</article>
|
</article>
|
||||||
}
|
}
|
||||||
@@ -57,7 +57,7 @@ templ UploadPage(u domain.UploadedTranscript, chatEnabled bool) {
|
|||||||
|
|
||||||
// uploadStatus is the region HTMX polls. Unfinished states carry hx-trigger so the
|
// uploadStatus is the region HTMX polls. Unfinished states carry hx-trigger so the
|
||||||
// page re-asks; done and failed render without it, so nothing polls forever.
|
// page re-asks; done and failed render without it, so nothing polls forever.
|
||||||
templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool) {
|
templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time) {
|
||||||
switch u.Status {
|
switch u.Status {
|
||||||
case domain.UploadDone:
|
case domain.UploadDone:
|
||||||
<div class="upload-status upload-done" role="status">
|
<div class="upload-status upload-done" role="status">
|
||||||
@@ -67,6 +67,12 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool) {
|
|||||||
of audio by <span class="chip">{ u.Endpoint }</span>.
|
of audio by <span class="chip">{ u.Endpoint }</span>.
|
||||||
</p>
|
</p>
|
||||||
@reconciliationNote(u.Reconciliation)
|
@reconciliationNote(u.Reconciliation)
|
||||||
|
if len(u.Progress) > 0 {
|
||||||
|
<details class="what-was-done" if hasNote(u.Progress) { open }>
|
||||||
|
<summary>What was done</summary>
|
||||||
|
@progressSteps(u.Progress, now, false)
|
||||||
|
</details>
|
||||||
|
}
|
||||||
if chatEnabled {
|
if chatEnabled {
|
||||||
<p><a class="btn" href={ templ.SafeURL("/uploads/" + u.ID + "/chat") }>Ask about this recording</a></p>
|
<p><a class="btn" href={ templ.SafeURL("/uploads/" + u.ID + "/chat") }>Ask about this recording</a></p>
|
||||||
}
|
}
|
||||||
@@ -79,6 +85,12 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool) {
|
|||||||
<div class="upload-status upload-failed flash flash-error" role="alert">
|
<div class="upload-status upload-failed flash flash-error" role="alert">
|
||||||
@videoNotice(u.HadVideo)
|
@videoNotice(u.HadVideo)
|
||||||
<p>{ uploadFailureMessage(u.ErrorKind) }</p>
|
<p>{ uploadFailureMessage(u.ErrorKind) }</p>
|
||||||
|
if len(u.Progress) > 0 {
|
||||||
|
<details class="what-was-done" open>
|
||||||
|
<summary>Where it stopped</summary>
|
||||||
|
@progressSteps(u.Progress, now, false)
|
||||||
|
</details>
|
||||||
|
}
|
||||||
</div>
|
</div>
|
||||||
default:
|
default:
|
||||||
<div
|
<div
|
||||||
@@ -90,7 +102,13 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool) {
|
|||||||
>
|
>
|
||||||
@videoNotice(u.HadVideo)
|
@videoNotice(u.HadVideo)
|
||||||
if u.Status == domain.UploadTranscribing {
|
if u.Status == domain.UploadTranscribing {
|
||||||
<p>Transcribing… a long recording takes a few minutes.</p>
|
{{ health, silent := uploadHealth(u, now) }}
|
||||||
|
if len(u.Progress) > 0 {
|
||||||
|
@progressSteps(u.Progress, now, health != healthLost)
|
||||||
|
@healthNote(health, silent)
|
||||||
|
} else {
|
||||||
|
<p>Transcribing… a long recording takes a few minutes.</p>
|
||||||
|
}
|
||||||
} else {
|
} else {
|
||||||
<p>Queued — waiting for its turn to be transcribed.</p>
|
<p>Queued — waiting for its turn to be transcribed.</p>
|
||||||
}
|
}
|
||||||
@@ -142,13 +160,17 @@ templ reconciliationNote(r *domain.Reconciliation) {
|
|||||||
{ strconv.Itoa(r.Adjudicated) } decided by the model ({ r.Reconciler }),
|
{ strconv.Itoa(r.Adjudicated) } decided by the model ({ r.Reconciler }),
|
||||||
{ strconv.Itoa(r.Unsure) } uncertain.
|
{ strconv.Itoa(r.Unsure) } uncertain.
|
||||||
</p>
|
</p>
|
||||||
if len(r.Contested) > 0 {
|
if len(shownSpots(r)) > 0 {
|
||||||
<details>
|
<details>
|
||||||
<summary>Spots to check against the audio</summary>
|
<summary>Spots to check against the audio</summary>
|
||||||
<ul>
|
<ul>
|
||||||
for _, c := range shownSpots(r) {
|
for _, c := range shownSpots(r) {
|
||||||
<li>
|
<li>
|
||||||
{ minuteMark(c.Seconds) }: { c.Base } / { c.Alt } —
|
{ minuteMark(c.Seconds) }:
|
||||||
|
if c.Before != "" {
|
||||||
|
<span class="muted">…{ c.Before }</span>
|
||||||
|
}
|
||||||
|
{ c.Base } / { c.Alt } —
|
||||||
if c.Unsure {
|
if c.Unsure {
|
||||||
<span class="chip chip-warn">uncertain</span> kept { c.Used }
|
<span class="chip chip-warn">uncertain</span> kept { c.Used }
|
||||||
} else {
|
} else {
|
||||||
@@ -170,3 +192,57 @@ templ reconciliationNote(r *domain.Reconciliation) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// progressSteps is the job's phases as a log that fills in as it runs: finished ones
|
||||||
|
// collapse to a line with how long they took, the running one has a spinner and its
|
||||||
|
// elapsed time, and what has not started is dimmed. live is false when the spinner would
|
||||||
|
// be a lie (the service is lost, or the job is over), so the running phase shows a
|
||||||
|
// warning mark instead. Only a counted phase gets a bar (ADR-035).
|
||||||
|
templ progressSteps(steps []domain.ProgressStep, now time.Time, live bool) {
|
||||||
|
<ol class="steps" aria-label="Progress">
|
||||||
|
for _, s := range steps {
|
||||||
|
<li
|
||||||
|
class={ "step step-" + string(s.State) }
|
||||||
|
if s.State == domain.StepRunning {
|
||||||
|
aria-current="step"
|
||||||
|
}
|
||||||
|
>
|
||||||
|
if s.State == domain.StepRunning && live {
|
||||||
|
<span class="step-mark" aria-hidden="true"><span class="spinner"></span></span>
|
||||||
|
} else if s.State == domain.StepRunning {
|
||||||
|
<span class="step-mark step-mark-warn" aria-hidden="true">!</span>
|
||||||
|
} else {
|
||||||
|
<span class="step-mark" aria-hidden="true">{ stepMark(s) }</span>
|
||||||
|
}
|
||||||
|
<span class="step-label">{ s.Label }</span>
|
||||||
|
if t := stepTime(s, now); t != "" {
|
||||||
|
<span class="step-time">{ t }</span>
|
||||||
|
}
|
||||||
|
if s.Note != "" {
|
||||||
|
<span class="step-note">{ s.Note }</span>
|
||||||
|
}
|
||||||
|
if s.State == domain.StepRunning && s.Total > 0 {
|
||||||
|
<progress class="step-bar" value={ strconv.Itoa(s.Done) } max={ strconv.Itoa(s.Total) }></progress>
|
||||||
|
<span class="step-count">{ strconv.Itoa(s.Done) } of { strconv.Itoa(s.Total) }</span>
|
||||||
|
}
|
||||||
|
</li>
|
||||||
|
}
|
||||||
|
</ol>
|
||||||
|
}
|
||||||
|
|
||||||
|
// healthNote says so when the job is not simply running: the service stopped reporting,
|
||||||
|
// or a counted phase has gone quiet. Running says nothing: the spinner does.
|
||||||
|
templ healthNote(h jobHealth, silent time.Duration) {
|
||||||
|
switch h {
|
||||||
|
case healthLost:
|
||||||
|
<p class="step-health step-health-lost" role="alert">
|
||||||
|
The service stopped reporting { minutesAgo(silent) } ago. It may have restarted or crashed.
|
||||||
|
This page keeps checking; if nothing changes, upload the recording again.
|
||||||
|
</p>
|
||||||
|
case healthQuiet:
|
||||||
|
<p class="step-health step-health-quiet">
|
||||||
|
Nothing new for { minutesAgo(silent) }: the model may be slow. The service is still running,
|
||||||
|
and the job gives up on its own if it takes too long.
|
||||||
|
</p>
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,93 @@
|
|||||||
|
package web_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Input that cannot be right is the caller's to fix: a 4xx, never a 500 (adversarial
|
||||||
|
// review F7). No cross-tenant oracle either way: unknown and foreign ids look the same.
|
||||||
|
|
||||||
|
func TestUploadRoutes_ANonUUIDIdIs404NotA500(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
for _, path := range []string{"/uploads/not-a-uuid", "/uploads/not-a-uuid/status", "/uploads/not-a-uuid/chat", "/uploads/%00/status", "/uploads/1/status"} {
|
||||||
|
t.Run(path, func(t *testing.T) {
|
||||||
|
rec := do(t, app, get(path))
|
||||||
|
|
||||||
|
assert.Equal(t, http.StatusNotFound, rec.Code)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadPost_ATitleWithControlCharactersOrInvalidUTF8IsCleanedNotA500(t *testing.T) {
|
||||||
|
tests := []struct{ name, title, filename string }{
|
||||||
|
{"a NUL in the title", "Sprint\x00review", "a.m4a"},
|
||||||
|
{"invalid UTF-8 in the title", "Sprint \xff\xfe review", "a.m4a"},
|
||||||
|
{"a newline in the title", "Sprint\nreview", "a.m4a"},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
app, sub := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
|
||||||
|
rec := do(t, app, uploadReq(t, tt.title, tt.filename, []byte("synthetic audio bytes")))
|
||||||
|
|
||||||
|
require.Equal(t, http.StatusSeeOther, rec.Code, "bad bytes in a cosmetic title must not fail the upload: %s", body(t, rec))
|
||||||
|
require.Len(t, sub.jobs, 1)
|
||||||
|
var title string
|
||||||
|
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT title FROM uploaded_transcripts WHERE id = $1`, sub.jobs[0].UploadID).Scan(&title))
|
||||||
|
assert.NotContains(t, title, "\x00")
|
||||||
|
assert.NotContains(t, title, "\n")
|
||||||
|
assert.True(t, len(title) > 0, "a title is still shown")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadPost_AMalformedMultipartHeaderIsA400AndItsTextIsNotLogged(t *testing.T) {
|
||||||
|
// A control byte in the part's file name makes the multipart body malformed. It is the
|
||||||
|
// sender's to fix, and the parser's error quotes the header, which carries the name.
|
||||||
|
app, sub := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
buf := &lockedBuf{}
|
||||||
|
app.Log = slog.New(slog.NewTextHandler(buf, nil))
|
||||||
|
|
||||||
|
rec := do(t, app, uploadReq(t, "", "zebrakaka\x00weekly.m4a", []byte("x")))
|
||||||
|
|
||||||
|
assert.Equal(t, http.StatusBadRequest, rec.Code)
|
||||||
|
assert.Empty(t, sub.jobs)
|
||||||
|
assert.NotContains(t, buf.String(), "zebrakaka", "a file name is private content")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadPost_EveryBidirectionalControlCharacterIsStrippedFromATitle(t *testing.T) {
|
||||||
|
// U+202E reverses the text that follows it, so a title could render reversed on its
|
||||||
|
// owner's own page (second adversarial review). Every one of the twelve is pinned.
|
||||||
|
controls := []rune{0x202A, 0x202B, 0x202C, 0x202D, 0x202E, 0x2066, 0x2067, 0x2068, 0x2069, 0x200E, 0x200F, 0x061C}
|
||||||
|
for _, c := range controls {
|
||||||
|
t.Run(fmt.Sprintf("U+%04X", c), func(t *testing.T) {
|
||||||
|
app, sub := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
|
||||||
|
rec := do(t, app, uploadReq(t, "Sprint"+string(c)+"review", "a.m4a", []byte("x")))
|
||||||
|
|
||||||
|
require.Equal(t, http.StatusSeeOther, rec.Code)
|
||||||
|
var title string
|
||||||
|
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT title FROM uploaded_transcripts WHERE id = $1`, sub.jobs[0].UploadID).Scan(&title))
|
||||||
|
assert.Equal(t, "Sprintreview", title, "dropped, not replaced by a space: a space would leave a visible gap")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadPost_SwedishLettersAndEmojiSurviveTheTitleCleaning(t *testing.T) {
|
||||||
|
app, sub := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
|
||||||
|
// Includes a ZWJ sequence (woman + ZWJ + laptop) and a ZWNJ: joiners are not direction controls.
|
||||||
|
rec := do(t, app, uploadReq(t, "Möte åäö 🎙️ 👩\u200d💻 a\u200cb", "a.m4a", []byte("x")))
|
||||||
|
|
||||||
|
require.Equal(t, http.StatusSeeOther, rec.Code)
|
||||||
|
var title string
|
||||||
|
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT title FROM uploaded_transcripts WHERE id = $1`, sub.jobs[0].UploadID).Scan(&title))
|
||||||
|
assert.Equal(t, "Möte åäö 🎙️ 👩\u200d💻 a\u200cb", title, "joiners and variation selectors belong to the text")
|
||||||
|
}
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
package web_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/uploads"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestUploadPost_ABusyRejectionIs429WithItsMessage(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
app.Uploads = failingUploader{err: &uploads.RejectError{Reason: uploads.ReasonBusy, Message: "Too many recordings are waiting. Try again in a few minutes."}}
|
||||||
|
|
||||||
|
rec := do(t, app, uploadReq(t, "t", "a.m4a", []byte("x")))
|
||||||
|
|
||||||
|
require.Equal(t, http.StatusTooManyRequests, rec.Code, "the user's to retry: not a 500, not a 422")
|
||||||
|
assert.Contains(t, body(t, rec), "Too many recordings are waiting")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadStatus_ABusyFailureIsExplainedInWords(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
id := seedUpload(t, userID, "m", "failed", "queue_full")
|
||||||
|
|
||||||
|
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||||
|
|
||||||
|
assert.Contains(t, b, "waiting")
|
||||||
|
assert.NotContains(t, b, "This upload failed. Please upload it again.")
|
||||||
|
}
|
||||||
@@ -31,6 +31,7 @@ func newUploadChatApp(t *testing.T) (*web.App, *fakeChatter) {
|
|||||||
}
|
}
|
||||||
app.Chat = fc
|
app.Chat = fc
|
||||||
app.UploadChatMaxChars = 120000
|
app.UploadChatMaxChars = 120000
|
||||||
|
app.UploadChatModels = fc.models
|
||||||
return app, fc
|
return app, fc
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,73 @@
|
|||||||
|
package web_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"net/url"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/adapters/chat"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/web"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The whole private transcript goes to the model chosen in the form, so the upload chat
|
||||||
|
// offers and accepts local models only, even when the shared chat service knows a cloud
|
||||||
|
// one (ADR-035 review). The cloud model here is only a name; no network is involved.
|
||||||
|
|
||||||
|
func newUploadChatAppWithCloud(t *testing.T) (*web.App, *fakeChatter) {
|
||||||
|
t.Helper()
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
fc := &fakeChatter{
|
||||||
|
models: []string{"koala/phi4-mini", "iguana/gemma4-26b", "berget/mistral-small"},
|
||||||
|
reply: chat.Reply{Answer: "Anders owns the timber order."},
|
||||||
|
}
|
||||||
|
app.Chat = fc
|
||||||
|
app.UploadChatMaxChars = 120000
|
||||||
|
app.UploadChatModels = []string{"koala/phi4-mini", "iguana/gemma4-26b"}
|
||||||
|
return app, fc
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadChatPage_DoesNotOfferACloudModelEvenThoughTheServiceKnowsOne(t *testing.T) {
|
||||||
|
app, _ := newUploadChatAppWithCloud(t)
|
||||||
|
id := seedUpload(t, userID, "Roadmap sync", "done", meetingText)
|
||||||
|
|
||||||
|
b := body(t, do(t, app, get("/uploads/"+id+"/chat")))
|
||||||
|
|
||||||
|
assert.Contains(t, b, "iguana/gemma4-26b")
|
||||||
|
assert.NotContains(t, b, "berget/mistral-small", "the switcher must not list a model the transcript must not be sent to")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadChatPost_AForgedCloudModelIsReplacedByALocalOneBeforeTheTranscriptIsSent(t *testing.T) {
|
||||||
|
app, fc := newUploadChatAppWithCloud(t)
|
||||||
|
id := seedUpload(t, userID, "Roadmap sync", "done", meetingText)
|
||||||
|
|
||||||
|
rec := do(t, app, chatPost("/uploads/"+id+"/chat", url.Values{"question": {"Who owns timber?"}, "model": {"berget/mistral-small"}}))
|
||||||
|
|
||||||
|
require.Equal(t, http.StatusOK, rec.Code)
|
||||||
|
require.Equal(t, 1, fc.callCount)
|
||||||
|
assert.NotEqual(t, "berget/mistral-small", fc.gotReq.Model, "a cloud model reached the chat service with the private transcript")
|
||||||
|
assert.Contains(t, []string{"koala/phi4-mini", "iguana/gemma4-26b"}, fc.gotReq.Model)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadChatPost_ALocalModelThatIsOfferedIsHonoured(t *testing.T) {
|
||||||
|
app, fc := newUploadChatAppWithCloud(t)
|
||||||
|
id := seedUpload(t, userID, "Roadmap sync", "done", meetingText)
|
||||||
|
|
||||||
|
rec := do(t, app, chatPost("/uploads/"+id+"/chat", url.Values{"question": {"q"}, "model": {"koala/phi4-mini"}}))
|
||||||
|
|
||||||
|
require.Equal(t, http.StatusOK, rec.Code)
|
||||||
|
assert.Equal(t, "koala/phi4-mini", fc.gotReq.Model)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadChat_WithNoLocalModelConfiguredNothingIsSentAnywhere(t *testing.T) {
|
||||||
|
app, fc := newUploadChatAppWithCloud(t)
|
||||||
|
app.UploadChatModels = nil
|
||||||
|
id := seedUpload(t, userID, "Roadmap sync", "done", meetingText)
|
||||||
|
|
||||||
|
rec := do(t, app, chatPost("/uploads/"+id+"/chat", url.Values{"question": {"q"}, "model": {"berget/mistral-small"}}))
|
||||||
|
|
||||||
|
require.Equal(t, http.StatusOK, rec.Code)
|
||||||
|
assert.Zero(t, fc.callCount, "failing closed: no local model offered means no model is called")
|
||||||
|
}
|
||||||
@@ -0,0 +1,124 @@
|
|||||||
|
package web_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
// A client that hangs up in the middle of an upload is routine, not a server fault. The
|
||||||
|
// in-process recorder cannot see it (its context never cancels), so this talks to a real
|
||||||
|
// listener (third adversarial review: over a socket every such case logged an ERROR and
|
||||||
|
// half-close returned a 500, because the error page was rendered on a cancelled context).
|
||||||
|
|
||||||
|
// body positions at which the client can hang up: inside the title field, inside the file
|
||||||
|
// part's own headers, and mid-file. Every one must be handled alike (fourth adversarial
|
||||||
|
// review: the title branch had no guard and still logged an ERROR).
|
||||||
|
const (
|
||||||
|
inTitle = "in the title"
|
||||||
|
inHeaders = "in the file part's headers"
|
||||||
|
midFile = "mid-file"
|
||||||
|
)
|
||||||
|
|
||||||
|
func partialUpload(t *testing.T, addr, where string, hangUp func(*net.TCPConn)) {
|
||||||
|
t.Helper()
|
||||||
|
c, err := net.Dial("tcp", addr)
|
||||||
|
require.NoError(t, err)
|
||||||
|
tc := c.(*net.TCPConn)
|
||||||
|
const boundary = "xBOUNDARYx"
|
||||||
|
var body string
|
||||||
|
switch where {
|
||||||
|
case inTitle:
|
||||||
|
body = "--" + boundary + "\r\nContent-Disposition: form-data; name=\"title\"\r\n\r\nhalf a tit"
|
||||||
|
case inHeaders:
|
||||||
|
body = "--" + boundary + "\r\nContent-Disposition: form-data; name=\"file\"; filena"
|
||||||
|
default:
|
||||||
|
body = "--" + boundary + "\r\nContent-Disposition: form-data; name=\"file\"; filename=\"a.m4a\"\r\nContent-Type: audio/mp4\r\n\r\n" + strings.Repeat("a", 4000)
|
||||||
|
}
|
||||||
|
_, err = fmt.Fprintf(tc, "POST /uploads HTTP/1.1\r\nHost: x\r\nContent-Type: multipart/form-data; boundary=%s\r\nContent-Length: 200000\r\n\r\n%s", boundary, body)
|
||||||
|
require.NoError(t, err)
|
||||||
|
time.Sleep(50 * time.Millisecond) // let the server start reading
|
||||||
|
hangUp(tc)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadPost_AClientThatHangsUpAnywhereInTheBodyIsLoggedAtInfoAndNeverAsAnError(t *testing.T) {
|
||||||
|
hangUps := map[string]func(*net.TCPConn){
|
||||||
|
"a half close": func(c *net.TCPConn) { _ = c.CloseWrite(); time.Sleep(100 * time.Millisecond); _ = c.Close() },
|
||||||
|
"a full close": func(c *net.TCPConn) { _ = c.Close() },
|
||||||
|
"a reset (RST)": func(c *net.TCPConn) { _ = c.SetLinger(0); _ = c.Close() },
|
||||||
|
}
|
||||||
|
for _, where := range []string{inTitle, inHeaders, midFile} {
|
||||||
|
for name, hangUp := range hangUps {
|
||||||
|
t.Run(where+", "+name, func(t *testing.T) {
|
||||||
|
app, sub := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
buf := &lockedBuf{}
|
||||||
|
app.Log = slog.New(slog.NewTextHandler(buf, nil))
|
||||||
|
srv := httptest.NewServer(app.Router())
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
partialUpload(t, strings.TrimPrefix(srv.URL, "http://"), where, hangUp)
|
||||||
|
|
||||||
|
require.Eventually(t, func() bool {
|
||||||
|
out := buf.String()
|
||||||
|
return strings.Contains(out, "upload abandoned") || strings.Contains(out, "upload rejected") || strings.Contains(out, "level=ERROR")
|
||||||
|
}, 3*time.Second, 20*time.Millisecond, "the server never finished handling the aborted upload")
|
||||||
|
time.Sleep(100 * time.Millisecond)
|
||||||
|
out := buf.String()
|
||||||
|
assert.NotContains(t, out, "level=ERROR", "a client hanging up is not a server fault:\n%s", out)
|
||||||
|
assert.Empty(t, sub.jobs)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// goneMidAccept is an Uploader during whose Accept the client disappears: the request's
|
||||||
|
// context is cancelled, then Accept fails the way a read from a dead connection does.
|
||||||
|
type goneMidAccept struct{ cancel context.CancelFunc }
|
||||||
|
|
||||||
|
func (g goneMidAccept) Accept(context.Context, string, string, io.Reader) (string, error) {
|
||||||
|
g.cancel()
|
||||||
|
return "", fmt.Errorf("receive: %w", io.ErrClosedPipe)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadPost_AnAbandonedRequestGetsA499NotAnImplicitEmpty200(t *testing.T) {
|
||||||
|
// If the context is cancelled but the connection can still be written to (a client that
|
||||||
|
// half-closed after sending), a missing status becomes 200 with an empty body: a false
|
||||||
|
// success (fourth adversarial review).
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
buf := &lockedBuf{}
|
||||||
|
app.Log = slog.New(slog.NewTextHandler(buf, nil))
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
app.Uploads = goneMidAccept{cancel: cancel}
|
||||||
|
|
||||||
|
rec := do(t, app, uploadReq(t, "t", "a.m4a", []byte("x")).WithContext(ctx))
|
||||||
|
|
||||||
|
assert.Equal(t, 499, rec.Code, "never an implicit 200 for an upload that was not accepted")
|
||||||
|
assert.Contains(t, buf.String(), "upload abandoned")
|
||||||
|
assert.NotContains(t, buf.String(), "level=ERROR")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadPost_ADatabaseDropThatLooksLikeAnEOFIsStillAServerError(t *testing.T) {
|
||||||
|
// pgx turns a connection that drops mid-message into io.ErrUnexpectedEOF. Mapping that
|
||||||
|
// sentinel to "your upload was cut off" (a 400) would hide a real fault: only an error
|
||||||
|
// from reading the CLIENT's body is theirs (third adversarial review).
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
buf := &lockedBuf{}
|
||||||
|
app.Log = slog.New(slog.NewTextHandler(buf, nil))
|
||||||
|
app.Uploads = failingUploader{err: fmt.Errorf("failed to receive message: %w", io.ErrUnexpectedEOF)}
|
||||||
|
|
||||||
|
rec := do(t, app, uploadReq(t, "t", "a.m4a", []byte("x")))
|
||||||
|
|
||||||
|
assert.Equal(t, http.StatusInternalServerError, rec.Code)
|
||||||
|
assert.Contains(t, buf.String(), "level=ERROR")
|
||||||
|
}
|
||||||
@@ -132,7 +132,7 @@ func TestUploadPost_AcceptedAudioQueuesARowAndRedirectsToIt(t *testing.T) {
|
|||||||
assert.Equal(t, "Sprint review", title)
|
assert.Equal(t, "Sprint review", title)
|
||||||
assert.Equal(t, "/uploads/"+id, rec.Header().Get("Location"))
|
assert.Equal(t, "/uploads/"+id, rec.Header().Get("Location"))
|
||||||
require.Len(t, sub.jobs, 1)
|
require.Len(t, sub.jobs, 1)
|
||||||
assert.Equal(t, uploads.Job{UserID: userID, UploadID: id, Path: sub.jobs[0].Path}, sub.jobs[0])
|
assert.Equal(t, uploads.Job{UserID: userID, UploadID: id, Path: sub.jobs[0].Path, Duration: audioOK.Duration}, sub.jobs[0])
|
||||||
spooled, err := os.ReadFile(sub.jobs[0].Path)
|
spooled, err := os.ReadFile(sub.jobs[0].Path)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
assert.Equal(t, "synthetic audio bytes", string(spooled))
|
assert.Equal(t, "synthetic audio bytes", string(spooled))
|
||||||
@@ -366,8 +366,8 @@ func setReconciliation(t *testing.T, id, json string) {
|
|||||||
const reconciledJSON = `{"status":"reconciled","base":"iguana/pianissimo","alt":"iguana/kb-whisper","reconciler":"iguana/gemma4-26b",
|
const reconciledJSON = `{"status":"reconciled","base":"iguana/pianissimo","alt":"iguana/kb-whisper","reconciler":"iguana/gemma4-26b",
|
||||||
"disputes":380,"auto_resolved":90,"adjudicated":285,"changed":120,"unsure":5,"errors":0,
|
"disputes":380,"auto_resolved":90,"adjudicated":285,"changed":120,"unsure":5,"errors":0,
|
||||||
"contested":[
|
"contested":[
|
||||||
{"seconds":2210.5,"base":"SEB","alt":"SCB","used":"alt","unsure":false},
|
{"seconds":2210.5,"base":"Axel","alt":"Alex","used":"alt","unsure":false},
|
||||||
{"seconds":3400,"base":"T+1","alt":"en T","used":"base","unsure":true}]}`
|
{"seconds":3400,"base":"A+2","alt":"en A","used":"base","unsure":true}]}`
|
||||||
|
|
||||||
func TestUploadStatus_ReconciledSaysWhatWasDoneInPlainWords(t *testing.T) {
|
func TestUploadStatus_ReconciledSaysWhatWasDoneInPlainWords(t *testing.T) {
|
||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
@@ -394,11 +394,11 @@ func TestUploadStatus_ListsTheContestedSpotsWithAMinuteMarkBothReadingsAndWhichW
|
|||||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||||
|
|
||||||
assert.Contains(t, b, "~37 min", "2210.5 s")
|
assert.Contains(t, b, "~37 min", "2210.5 s")
|
||||||
assert.Contains(t, b, "SEB")
|
assert.Contains(t, b, "Axel")
|
||||||
assert.Contains(t, b, "SCB")
|
assert.Contains(t, b, "Alex")
|
||||||
assert.Contains(t, b, "~57 min", "3400 s")
|
assert.Contains(t, b, "~57 min", "3400 s")
|
||||||
assert.Contains(t, b, "T+1")
|
assert.Contains(t, b, "A+2")
|
||||||
assert.Contains(t, b, "en T")
|
assert.Contains(t, b, "en A")
|
||||||
assert.Contains(t, b, "uncertain", "an unsure spot is marked so the reader knows to check it")
|
assert.Contains(t, b, "uncertain", "an unsure spot is marked so the reader knows to check it")
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -460,3 +460,20 @@ func TestUploadStatus_AVeryLongListOfContestedSpotsIsShortenedWithItsTrueCount(t
|
|||||||
assert.NotContains(t, b, ">b119<", "the page does not render every one")
|
assert.NotContains(t, b, ">b119<", "the page does not render every one")
|
||||||
assert.Contains(t, b, "70 more", "120 stored, 50 shown")
|
assert.Contains(t, b, "70 more", "120 stored, 50 shown")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestUploadStatus_ALengthFailureIsExplainedInWordsAndNotAsAGenericError(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
tests := []struct{ kind, want string }{
|
||||||
|
{"audio_too_long", "longer than the limit"},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.kind, func(t *testing.T) {
|
||||||
|
id := seedUpload(t, userID, "m-"+tt.kind, "failed", tt.kind)
|
||||||
|
|
||||||
|
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||||
|
|
||||||
|
assert.Contains(t, b, tt.want)
|
||||||
|
assert.NotContains(t, b, "This upload failed. Please upload it again.")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,72 @@
|
|||||||
|
package web_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/url"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Error paths of the upload routes log a kind, never the error's text, which can carry
|
||||||
|
// a file name, a path, a database value or a provider's output (adversarial review F5).
|
||||||
|
|
||||||
|
type lockedBuf struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
b bytes.Buffer
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *lockedBuf) Write(p []byte) (int, error) {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
return l.b.Write(p)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *lockedBuf) String() string {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
return l.b.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
type failingUploader struct{ err error }
|
||||||
|
|
||||||
|
func (f failingUploader) Accept(context.Context, string, string, io.Reader) (string, error) {
|
||||||
|
return "", f.err
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadPost_AnUnexpectedFailureLogsAKindAndNotItsText(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
buf := &lockedBuf{}
|
||||||
|
app.Log = slog.New(slog.NewTextHandler(buf, nil))
|
||||||
|
app.Uploads = failingUploader{err: errors.New(`ffprobe: Error opening input file /spool/upload-555 for "zebrakaka-meeting.m4a"`)}
|
||||||
|
|
||||||
|
rec := do(t, app, uploadReq(t, "t", "zebrakaka-meeting.m4a", []byte("x")))
|
||||||
|
|
||||||
|
assert.Equal(t, http.StatusInternalServerError, rec.Code)
|
||||||
|
out := buf.String()
|
||||||
|
assert.NotContains(t, out, "zebrakaka", "a file name is private content")
|
||||||
|
assert.NotContains(t, out, "/spool/", "a spool path carries the upload's identity")
|
||||||
|
assert.Contains(t, out, "kind=error")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadChatPost_AModelFailureLogsAKindAndNotItsText(t *testing.T) {
|
||||||
|
app, fc := newUploadChatApp(t)
|
||||||
|
buf := &lockedBuf{}
|
||||||
|
app.Log = slog.New(slog.NewTextHandler(buf, nil))
|
||||||
|
fc.err = errors.New("LLM returned 400: the prompt was zebrakaka secret words")
|
||||||
|
id := seedUpload(t, userID, "m", "done", meetingText)
|
||||||
|
|
||||||
|
rec := do(t, app, chatPost("/uploads/"+id+"/chat", url.Values{"question": {"q"}}))
|
||||||
|
|
||||||
|
require.Equal(t, http.StatusOK, rec.Code)
|
||||||
|
out := buf.String()
|
||||||
|
assert.NotContains(t, out, "zebrakaka")
|
||||||
|
assert.Contains(t, out, "kind=error")
|
||||||
|
}
|
||||||
@@ -0,0 +1,287 @@
|
|||||||
|
package web_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/web"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The page's progress display (ADR-035, #42): real phases from the row, and the three
|
||||||
|
// honest states the page can tell apart -- running, quiet (counted phase not advancing)
|
||||||
|
// and lost (the service stopped reporting). Real router, real Postgres, a fixed clock.
|
||||||
|
|
||||||
|
var progressNow = time.Date(2026, 10, 2, 9, 0, 0, 0, time.UTC)
|
||||||
|
|
||||||
|
func fixedClock(app *web.App) { app.Now = func() time.Time { return progressNow } }
|
||||||
|
|
||||||
|
// seedProgress writes the job's phases and the two timestamps onto an existing row.
|
||||||
|
func seedProgress(t *testing.T, id string, steps []domain.ProgressStep, progressAt, aliveAt time.Time) {
|
||||||
|
t.Helper()
|
||||||
|
b, err := json.Marshal(steps)
|
||||||
|
require.NoError(t, err)
|
||||||
|
var alive any
|
||||||
|
if !aliveAt.IsZero() {
|
||||||
|
alive = aliveAt
|
||||||
|
}
|
||||||
|
_, err = rawPool(t).Exec(context.Background(),
|
||||||
|
`UPDATE uploaded_transcripts SET progress = $2, progress_at = $3, alive_at = $4 WHERE id = $1`,
|
||||||
|
id, b, progressAt, alive)
|
||||||
|
require.NoError(t, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
func step(key, label string, state domain.StepState, startedAgo, ranFor time.Duration, done, total int) domain.ProgressStep {
|
||||||
|
s := domain.ProgressStep{Key: key, Label: label, State: state, Done: done, Total: total}
|
||||||
|
if state != domain.StepPending {
|
||||||
|
s.StartedAt = progressNow.Add(-startedAgo)
|
||||||
|
}
|
||||||
|
if state == domain.StepDone || state == domain.StepFailed {
|
||||||
|
s.FinishedAt = s.StartedAt.Add(ranFor)
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// twoEngineRun is a job in its second engine: three steps done, one running, three to come.
|
||||||
|
func twoEngineRun() []domain.ProgressStep {
|
||||||
|
return []domain.ProgressStep{
|
||||||
|
step("received", "Received", domain.StepDone, 5*time.Minute, 0, 0, 0),
|
||||||
|
step("audio", "Preparing the audio", domain.StepDone, 5*time.Minute, 40*time.Second, 0, 0),
|
||||||
|
step("engine_a", "Transcribing with iguana/pianissimo", domain.StepDone, 4*time.Minute, 38*time.Second, 0, 0),
|
||||||
|
step("engine_b", "Transcribing with iguana/kb-whisper", domain.StepRunning, 3*time.Minute+12*time.Second, 0, 0, 0),
|
||||||
|
step("compare", "Comparing the two transcripts", domain.StepPending, 0, 0, 0, 0),
|
||||||
|
step("decide", "Deciding the differences", domain.StepPending, 0, 0, 0, 0),
|
||||||
|
step("save", "Saving the transcript", domain.StepPending, 0, 0, 0, 0),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func statusBody(t *testing.T, app *web.App, id string) string {
|
||||||
|
t.Helper()
|
||||||
|
rec := do(t, app, get("/uploads/"+id+"/status"))
|
||||||
|
require.Equal(t, 200, rec.Code)
|
||||||
|
return body(t, rec)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_ARunningJobShowsEveryPhaseInOrderWithTheRunningOneMarked(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
fixedClock(app)
|
||||||
|
id := seedUpload(t, userID, "m", "transcribing", "")
|
||||||
|
seedProgress(t, id, twoEngineRun(), progressNow.Add(-10*time.Second), progressNow.Add(-4*time.Second))
|
||||||
|
|
||||||
|
b := statusBody(t, app, id)
|
||||||
|
|
||||||
|
last := -1
|
||||||
|
for _, label := range []string{
|
||||||
|
"Received", "Preparing the audio", "Transcribing with iguana/pianissimo", "Transcribing with iguana/kb-whisper",
|
||||||
|
"Comparing the two transcripts", "Deciding the differences", "Saving the transcript",
|
||||||
|
} {
|
||||||
|
i := strings.Index(b, label)
|
||||||
|
require.Greater(t, i, last, "phase %q missing or out of order:\n%s", label, b)
|
||||||
|
last = i
|
||||||
|
}
|
||||||
|
assert.Equal(t, 3, strings.Count(b, "step-done"), "three finished phases collapse to a done line")
|
||||||
|
assert.Equal(t, 1, strings.Count(b, `aria-current="step"`), "exactly one phase is the current one")
|
||||||
|
assert.Contains(t, b, "3m 12s", "the running phase shows how long it has run")
|
||||||
|
assert.Contains(t, b, "38s", "a finished phase shows how long it took")
|
||||||
|
assert.Contains(t, b, `hx-trigger="every 2s"`, "an unfinished job re-asks every two seconds")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_OnlyACountedPhaseGetsABarAndItShowsRealCounts(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
fixedClock(app)
|
||||||
|
id := seedUpload(t, userID, "m", "transcribing", "")
|
||||||
|
steps := twoEngineRun()
|
||||||
|
steps[3] = step("engine_b", "Transcribing with iguana/kb-whisper", domain.StepDone, 3*time.Minute, 2*time.Minute, 0, 0)
|
||||||
|
steps[4] = step("compare", "Comparing the two transcripts", domain.StepDone, 1*time.Minute, time.Second, 0, 0)
|
||||||
|
steps[5] = step("decide", "Deciding the differences", domain.StepRunning, 50*time.Second, 0, 142, 282)
|
||||||
|
seedProgress(t, id, steps, progressNow.Add(-5*time.Second), progressNow.Add(-3*time.Second))
|
||||||
|
|
||||||
|
b := statusBody(t, app, id)
|
||||||
|
|
||||||
|
assert.Contains(t, b, `<progress`)
|
||||||
|
assert.Contains(t, b, `value="142"`)
|
||||||
|
assert.Contains(t, b, `max="282"`)
|
||||||
|
assert.Contains(t, b, "142 of 282")
|
||||||
|
|
||||||
|
// ...and a phase with no honest count never gets one: no percentage out of thin air.
|
||||||
|
app2, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
fixedClock(app2)
|
||||||
|
id2 := seedUpload(t, userID, "m2", "transcribing", "")
|
||||||
|
seedProgress(t, id2, twoEngineRun(), progressNow.Add(-10*time.Second), progressNow.Add(-4*time.Second))
|
||||||
|
assert.NotContains(t, statusBody(t, app2, id2), "<progress", "an engine call is one request: there is nothing to count")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_ALongEngineCallWithAFreshHeartbeatIsStillJustRunning(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
fixedClock(app)
|
||||||
|
id := seedUpload(t, userID, "m", "transcribing", "")
|
||||||
|
steps := twoEngineRun()
|
||||||
|
steps[3] = step("engine_b", "Transcribing with iguana/kb-whisper", domain.StepRunning, 25*time.Minute, 0, 0, 0)
|
||||||
|
seedProgress(t, id, steps, progressNow.Add(-25*time.Minute), progressNow.Add(-3*time.Second))
|
||||||
|
|
||||||
|
b := statusBody(t, app, id)
|
||||||
|
|
||||||
|
assert.Contains(t, b, "25m 00s")
|
||||||
|
assert.NotContains(t, b, "step-health-lost")
|
||||||
|
assert.NotContains(t, b, "step-health-quiet",
|
||||||
|
"a single blocking request has no sub-progress to go quiet about; the heartbeat says the service is alive")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_AHeartbeatThatStoppedMeansTheServiceIsLostAndTheJobIsNotCalledRunning(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
fixedClock(app)
|
||||||
|
id := seedUpload(t, userID, "m", "transcribing", "")
|
||||||
|
seedProgress(t, id, twoEngineRun(), progressNow.Add(-3*time.Minute), progressNow.Add(-3*time.Minute))
|
||||||
|
|
||||||
|
b := statusBody(t, app, id)
|
||||||
|
|
||||||
|
assert.Contains(t, b, "step-health-lost")
|
||||||
|
assert.Contains(t, b, "stopped reporting")
|
||||||
|
assert.Contains(t, b, "3 min", "how long ago it was last heard from")
|
||||||
|
assert.NotContains(t, b, "spinner", "a lost job must not wear the running spinner")
|
||||||
|
assert.Contains(t, b, "step-mark-warn", "the running phase of a lost job carries a warning mark, in the error colour, not the accent")
|
||||||
|
assert.Contains(t, b, `hx-trigger="every`, "it keeps asking: a restarted service comes back, or the start-up sweep fails the row")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_ACountedPhaseThatStopsAdvancingGoesQuietNotLost(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
fixedClock(app)
|
||||||
|
id := seedUpload(t, userID, "m", "transcribing", "")
|
||||||
|
steps := twoEngineRun()
|
||||||
|
steps[5] = step("decide", "Deciding the differences", domain.StepRunning, 20*time.Minute, 0, 142, 282)
|
||||||
|
seedProgress(t, id, steps, progressNow.Add(-6*time.Minute), progressNow.Add(-3*time.Second))
|
||||||
|
|
||||||
|
b := statusBody(t, app, id)
|
||||||
|
|
||||||
|
assert.Contains(t, b, "step-health-quiet")
|
||||||
|
assert.Contains(t, b, "Nothing new for 6 min")
|
||||||
|
assert.NotContains(t, b, "stopped reporting", "the service is alive; it is the model that is slow")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_BeforeTheFirstHeartbeatAJobIsRunningUntilItHasBeenSilentTooLong(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
fixedClock(app)
|
||||||
|
fresh := seedUpload(t, userID, "fresh", "transcribing", "")
|
||||||
|
seedProgress(t, fresh, twoEngineRun(), progressNow.Add(-5*time.Second), time.Time{})
|
||||||
|
silent := seedUpload(t, userID, "silent", "transcribing", "")
|
||||||
|
seedProgress(t, silent, twoEngineRun(), progressNow.Add(-2*time.Minute), time.Time{})
|
||||||
|
|
||||||
|
assert.NotContains(t, statusBody(t, app, fresh), "step-health-lost", "the first beat has not been due yet")
|
||||||
|
assert.Contains(t, statusBody(t, app, silent), "step-health-lost", "no beat and no progress for two minutes is a lost service")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_AQueuedJobShowsNoPhasesAndNoLostWarning(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
fixedClock(app)
|
||||||
|
id := seedUpload(t, userID, "m", "queued", "")
|
||||||
|
|
||||||
|
b := statusBody(t, app, id)
|
||||||
|
|
||||||
|
assert.Contains(t, b, "Queued")
|
||||||
|
assert.NotContains(t, b, `class="steps"`)
|
||||||
|
assert.NotContains(t, b, "step-health")
|
||||||
|
assert.Contains(t, b, `hx-trigger="every 2s"`)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_AFailedJobShowsWhichPhaseFailedAndWhatNeverStarted(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
fixedClock(app)
|
||||||
|
id := seedUpload(t, userID, "m", "failed", "transcription_failed")
|
||||||
|
steps := twoEngineRun()
|
||||||
|
steps[3] = step("engine_b", "Transcribing with iguana/kb-whisper", domain.StepFailed, 3*time.Minute, time.Minute, 0, 0)
|
||||||
|
seedProgress(t, id, steps, progressNow.Add(-2*time.Minute), progressNow.Add(-2*time.Minute))
|
||||||
|
|
||||||
|
b := statusBody(t, app, id)
|
||||||
|
|
||||||
|
assert.Contains(t, b, "Transcription failed")
|
||||||
|
assert.Contains(t, b, "step-failed")
|
||||||
|
assert.Contains(t, b, "step-pending", "what never started says where the job stopped")
|
||||||
|
assert.NotContains(t, b, "hx-trigger", "a failed job stops polling")
|
||||||
|
assert.NotContains(t, b, "step-health", "a finished job is not 'lost'")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_ADoneJobKeepsItsRecordOfWhatWasDoneAndHowLongEachTook(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
fixedClock(app)
|
||||||
|
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
||||||
|
steps := []domain.ProgressStep{
|
||||||
|
step("received", "Received", domain.StepDone, 20*time.Minute, 0, 0, 0),
|
||||||
|
step("audio", "Preparing the audio", domain.StepDone, 20*time.Minute, 40*time.Second, 0, 0),
|
||||||
|
step("decide", "Deciding the differences", domain.StepDone, 15*time.Minute, 9*time.Minute+30*time.Second, 282, 282),
|
||||||
|
step("save", "Saving the transcript", domain.StepSkipped, 0, 0, 0, 0),
|
||||||
|
}
|
||||||
|
seedProgress(t, id, steps, progressNow.Add(-time.Minute), progressNow.Add(-time.Minute))
|
||||||
|
|
||||||
|
b := statusBody(t, app, id)
|
||||||
|
|
||||||
|
assert.Contains(t, b, "Synthetic transcript text.")
|
||||||
|
assert.Contains(t, b, "What was done")
|
||||||
|
assert.Contains(t, b, "9m 30s")
|
||||||
|
assert.Contains(t, b, "step-skipped")
|
||||||
|
assert.NotContains(t, b, "hx-trigger", "a finished upload must not poll forever")
|
||||||
|
assert.NotContains(t, b, "step-health", "a finished job is never 'lost' however old its heartbeat")
|
||||||
|
assert.NotContains(t, b, "spinner")
|
||||||
|
}
|
||||||
|
|
||||||
|
// stepsBlock is the progress list of a page: from its opening tag to its closing one.
|
||||||
|
func stepsBlock(t *testing.T, html string) string {
|
||||||
|
t.Helper()
|
||||||
|
start := strings.Index(html, `<ol class="steps"`)
|
||||||
|
require.GreaterOrEqual(t, start, 0, "no progress list in:\n%s", html)
|
||||||
|
end := strings.Index(html[start:], "</ol>")
|
||||||
|
require.GreaterOrEqual(t, end, 0)
|
||||||
|
return html[start : start+end+len("</ol>")]
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_TheFullPageCarriesTheVerySameProgressListAsTheFragment(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
fixedClock(app)
|
||||||
|
id := seedUpload(t, userID, "m", "transcribing", "")
|
||||||
|
seedProgress(t, id, twoEngineRun(), progressNow.Add(-10*time.Second), progressNow.Add(-4*time.Second))
|
||||||
|
|
||||||
|
full := body(t, do(t, app, get("/uploads/"+id)))
|
||||||
|
fragment := statusBody(t, app, id)
|
||||||
|
|
||||||
|
assert.Equal(t, stepsBlock(t, fragment), stepsBlock(t, full),
|
||||||
|
"the page a user opens and the fragment HTMX swaps in must show the same list, or the display jumps on the first poll")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_AStepNoteIsShownDirectlyUnderItsStepAndEscaped(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
fixedClock(app)
|
||||||
|
id := seedUpload(t, userID, "m", "transcribing", "")
|
||||||
|
steps := twoEngineRun()
|
||||||
|
steps[1].Note = "The audio is 20 min long; the file claimed 1 h 19 min. <b>cut off</b>?"
|
||||||
|
seedProgress(t, id, steps, progressNow.Add(-10*time.Second), progressNow.Add(-4*time.Second))
|
||||||
|
|
||||||
|
b := statusBody(t, app, id)
|
||||||
|
|
||||||
|
audio := strings.Index(b, "Preparing the audio")
|
||||||
|
note := strings.Index(b, "The audio is 20 min long")
|
||||||
|
next := strings.Index(b, "Transcribing with iguana/pianissimo")
|
||||||
|
require.True(t, audio >= 0 && note > audio && next > note, "the note must sit between its own step and the next one:\n%s", b)
|
||||||
|
assert.Contains(t, b, "step-note")
|
||||||
|
assert.NotContains(t, b, "<b>cut off</b>", "a note is words, never markup")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_ADonePageShowsItsRecordOpenWhenAStepHasANote(t *testing.T) {
|
||||||
|
// A truncated recording must not look like a clean one: the note that replaced the
|
||||||
|
// refusal is the only warning, so it cannot sit inside a collapsed list (third review).
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
fixedClock(app)
|
||||||
|
noted := seedUpload(t, userID, "noted", "done", "Synthetic transcript text.")
|
||||||
|
steps := []domain.ProgressStep{step("audio", "Preparing the audio", domain.StepDone, 20*time.Minute, 40*time.Second, 0, 0)}
|
||||||
|
steps[0].Note = "The audio is 20 min long; the file claimed 1 h 19 min."
|
||||||
|
seedProgress(t, noted, steps, progressNow.Add(-time.Minute), progressNow.Add(-time.Minute))
|
||||||
|
clean := seedUpload(t, userID, "clean", "done", "Synthetic transcript text.")
|
||||||
|
seedProgress(t, clean, []domain.ProgressStep{step("audio", "Preparing the audio", domain.StepDone, 20*time.Minute, 40*time.Second, 0, 0)}, progressNow.Add(-time.Minute), progressNow.Add(-time.Minute))
|
||||||
|
|
||||||
|
assert.Contains(t, statusBody(t, app, noted), `<details class="what-was-done" open>`)
|
||||||
|
assert.NotContains(t, statusBody(t, app, clean), `<details class="what-was-done" open>`, "a clean record stays collapsed")
|
||||||
|
}
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
package web_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestRenderUploadProgressStatesForReview writes the upload page in each progress state
|
||||||
|
// to $TAPIR_SHOT_DIR as standalone HTML (light and dark), for a screenshot review of the
|
||||||
|
// visual claims tests cannot assert: that it looks like tapir and reads like a log of
|
||||||
|
// phases (ADR-035, #42). Opt-in: it does nothing unless the variable is set. The text is
|
||||||
|
// synthetic. Screenshot with any browser, for example:
|
||||||
|
//
|
||||||
|
// TAPIR_SHOT_DIR=/tmp/shots go test -run TestRenderUploadProgressStatesForReview ./internal/web
|
||||||
|
// chromium --headless --screenshot=/tmp/shots/running.png --window-size=900,900 /tmp/shots/running-light.html
|
||||||
|
func TestRenderUploadProgressStatesForReview(t *testing.T) {
|
||||||
|
dir := os.Getenv("TAPIR_SHOT_DIR")
|
||||||
|
if dir == "" {
|
||||||
|
t.Skip("set TAPIR_SHOT_DIR to render the progress states for a screenshot review")
|
||||||
|
}
|
||||||
|
require.NoError(t, os.MkdirAll(dir, 0o750))
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
fixedClock(app)
|
||||||
|
|
||||||
|
decide := twoEngineRun()
|
||||||
|
decide[3] = step("engine_b", "Transcribing with iguana/kb-whisper", domain.StepDone, 9*time.Minute, 4*time.Minute, 0, 0)
|
||||||
|
decide[4] = step("compare", "Comparing the two transcripts", domain.StepDone, 5*time.Minute, time.Second, 0, 0)
|
||||||
|
decide[5] = step("decide", "Deciding the differences", domain.StepRunning, 4*time.Minute+30*time.Second, 0, 142, 282)
|
||||||
|
|
||||||
|
failed := twoEngineRun()
|
||||||
|
failed[3] = step("engine_b", "Transcribing with iguana/kb-whisper", domain.StepFailed, 3*time.Minute, time.Minute, 0, 0)
|
||||||
|
|
||||||
|
done := []domain.ProgressStep{
|
||||||
|
step("received", "Received", domain.StepDone, 22*time.Minute, 0, 0, 0),
|
||||||
|
step("audio", "Preparing the audio", domain.StepDone, 22*time.Minute, 41*time.Second, 0, 0),
|
||||||
|
step("engine_a", "Transcribing with iguana/pianissimo", domain.StepDone, 21*time.Minute, 2*time.Minute+10*time.Second, 0, 0),
|
||||||
|
step("engine_b", "Transcribing with iguana/kb-whisper", domain.StepDone, 19*time.Minute, 4*time.Minute+5*time.Second, 0, 0),
|
||||||
|
step("compare", "Comparing the two transcripts", domain.StepDone, 15*time.Minute, 2*time.Second, 0, 0),
|
||||||
|
step("decide", "Deciding the differences", domain.StepDone, 15*time.Minute, 14*time.Minute, 282, 282),
|
||||||
|
step("save", "Saving the transcript", domain.StepDone, time.Minute, time.Second, 0, 0),
|
||||||
|
}
|
||||||
|
|
||||||
|
noted := twoEngineRun()
|
||||||
|
noted[1].Note = "The audio is 20 min long; the file claimed 1 h 19 min. If the recording was cut off, upload the complete one."
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
name, status, content string
|
||||||
|
steps []domain.ProgressStep
|
||||||
|
progressAgo, aliveAgo time.Duration
|
||||||
|
}{
|
||||||
|
{"running", "transcribing", "", twoEngineRun(), 10 * time.Second, 4 * time.Second},
|
||||||
|
{"deciding", "transcribing", "", decide, 3 * time.Second, 3 * time.Second},
|
||||||
|
{"quiet", "transcribing", "", decide, 6 * time.Minute, 4 * time.Second},
|
||||||
|
{"lost", "transcribing", "", twoEngineRun(), 3 * time.Minute, 3 * time.Minute},
|
||||||
|
{"failed", "failed", "transcription_failed", failed, 2 * time.Minute, 2 * time.Minute},
|
||||||
|
{"done", "done", "Synthetic transcript text for the review.", done, time.Minute, time.Minute},
|
||||||
|
{"noted", "transcribing", "", noted, 10 * time.Second, 4 * time.Second},
|
||||||
|
{"queued", "queued", "", nil, 0, 0},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
id := seedUpload(t, userID, "Project notes (synthetic)", c.status, c.content)
|
||||||
|
if c.steps != nil {
|
||||||
|
seedProgress(t, id, c.steps, progressNow.Add(-c.progressAgo), progressNow.Add(-c.aliveAgo))
|
||||||
|
}
|
||||||
|
page := body(t, do(t, app, get("/uploads/"+id)))
|
||||||
|
for _, theme := range []string{"light", "dark"} {
|
||||||
|
html := page
|
||||||
|
if theme == "dark" {
|
||||||
|
html = strings.Replace(html, "<html", `<html data-theme="dark"`, 1)
|
||||||
|
}
|
||||||
|
require.NoError(t, os.WriteFile(filepath.Join(dir, c.name+"-"+theme+".html"), []byte(html), 0o600))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
package web_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The stored judged spots include the ones the model kept as the base (ADR-035), for a
|
||||||
|
// later evaluation. The page lists only the ones a reader should check: changed and
|
||||||
|
// uncertain.
|
||||||
|
|
||||||
|
func TestUploadStatus_OnlyChangedAndUncertainSpotsAreListedNotTheKeptOnes(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
id := seedUpload(t, userID, "m", "done", "Synthetic transcript.")
|
||||||
|
setReconciliation(t, id, `{"status":"reconciled","base":"a","alt":"b","reconciler":"c","disputes":3,"adjudicated":3,"changed":1,"unsure":1,
|
||||||
|
"contested":[
|
||||||
|
{"seconds":60,"base":"CHANGEDBASE","alt":"CHANGEDALT","used":"alt","unsure":false},
|
||||||
|
{"seconds":120,"base":"KEPTBASE","alt":"KEPTALT","used":"base","unsure":false},
|
||||||
|
{"seconds":180,"base":"UNSUREBASE","alt":"UNSUREALT","used":"base","unsure":true}]}`)
|
||||||
|
|
||||||
|
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||||
|
|
||||||
|
assert.Contains(t, b, "CHANGEDBASE")
|
||||||
|
assert.Contains(t, b, "UNSUREALT")
|
||||||
|
assert.NotContains(t, b, "KEPTBASE", "a spot the model confidently kept is not one for the reader to check")
|
||||||
|
assert.NotContains(t, b, "KEPTALT")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadStatus_TheMoreCountIgnoresKeptSpotsAndMatchesTheListedKind(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
id := seedUpload(t, userID, "m", "done", "Synthetic transcript.")
|
||||||
|
var spots []string
|
||||||
|
for i := 0; i < 60; i++ {
|
||||||
|
spots = append(spots, fmt.Sprintf(`{"seconds":%d,"base":"b%d","alt":"a%d","used":"alt","unsure":false}`, i*10, i, i))
|
||||||
|
spots = append(spots, fmt.Sprintf(`{"seconds":%d,"base":"k%d","alt":"j%d","used":"base","unsure":false}`, i*10+5, i, i))
|
||||||
|
}
|
||||||
|
setReconciliation(t, id, `{"status":"reconciled","base":"a","alt":"b","reconciler":"c","disputes":400,"changed":60,"contested":[`+strings.Join(spots, ",")+`]}`)
|
||||||
|
|
||||||
|
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||||
|
|
||||||
|
assert.Contains(t, b, "10 more", "60 changed, 50 shown: the 60 kept ones do not count")
|
||||||
|
assert.NotContains(t, b, "k0 / j0", "a kept spot is not rendered even when it is among the first stored")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadStatus_WhenNothingNeedsCheckingThereIsNoSpotsSection(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
id := seedUpload(t, userID, "m", "done", "Synthetic transcript.")
|
||||||
|
setReconciliation(t, id, `{"status":"reconciled","base":"a","alt":"b","reconciler":"c","disputes":1,"adjudicated":1,
|
||||||
|
"contested":[{"seconds":60,"base":"KEPTBASE","alt":"KEPTALT","used":"base","unsure":false}]}`)
|
||||||
|
|
||||||
|
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||||
|
|
||||||
|
require.Contains(t, b, "Reconciled from two transcripts")
|
||||||
|
assert.NotContains(t, b, "Spots to check", "an empty list would invite checking nothing")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadStatus_ASpotShowsTheWordsJustBeforeIt(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
id := seedUpload(t, userID, "m", "done", "Synthetic transcript.")
|
||||||
|
setReconciliation(t, id, `{"status":"reconciled","base":"a","alt":"b","reconciler":"c","disputes":1,"changed":1,
|
||||||
|
"contested":[{"seconds":2640,"base":"vanligt","alt":"KORT","used":"alt","unsure":false,"before":"grannen målar ett"}]}`)
|
||||||
|
|
||||||
|
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||||
|
|
||||||
|
assert.Contains(t, b, "grannen målar ett", "the words before the spot are how a reader finds it in the audio")
|
||||||
|
assert.Contains(t, b, "vanligt")
|
||||||
|
assert.Contains(t, b, "KORT")
|
||||||
|
}
|
||||||
+475
-115
@@ -176,7 +176,7 @@ func UploadsPage(v uploadsView) templ.Component {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// UploadPage is one upload's page: its title and the polling status region.
|
// UploadPage is one upload's page: its title and the polling status region.
|
||||||
func UploadPage(u domain.UploadedTranscript, chatEnabled bool) templ.Component {
|
func UploadPage(u domain.UploadedTranscript, chatEnabled bool, now time.Time) templ.Component {
|
||||||
return templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
|
return templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
|
||||||
templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
|
templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
|
||||||
if templ_7745c5c3_CtxErr := ctx.Err(); templ_7745c5c3_CtxErr != nil {
|
if templ_7745c5c3_CtxErr := ctx.Err(); templ_7745c5c3_CtxErr != nil {
|
||||||
@@ -226,7 +226,7 @@ func UploadPage(u domain.UploadedTranscript, chatEnabled bool) templ.Component {
|
|||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = uploadStatus(u, chatEnabled).Render(ctx, templ_7745c5c3_Buffer)
|
templ_7745c5c3_Err = uploadStatus(u, chatEnabled, now).Render(ctx, templ_7745c5c3_Buffer)
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
@@ -246,7 +246,7 @@ func UploadPage(u domain.UploadedTranscript, chatEnabled bool) templ.Component {
|
|||||||
|
|
||||||
// uploadStatus is the region HTMX polls. Unfinished states carry hx-trigger so the
|
// uploadStatus is the region HTMX polls. Unfinished states carry hx-trigger so the
|
||||||
// page re-asks; done and failed render without it, so nothing polls forever.
|
// page re-asks; done and failed render without it, so nothing polls forever.
|
||||||
func uploadStatus(u domain.UploadedTranscript, chatEnabled bool) templ.Component {
|
func uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time) templ.Component {
|
||||||
return templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
|
return templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
|
||||||
templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
|
templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
|
||||||
if templ_7745c5c3_CtxErr := ctx.Err(); templ_7745c5c3_CtxErr != nil {
|
if templ_7745c5c3_CtxErr := ctx.Err(); templ_7745c5c3_CtxErr != nil {
|
||||||
@@ -311,44 +311,68 @@ func uploadStatus(u domain.UploadedTranscript, chatEnabled bool) templ.Component
|
|||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
|
if len(u.Progress) > 0 {
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 21, "<details class=\"what-was-done\"")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
if hasNote(u.Progress) {
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 22, " open")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 23, "><summary>What was done</summary>")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = progressSteps(u.Progress, now, false).Render(ctx, templ_7745c5c3_Buffer)
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 24, "</details> ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
}
|
||||||
if chatEnabled {
|
if chatEnabled {
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 21, "<p><a class=\"btn\" href=\"")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 25, "<p><a class=\"btn\" href=\"")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var15 templ.SafeURL
|
var templ_7745c5c3_Var15 templ.SafeURL
|
||||||
templ_7745c5c3_Var15, templ_7745c5c3_Err = templ.JoinURLErrs(templ.SafeURL("/uploads/" + u.ID + "/chat"))
|
templ_7745c5c3_Var15, templ_7745c5c3_Err = templ.JoinURLErrs(templ.SafeURL("/uploads/" + u.ID + "/chat"))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 71, Col: 73}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 77, Col: 73}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var15))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var15))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 22, "\">Ask about this recording</a></p>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 26, "\">Ask about this recording</a></p>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 23, "<details open><summary>Transcript</summary><pre class=\"transcript\">")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 27, "<details open><summary>Transcript</summary><pre class=\"transcript\">")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var16 string
|
var templ_7745c5c3_Var16 string
|
||||||
templ_7745c5c3_Var16, templ_7745c5c3_Err = templ.JoinStringErrs(u.Content)
|
templ_7745c5c3_Var16, templ_7745c5c3_Err = templ.JoinStringErrs(u.Content)
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 75, Col: 40}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 81, Col: 40}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var16))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var16))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 24, "</pre></details></div>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 28, "</pre></details></div>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
case domain.UploadFailed:
|
case domain.UploadFailed:
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 25, "<div class=\"upload-status upload-failed flash flash-error\" role=\"alert\">")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 29, "<div class=\"upload-status upload-failed flash flash-error\" role=\"alert\">")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
@@ -356,51 +380,69 @@ func uploadStatus(u domain.UploadedTranscript, chatEnabled bool) templ.Component
|
|||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 26, "<p>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 30, "<p>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var17 string
|
var templ_7745c5c3_Var17 string
|
||||||
templ_7745c5c3_Var17, templ_7745c5c3_Err = templ.JoinStringErrs(uploadFailureMessage(u.ErrorKind))
|
templ_7745c5c3_Var17, templ_7745c5c3_Err = templ.JoinStringErrs(uploadFailureMessage(u.ErrorKind))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 81, Col: 42}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 87, Col: 42}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var17))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var17))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 27, "</p></div>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 31, "</p>")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
if len(u.Progress) > 0 {
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 32, "<details class=\"what-was-done\" open><summary>Where it stopped</summary>")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = progressSteps(u.Progress, now, false).Render(ctx, templ_7745c5c3_Buffer)
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 33, "</details>")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 34, "</div>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
default:
|
default:
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 28, "<div class=\"upload-status upload-working\" role=\"status\" hx-get=\"")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 35, "<div class=\"upload-status upload-working\" role=\"status\" hx-get=\"")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var18 string
|
var templ_7745c5c3_Var18 string
|
||||||
templ_7745c5c3_Var18, templ_7745c5c3_Err = templ.ResolveAttributeValue("/uploads/" + u.ID + "/status")
|
templ_7745c5c3_Var18, templ_7745c5c3_Err = templ.ResolveAttributeValue("/uploads/" + u.ID + "/status")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 87, Col: 43}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 99, Col: 43}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var18)
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var18)
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 29, "\" hx-trigger=\"")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 36, "\" hx-trigger=\"")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var19 string
|
var templ_7745c5c3_Var19 string
|
||||||
templ_7745c5c3_Var19, templ_7745c5c3_Err = templ.ResolveAttributeValue("every " + uploadPollEvery)
|
templ_7745c5c3_Var19, templ_7745c5c3_Err = templ.ResolveAttributeValue("every " + uploadPollEvery)
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 88, Col: 43}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 100, Col: 43}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var19)
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var19)
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 30, "\" hx-swap=\"outerHTML\">")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 37, "\" hx-swap=\"outerHTML\">")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
@@ -409,17 +451,33 @@ func uploadStatus(u domain.UploadedTranscript, chatEnabled bool) templ.Component
|
|||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
if u.Status == domain.UploadTranscribing {
|
if u.Status == domain.UploadTranscribing {
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 31, "<p>Transcribing… a long recording takes a few minutes.</p>")
|
health, silent := uploadHealth(u, now)
|
||||||
if templ_7745c5c3_Err != nil {
|
if len(u.Progress) > 0 {
|
||||||
return templ_7745c5c3_Err
|
templ_7745c5c3_Err = progressSteps(u.Progress, now, health != healthLost).Render(ctx, templ_7745c5c3_Buffer)
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 38, " ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = healthNote(health, silent).Render(ctx, templ_7745c5c3_Buffer)
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 39, "<p>Transcribing… a long recording takes a few minutes.</p>")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 32, "<p>Queued — waiting for its turn to be transcribed.</p>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 40, "<p>Queued — waiting for its turn to be transcribed.</p>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 33, "</div>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 41, "</div>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
@@ -463,20 +521,20 @@ func UploadChatPage(u domain.UploadedTranscript, v chatView) templ.Component {
|
|||||||
}()
|
}()
|
||||||
}
|
}
|
||||||
ctx = templ.InitializeContext(ctx)
|
ctx = templ.InitializeContext(ctx)
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 34, "<article class=\"account\"><h1>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 42, "<article class=\"account\"><h1>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var22 string
|
var templ_7745c5c3_Var22 string
|
||||||
templ_7745c5c3_Var22, templ_7745c5c3_Err = templ.JoinStringErrs(u.Title)
|
templ_7745c5c3_Var22, templ_7745c5c3_Err = templ.JoinStringErrs(u.Title)
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 106, Col: 16}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 124, Col: 16}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var22))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var22))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 35, "</h1><section id=\"chat-section\" class=\"chat-dock chat-dock-open\"><h2 class=\"chat-heading\">Ask about this recording</h2><p class=\"chat-scope muted\">Answers come only from this recording's transcript, which is private to you.</p>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 43, "</h1><section id=\"chat-section\" class=\"chat-dock chat-dock-open\"><h2 class=\"chat-heading\">Ask about this recording</h2><p class=\"chat-scope muted\">Answers come only from this recording's transcript, which is private to you.</p>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
@@ -484,20 +542,20 @@ func UploadChatPage(u domain.UploadedTranscript, v chatView) templ.Component {
|
|||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 36, "</section><p><a href=\"")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 44, "</section><p><a href=\"")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var23 templ.SafeURL
|
var templ_7745c5c3_Var23 templ.SafeURL
|
||||||
templ_7745c5c3_Var23, templ_7745c5c3_Err = templ.JoinURLErrs(templ.SafeURL("/uploads/" + u.ID))
|
templ_7745c5c3_Var23, templ_7745c5c3_Err = templ.JoinURLErrs(templ.SafeURL("/uploads/" + u.ID))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 112, Col: 49}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 130, Col: 49}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var23))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var23))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 37, "\">Back to the transcript</a></p></article>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 45, "\">Back to the transcript</a></p></article>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
@@ -537,7 +595,7 @@ func videoNotice(hadVideo bool) templ.Component {
|
|||||||
}
|
}
|
||||||
ctx = templ.ClearChildren(ctx)
|
ctx = templ.ClearChildren(ctx)
|
||||||
if hadVideo {
|
if hadVideo {
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 38, "<p class=\"muted video-notice\">This file contained a video track. Only its audio was used; the video was discarded and nothing from it was kept.</p>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 46, "<p class=\"muted video-notice\">This file contained a video track. Only its audio was used; the video was discarded and nothing from it was kept.</p>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
@@ -574,262 +632,281 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
|
|||||||
if r != nil {
|
if r != nil {
|
||||||
switch r.Status {
|
switch r.Status {
|
||||||
case domain.ReconcileDone:
|
case domain.ReconcileDone:
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 39, "<div class=\"reconciliation\"><p>Reconciled from two transcripts (")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 47, "<div class=\"reconciliation\"><p>Reconciled from two transcripts (")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var26 string
|
var templ_7745c5c3_Var26 string
|
||||||
templ_7745c5c3_Var26, templ_7745c5c3_Err = templ.JoinStringErrs(r.Base)
|
templ_7745c5c3_Var26, templ_7745c5c3_Err = templ.JoinStringErrs(r.Base)
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 139, Col: 47}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 157, Col: 47}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var26))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var26))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 40, " and ")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 48, " and ")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var27 string
|
var templ_7745c5c3_Var27 string
|
||||||
templ_7745c5c3_Var27, templ_7745c5c3_Err = templ.JoinStringErrs(r.Alt)
|
templ_7745c5c3_Var27, templ_7745c5c3_Err = templ.JoinStringErrs(r.Alt)
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 139, Col: 61}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 157, Col: 61}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var27))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var27))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 41, "): ")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 49, "): ")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var28 string
|
var templ_7745c5c3_Var28 string
|
||||||
templ_7745c5c3_Var28, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(r.Disputes))
|
templ_7745c5c3_Var28, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(r.Disputes))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 140, Col: 32}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 158, Col: 32}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var28))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var28))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 42, " differences, ")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 50, " differences, ")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var29 string
|
var templ_7745c5c3_Var29 string
|
||||||
templ_7745c5c3_Var29, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(r.AutoResolved))
|
templ_7745c5c3_Var29, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(r.AutoResolved))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 141, Col: 36}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 159, Col: 36}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var29))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var29))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 43, " settled automatically, ")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 51, " settled automatically, ")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var30 string
|
var templ_7745c5c3_Var30 string
|
||||||
templ_7745c5c3_Var30, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(r.Adjudicated))
|
templ_7745c5c3_Var30, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(r.Adjudicated))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 142, Col: 35}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 160, Col: 35}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var30))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var30))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 44, " decided by the model (")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 52, " decided by the model (")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var31 string
|
var templ_7745c5c3_Var31 string
|
||||||
templ_7745c5c3_Var31, templ_7745c5c3_Err = templ.JoinStringErrs(r.Reconciler)
|
templ_7745c5c3_Var31, templ_7745c5c3_Err = templ.JoinStringErrs(r.Reconciler)
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 142, Col: 74}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 160, Col: 74}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var31))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var31))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 45, "), ")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 53, "), ")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var32 string
|
var templ_7745c5c3_Var32 string
|
||||||
templ_7745c5c3_Var32, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(r.Unsure))
|
templ_7745c5c3_Var32, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(r.Unsure))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 143, Col: 30}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 161, Col: 30}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var32))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var32))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 46, " uncertain.</p>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 54, " uncertain.</p>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
if len(r.Contested) > 0 {
|
if len(shownSpots(r)) > 0 {
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 47, "<details><summary>Spots to check against the audio</summary><ul>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 55, "<details><summary>Spots to check against the audio</summary><ul>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
for _, c := range shownSpots(r) {
|
for _, c := range shownSpots(r) {
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 48, "<li>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 56, "<li>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var33 string
|
var templ_7745c5c3_Var33 string
|
||||||
templ_7745c5c3_Var33, templ_7745c5c3_Err = templ.JoinStringErrs(minuteMark(c.Seconds))
|
templ_7745c5c3_Var33, templ_7745c5c3_Err = templ.JoinStringErrs(minuteMark(c.Seconds))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 151, Col: 33}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 169, Col: 33}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var33))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var33))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 49, ": ")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 57, ": ")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var34 string
|
if c.Before != "" {
|
||||||
templ_7745c5c3_Var34, templ_7745c5c3_Err = templ.JoinStringErrs(c.Base)
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 58, "<span class=\"muted\">…")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 151, Col: 45}
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var34))
|
var templ_7745c5c3_Var34 string
|
||||||
if templ_7745c5c3_Err != nil {
|
templ_7745c5c3_Var34, templ_7745c5c3_Err = templ.JoinStringErrs(c.Before)
|
||||||
return templ_7745c5c3_Err
|
if templ_7745c5c3_Err != nil {
|
||||||
}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 171, Col: 44}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 50, " / ")
|
}
|
||||||
if templ_7745c5c3_Err != nil {
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var34))
|
||||||
return templ_7745c5c3_Err
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 59, "</span> ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var35 string
|
var templ_7745c5c3_Var35 string
|
||||||
templ_7745c5c3_Var35, templ_7745c5c3_Err = templ.JoinStringErrs(c.Alt)
|
templ_7745c5c3_Var35, templ_7745c5c3_Err = templ.JoinStringErrs(c.Base)
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 151, Col: 57}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 173, Col: 18}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var35))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var35))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 51, " — ")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 60, " / ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
var templ_7745c5c3_Var36 string
|
||||||
|
templ_7745c5c3_Var36, templ_7745c5c3_Err = templ.JoinStringErrs(c.Alt)
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 173, Col: 30}
|
||||||
|
}
|
||||||
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var36))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 61, " — ")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
if c.Unsure {
|
if c.Unsure {
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 52, "<span class=\"chip chip-warn\">uncertain</span> kept ")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 62, "<span class=\"chip chip-warn\">uncertain</span> kept ")
|
||||||
if templ_7745c5c3_Err != nil {
|
|
||||||
return templ_7745c5c3_Err
|
|
||||||
}
|
|
||||||
var templ_7745c5c3_Var36 string
|
|
||||||
templ_7745c5c3_Var36, templ_7745c5c3_Err = templ.JoinStringErrs(c.Used)
|
|
||||||
if templ_7745c5c3_Err != nil {
|
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 153, Col: 70}
|
|
||||||
}
|
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var36))
|
|
||||||
if templ_7745c5c3_Err != nil {
|
|
||||||
return templ_7745c5c3_Err
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 53, "<span class=\"chip\">changed</span> used ")
|
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var37 string
|
var templ_7745c5c3_Var37 string
|
||||||
templ_7745c5c3_Var37, templ_7745c5c3_Err = templ.JoinStringErrs(c.Used)
|
templ_7745c5c3_Var37, templ_7745c5c3_Err = templ.JoinStringErrs(c.Used)
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 155, Col: 58}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 175, Col: 70}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var37))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var37))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
|
} else {
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 63, "<span class=\"chip\">changed</span> used ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
var templ_7745c5c3_Var38 string
|
||||||
|
templ_7745c5c3_Var38, templ_7745c5c3_Err = templ.JoinStringErrs(c.Used)
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 177, Col: 58}
|
||||||
|
}
|
||||||
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var38))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 54, "</li>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 64, "</li>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 55, "</ul>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 65, "</ul>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
if moreSpots(r) > 0 {
|
if moreSpots(r) > 0 {
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 56, "<p class=\"muted\">and ")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 66, "<p class=\"muted\">and ")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var38 string
|
var templ_7745c5c3_Var39 string
|
||||||
templ_7745c5c3_Var38, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(moreSpots(r)))
|
templ_7745c5c3_Var39, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(moreSpots(r)))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 161, Col: 57}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 183, Col: 57}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var38))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var39))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 57, " more not listed</p>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 67, " more not listed</p>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 58, "</details>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 68, "</details>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 59, "</div>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 69, "</div>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
case domain.ReconcileSingle:
|
case domain.ReconcileSingle:
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 60, "<p class=\"muted reconciliation\">Only one engine produced this transcript (")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 70, "<p class=\"muted reconciliation\">Only one engine produced this transcript (")
|
||||||
if templ_7745c5c3_Err != nil {
|
|
||||||
return templ_7745c5c3_Err
|
|
||||||
}
|
|
||||||
var templ_7745c5c3_Var39 string
|
|
||||||
templ_7745c5c3_Var39, templ_7745c5c3_Err = templ.JoinStringErrs(r.Reason)
|
|
||||||
if templ_7745c5c3_Err != nil {
|
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 167, Col: 88}
|
|
||||||
}
|
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var39))
|
|
||||||
if templ_7745c5c3_Err != nil {
|
|
||||||
return templ_7745c5c3_Err
|
|
||||||
}
|
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 61, "), so there was no second opinion to reconcile it with.</p>")
|
|
||||||
if templ_7745c5c3_Err != nil {
|
|
||||||
return templ_7745c5c3_Err
|
|
||||||
}
|
|
||||||
case domain.ReconcileSkipped:
|
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 62, "<p class=\"muted reconciliation\">Two transcripts were made but could not be reconciled (")
|
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var40 string
|
var templ_7745c5c3_Var40 string
|
||||||
templ_7745c5c3_Var40, templ_7745c5c3_Err = templ.JoinStringErrs(r.Reason)
|
templ_7745c5c3_Var40, templ_7745c5c3_Err = templ.JoinStringErrs(r.Reason)
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 169, Col: 101}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 189, Col: 88}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var40))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var40))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 63, "); the text shown is ")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 71, "), so there was no second opinion to reconcile it with.</p>")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
case domain.ReconcileSkipped:
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 72, "<p class=\"muted reconciliation\">Two transcripts were made but could not be reconciled (")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var41 string
|
var templ_7745c5c3_Var41 string
|
||||||
templ_7745c5c3_Var41, templ_7745c5c3_Err = templ.JoinStringErrs(r.Base)
|
templ_7745c5c3_Var41, templ_7745c5c3_Err = templ.JoinStringErrs(r.Reason)
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 169, Col: 132}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 191, Col: 101}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var41))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var41))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 64, "'s, unreconciled.</p>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 73, "); the text shown is ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
var templ_7745c5c3_Var42 string
|
||||||
|
templ_7745c5c3_Var42, templ_7745c5c3_Err = templ.JoinStringErrs(r.Base)
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 191, Col: 132}
|
||||||
|
}
|
||||||
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var42))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 74, "'s, unreconciled.</p>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
@@ -839,4 +916,287 @@ func reconciliationNote(r *domain.Reconciliation) templ.Component {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// progressSteps is the job's phases as a log that fills in as it runs: finished ones
|
||||||
|
// collapse to a line with how long they took, the running one has a spinner and its
|
||||||
|
// elapsed time, and what has not started is dimmed. live is false when the spinner would
|
||||||
|
// be a lie (the service is lost, or the job is over), so the running phase shows a
|
||||||
|
// warning mark instead. Only a counted phase gets a bar (ADR-035).
|
||||||
|
func progressSteps(steps []domain.ProgressStep, now time.Time, live bool) templ.Component {
|
||||||
|
return templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
|
||||||
|
templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
|
||||||
|
if templ_7745c5c3_CtxErr := ctx.Err(); templ_7745c5c3_CtxErr != nil {
|
||||||
|
return templ_7745c5c3_CtxErr
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Buffer, templ_7745c5c3_IsBuffer := templruntime.GetBuffer(templ_7745c5c3_W)
|
||||||
|
if !templ_7745c5c3_IsBuffer {
|
||||||
|
defer func() {
|
||||||
|
templ_7745c5c3_BufErr := templruntime.ReleaseBuffer(templ_7745c5c3_Buffer)
|
||||||
|
if templ_7745c5c3_Err == nil {
|
||||||
|
templ_7745c5c3_Err = templ_7745c5c3_BufErr
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
ctx = templ.InitializeContext(ctx)
|
||||||
|
templ_7745c5c3_Var43 := templ.GetChildren(ctx)
|
||||||
|
if templ_7745c5c3_Var43 == nil {
|
||||||
|
templ_7745c5c3_Var43 = templ.NopComponent
|
||||||
|
}
|
||||||
|
ctx = templ.ClearChildren(ctx)
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 75, "<ol class=\"steps\" aria-label=\"Progress\">")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
for _, s := range steps {
|
||||||
|
templ_7745c5c3_Var44 := []any{"step step-" + string(s.State)}
|
||||||
|
templ_7745c5c3_Err = templ.RenderCSSItems(ctx, templ_7745c5c3_Buffer, templ_7745c5c3_Var44...)
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 76, "<li class=\"")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
var templ_7745c5c3_Var45 string
|
||||||
|
templ_7745c5c3_Var45, templ_7745c5c3_Err = templ.ResolveAttributeValue(templ.CSSClasses(templ_7745c5c3_Var44).String())
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 1, Col: 0}
|
||||||
|
}
|
||||||
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var45)
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 77, "\"")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
if s.State == domain.StepRunning {
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 78, " aria-current=\"step\"")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 79, ">")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
if s.State == domain.StepRunning && live {
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 80, "<span class=\"step-mark\" aria-hidden=\"true\"><span class=\"spinner\"></span></span> ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
} else if s.State == domain.StepRunning {
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 81, "<span class=\"step-mark step-mark-warn\" aria-hidden=\"true\">!</span> ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 82, "<span class=\"step-mark\" aria-hidden=\"true\">")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
var templ_7745c5c3_Var46 string
|
||||||
|
templ_7745c5c3_Var46, templ_7745c5c3_Err = templ.JoinStringErrs(stepMark(s))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 215, Col: 61}
|
||||||
|
}
|
||||||
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var46))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 83, "</span> ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 84, "<span class=\"step-label\">")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
var templ_7745c5c3_Var47 string
|
||||||
|
templ_7745c5c3_Var47, templ_7745c5c3_Err = templ.JoinStringErrs(s.Label)
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 217, Col: 38}
|
||||||
|
}
|
||||||
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var47))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 85, "</span> ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
if t := stepTime(s, now); t != "" {
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 86, "<span class=\"step-time\">")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
var templ_7745c5c3_Var48 string
|
||||||
|
templ_7745c5c3_Var48, templ_7745c5c3_Err = templ.JoinStringErrs(t)
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 219, Col: 32}
|
||||||
|
}
|
||||||
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var48))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 87, "</span> ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if s.Note != "" {
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 88, "<span class=\"step-note\">")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
var templ_7745c5c3_Var49 string
|
||||||
|
templ_7745c5c3_Var49, templ_7745c5c3_Err = templ.JoinStringErrs(s.Note)
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 222, Col: 37}
|
||||||
|
}
|
||||||
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var49))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 89, "</span> ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if s.State == domain.StepRunning && s.Total > 0 {
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 90, "<progress class=\"step-bar\" value=\"")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
var templ_7745c5c3_Var50 string
|
||||||
|
templ_7745c5c3_Var50, templ_7745c5c3_Err = templ.ResolveAttributeValue(strconv.Itoa(s.Done))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 225, Col: 60}
|
||||||
|
}
|
||||||
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var50)
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 91, "\" max=\"")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
var templ_7745c5c3_Var51 string
|
||||||
|
templ_7745c5c3_Var51, templ_7745c5c3_Err = templ.ResolveAttributeValue(strconv.Itoa(s.Total))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 225, Col: 90}
|
||||||
|
}
|
||||||
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var51)
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 92, "\"></progress> <span class=\"step-count\">")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
var templ_7745c5c3_Var52 string
|
||||||
|
templ_7745c5c3_Var52, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(s.Done))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 226, Col: 52}
|
||||||
|
}
|
||||||
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var52))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 93, " of ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
var templ_7745c5c3_Var53 string
|
||||||
|
templ_7745c5c3_Var53, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(s.Total))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 226, Col: 81}
|
||||||
|
}
|
||||||
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var53))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 94, "</span>")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 95, "</li>")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 96, "</ol>")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// healthNote says so when the job is not simply running: the service stopped reporting,
|
||||||
|
// or a counted phase has gone quiet. Running says nothing: the spinner does.
|
||||||
|
func healthNote(h jobHealth, silent time.Duration) templ.Component {
|
||||||
|
return templruntime.GeneratedTemplate(func(templ_7745c5c3_Input templruntime.GeneratedComponentInput) (templ_7745c5c3_Err error) {
|
||||||
|
templ_7745c5c3_W, ctx := templ_7745c5c3_Input.Writer, templ_7745c5c3_Input.Context
|
||||||
|
if templ_7745c5c3_CtxErr := ctx.Err(); templ_7745c5c3_CtxErr != nil {
|
||||||
|
return templ_7745c5c3_CtxErr
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Buffer, templ_7745c5c3_IsBuffer := templruntime.GetBuffer(templ_7745c5c3_W)
|
||||||
|
if !templ_7745c5c3_IsBuffer {
|
||||||
|
defer func() {
|
||||||
|
templ_7745c5c3_BufErr := templruntime.ReleaseBuffer(templ_7745c5c3_Buffer)
|
||||||
|
if templ_7745c5c3_Err == nil {
|
||||||
|
templ_7745c5c3_Err = templ_7745c5c3_BufErr
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
ctx = templ.InitializeContext(ctx)
|
||||||
|
templ_7745c5c3_Var54 := templ.GetChildren(ctx)
|
||||||
|
if templ_7745c5c3_Var54 == nil {
|
||||||
|
templ_7745c5c3_Var54 = templ.NopComponent
|
||||||
|
}
|
||||||
|
ctx = templ.ClearChildren(ctx)
|
||||||
|
switch h {
|
||||||
|
case healthLost:
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 97, "<p class=\"step-health step-health-lost\" role=\"alert\">The service stopped reporting ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
var templ_7745c5c3_Var55 string
|
||||||
|
templ_7745c5c3_Var55, templ_7745c5c3_Err = templ.JoinStringErrs(minutesAgo(silent))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 239, Col: 54}
|
||||||
|
}
|
||||||
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var55))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 98, " ago. It may have restarted or crashed. This page keeps checking; if nothing changes, upload the recording again.</p>")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
case healthQuiet:
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 99, "<p class=\"step-health step-health-quiet\">Nothing new for ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
var templ_7745c5c3_Var56 string
|
||||||
|
templ_7745c5c3_Var56, templ_7745c5c3_Err = templ.JoinStringErrs(minutesAgo(silent))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 244, Col: 40}
|
||||||
|
}
|
||||||
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var56))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 100, ": the model may be slow. The service is still running, and the job gives up on its own if it takes too long.</p>")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
var _ = templruntime.GeneratedTemplate
|
var _ = templruntime.GeneratedTemplate
|
||||||
|
|||||||
@@ -765,6 +765,32 @@ a.btn, a.btn:visited { color: var(--accent-fg); }
|
|||||||
.flash-success { background: var(--accent-weak); color: var(--accent); border-color: var(--accent); }
|
.flash-success { background: var(--accent-weak); color: var(--accent); border-color: var(--accent); }
|
||||||
.flash-error { background: var(--err-bg); color: var(--err-fg); border-color: var(--err-line); }
|
.flash-error { background: var(--err-bg); color: var(--err-fg); border-color: var(--err-line); }
|
||||||
|
|
||||||
|
/* upload progress (ADR-035): a log of phases that fills in as the job runs */
|
||||||
|
.steps { list-style: none; margin: var(--s3) 0; padding: 0; font-family: var(--mono); font-size: .86rem; }
|
||||||
|
.step { display: flex; align-items: baseline; gap: var(--s2); padding: .2rem 0; color: var(--muted); flex-wrap: wrap; }
|
||||||
|
.step-mark { flex: none; width: 1.1rem; text-align: center; }
|
||||||
|
.step-label { flex: 1 1 auto; }
|
||||||
|
.step-time { margin-left: auto; color: var(--muted); font-weight: 400; font-variant-numeric: tabular-nums; }
|
||||||
|
.step-done { color: var(--fg); }
|
||||||
|
.step-done .step-mark { color: var(--mint); }
|
||||||
|
.step-running { color: var(--fg); font-weight: 600; }
|
||||||
|
.step-running .step-mark { color: var(--accent); }
|
||||||
|
.step-failed { color: var(--err-fg); }
|
||||||
|
.step-skipped, .step-pending { color: var(--muted); }
|
||||||
|
.step-skipped { opacity: .7; }
|
||||||
|
.step-pending { opacity: .75; }
|
||||||
|
.step-bar { flex: none; width: calc(100% - 1.6rem); margin: 0 0 0 1.6rem; height: .5rem; accent-color: var(--accent); }
|
||||||
|
.step-note { flex: none; width: calc(100% - 1.6rem); margin-left: 1.6rem; font-weight: 400; color: var(--muted); font-family: system-ui, -apple-system, sans-serif; font-size: .82rem; }
|
||||||
|
.step-count { flex: none; width: calc(100% - 1.6rem); margin-left: 1.6rem; font-weight: 400; color: var(--muted); }
|
||||||
|
.step-running .step-mark-warn { color: var(--err-fg); }
|
||||||
|
.spinner { display: inline-block; vertical-align: -.1em; width: .8rem; height: .8rem; box-sizing: border-box; border: 2px solid var(--line); border-top-color: var(--accent); border-radius: 50%; animation: tapir-spin 1s linear infinite; }
|
||||||
|
@keyframes tapir-spin { to { transform: rotate(360deg); } }
|
||||||
|
@media (prefers-reduced-motion: reduce) { .spinner { animation: none; border-color: var(--accent); } }
|
||||||
|
.step-health { margin: var(--s2) 0; padding: var(--s2) var(--s3); border-radius: var(--radius); font-size: .9rem; border: 1px solid var(--line); }
|
||||||
|
.step-health-lost { background: var(--err-bg); color: var(--err-fg); border-color: var(--err-line); }
|
||||||
|
.step-health-quiet { background: var(--accent-weak); color: var(--fg); border-color: var(--accent); }
|
||||||
|
.what-was-done summary { cursor: pointer; color: var(--muted); }
|
||||||
|
|
||||||
/* htmx loading feedback */
|
/* htmx loading feedback */
|
||||||
.htmx-indicator { opacity: 0; transition: opacity .2s; color: var(--muted); font-size: .8rem; }
|
.htmx-indicator { opacity: 0; transition: opacity .2s; color: var(--muted); font-size: .8rem; }
|
||||||
.htmx-request .htmx-indicator, .htmx-request.htmx-indicator { opacity: 1; }
|
.htmx-request .htmx-indicator, .htmx-request.htmx-indicator { opacity: 1; }
|
||||||
|
|||||||
@@ -134,6 +134,28 @@ var scenarioCoverage = map[string]string{
|
|||||||
"A cancelled job never reaches the cloud": "TestEnsemble_ACancelledJobNeverReachesTheCloud",
|
"A cancelled job never reaches the cloud": "TestEnsemble_ACancelledJobNeverReachesTheCloud",
|
||||||
"A single-engine transcript claims no second opinion": "TestUploadStatus_ASingleEngineTranscriptDoesNotClaimASecondOpinion",
|
"A single-engine transcript claims no second opinion": "TestUploadStatus_ASingleEngineTranscriptDoesNotClaimASecondOpinion",
|
||||||
|
|
||||||
|
// audio_upload.feature, ADR-035
|
||||||
|
"I can see which phase my upload is in": "TestUploadProgress_ARunningJobShowsEveryPhaseInOrderWithTheRunningOneMarked",
|
||||||
|
"A bar appears only where the count is real": "TestUploadProgress_OnlyACountedPhaseGetsABarAndItShowsRealCounts",
|
||||||
|
"A slow engine call is not mistaken for a dead service": "TestUploadProgress_ALongEngineCallWithAFreshHeartbeatIsStillJustRunning",
|
||||||
|
"A service that stopped reporting is called lost": "TestUploadProgress_AHeartbeatThatStoppedMeansTheServiceIsLostAndTheJobIsNotCalledRunning",
|
||||||
|
"A model that stops advancing is called quiet": "TestUploadProgress_ACountedPhaseThatStopsAdvancingGoesQuietNotLost",
|
||||||
|
"A failed job shows where it stopped": "TestUploadProgress_AFailedJobShowsWhichPhaseFailedAndWhatNeverStarted",
|
||||||
|
"A finished job keeps its record of what was done": "TestUploadProgress_ADoneJobKeepsItsRecordOfWhatWasDoneAndHowLongEachTook",
|
||||||
|
"The heartbeat runs on its own while an engine call is stuck": "TestWorkerProgress_TheHeartbeatBeatsWhileAnEngineCallIsStuckAndStopsWhenTheJobEnds",
|
||||||
|
"The log names every phase and never the transcript": "TestWorkerProgress_OnASuccessfulJobTheLogNamesEveryPhaseInOrderAndNeverTheTranscriptOrThePath",
|
||||||
|
"Every judged spot is stored but only the ones to check are listed": "TestUploadStatus_OnlyChangedAndUncertainSpotsAreListedNotTheKeptOnes",
|
||||||
|
"Chat about a recording is offered with local models only": "TestUploadChatPage_DoesNotOfferACloudModelEvenThoughTheServiceKnowsOne",
|
||||||
|
"A forged cloud model is never sent the transcript": "TestUploadChatPost_AForgedCloudModelIsReplacedByALocalOneBeforeTheTranscriptIsSent",
|
||||||
|
"A wedged engine does not use up the whole job": "TestChain_AHungPrimaryReachesTheFallbackWhenItHasItsOwnDeadline",
|
||||||
|
"A file that lies about its length is refused": "TestWorkerDuration_AConvertedLengthOverTheLimitFailsTheJobBeforeAnyEngineSeesIt",
|
||||||
|
"A recording much shorter than its file claimed is transcribed and noted": "TestWorkerDuration_AFileThatEndsFarEarlierThanItClaimedIsStillTranscribedAndSaysSo",
|
||||||
|
"A full queue is a retry, not an error": "TestAccept_AFullQueueIsABusyRejectionNotAServerErrorAndLeavesNothingBehind",
|
||||||
|
"One user cannot fill the queue for everyone": "TestWorkerSubmit_OneUserCannotFillTheQueueForEveryoneElse",
|
||||||
|
"Errors are shown as kinds, never their text": "TestReconcile_TheBreakerReasonCarriesAKindNotTheModelsText",
|
||||||
|
"Input that cannot be right is a client error": "TestUploadRoutes_ANonUUIDIdIs404NotA500",
|
||||||
|
"The saved segments carry the merged words": "TestReconcileSegments_AChangedWordReplacesTheBaseWordInItsOwnSegmentOnly",
|
||||||
|
|
||||||
// summarize_new_video.feature
|
// summarize_new_video.feature
|
||||||
"A subscribed channel posts a video that has captions": "TestSubscribedVideoWithCaptionsIsSummarizedAndDelivered",
|
"A subscribed channel posts a video that has captions": "TestSubscribedVideoWithCaptionsIsSummarizedAndDelivered",
|
||||||
"A subscribed channel posts a video with no usable transcript": "TestVideoWithNoTranscriptIsSkipped",
|
"A subscribed channel posts a video with no usable transcript": "TestVideoWithNoTranscriptIsSkipped",
|
||||||
|
|||||||
Reference in New Issue
Block a user