Upload a recording (audio, or video with the camera off), get a private Swedish transcript, and chat about it. - ADR-033: RLS-scoped private transcript store; ffprobe/ffmpeg intake; gateway transcription (iguana kb-whisper, berget fallback); in-process single-replica worker; chat over the transcript (local models only). - ADR-034: optional two-engine mode (pianissimo + kb-whisper) reconciled by a LOCAL model; off unless TAPIR_RECONCILE_BASE_MODEL is set. - ADR-035: real progress (phases, running/quiet/lost), phase logging, every judged spot stored, merged segments. - Hardening from four adversarial reviews: errors logged and stored as kinds never text; per-engine deadlines; converted-length check; bounded queue; client hang-ups are not errors; strict local-alias check. - Dockerfile: static ffmpeg/ffprobe 7.0.2 pinned by sha256 (#34). Built and inspected by hand; CI does not build the image on a PR, so its first CI build is this merge. Uploads stay OFF until TAPIR_UPLOADS_ENABLED is set in the deployment. Closes #33, #34, #42, #43, #44. Co-authored-by: Mathias <mthbqv@gmail.com>
This commit was merged in pull request #39.
This commit is contained in:
+40
-3
@@ -18,7 +18,12 @@ only opaque references to them; the secret material lives in ESO/1Password (ADR-
|
||||
not LLM re-summarization but a rate-gated, reputation-risky caption fetch (ADR-010/014), which
|
||||
is paid per re-fetch regardless of user count — so persisting public caption content once and
|
||||
sharing it strictly beats the coupling it removes. Everything else each user owns is
|
||||
self-contained; `rls_test.go` proves transcripts is the single exception.
|
||||
self-contained; `rls_test.go` proves transcripts is the single exception for *content*.
|
||||
**Uploaded-audio transcripts are the opposite of public caption content** (private, owned by
|
||||
one user) and therefore live in their own RLS-scoped `uploaded_transcripts` table, never in
|
||||
`transcripts` (ADR-033). The isolation test now also pins the **exact set** of tables with row
|
||||
security off (`transcripts`, `user_identities`, `invitations`, `schema_migrations`), so a new
|
||||
table added without RLS fails CI.
|
||||
- **Secrets by reference only.** Tables hold a `secret_ref` (opaque string/UUID resolved via
|
||||
the `SecretStore` port), never tokens or keys.
|
||||
- **The brain sink is just a delivery target.** No brain-specific tables. Whether a summary
|
||||
@@ -41,6 +46,7 @@ erDiagram
|
||||
VIDEO ||--o| SUMMARY : "has at most one"
|
||||
SUMMARY ||--o{ SINK_DELIVERY : "delivered via"
|
||||
USER ||--o{ CHANNEL_ERROR : "reports unavailable channels"
|
||||
USER ||--o{ UPLOADED_TRANSCRIPT : "uploads (private, ADR-033)"
|
||||
|
||||
USER {
|
||||
uuid id PK
|
||||
@@ -136,12 +142,32 @@ erDiagram
|
||||
timestamptz first_seen
|
||||
timestamptz last_seen
|
||||
}
|
||||
UPLOADED_TRANSCRIPT {
|
||||
uuid id PK
|
||||
uuid user_id FK "-> USER, ON DELETE CASCADE, NOT NULL; RLS ENABLE+FORCE (migration 017)"
|
||||
text title
|
||||
text status "queued | transcribing | done | failed"
|
||||
text content "nullable until done; the transcript text"
|
||||
jsonb segments "nullable until done; [{start,end,text}] in seconds"
|
||||
text endpoint "which backend served it: iguana/... | berget/... (berget = audio left the homelab)"
|
||||
float audio_seconds
|
||||
text error_kind "typed: interrupted | transcription_failed — never a provider string"
|
||||
bool had_video "the file carried a video track; only its audio was used (migration 018)"
|
||||
jsonb reconciliation "NULL for a single engine; else status, exact counts and a bounded list of every judged spot (migration 019, ADR-034; all spots stored, ADR-035)"
|
||||
jsonb progress "the job's ordered phases: key, label, state, times, done/total; NULL until the job starts (migration 020, ADR-035)"
|
||||
timestamptz progress_at "when a phase last really changed or advanced (migration 020)"
|
||||
timestamptz alive_at "heartbeat from the running job, every ~10 s; not progress (migration 020)"
|
||||
timestamptz created_at
|
||||
timestamptz finished_at
|
||||
}
|
||||
```
|
||||
|
||||
`SUMMARY_ACTION` has `UNIQUE (user_id, video_id, action)`; `VIDEO_CONNECTION` has
|
||||
`UNIQUE (user_id, provider)` (one connection per provider — reconnect upserts in place).
|
||||
RLS (`ENABLE` + `FORCE`) is on **every solid user-owned table above** — `users`, `videos`,
|
||||
`transcripts`, `summaries`, `summary_actions`, `video_connections`, `channel_errors`.
|
||||
`summaries`, `summary_actions`, `video_connections`, `channel_errors`, `uploaded_transcripts`.
|
||||
(`transcripts` is deliberately **not** in this list: since ADR-021 it is shared public caption
|
||||
content with no `user_id`.)
|
||||
`sink_deliveries` is RLS'd via an `EXISTS` on its parent summary; `user_identities` is
|
||||
intentionally **not** RLS'd (auth plumbing). See the *Isolation invariant* section for the
|
||||
mechanism.
|
||||
@@ -197,6 +223,14 @@ mechanism.
|
||||
the most recent discovery pass. Upserted per scheduler pass (`last_seen` refreshed each run);
|
||||
surfaced on the account page as a warning. Cascades on user deletion. Primary key is
|
||||
`(user_id, channel_id)`. FORCE RLS'd.
|
||||
- **UPLOADED_TRANSCRIPTS** (migration 017, ADR-033) — the **private** transcript of audio a user
|
||||
uploaded. Deliberately a separate table from the shared `transcripts`: that one is non-RLS on the
|
||||
grounds that it holds only public caption content, and a recorded meeting is the opposite.
|
||||
`user_id NOT NULL ... ON DELETE CASCADE`, FORCE RLS'd. `status` runs `queued → transcribing →
|
||||
done | failed`; `done` requires `content`, `failed` requires a typed `error_kind` (CHECK
|
||||
constraints). `reconciliation` (migration 019, ADR-034) is NULL when one engine produced the text, so the page can never claim a second opinion that was not there; otherwise it holds the status (`reconciled` / `single_engine` / `skipped`), exact counts (differences, settled automatically, decided by the model, changed, uncertain, model errors) and up to 2,000 judged spots (ADR-035: every one the model decided, kept or changed or unsure, each a position in seconds, both readings, which was used, and the few words before it; the page lists only the changed and uncertain ones). `progress`, `progress_at` and `alive_at` (migration 020, ADR-035) let the page show the job's phases and tell a running job from a quiet one from a lost service: `alive_at` is a heartbeat from a goroutine independent of any engine call, `progress_at` moves only when work really moved; the saved `segments` carry the reconciled words with the base engine's timing. The spots hold fragments of the private transcript, which is why they live in this RLS-scoped row. `had_video` (migration 018) records that the file carried a video track — a Teams meeting with the camera off is still a video file — so the page can say only its audio was used and the video was discarded. `endpoint` records which backend served it — `berget/...` means the audio left the
|
||||
homelab. The audio itself is never stored. `segments` keeps the timestamps (re-transcribing 90
|
||||
minutes costs ~7). Unlike `summary_actions`/`login_events` it **does** cascade on user deletion.
|
||||
- **LOGIN_EVENTS** (migration 010) — throttled one-row-per-(user, date) login stamp. Used by the
|
||||
Stage-0 gate query (VISION §Stage 0: "returned and used in ≥2 distinct weeks").
|
||||
|
||||
@@ -209,7 +243,10 @@ enforcement dormant); **ADR-012 opened Stage 1 and turned enforcement on in the
|
||||
Enforcement is **Postgres Row-Level Security** (migration `003_rls.up.sql`):
|
||||
|
||||
- RLS is `ENABLE`d **and** `FORCE`d on every user-owned table — `users`, `videos`,
|
||||
`transcripts`, `summaries`, `summary_actions`, `video_connections`, `channel_errors`.
|
||||
`summaries`, `summary_actions`, `video_connections`, `channel_errors`, and (migration 017)
|
||||
`uploaded_transcripts`. `transcripts` is **not** one of them: ADR-021 made it shared public
|
||||
content, and migration 003's original per-user `transcripts` policy was dropped with that
|
||||
table in migration 015.
|
||||
`FORCE` is load-bearing: the app connects as the table **owner** (`tapir` role), and owners
|
||||
bypass RLS unless forced.
|
||||
- Each policy keys off the per-request GUC `tapir.current_user_id`, set transaction-locally by
|
||||
|
||||
@@ -12,6 +12,28 @@ it** — endpoints and aliases drift, and this file is a snapshot (2026-06-06),
|
||||
- **Off-cluster (host / NodePort):** `http://koala:30401/v1/` — service `litellm-nodeport` (`4000:30401`).
|
||||
- NOTE: the earlier `31234` was **wrong** — `31234` is the `llama-swap` NodePort (`8080:31234`),
|
||||
not LiteLLM. Use `30401` for off-cluster LiteLLM. `confirm` resolved.
|
||||
- **Speech-to-text (ADR-033):** the same gateway serves `POST /v1/audio/transcriptions` for
|
||||
`iguana/kb-whisper` (primary, local) and `berget/kb-whisper` (fallback, EU cloud — private audio
|
||||
leaves the homelab if it is used). **Call it via the NodePort / in-cluster address above, NOT
|
||||
`https://llm-api.d-ma.be`:** that public ingress (openresty) returns `413` for bodies over
|
||||
roughly 8–10 MB (~5 minutes of 16 kHz WAV), while the NodePort took 174 MB (90 minutes) in one
|
||||
request (measured 2026-10-01, `docs/research/audio-length-ceiling-2026-10-01.md`). Send multipart
|
||||
field `file`, `response_format=verbose_json`, no `Expect: 100-continue`. iguana's whisper-server
|
||||
can wedge permanently after a Metal OOM (infra#273); never branch on its
|
||||
`{"error":"failed to process audio"}` string — it has had at least five causes. Tapir's knobs:
|
||||
`TAPIR_UPLOADS_ENABLED` (off by default), `TAPIR_UPLOAD_MAX_MB` (200), `TAPIR_UPLOAD_MAX_MINUTES`
|
||||
(90), `TAPIR_TRANSCRIBE_MODEL`, `TAPIR_TRANSCRIBE_FALLBACK_MODEL` (empty disables the cloud
|
||||
fallback), `TAPIR_TRANSCRIBE_TIMEOUT` (60m, the whole job), `TAPIR_TRANSCRIBE_ENGINE_TIMEOUT` (20m, one engine call: a wedged engine then fails while the job lives and the fallback can run), `TAPIR_UPLOAD_TMP_DIR`, `TAPIR_FFMPEG_BIN`,
|
||||
`TAPIR_FFPROBE_BIN`, `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS` (90000 — the measured ceiling of
|
||||
`iguana/gemma4-26b`, which silently drops the head above it), `TAPIR_UPLOAD_CHAT_MODEL`
|
||||
(`iguana/gemma4-26b`; must be a local `koala/` or `iguana/` alias, since the whole private transcript is sent to it; `koala/phi4-mini`'s 8,192-token context 400s on an hour of speech). `ffmpeg` + `ffprobe` are in the runtime image (static, pinned by sha256; #34 decided A on 2026-10-02); Tapir's own ingress body limit for a 200 MB upload is unverified (infra).
|
||||
**Two-engine mode (ADR-034), off unless `TAPIR_RECONCILE_BASE_MODEL` is set:** it runs
|
||||
`iguana/pianissimo` (the base text) and `TAPIR_TRANSCRIBE_MODEL` (kb-whisper) on the same audio, then a
|
||||
LOCAL chat model (`TAPIR_RECONCILE_MODEL`) chooses between their disagreements. **Deployment choice (2026-10-02): `iguana/gemma4-31b`** (97% against a speaker's blind answers; 92% for `iguana/gemma4-26b`). With iguana's macOS **Low Power Mode off** the 31B decodes at 22 tok/s and reconciled a 63.6-minute recording in 21 min 49 s end to end; with it on it ran at 8 tok/s. Check `pmset -g | grep lowpowermode` before blaming a model or an engine for slowness.
|
||||
`TAPIR_RECONCILE_CHECK_ORDER=true` asks twice per dispute with the order swapped (doubles the time).
|
||||
Every model must start `koala/` or `iguana/`: a cloud alias is a start-up error. The default job timeout is
|
||||
60 minutes. `iguana/pianissimo` is KlangAI's Swedish ASR (a Parakeet/FastConformer on MLX, served by a small
|
||||
FastAPI shell on iguana:8102) and speaks the same `/v1/audio/transcriptions` API.
|
||||
- **Auth key:** treat as a secret via the SecretStore port; do not hardcode in committed code.
|
||||
NOTE: the documented `sk-local-123` returned `401` on `30401/v1/models` (2026-06-02) — it is
|
||||
stale or the gateway now enforces the `LITELLM_MASTER_KEY` (1Password `op://HomeLab/LITELLM_MASTER_KEY`).
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
# kb-whisper vs pianissimo on one real hour of Swedish speech (tapir#33, 2026-10-01)
|
||||
|
||||
One recording, one speaker: a 63.7-minute Teams session of the maintainer dictating project
|
||||
notes (domain-heavy, English loanwords and product names mixed into Swedish). **No transcript
|
||||
text and no domain terms are recorded here**: this repo is public and the content is private.
|
||||
Only aggregate numbers.
|
||||
|
||||
Backends, both through the LiteLLM gateway and Tapir's own adapter, unchanged:
|
||||
`iguana/kb-whisper` (whisper.cpp, KB's Swedish fine-tune) and `iguana/pianissimo` (KlangAI
|
||||
Swedish Parakeet/FastConformer-TDT, MLX 8-bit, served by a small FastAPI shell on iguana:8102).
|
||||
|
||||
## Result: a near tie on accuracy, a large difference in speed
|
||||
|
||||
| | kb-whisper | pianissimo |
|
||||
|---|---|---|
|
||||
| Wall time for 63.7 min | 4m08s (~15x realtime) | **40 s (~95x realtime)** |
|
||||
| Segments | 1,302 (timed pieces, wrapped at ~60 chars) | 434 (sentences) |
|
||||
| Words (normalised) | 6,606 | 6,510 |
|
||||
| Punctuation | light, every segment ends in a full stop | heavier (138 vs 82 commas) |
|
||||
| Repetition loops | none | none |
|
||||
|
||||
Word-level agreement between them: **92.4%** as first computed, **93.1%** after the segment
|
||||
rejoin fix below. 380 aligned disagreement spans, median one word each.
|
||||
|
||||
**Blind A/B by the speaker** (the only ground truth available), on the 30 spans a model
|
||||
(Claude Opus 5.5) had picked as changing meaning, shuffled and unlabelled: **pianissimo 17,
|
||||
kb-whisper 13**, no "neither", no "can't tell".
|
||||
|
||||
| Category of the 30 | pianissimo | kb-whisper |
|
||||
|---|---|---|
|
||||
| Technical term / English loanword | 8 | 4 |
|
||||
| Number / date / unit | 4 | 1 |
|
||||
| Name / proper noun | 3 | 3 |
|
||||
| Function word / small word | 1 | 2 |
|
||||
| Other (everyday words) | 1 | 3 |
|
||||
|
||||
The reading: pianissimo is better on terminology and numbers, kb-whisper on ordinary Swedish
|
||||
words; names are even. Their errors differ in kind: kb-whisper's tend to be garbled words, a
|
||||
couple were wrong numbers; pianissimo's tend to be fluent substitutions that read as plausible
|
||||
(a wrong institution, an invented person, a dropped sign), which a reader or an LLM is less
|
||||
likely to catch.
|
||||
|
||||
Opus's own lean agreed with the speaker's blind answer on **30 of 30**. That is not a measure
|
||||
of Opus over all 380 spans: it chose spans it could decide, and every one of the 30 carried a
|
||||
lean. Its full classification of the 380 (counts only): function/small word 81, filler or
|
||||
false start 65, technical term 58, compound split/join 46, other 39, name 28, casing or
|
||||
punctuation residue 26, inflection 23, number/date/unit 10, content omission or insertion
|
||||
(>= 3 words) 4.
|
||||
|
||||
## A bug this comparison found in Tapir (fixed)
|
||||
|
||||
kb-whisper's output is wrapped at ~60 characters and **a wrap can fall inside a word**; the
|
||||
continuation segment has no leading space, the response's `text` field joins segments with
|
||||
newlines, and a segment can be a lone `.`. Tapir stored that joined text, so chat received
|
||||
broken words. Measured on real gateway output: 7% of segments (96/1,332) in a synthetic hour,
|
||||
42/1,302 lines in the real one. berget's segments are whole sentences with no leading space, so
|
||||
the two need different joins. `transcribe.joinSegments` now rebuilds the text from the segments.
|
||||
Checked against the real transcript with pianissimo's vocabulary as an independent yardstick:
|
||||
36 spans changed, 31 now entirely known words, words unknown to pianissimo 224 -> 177.
|
||||
Five rejoined spans still contain an unknown word (a name or an unlucky glue): small residual
|
||||
risk. Earlier tests used short sentences and could not see this.
|
||||
|
||||
## What this does not show
|
||||
|
||||
- **One recording, one speaker, one domain.** No claim about other voices, accents, crosstalk,
|
||||
or room noise. 17 vs 13 is not a significant difference.
|
||||
- **The other ~350 spans** were not judged by the speaker. They are mostly spelling, compounds,
|
||||
fillers and small words; Opus's counts there are a model's judgement, not ground truth.
|
||||
- Errors **both** models share in text they agree on are invisible to this method.
|
||||
- Speed was measured once each, on an otherwise idle iguana; pianissimo's 40 s includes the
|
||||
gateway hop and a 122 MB WAV upload.
|
||||
- Whether pianissimo survives the infra#273 Metal-OOM contention the way whisper does not was
|
||||
not tested; both run on iguana.
|
||||
|
||||
## What it suggests for Tapir (not decided, filed as an issue)
|
||||
|
||||
Pianissimo is a **second local backend on the same host, through the same adapter**. That
|
||||
makes it a natural middle tier ahead of the cloud fallback (kb-whisper -> pianissimo ->
|
||||
berget), so a wedged whisper no longer sends private audio off the homelab. At 40 s an hour it
|
||||
is also cheap enough to run **both** on every upload and give chat both texts, or flag where
|
||||
they disagree; the maintainer's blind answers show they err in different places.
|
||||
@@ -0,0 +1,110 @@
|
||||
# Audio length ceiling — measured before designing chunking (tapir#33, 2026-10-01)
|
||||
|
||||
Question (#33, "the main technical risk"): at what recording length does Swedish
|
||||
transcription through the LiteLLM gateway first fail, and with what failure mode?
|
||||
Measured **before** any chunking design, with iguana not holding a large ollama model,
|
||||
so the result is the ceiling and not the Metal OOM of infra#273.
|
||||
|
||||
## Answer
|
||||
|
||||
**No transcription ceiling was found at or below 90 minutes.** A single request carrying
|
||||
90 minutes of 16 kHz mono PCM (174 MB) transcribed in one pass in 440 s. The first thing
|
||||
that actually fails is **not whisper**: it is the public gateway ingress, which rejects the
|
||||
upload with `413` at roughly 5.5 minutes of WAV.
|
||||
|
||||
| Route | First failure | Failure mode |
|
||||
|---|---|---|
|
||||
| `https://llm-api.d-ma.be` (openresty ingress) | between 8.0 MB and 10.5 MB body (~4–5.5 min of WAV) | `413 Request Entity Too Large`, HTML body from `openresty`, returned instantly |
|
||||
| `http://koala:30401` (LiteLLM NodePort) → `iguana/kb-whisper` | **none up to 90 min / 174 MB** | — |
|
||||
| `http://koala:30401` → `berget/kb-whisper` | **none up to 22 min / 42 MB** (largest tried) | — |
|
||||
|
||||
The ingress limit is the reason the 8:20 / 15 MB recording in #33 "failed repeatedly" and a
|
||||
30-second slice succeeded. That failure was attributed to a Metal OOM; at least in the
|
||||
size-dependent form described, a body-size cap explains it without one. (The OOM
|
||||
mechanism in infra#273 is separately real; this run did not provoke it.)
|
||||
|
||||
The exact ingress cap was bracketed (8.0 MB passed, 10.5 MB failed), not read from the
|
||||
ingress configuration, which lives in `infra`. "About 10 MB" is an inference.
|
||||
|
||||
## Method
|
||||
|
||||
- **Audio.** No known-good Swedish recording is available that is safe to use: the only real
|
||||
one is a private recording (#31 custody still open) and was
|
||||
deliberately not touched. Instead: macOS `say -v Alva` (Swedish) on iguana, a 1,171-char
|
||||
five-paragraph meeting script → 82.5 s of 16 kHz mono PCM WAV (`ffmpeg -ar 16000 -ac 1`).
|
||||
Concatenated ×4, ×8, ×16, ×32, ×44, ×66 with Python `wave` (the issue's own recipe).
|
||||
Nothing here is a real transcript; none of it is committed.
|
||||
- **Request.** `POST /v1/audio/transcriptions`, multipart `file=`, `model=iguana/kb-whisper`,
|
||||
`response_format=verbose_json`, `-H 'Expect:'` (per #33's traps), master key supplied only
|
||||
through `op run` and never on argv.
|
||||
- **Confound control.** `ollama /api/ps` on iguana was read before every step. Resident:
|
||||
nothing, or `nomic-embed-text` (0.37 GB). Never a large model.
|
||||
|
||||
## Results (`iguana/kb-whisper`, NodePort)
|
||||
|
||||
| Audio | WAV size | HTTP | Wall | Realtime factor | Segments | Chars |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0:30 | 0.96 MB | 200 | 2.8 s | 11× | — | — |
|
||||
| 2:30 / 3:20 / 4:10 | 4.8 / 6.4 / 8.0 MB | 200 (ingress) | 12.9 / 16.0 / 19.9 s | 12× | — | — |
|
||||
| 5:30 (×4) | 10.5 MB | **413 ingress**, 200 NodePort in 27 s | 27 s | 12× | 121 | 4,557 |
|
||||
| 11:00 (×8) | 21.1 MB | 200 | 53 s | 12× | 241 | 9,094 |
|
||||
| 22:00 (×16) | 42.3 MB | 200 | 105 s | 13× | 482 | 18,201 |
|
||||
| 44:01 (×32) | 84.5 MB | 200 | 209 s | 13× | 966 | 36,438 |
|
||||
| 60:32 (×44) | 116.2 MB | 200 | 291 s | 12× | 1,332 | 50,145 |
|
||||
| 90:48 (×66) | 174.3 MB | 200 | 440 s | 12× | 2,005 | 75,165 |
|
||||
|
||||
- Throughput is flat at ~12× realtime. 90 minutes costs ~7.5 minutes of wall time. This is
|
||||
a minutes-long job, not a request/response one (see ADR-033 on ADR-018).
|
||||
- Segments per source repetition stay at 30.1–30.4 from ×4 to ×66, and segment times are
|
||||
monotonic with `last_end` within 0.2 s of `duration`. No drift and no repetition-loop
|
||||
hallucination appeared across 90 minutes. This is a weak check — the input is a repeated
|
||||
loop and synthetic voice — not evidence of accuracy.
|
||||
- **Only `iguana/kb-whisper`'s `x4` run reported `last_end` 360.0 s against `duration`
|
||||
330.2 s.** Whisper's final segment can end past the audio; clamp to `duration`.
|
||||
|
||||
## Findings that change the design
|
||||
|
||||
1. **Use the NodePort route, not the public hostname.** Tapir runs in-cluster beside the
|
||||
gateway; `llm-api.d-ma.be` is the external door and has a body cap. #33 said "use the
|
||||
gateway, never the host" and named the public URL; the gateway is right, that URL is not.
|
||||
`TAPIR_GATEWAY_URL` already points at `koala:30401/v1`.
|
||||
2. **`response_format=srt` survives the LiteLLM hop, but wrapped.** The response is JSON
|
||||
`{"text": "<the SRT>", "usage": …}`, not a bare SRT body — a client that parses the body
|
||||
as SRT gets a parse error on `{`. #33 question 2 is answered: it passes through, inside
|
||||
an envelope.
|
||||
3. **The two backends return different timestamp granularity.** `iguana/kb-whisper`
|
||||
segments are **whole seconds** (`0→1`, `1→4`, `4→7`); `berget/kb-whisper` returns real
|
||||
sub-second times (`0.051→1.192`) and a different `language` token (`swedish` vs `sv`).
|
||||
`verbose_json` and the default JSON were byte-identical on iguana. An adapter that
|
||||
falls back between the two must normalise both, and "which endpoint served this" is
|
||||
a quality fact, not just a log line.
|
||||
4. **The existing chat path cannot answer about most of a long recording.** At
|
||||
828 chars/min (this fixture; real speech is plausibly a little denser),
|
||||
`TAPIR_MAX_TRANSCRIPT_CHARS`'s default of **18,000** is ~21.7 minutes. A 60-minute
|
||||
transcript is 50 k chars, a 90-minute one 75 k. `chat.Service` truncates by keeping
|
||||
the head (`internal/adapters/chat/chat.go`), so questions about the last two-thirds of
|
||||
a one-hour meeting get an answer from a transcript that does not contain it. #33 says
|
||||
chat "retrieves rather than generates" and therefore has no ceiling; it **stuffs and
|
||||
truncates**, and its ceiling is the model context. See ADR-033.
|
||||
5. **Chunking is not needed for v1 on length.** Nothing failed up to 90 minutes. The
|
||||
chunking-and-stitching recipe in #33 (offset timestamps, renumber cues, strip `\r`) is
|
||||
not built. It stays the answer *if* a longer cap is wanted or the OOM mechanism forces
|
||||
smaller requests — see ADR-033's resolution triggers.
|
||||
|
||||
## What this did not measure
|
||||
|
||||
- **Metal OOM under contention.** Deliberately avoided. A big resident ollama model
|
||||
during a long transcription is still the known way to wedge iguana permanently.
|
||||
- **Real speech.** TTS is clean, single-voice, no crosstalk or room noise. WER and
|
||||
speaker overlap in a Teams export are untested. Speed is probably representative;
|
||||
accuracy is not claimed.
|
||||
- **Compressed input.** Only WAV was sent. Whether `m4a`/`mp3`/`opus` can reach whisper
|
||||
without a server-side transcode is not established here (ADR-033 treats it as needing one).
|
||||
- **Concurrency.** One request at a time. Two simultaneous 90-minute jobs on iguana were
|
||||
not tried.
|
||||
- **`berget/kb-whisper` beyond 22 min / 42 MB.** Enough to know the fallback is not
|
||||
size-capped at meeting length for 22 minutes; not proven at 90.
|
||||
- **Time-of-day / other tenants.** One afternoon, one run per length.
|
||||
|
||||
Scripts were throwaway shell/Python in a session scratchpad and are not committed; the
|
||||
method above reproduces them.
|
||||
@@ -0,0 +1,66 @@
|
||||
# Chat over a long transcript — what the offered models really hold (tapir#33, 2026-10-01)
|
||||
|
||||
Question: ADR-033 reuses `chat.Service`, which stuffs the transcript into the prompt and
|
||||
cuts the tail. #33 assumed chat "retrieves rather than generates" and so has no length
|
||||
ceiling. It does not retrieve, so its ceiling is the model's context. What is it, for the
|
||||
models chat actually offers?
|
||||
|
||||
## Answer
|
||||
|
||||
| Model (via the NodePort gateway) | Result | Limit found |
|
||||
|---|---|---|
|
||||
| `koala/phi4-mini` | correct to 30,000 chars; **HTTP 400 at 50,000** (13,332 tokens vs an 8,192-token context) | ~22,000 chars in production, where the client adds `max_tokens: 1500` |
|
||||
| `iguana/gemma4-26b` | correct at 18k, 50k and **90,000 chars** (26.5 k tokens); **at 120,000 chars it silently loses the head** | **90,000 chars** |
|
||||
| `berget/mistral-small` | answered "the transcript does not contain it" for a fact **at the end** of an 18,000-char and a 50,000-char transcript | unreliable here, one run each — see caveats |
|
||||
|
||||
The dangerous one is gemma at 120,000 chars: **no error, no truncation flag** — the reported
|
||||
prompt tokens fell from 26,548 (90 k chars) to 16,387 (120 k chars), and the model answered
|
||||
"it does not say who the chair is" about a sentence at the very start of the transcript. The
|
||||
serving layer truncated the prompt from the front. `chat.Service` reports `Truncated: false`
|
||||
because *it* did not cut anything.
|
||||
|
||||
## What this changes
|
||||
|
||||
1. **The budget for an uploaded transcript is 90,000 chars, not a round 120,000.** That is the
|
||||
best offered model's measured limit, ~100 minutes of speech at the fixture's density.
|
||||
`TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS` defaults to it.
|
||||
2. **Chat over an upload must not open on the summarizer default.** The offered list is
|
||||
`[SummarizerModel, FallbackModel, CloudFallbackModel]`, so `koala/phi4-mini` is first — and
|
||||
400s on an hour of speech. `TAPIR_UPLOAD_CHAT_MODEL` (default `iguana/gemma4-26b`) is what
|
||||
the upload chat opens with; the switcher still lists every model.
|
||||
3. **A per-model budget does not exist yet.** Someone who switches the dropdown to
|
||||
`phi4-mini` on a long recording gets a loud HTTP 400, shown inline as "couldn't answer".
|
||||
That fails honestly but unhelpfully. Filed as a follow-up.
|
||||
4. **The "retrieval chat" trigger in ADR-033 is now concrete:** any recording past 90,000 chars
|
||||
(≈100 min) cannot be answered by stuffing, for any offered model.
|
||||
|
||||
## Method
|
||||
|
||||
`POST /v1/chat/completions` through `http://koala:30401/v1`, `temperature: 0`,
|
||||
`max_tokens: 200` (production sends 1500 for chat via the summary token cap), the same system
|
||||
framing as `chat.Service`. The transcript is the 60-minute synthetic fixture from
|
||||
`audio-length-ceiling-2026-10-01.md` truncated/repeated to N chars, with one unique sentence
|
||||
planted either at the **end** ("lagret på Kvarngatan ska tömmas … av Pelle", question: who/when)
|
||||
or at the **head** ("kvällens ordförande är Ingrid", question: who chairs). Needle at the
|
||||
end alone would have missed the gemma failure — a serving layer that truncates from the front
|
||||
keeps the tail, so the end-needle passed at 120,000 chars; only the head-needle exposed it.
|
||||
Grounded = the answer contains the planted fact.
|
||||
|
||||
## What this did not measure
|
||||
|
||||
- **Real speech**: the fixture is a repeated 82-second script, so retrieval "difficulty" is
|
||||
artificially easy except for the planted needle. One question per run, one run per cell.
|
||||
- **`berget/mistral-small`'s failure**: it missed an end-needle at 18,000 chars where the
|
||||
others succeeded. One run; could be the model, the prompt framing, or position bias. Not
|
||||
investigated; do not read it as a ranking.
|
||||
- **Latency under load**, **concurrent** chat while whisper is transcribing, and in particular
|
||||
the hazard below.
|
||||
- **`koala/qwen36-35b-a3b` or any frontier model** — not in the offered list.
|
||||
|
||||
## A hazard this surfaces and does not solve
|
||||
|
||||
`iguana/gemma4-26b` is a large ollama model on the same host as `iguana/kb-whisper`. infra#273
|
||||
is precisely that a large resident ollama model can Metal-OOM whisper-server permanently. Making
|
||||
gemma the default *chat* model for uploads means chatting with a transcript can load the model
|
||||
that wedges the transcriber. The primary-fails-over-to-berget design limits the blast radius but
|
||||
does not remove it. Recorded in ADR-033 and filed.
|
||||
@@ -0,0 +1,85 @@
|
||||
# Can a local model adjudicate two ASR transcripts? (tapir#33, 2026-10-01)
|
||||
|
||||
Question: when `iguana/kb-whisper` and `iguana/pianissimo` disagree about a word or phrase,
|
||||
can a **local** model say which reading is right, well enough to merge the two transcripts?
|
||||
Ground truth: the speaker's own blind answers on 30 disagreements of one real recording
|
||||
(see `asr-kb-whisper-vs-pianissimo-2026-10-01.md`; no transcript text is recorded here,
|
||||
this repo is public). Reference: Claude Opus 5.5 agreed with the speaker on all 30.
|
||||
|
||||
## Method
|
||||
|
||||
- **Items:** 30 disagreement spans that Opus had picked as changing meaning, the speaker's
|
||||
verdict as truth (pianissimo right 17, kb-whisper right 13).
|
||||
- **Prompt:** one fixed English instruction (two transcriptions of Swedish speech differ at
|
||||
ONE marked spot; say which reading the speaker most likely said; reply `ANSWER: 1|2|UNSURE`
|
||||
plus one sentence). Each item was asked in **both reading orders** and in **two contexts**:
|
||||
narrow (+-8 words) and wide (+-100 words). 120 calls per model, temperature 0.
|
||||
- **Models:** 11, all local (`koala/*`, `iguana/*`): nothing left the homelab. Run one iguana
|
||||
model at a time, each unloaded afterwards, to avoid the memory pressure behind infra#273;
|
||||
whisper was confirmed healthy afterwards.
|
||||
- **Pass criteria, fixed before the run:** wide context, strict accuracy >= 80% (an
|
||||
unparseable or "unsure" answer counts as wrong), order-swap consistency >= 85%,
|
||||
unparseable <= 10%.
|
||||
|
||||
## Results (wide context, 60 calls per model)
|
||||
|
||||
| Model | Correct | 95% CI | Same pick when order swapped | Unparseable | s/call | Pass |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `iguana/gemma4-31b` | 58 = 97% | 89-99% | 93% | 0 | 10.4 | **yes** |
|
||||
| `iguana/gemma4-26b` | 55 = 92% | 82-96% | 93% | 0 | 2.0 | **yes** |
|
||||
| `iguana/ornith-35b` | 51 = 85% | 74-92% | 77% | 0 | 2.5 | no (consistency) |
|
||||
| `koala/phi4-14b` | 50 = 83% | 72-91% | 80% | 0 | 1.0 | no (consistency) |
|
||||
| `koala/qwen36-35b-a3b` | 49 = 82% | 70-89% | 83% | 0 | 3.3 | no (consistency) |
|
||||
| `iguana/granite4` | 47 = 78% | 66-87% | 83% | 0 | 4.6 | no |
|
||||
| `iguana/qwen3-14b` | 44 = 73% | 61-83% | 87% | 0 | 4.4 | no (accuracy) |
|
||||
| `iguana/gpt-oss-20b` | 40 = 67% | 54-77% | 100% | 17 | 11.2 | no (unparseable) |
|
||||
| `koala/phi4-mini` | 37 = 62% | 49-73% | 24% | 1 | 0.5 | no |
|
||||
| `iguana/gemma4-26b-think` | 23 = 38% | 27-51% | 100% | 37 | 15.3 | no (unparseable) |
|
||||
| `koala/qwen35-9b-fast` | 0 | n/a | n/a | 60 | 7.2 | no (no output) |
|
||||
|
||||
Narrow context only (+-8 words) costs 6-12 points for the models that pass: gemma4-31b 92%,
|
||||
gemma4-26b 83%, ornith-35b 80%. Context is not decoration: Opus also used evidence from
|
||||
elsewhere in the recording (an amount stated earlier, a term that recurs).
|
||||
|
||||
Position bias: the passing models pick "Reading 1" 47-51% of the time. `phi4-mini` picks it 15%
|
||||
and flips its answer when the order is swapped 76% of the time: it carries almost no signal.
|
||||
|
||||
## Reading it
|
||||
|
||||
- **Two models pass:** the gemma4 pair. Both clear the bar and neither shows position bias.
|
||||
26b is ~5x faster; their intervals overlap, so 30 items cannot separate 92% from 97%.
|
||||
- **The thinking models were not tested fairly.** `gpt-oss-20b` and `gemma4-26b-think` spend
|
||||
the 600-token budget reasoning and often never reach the answer line. When gemma4-26b-think
|
||||
did answer, it was right 23 of 23; gpt-oss-20b 93% of the answered. A larger budget would
|
||||
make them eligible; at 15 s a call it would not make them practical.
|
||||
|
||||
## What this does not show
|
||||
|
||||
- **Thirty items, one recording, one speaker, one domain.** The 30 were chosen by Opus as
|
||||
decidable, so they overstate how easy the typical disagreement is; ~350 others are mostly
|
||||
spelling, compounds and fillers, which no ground truth covers.
|
||||
- Opus agreeing 30/30 is not a measure of Opus over all 380 disagreements.
|
||||
- Accuracy was measured at temperature 0 on one prompt; no prompt tuning was tried.
|
||||
- The speaker's answers are the only truth; they are one person's recollection of what they said.
|
||||
|
||||
## Why gemma4-31b is slow, and what was ruled out
|
||||
|
||||
On iguana (M2 Ultra, 64 GB) the dense 31B runs at **103 tok/s prefill, 7.9 tok/s decode**; the
|
||||
26B MoE at 641 and 39.6. Ruled out as the cause: memory pressure (no swap, 45% free, model 100%
|
||||
on GPU), context length (32k vs 4k made no difference), and the **inference engine** — MLX 4-bit
|
||||
of the same model measured 76 tok/s prefill and 8.6 decode, the same within noise. (Terse
|
||||
output saves only ~3.6 s of ~9.3 s because prefill dominates.) Two unrelated engines at the same
|
||||
speed points at the machine: ~140 GB/s effective memory bandwidth on a part rated for 800, and
|
||||
**iguana had macOS Low Power Mode on** (`pmset` `lowpowermode 1`). A first MLX benchmark was
|
||||
wrong because `mlx_lm.server` caches prompts; the figures above are from the corrected run.
|
||||
|
||||
**Confirmed 2026-10-02.** The owner turned Low Power Mode off (`lowpowermode 0`, read back over
|
||||
ssh) and `gemma4:31b` was re-measured with nothing else resident: **217 tok/s prefill, 22.3
|
||||
tok/s decode** (4 runs, spread 22.2-22.6), against 103 and 7.9 with it on, so 2.1x and 2.8x. The
|
||||
slowness was the power mode, not the engine or the model. Caveats on the comparison: the new run
|
||||
used a different, synthetic Swedish prompt (~700 words, the original items held real transcript
|
||||
text and were deleted) and the model answered in 26-43 tokens, not the ~200 of the first run, so
|
||||
it is a same-machine before/after, not a controlled A/B. `gemma4:26b` and the MLX figures were
|
||||
not re-measured. The 31B's 97% judge accuracy is therefore usable at roughly a third of the
|
||||
latency the sweep implied; the 26B-vs-31B choice in ADR-034 was made on the slow numbers and is
|
||||
reopened by this.
|
||||
@@ -0,0 +1,68 @@
|
||||
# Two-engine reconcile, first check on a real recording (tapir#33, ADR-034, 2026-10-02)
|
||||
|
||||
Aggregate numbers only. This repo is public and the recording is private (#31 custody is open), so no
|
||||
transcript text, no disputed words and no per-item reading appears here.
|
||||
|
||||
## What was run
|
||||
|
||||
One real 63.6-minute Teams recording of one speaker dictating project notes (the video track, camera
|
||||
off, was discarded). Build: PR #39 head `7f8e367`. Everything local: `iguana/pianissimo` as base,
|
||||
`iguana/kb-whisper` as the second engine, `iguana/gemma4-31b` as reconciler (iguana Low Power Mode
|
||||
off), cloud fallbacks off, a throwaway Postgres.
|
||||
|
||||
| Measure | Value |
|
||||
|---|---|
|
||||
| Job time, upload to saved | 21 min 49 s for 63.6 min of audio (20.6 min per hour of audio) |
|
||||
| Differences between the engines | 362 |
|
||||
| Settled automatically (compound splits, up to two filler words) | 80 |
|
||||
| Decided by the model | 282 |
|
||||
| Of those: alternative reading used / base kept / uncertain | 130 / 151 / 1 |
|
||||
| Model call errors | 0 |
|
||||
|
||||
20.6 min per hour is inside the 25-minute resolution trigger in ADR-034, so no smaller adjudicator is
|
||||
needed on speed grounds. The earlier estimate was about 10 min per hour on gemma4-26b; it was not
|
||||
measured and this run used the 31B.
|
||||
|
||||
## Blind check
|
||||
|
||||
30 spots drawn at random (fixed seed) from the 89 replacement spots that could be placed in the base
|
||||
text, across the whole recording (0:44 to 60:57). The two readings were shown as x and y in random
|
||||
order, with the key kept apart from the judge. Judge: Claude Opus, text only, no audio, at the
|
||||
speaker's request ("I trust Opus"). The speaker did not judge these items themselves.
|
||||
|
||||
| Verdict | Items |
|
||||
|---|---|
|
||||
| The reconciler's reading (the alternative) is right | 21 |
|
||||
| The base reading (pianissimo) is right, so the change made it worse | 1 |
|
||||
| Neither reading is right | 3 |
|
||||
| Cannot tell from text (compound, hyphen and number-format differences) | 5 |
|
||||
|
||||
- Right of the items with a verdict: 21 of 22, 95% (95% interval 78% to 99%).
|
||||
- Right of all 30, counting "neither" and "cannot tell" as not right: 21 of 30, 70% (52% to 83%).
|
||||
- Right of the 25 that were not "cannot tell": 21 of 25, 84% (65% to 94%).
|
||||
- Net effect against the base on these 30: 21 better, 1 worse, 8 unchanged in quality.
|
||||
|
||||
## Reading the result
|
||||
|
||||
ADR-034's bar was "right at least 85% of the time". It did not say how ties count, and that was a
|
||||
flaw in the bar, not in the data. The literal reading (all 30) gives 70% and **does not meet** the bar;
|
||||
counting only decided items gives 95% and meets it. Neither is a hidden pass or a hidden failure; both
|
||||
are reported. What is not in doubt: on a random sample the reconciler improved the text far more often
|
||||
than it harmed it (21 to 1), and the 8 unchanged cases are mostly spellings the text cannot settle.
|
||||
|
||||
## What this did not measure
|
||||
|
||||
- **Better single engine.** The second ADR-034 condition, "no worse than the better single engine", is
|
||||
**not answered**. Only changed spots are stored, not the 151 where the model kept pianissimo, so the
|
||||
reconciler cannot be compared with kb-whisper alone. The 21-to-1 result is against pianissimo.
|
||||
- **The speaker as judge.** Opus judged from text. It cannot hear the audio, so it can be wrong where
|
||||
the right word is audible but not inferable (items such as a mis-heard number or a name). Earlier,
|
||||
Opus and the speaker agreed 30 of 30, but on items chosen as decidable; that does not transfer to a
|
||||
random sample.
|
||||
- **Insertions and deletions.** 42 of the 131 stored spots were not sampled: 39 are an added or
|
||||
removed word, which cannot be placed in the base text, and 3 are the uncertain spot or could not be
|
||||
located. The 39 are mostly small words ("och", "det"), so this sample says little about them.
|
||||
- **One recording, one speaker, 30 items.** Wide intervals, and neighbouring spots (three pairs within
|
||||
seconds of each other) are not independent.
|
||||
- **Segments keep the base text.** The saved timed segments hold pianissimo's wording, not the merged
|
||||
text; only the stored full text is merged. Found while building this sample; not yet an issue.
|
||||
@@ -0,0 +1,258 @@
|
||||
Feature: Upload an audio recording, get a private Swedish transcript, discuss it
|
||||
As someone with a recorded meeting on my computer
|
||||
I want to upload it and then ask questions about what was said
|
||||
So that I can find out what was decided, what is open and who owes what
|
||||
|
||||
# ADR-033, tapir#33. The load-bearing constraint: an uploaded transcript is
|
||||
# PRIVATE content, so it lives in its own RLS-scoped store and never in the
|
||||
# shared public-caption `transcripts` table (ADR-021). Rejection is synchronous,
|
||||
# the audio is never retained, and a failed job ends in a terminal state with a
|
||||
# message — never a spinner (ADR-025). A file with a video track (a Teams meeting
|
||||
# with the camera off is one) is accepted: only its audio is used and the user is
|
||||
# told. Video playback, keyframes and analysis of what is shown remain #28.
|
||||
|
||||
Background:
|
||||
Given I am a registered user with audio upload enabled
|
||||
|
||||
Scenario: The upload form states the limits up front
|
||||
When I open the upload page
|
||||
Then I see the size limit and the length limit before I choose a file
|
||||
|
||||
Scenario: An accepted recording is queued and I am taken to its page
|
||||
When I upload an audio file within the limits
|
||||
Then it is recorded as queued for me
|
||||
And I am redirected to its page
|
||||
|
||||
Scenario: A recording with no title is named after its file
|
||||
When I upload an audio file and give no title
|
||||
Then its title is the filename without its extension
|
||||
|
||||
Scenario: A file with a video track is accepted and I am told the video was ignored
|
||||
When I upload a recording that also has a video track
|
||||
Then it is accepted and queued like any other
|
||||
And its page says only its audio was used and the video was discarded
|
||||
And it says so while queued, while transcribing, when done and when failed
|
||||
|
||||
Scenario: A video file with no audio is rejected for having no audio
|
||||
When I upload a video that has no audio track
|
||||
Then it is rejected with a message that it has no audio
|
||||
And nothing is recorded
|
||||
|
||||
Scenario: A recording over the length limit is rejected with both lengths
|
||||
When I upload a recording longer than the limit
|
||||
Then it is rejected stating the recording's length and the limit
|
||||
And nothing is recorded
|
||||
|
||||
Scenario: A file over the size limit is rejected as too large
|
||||
When I upload a file larger than the size limit
|
||||
Then it is rejected with 413 and a message about the limit
|
||||
And nothing is recorded
|
||||
|
||||
Scenario: Cover art on an audio file is not a video track
|
||||
When I upload an mp3 whose only video stream is attached cover art
|
||||
Then it is treated as audio and no video notice is shown
|
||||
|
||||
Scenario: A file that is not media is rejected as unreadable
|
||||
When I upload a file ffprobe cannot read
|
||||
Then it is rejected as the file's fault
|
||||
|
||||
Scenario: A broken ffprobe is reported as our fault, not as a bad file
|
||||
When the prober itself fails
|
||||
Then the upload is an error rather than a rejection of my file
|
||||
|
||||
Scenario: A recording becomes a transcript and the audio is deleted
|
||||
Given I have uploaded a recording
|
||||
When it has been transcribed
|
||||
Then the transcript is stored privately for me
|
||||
And the audio is deleted
|
||||
|
||||
Scenario: The endpoint that served a transcript is recorded and shown
|
||||
Given a recording was transcribed by the fallback endpoint
|
||||
When I view its page
|
||||
Then it names the fallback endpoint, because that means the audio left the homelab
|
||||
|
||||
Scenario: The local endpoint is tried first and the fallback only on failure
|
||||
When the primary transcription endpoint fails for any reason
|
||||
Then the fallback endpoint transcribes it
|
||||
But when the primary succeeds the fallback is never contacted
|
||||
|
||||
Scenario: A failed transcription ends in a message, never a spinner
|
||||
When every transcription endpoint fails
|
||||
Then the upload is failed with a typed reason
|
||||
And its page shows that reason and stops polling
|
||||
|
||||
Scenario: A restart fails the jobs it interrupted
|
||||
Given the service restarted while a recording was queued or transcribing
|
||||
When it starts again
|
||||
Then those uploads are failed as interrupted
|
||||
And any audio left in the spool is deleted
|
||||
|
||||
Scenario: Only one recording is transcribed at a time
|
||||
When several recordings are queued
|
||||
Then they are transcribed one after another
|
||||
|
||||
Scenario: My uploads are invisible to other users
|
||||
Given another user has uploaded a recording
|
||||
When I ask for it by its id
|
||||
Then I get 404 and no part of its title or text
|
||||
And the chat for it calls no model
|
||||
|
||||
Scenario: I can ask about my recording and get a grounded answer
|
||||
Given my recording has been transcribed
|
||||
When I ask a question about it
|
||||
Then the answer comes from my transcript through the same chat as a video's
|
||||
|
||||
Scenario: A long recording is not cut to its first third
|
||||
Given my recording's transcript is longer than the caption default
|
||||
When I ask a question about it
|
||||
Then the transcript is sent with the upload budget, not the 18,000-character caption default
|
||||
|
||||
# ADR-034, two engines and a local reconciler (off unless TAPIR_RECONCILE_BASE_MODEL is set)
|
||||
|
||||
Scenario: Two engines transcribe the audio and their disagreements are reconciled
|
||||
Given two-engine transcription is enabled
|
||||
When a recording is transcribed
|
||||
Then both engines run on the same audio, the fast one first
|
||||
And a local model chooses between their disagreements
|
||||
And the endpoint names every engine and the model that took part
|
||||
|
||||
Scenario: Trivial disagreements never reach the model
|
||||
When the engines differ only by a compound written as one word or two, or by a filler word
|
||||
Then the base reading is kept without a model call
|
||||
|
||||
Scenario: Differences close together are one decision
|
||||
When two differences are separated by a single shared word
|
||||
Then they are asked about as one dispute
|
||||
|
||||
Scenario: An unsure or changed reading is flagged for me
|
||||
When the model changes a reading or cannot decide
|
||||
Then the page lists the spot with a minute mark, both readings and which was used
|
||||
And an unsure spot keeps the base reading and is marked uncertain
|
||||
|
||||
Scenario: The model cannot be a cloud model
|
||||
When the reconciliation model or base engine is not a koala/ or iguana/ alias
|
||||
Then the service refuses to start
|
||||
And a cloud alias that merely contains a local host name is refused too
|
||||
|
||||
Scenario: One engine failing does not fail the upload
|
||||
When one of the two engines fails
|
||||
Then the other engine's transcript is used
|
||||
And the page says only one engine produced it and why
|
||||
And the cloud fallback is not used
|
||||
|
||||
Scenario: Only when both local engines fail does the cloud fallback run
|
||||
When both local engines fail and a fallback is configured
|
||||
Then the fallback transcribes it and the row records that it did
|
||||
|
||||
Scenario: A model that is down gives an unreconciled transcript, not a failed job
|
||||
When the reconciliation model fails three times in a row
|
||||
Then the base transcript is kept unreconciled with the reason
|
||||
And the page says it could not be reconciled
|
||||
|
||||
Scenario: A cancelled job never reaches the cloud
|
||||
When the job is cancelled during either engine
|
||||
Then neither the second engine nor the cloud fallback is called
|
||||
|
||||
Scenario: A single-engine transcript claims no second opinion
|
||||
When only one engine was configured
|
||||
Then the page does not mention reconciliation
|
||||
|
||||
# ADR-035: progress, the log, and the judged spots
|
||||
|
||||
Scenario: I can see which phase my upload is in
|
||||
Given a recording is being transcribed
|
||||
When I open its page
|
||||
Then it lists every phase in order, the finished ones with how long each took
|
||||
And the running phase is marked, with how long it has run
|
||||
And the page updates in place every two seconds until the job ends
|
||||
|
||||
Scenario: A bar appears only where the count is real
|
||||
When the deciding phase has decided 142 of 282 differences
|
||||
Then the page shows a bar with "142 of 282"
|
||||
And an engine call, which is one request with nothing to count, never gets a bar or a percentage
|
||||
|
||||
Scenario: A slow engine call is not mistaken for a dead service
|
||||
When an engine call has run for 25 minutes and the service is still reporting
|
||||
Then the page shows the elapsed time and says nothing is wrong
|
||||
|
||||
Scenario: A service that stopped reporting is called lost
|
||||
When nothing has been heard from the service for longer than the heartbeat allows
|
||||
Then the page says the service stopped reporting and may have restarted or crashed
|
||||
And the running phase carries a warning mark and no spinner
|
||||
And the page keeps checking
|
||||
|
||||
Scenario: A model that stops advancing is called quiet
|
||||
When the deciding phase has not advanced for five minutes and the service is still reporting
|
||||
Then the page says nothing new has happened for that long and that the service is still running
|
||||
And it does not say the service is lost
|
||||
|
||||
Scenario: A failed job shows where it stopped
|
||||
When a job fails
|
||||
Then the failed phase is marked and what never started stays pending
|
||||
And the page stops checking
|
||||
|
||||
Scenario: A finished job keeps its record of what was done
|
||||
When a job is done
|
||||
Then the page keeps a collapsed list of the phases with how long each took
|
||||
|
||||
Scenario: The heartbeat runs on its own while an engine call is stuck
|
||||
When an engine call does not return
|
||||
Then the row is still stamped alive every few seconds
|
||||
And no heartbeat outlives its job
|
||||
|
||||
Scenario: The log names every phase and never the transcript
|
||||
When a job runs
|
||||
Then the log has a line per phase in the order it ran, each naming the upload
|
||||
And no line contains transcript text or the spool path
|
||||
|
||||
Scenario: Every judged spot is stored but only the ones to check are listed
|
||||
When the model has changed, kept and been unsure about spots
|
||||
Then every one is stored with its verdict and the words before it
|
||||
And the page lists only the changed and uncertain ones, with the words before each
|
||||
|
||||
Scenario: The saved segments carry the merged words
|
||||
When the model changes a word
|
||||
Then the segment containing it carries the reconciled word
|
||||
And the segment's timing is unchanged
|
||||
|
||||
# ADR-035 decision 8: hardening from the adversarial review
|
||||
|
||||
Scenario: Chat about a recording is offered with local models only
|
||||
When I open the chat for my recording
|
||||
Then the model list holds only models that run on the homelab
|
||||
And a cloud model the shared chat service knows is not in it
|
||||
|
||||
Scenario: A forged cloud model is never sent the transcript
|
||||
When a chat request names a cloud model
|
||||
Then a local model answers instead, and nothing was sent to the cloud one
|
||||
|
||||
Scenario: A wedged engine does not use up the whole job
|
||||
When the primary engine never answers
|
||||
Then it fails at its own deadline while the job is still alive
|
||||
And the fallback runs, unless the job itself was cancelled or timed out
|
||||
|
||||
Scenario: A file that lies about its length is refused
|
||||
When a file claims to be short but converts to more than the limit
|
||||
Then the job fails with a plain explanation and no engine receives the audio
|
||||
|
||||
Scenario: A recording much shorter than its file claimed is transcribed and noted
|
||||
When a file converts to under 80 percent of the length it claimed and more than two minutes short
|
||||
Then it is transcribed anyway and its audio step says how long the audio really is
|
||||
And a complete recording whose header over-estimated its length is not refused
|
||||
|
||||
Scenario: A full queue is a retry, not an error
|
||||
When too many recordings are already waiting
|
||||
Then the upload is answered as busy, its row is failed honestly and its audio is deleted
|
||||
|
||||
Scenario: One user cannot fill the queue for everyone
|
||||
When one user has several recordings waiting
|
||||
Then further uploads from that user are refused and other users are not affected
|
||||
|
||||
Scenario: Errors are shown as kinds, never their text
|
||||
When a model or an engine fails
|
||||
Then the log, the stored reason and the page carry a kind such as timeout or http_502
|
||||
And never the provider's response, a file path or a word from the transcript
|
||||
|
||||
Scenario: Input that cannot be right is a client error
|
||||
When the id is not an id, or a header or title holds control bytes
|
||||
Then the answer is a 4xx or the title is cleaned, never a server error
|
||||
Reference in New Issue
Block a user