mathias
4a2067063d
Upload a recording (audio, or video with the camera off), get a private Swedish transcript, and chat about it. - ADR-033: RLS-scoped private transcript store; ffprobe/ffmpeg intake; gateway transcription (iguana kb-whisper, berget fallback); in-process single-replica worker; chat over the transcript (local models only). - ADR-034: optional two-engine mode (pianissimo + kb-whisper) reconciled by a LOCAL model; off unless TAPIR_RECONCILE_BASE_MODEL is set. - ADR-035: real progress (phases, running/quiet/lost), phase logging, every judged spot stored, merged segments. - Hardening from four adversarial reviews: errors logged and stored as kinds never text; per-engine deadlines; converted-length check; bounded queue; client hang-ups are not errors; strict local-alias check. - Dockerfile: static ffmpeg/ffprobe 7.0.2 pinned by sha256 (#34). Built and inspected by hand; CI does not build the image on a PR, so its first CI build is this merge. Uploads stay OFF until TAPIR_UPLOADS_ENABLED is set in the deployment. Closes #33, #34, #42, #43, #44. Co-authored-by: Mathias <mthbqv@gmail.com>
67 lines
4.2 KiB
Markdown
67 lines
4.2 KiB
Markdown
# Chat over a long transcript — what the offered models really hold (tapir#33, 2026-10-01)
|
|
|
|
Question: ADR-033 reuses `chat.Service`, which stuffs the transcript into the prompt and
|
|
cuts the tail. #33 assumed chat "retrieves rather than generates" and so has no length
|
|
ceiling. It does not retrieve, so its ceiling is the model's context. What is it, for the
|
|
models chat actually offers?
|
|
|
|
## Answer
|
|
|
|
| Model (via the NodePort gateway) | Result | Limit found |
|
|
|---|---|---|
|
|
| `koala/phi4-mini` | correct to 30,000 chars; **HTTP 400 at 50,000** (13,332 tokens vs an 8,192-token context) | ~22,000 chars in production, where the client adds `max_tokens: 1500` |
|
|
| `iguana/gemma4-26b` | correct at 18k, 50k and **90,000 chars** (26.5 k tokens); **at 120,000 chars it silently loses the head** | **90,000 chars** |
|
|
| `berget/mistral-small` | answered "the transcript does not contain it" for a fact **at the end** of an 18,000-char and a 50,000-char transcript | unreliable here, one run each — see caveats |
|
|
|
|
The dangerous one is gemma at 120,000 chars: **no error, no truncation flag** — the reported
|
|
prompt tokens fell from 26,548 (90 k chars) to 16,387 (120 k chars), and the model answered
|
|
"it does not say who the chair is" about a sentence at the very start of the transcript. The
|
|
serving layer truncated the prompt from the front. `chat.Service` reports `Truncated: false`
|
|
because *it* did not cut anything.
|
|
|
|
## What this changes
|
|
|
|
1. **The budget for an uploaded transcript is 90,000 chars, not a round 120,000.** That is the
|
|
best offered model's measured limit, ~100 minutes of speech at the fixture's density.
|
|
`TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS` defaults to it.
|
|
2. **Chat over an upload must not open on the summarizer default.** The offered list is
|
|
`[SummarizerModel, FallbackModel, CloudFallbackModel]`, so `koala/phi4-mini` is first — and
|
|
400s on an hour of speech. `TAPIR_UPLOAD_CHAT_MODEL` (default `iguana/gemma4-26b`) is what
|
|
the upload chat opens with; the switcher still lists every model.
|
|
3. **A per-model budget does not exist yet.** Someone who switches the dropdown to
|
|
`phi4-mini` on a long recording gets a loud HTTP 400, shown inline as "couldn't answer".
|
|
That fails honestly but unhelpfully. Filed as a follow-up.
|
|
4. **The "retrieval chat" trigger in ADR-033 is now concrete:** any recording past 90,000 chars
|
|
(≈100 min) cannot be answered by stuffing, for any offered model.
|
|
|
|
## Method
|
|
|
|
`POST /v1/chat/completions` through `http://koala:30401/v1`, `temperature: 0`,
|
|
`max_tokens: 200` (production sends 1500 for chat via the summary token cap), the same system
|
|
framing as `chat.Service`. The transcript is the 60-minute synthetic fixture from
|
|
`audio-length-ceiling-2026-10-01.md` truncated/repeated to N chars, with one unique sentence
|
|
planted either at the **end** ("lagret på Kvarngatan ska tömmas … av Pelle", question: who/when)
|
|
or at the **head** ("kvällens ordförande är Ingrid", question: who chairs). Needle at the
|
|
end alone would have missed the gemma failure — a serving layer that truncates from the front
|
|
keeps the tail, so the end-needle passed at 120,000 chars; only the head-needle exposed it.
|
|
Grounded = the answer contains the planted fact.
|
|
|
|
## What this did not measure
|
|
|
|
- **Real speech**: the fixture is a repeated 82-second script, so retrieval "difficulty" is
|
|
artificially easy except for the planted needle. One question per run, one run per cell.
|
|
- **`berget/mistral-small`'s failure**: it missed an end-needle at 18,000 chars where the
|
|
others succeeded. One run; could be the model, the prompt framing, or position bias. Not
|
|
investigated; do not read it as a ranking.
|
|
- **Latency under load**, **concurrent** chat while whisper is transcribing, and in particular
|
|
the hazard below.
|
|
- **`koala/qwen36-35b-a3b` or any frontier model** — not in the offered list.
|
|
|
|
## A hazard this surfaces and does not solve
|
|
|
|
`iguana/gemma4-26b` is a large ollama model on the same host as `iguana/kb-whisper`. infra#273
|
|
is precisely that a large resident ollama model can Metal-OOM whisper-server permanently. Making
|
|
gemma the default *chat* model for uploads means chatting with a transcript can load the model
|
|
that wedges the transcriber. The primary-fails-over-to-berget design limits the blast radius but
|
|
does not remove it. Recorded in ADR-033 and filed.
|