diff --git a/docs/research/chat-context-over-long-transcripts-2026-10-01.md b/docs/research/chat-context-over-long-transcripts-2026-10-01.md new file mode 100644 index 0000000..b9fc97d --- /dev/null +++ b/docs/research/chat-context-over-long-transcripts-2026-10-01.md @@ -0,0 +1,66 @@ +# Chat over a long transcript — what the offered models really hold (tapir#33, 2026-10-01) + +Question: ADR-033 reuses `chat.Service`, which stuffs the transcript into the prompt and +cuts the tail. #33 assumed chat "retrieves rather than generates" and so has no length +ceiling. It does not retrieve, so its ceiling is the model's context. What is it, for the +models chat actually offers? + +## Answer + +| Model (via the NodePort gateway) | Result | Limit found | +|---|---|---| +| `koala/phi4-mini` | correct to 30,000 chars; **HTTP 400 at 50,000** (13,332 tokens vs an 8,192-token context) | ~22,000 chars in production, where the client adds `max_tokens: 1500` | +| `iguana/gemma4-26b` | correct at 18k, 50k and **90,000 chars** (26.5 k tokens); **at 120,000 chars it silently loses the head** | **90,000 chars** | +| `berget/mistral-small` | answered "the transcript does not contain it" for a fact **at the end** of an 18,000-char and a 50,000-char transcript | unreliable here, one run each — see caveats | + +The dangerous one is gemma at 120,000 chars: **no error, no truncation flag** — the reported +prompt tokens fell from 26,548 (90 k chars) to 16,387 (120 k chars), and the model answered +"it does not say who the chair is" about a sentence at the very start of the transcript. The +serving layer truncated the prompt from the front. `chat.Service` reports `Truncated: false` +because *it* did not cut anything. + +## What this changes + +1. **The budget for an uploaded transcript is 90,000 chars, not a round 120,000.** That is the + best offered model's measured limit, ~100 minutes of speech at the fixture's density. + `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS` defaults to it. +2. **Chat over an upload must not open on the summarizer default.** The offered list is + `[SummarizerModel, FallbackModel, CloudFallbackModel]`, so `koala/phi4-mini` is first — and + 400s on an hour of speech. `TAPIR_UPLOAD_CHAT_MODEL` (default `iguana/gemma4-26b`) is what + the upload chat opens with; the switcher still lists every model. +3. **A per-model budget does not exist yet.** Someone who switches the dropdown to + `phi4-mini` on a long recording gets a loud HTTP 400, shown inline as "couldn't answer". + That fails honestly but unhelpfully. Filed as a follow-up. +4. **The "retrieval chat" trigger in ADR-033 is now concrete:** any recording past 90,000 chars + (≈100 min) cannot be answered by stuffing, for any offered model. + +## Method + +`POST /v1/chat/completions` through `http://koala:30401/v1`, `temperature: 0`, +`max_tokens: 200` (production sends 1500 for chat via the summary token cap), the same system +framing as `chat.Service`. The transcript is the 60-minute synthetic fixture from +`audio-length-ceiling-2026-10-01.md` truncated/repeated to N chars, with one unique sentence +planted either at the **end** ("lagret på Kvarngatan ska tömmas … av Pelle", question: who/when) +or at the **head** ("kvällens ordförande är Ingrid", question: who chairs). Needle at the +end alone would have missed the gemma failure — a serving layer that truncates from the front +keeps the tail, so the end-needle passed at 120,000 chars; only the head-needle exposed it. +Grounded = the answer contains the planted fact. + +## What this did not measure + +- **Real speech**: the fixture is a repeated 82-second script, so retrieval "difficulty" is + artificially easy except for the planted needle. One question per run, one run per cell. +- **`berget/mistral-small`'s failure**: it missed an end-needle at 18,000 chars where the + others succeeded. One run; could be the model, the prompt framing, or position bias. Not + investigated; do not read it as a ranking. +- **Latency under load**, **concurrent** chat while whisper is transcribing, and in particular + the hazard below. +- **`koala/qwen36-35b-a3b` or any frontier model** — not in the offered list. + +## A hazard this surfaces and does not solve + +`iguana/gemma4-26b` is a large ollama model on the same host as `iguana/kb-whisper`. infra#273 +is precisely that a large resident ollama model can Metal-OOM whisper-server permanently. Making +gemma the default *chat* model for uploads means chatting with a transcript can load the model +that wedges the transcriber. The primary-fails-over-to-berget design limits the blast radius but +does not remove it. Recorded in ADR-033 and filed.