Files

T

mathiasandClaude Sonnet 5.5 4d3e2a471a docs(research): chat over a long transcript -- what the offered models really hold

phi4-mini 400s at ~13k tokens; gemma4-26b is correct to 90,000 chars and
silently loses the HEAD at 120,000 (no error, no truncation flag). The
end-needle test alone passed at 120k -- only a head-needle exposed it.
Records the default-model trap and the iguana contention hazard.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

2026-10-01 17:08:13 +02:00

4.2 KiB

Raw Blame History

Chat over a long transcript — what the offered models really hold (tapir#33, 2026-10-01)

Question: ADR-033 reuses chat.Service, which stuffs the transcript into the prompt and cuts the tail. #33 assumed chat "retrieves rather than generates" and so has no length ceiling. It does not retrieve, so its ceiling is the model's context. What is it, for the models chat actually offers?

Answer

Model (via the NodePort gateway)	Result	Limit found
`koala/phi4-mini`	correct to 30,000 chars; HTTP 400 at 50,000 (13,332 tokens vs an 8,192-token context)	~22,000 chars in production, where the client adds `max_tokens: 1500`
`iguana/gemma4-26b`	correct at 18k, 50k and 90,000 chars (26.5 k tokens); at 120,000 chars it silently loses the head	90,000 chars
`berget/mistral-small`	answered "the transcript does not contain it" for a fact at the end of an 18,000-char and a 50,000-char transcript	unreliable here, one run each — see caveats

The dangerous one is gemma at 120,000 chars: no error, no truncation flag — the reported prompt tokens fell from 26,548 (90 k chars) to 16,387 (120 k chars), and the model answered "it does not say who the chair is" about a sentence at the very start of the transcript. The serving layer truncated the prompt from the front. chat.Service reports Truncated: false because it did not cut anything.

What this changes

The budget for an uploaded transcript is 90,000 chars, not a round 120,000. That is the best offered model's measured limit, ~100 minutes of speech at the fixture's density. TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS defaults to it.
Chat over an upload must not open on the summarizer default. The offered list is [SummarizerModel, FallbackModel, CloudFallbackModel], so koala/phi4-mini is first — and 400s on an hour of speech. TAPIR_UPLOAD_CHAT_MODEL (default iguana/gemma4-26b) is what the upload chat opens with; the switcher still lists every model.
A per-model budget does not exist yet. Someone who switches the dropdown to phi4-mini on a long recording gets a loud HTTP 400, shown inline as "couldn't answer". That fails honestly but unhelpfully. Filed as a follow-up.
The "retrieval chat" trigger in ADR-033 is now concrete: any recording past 90,000 chars (≈100 min) cannot be answered by stuffing, for any offered model.

Method

POST /v1/chat/completions through http://koala:30401/v1, temperature: 0, max_tokens: 200 (production sends 1500 for chat via the summary token cap), the same system framing as chat.Service. The transcript is the 60-minute synthetic fixture from audio-length-ceiling-2026-10-01.md truncated/repeated to N chars, with one unique sentence planted either at the end ("lagret på Kvarngatan ska tömmas … av Pelle", question: who/when) or at the head ("kvällens ordförande är Ingrid", question: who chairs). Needle at the end alone would have missed the gemma failure — a serving layer that truncates from the front keeps the tail, so the end-needle passed at 120,000 chars; only the head-needle exposed it. Grounded = the answer contains the planted fact.

What this did not measure

Real speech: the fixture is a repeated 82-second script, so retrieval "difficulty" is artificially easy except for the planted needle. One question per run, one run per cell.
berget/mistral-small's failure: it missed an end-needle at 18,000 chars where the others succeeded. One run; could be the model, the prompt framing, or position bias. Not investigated; do not read it as a ranking.
Latency under load, concurrent chat while whisper is transcribing, and in particular the hazard below.
koala/qwen36-35b-a3b or any frontier model — not in the offered list.

A hazard this surfaces and does not solve

iguana/gemma4-26b is a large ollama model on the same host as iguana/kb-whisper. infra#273 is precisely that a large resident ollama model can Metal-OOM whisper-server permanently. Making gemma the default chat model for uploads means chatting with a transcript can load the model that wedges the transcriber. The primary-fails-over-to-berget design limits the blast radius but does not remove it. Recorded in ADR-033 and filed.

4.2 KiB Raw Blame History