From 77fe6954e543c3b397e82ae36e18a1bf76e35eee Mon Sep 17 00:00:00 2001 From: Mathias Date: Thu, 1 Oct 2026 17:08:28 +0200 Subject: [PATCH] docs(adr): ADR-033 decision 8 -- chat budget and model from measurement The proposed 120,000-char budget was a guess; measured, gemma4-26b silently loses the head at that size and phi4-mini (first in the offered list) 400s on an hour of speech. Budget is now 90,000, uploads open on gemma4-26b, the retrieval trigger is concrete, and the iguana contention hazard is recorded. ADR-033 is still Proposed, so this is an in-place correction, not a supersession. Co-Authored-By: Claude Sonnet 5.5 --- DECISIONS.md | 44 +++++++++++++++++++++++++++++++------------- 1 file changed, 31 insertions(+), 13 deletions(-) diff --git a/DECISIONS.md b/DECISIONS.md index 12dc6a9..a0710cf 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -1353,11 +1353,13 @@ would leak across tenants **and pass CI green**. `berget/kb-whisper` returns sub-second times and a different `language` token. - **The existing chat path truncates at 18,000 chars (~22 min of speech), keeping the head.** #33 assumed chat "retrieves rather than generates" and so has no length ceiling. It does - not retrieve; it stuffs the transcript into the prompt and cuts the tail. + not retrieve; it stuffs the transcript into the prompt and cuts the tail. Measured against + the offered models: `phi4-mini` 400s at ~13k tokens; `gemma4-26b` is right to 90,000 chars and + silently drops the head at 120,000. **Unknowns, stated.** Real-speech accuracy; compressed-input behaviour; concurrent long jobs on -iguana; whether the models chat offers can hold an hour of Swedish (~50 k chars); and the Metal -OOM, deliberately not provoked. +iguana; and the Metal OOM, deliberately not provoked. (Whether the offered chat models can hold +an hour of Swedish *was* an unknown here; it is now measured, see decision 8.) **Decision.** @@ -1436,13 +1438,21 @@ OOM, deliberately not provoked. 8. **Chat reuses `chat.Service`; it is not forked.** The uploaded-transcript chat handlers look the row up through the RLS-scoped store, so a non-owner gets 404 by the same mechanism - as ADR-027's summary gate. ADR-027's decisions (ephemeral, switchable models, trust-the-model) - carry over unchanged. **One thing must differ:** the 18,000-char default cuts a one-hour - meeting to its first third, so uploaded transcripts take their own cap, - `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS`. **Provisional default 120,000 chars (~2.4 h at - measured density)**, with the existing `Truncated` flag surfaced honestly. The default is a - guess until the chat slice measures what the offered models actually accept; it is not - evidence that they can. + as ADR-027's summary gate, **before any model is called**. ADR-027's decisions (ephemeral, + switchable models, trust-the-model) carry over unchanged. **Two things must differ, and both + were measured** (`docs/research/chat-context-over-long-transcripts-2026-10-01.md`): + - **The transcript budget.** The 18,000-char caption default cuts a one-hour meeting to its + first third. Uploaded transcripts take their own `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS`, + default **90,000** — the measured ceiling of the best offered model (`iguana/gemma4-26b`), + which answers correctly there but **silently loses the head of a 120,000-char transcript** + with no error and no truncation flag. (An earlier draft of this ADR proposed 120,000 as a + guess; the measurement refuted it.) The mechanism is a per-request `chat.Request.MaxChars`. + - **The opening model.** The offered list leads with the summarizer default `koala/phi4-mini`, + whose 8,192-token context returns HTTP 400 on an hour of speech (~13,000 tokens). Chat over + an upload opens on `TAPIR_UPLOAD_CHAT_MODEL` (default `iguana/gemma4-26b`); the switcher + still lists every model, and picking `phi4-mini` on a long recording fails loudly and + inline. A per-model budget is not built. + Past ~90,000 chars (≈100 minutes) no offered model can answer by stuffing; see triggers. 9. **Not decided here.** Rich `AVSNITT`/`ALTERNATIV` analysis (gated on #30 — `analyze_srt.py` hardcodes `max_tokens: 6000`); video, keyframes, synced playback (#28); language detection; @@ -1465,7 +1475,13 @@ deferred. The scope cut — chat first, no rich analysis, no video — is what k dependency that matters) — a Metal OOM now means *every upload fails over to berget*, i.e. private audio leaving the homelab, silently unless the per-row endpoint is surfaced. - A 90-minute job holds the single slot for ~7.5 minutes; a second user's upload queues behind it. - - Chat over a long transcript is bounded by model context, not by anything in Tapir. + - Chat over a long transcript is bounded by model context, not by anything in Tapir, and the + failure at the edge is *silent* for gemma (the head is dropped, no error). The 90,000 default + sits just inside the measured limit and is only as good as that single measurement. + - **The default chat model for uploads is a large ollama model on iguana — the same host as + `iguana/kb-whisper`.** infra#273 is exactly that a large resident ollama model can Metal-OOM + whisper-server permanently. Chatting with a transcript can therefore load the model that + wedges the transcriber. Not solved here; the berget fallback limits the blast radius. - Tapir's own ingress body limit for a 200 MB upload is **unverified** (it lives in `infra`). **Validation / resolution triggers** (each observable by someone other than the author): @@ -1473,8 +1489,10 @@ deferred. The scope cut — chat first, no rich analysis, no video — is what k request at the current cap fails with whisper-side (not ingress) errors. - *Berget-first*: if more than a quarter of uploads in any week are served by the fallback, swap the primary/fallback order and revisit infra#273. -- *Retrieval chat*: if the chat slice finds an offered model cannot answer a question whose - answer is in the last third of a 60-minute transcript, chunked retrieval replaces the larger cap. +- *Retrieval chat*: build chunked retrieval when a recording's transcript exceeds the measured + 90,000-char ceiling (≈100 minutes) — i.e. the moment the duration cap is raised above that, or a + real recording turns out denser than the fixture — or when the offered model list changes and + the ceiling is re-measured lower. - *Worker*: if a transcription job ever degrades the reading UI, or Tapir runs more than one replica, ADR-018 is revisited. - *The decision itself*: if the maintainer has not used the feature unprompted within the