docs(adr): ADR-033 decision 8 -- chat budget and model from measurement
The proposed 120,000-char budget was a guess; measured, gemma4-26b silently loses the head at that size and phi4-mini (first in the offered list) 400s on an hour of speech. Budget is now 90,000, uploads open on gemma4-26b, the retrieval trigger is concrete, and the iguana contention hazard is recorded. ADR-033 is still Proposed, so this is an in-place correction, not a supersession. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
+31
-13
@@ -1353,11 +1353,13 @@ would leak across tenants **and pass CI green**.
|
|||||||
`berget/kb-whisper` returns sub-second times and a different `language` token.
|
`berget/kb-whisper` returns sub-second times and a different `language` token.
|
||||||
- **The existing chat path truncates at 18,000 chars (~22 min of speech), keeping the head.**
|
- **The existing chat path truncates at 18,000 chars (~22 min of speech), keeping the head.**
|
||||||
#33 assumed chat "retrieves rather than generates" and so has no length ceiling. It does
|
#33 assumed chat "retrieves rather than generates" and so has no length ceiling. It does
|
||||||
not retrieve; it stuffs the transcript into the prompt and cuts the tail.
|
not retrieve; it stuffs the transcript into the prompt and cuts the tail. Measured against
|
||||||
|
the offered models: `phi4-mini` 400s at ~13k tokens; `gemma4-26b` is right to 90,000 chars and
|
||||||
|
silently drops the head at 120,000.
|
||||||
|
|
||||||
**Unknowns, stated.** Real-speech accuracy; compressed-input behaviour; concurrent long jobs on
|
**Unknowns, stated.** Real-speech accuracy; compressed-input behaviour; concurrent long jobs on
|
||||||
iguana; whether the models chat offers can hold an hour of Swedish (~50 k chars); and the Metal
|
iguana; and the Metal OOM, deliberately not provoked. (Whether the offered chat models can hold
|
||||||
OOM, deliberately not provoked.
|
an hour of Swedish *was* an unknown here; it is now measured, see decision 8.)
|
||||||
|
|
||||||
**Decision.**
|
**Decision.**
|
||||||
|
|
||||||
@@ -1436,13 +1438,21 @@ OOM, deliberately not provoked.
|
|||||||
|
|
||||||
8. **Chat reuses `chat.Service`; it is not forked.** The uploaded-transcript chat handlers
|
8. **Chat reuses `chat.Service`; it is not forked.** The uploaded-transcript chat handlers
|
||||||
look the row up through the RLS-scoped store, so a non-owner gets 404 by the same mechanism
|
look the row up through the RLS-scoped store, so a non-owner gets 404 by the same mechanism
|
||||||
as ADR-027's summary gate. ADR-027's decisions (ephemeral, switchable models, trust-the-model)
|
as ADR-027's summary gate, **before any model is called**. ADR-027's decisions (ephemeral,
|
||||||
carry over unchanged. **One thing must differ:** the 18,000-char default cuts a one-hour
|
switchable models, trust-the-model) carry over unchanged. **Two things must differ, and both
|
||||||
meeting to its first third, so uploaded transcripts take their own cap,
|
were measured** (`docs/research/chat-context-over-long-transcripts-2026-10-01.md`):
|
||||||
`TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS`. **Provisional default 120,000 chars (~2.4 h at
|
- **The transcript budget.** The 18,000-char caption default cuts a one-hour meeting to its
|
||||||
measured density)**, with the existing `Truncated` flag surfaced honestly. The default is a
|
first third. Uploaded transcripts take their own `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS`,
|
||||||
guess until the chat slice measures what the offered models actually accept; it is not
|
default **90,000** — the measured ceiling of the best offered model (`iguana/gemma4-26b`),
|
||||||
evidence that they can.
|
which answers correctly there but **silently loses the head of a 120,000-char transcript**
|
||||||
|
with no error and no truncation flag. (An earlier draft of this ADR proposed 120,000 as a
|
||||||
|
guess; the measurement refuted it.) The mechanism is a per-request `chat.Request.MaxChars`.
|
||||||
|
- **The opening model.** The offered list leads with the summarizer default `koala/phi4-mini`,
|
||||||
|
whose 8,192-token context returns HTTP 400 on an hour of speech (~13,000 tokens). Chat over
|
||||||
|
an upload opens on `TAPIR_UPLOAD_CHAT_MODEL` (default `iguana/gemma4-26b`); the switcher
|
||||||
|
still lists every model, and picking `phi4-mini` on a long recording fails loudly and
|
||||||
|
inline. A per-model budget is not built.
|
||||||
|
Past ~90,000 chars (≈100 minutes) no offered model can answer by stuffing; see triggers.
|
||||||
|
|
||||||
9. **Not decided here.** Rich `AVSNITT`/`ALTERNATIV` analysis (gated on #30 — `analyze_srt.py`
|
9. **Not decided here.** Rich `AVSNITT`/`ALTERNATIV` analysis (gated on #30 — `analyze_srt.py`
|
||||||
hardcodes `max_tokens: 6000`); video, keyframes, synced playback (#28); language detection;
|
hardcodes `max_tokens: 6000`); video, keyframes, synced playback (#28); language detection;
|
||||||
@@ -1465,7 +1475,13 @@ deferred. The scope cut — chat first, no rich analysis, no video — is what k
|
|||||||
dependency that matters) — a Metal OOM now means *every upload fails over to berget*, i.e.
|
dependency that matters) — a Metal OOM now means *every upload fails over to berget*, i.e.
|
||||||
private audio leaving the homelab, silently unless the per-row endpoint is surfaced.
|
private audio leaving the homelab, silently unless the per-row endpoint is surfaced.
|
||||||
- A 90-minute job holds the single slot for ~7.5 minutes; a second user's upload queues behind it.
|
- A 90-minute job holds the single slot for ~7.5 minutes; a second user's upload queues behind it.
|
||||||
- Chat over a long transcript is bounded by model context, not by anything in Tapir.
|
- Chat over a long transcript is bounded by model context, not by anything in Tapir, and the
|
||||||
|
failure at the edge is *silent* for gemma (the head is dropped, no error). The 90,000 default
|
||||||
|
sits just inside the measured limit and is only as good as that single measurement.
|
||||||
|
- **The default chat model for uploads is a large ollama model on iguana — the same host as
|
||||||
|
`iguana/kb-whisper`.** infra#273 is exactly that a large resident ollama model can Metal-OOM
|
||||||
|
whisper-server permanently. Chatting with a transcript can therefore load the model that
|
||||||
|
wedges the transcriber. Not solved here; the berget fallback limits the blast radius.
|
||||||
- Tapir's own ingress body limit for a 200 MB upload is **unverified** (it lives in `infra`).
|
- Tapir's own ingress body limit for a 200 MB upload is **unverified** (it lives in `infra`).
|
||||||
|
|
||||||
**Validation / resolution triggers** (each observable by someone other than the author):
|
**Validation / resolution triggers** (each observable by someone other than the author):
|
||||||
@@ -1473,8 +1489,10 @@ deferred. The scope cut — chat first, no rich analysis, no video — is what k
|
|||||||
request at the current cap fails with whisper-side (not ingress) errors.
|
request at the current cap fails with whisper-side (not ingress) errors.
|
||||||
- *Berget-first*: if more than a quarter of uploads in any week are served by the fallback,
|
- *Berget-first*: if more than a quarter of uploads in any week are served by the fallback,
|
||||||
swap the primary/fallback order and revisit infra#273.
|
swap the primary/fallback order and revisit infra#273.
|
||||||
- *Retrieval chat*: if the chat slice finds an offered model cannot answer a question whose
|
- *Retrieval chat*: build chunked retrieval when a recording's transcript exceeds the measured
|
||||||
answer is in the last third of a 60-minute transcript, chunked retrieval replaces the larger cap.
|
90,000-char ceiling (≈100 minutes) — i.e. the moment the duration cap is raised above that, or a
|
||||||
|
real recording turns out denser than the fixture — or when the offered model list changes and
|
||||||
|
the ceiling is re-measured lower.
|
||||||
- *Worker*: if a transcription job ever degrades the reading UI, or Tapir runs more than one
|
- *Worker*: if a transcription job ever degrades the reading UI, or Tapir runs more than one
|
||||||
replica, ADR-018 is revisited.
|
replica, ADR-018 is revisited.
|
||||||
- *The decision itself*: if the maintainer has not used the feature unprompted within the
|
- *The decision itself*: if the maintainer has not used the feature unprompted within the
|
||||||
|
|||||||
Reference in New Issue
Block a user