Rewritten 2026-08-13 after review. The original version would have returned a false negative and recommended buying cloud models to fix a hardcoded constant. What changed is recorded at the bottom.
Blocked on #32 (demand pre-spike). Runs against IMG_1233.sv.srt, which exists — no new file, no ffmpeg, no whisper, no dependency on #29.
Question to answer
scripts/spike-media/analyze_srt.py produces the /bygge analysis today using berget/mistral-medium — a mid-tier cloud model. #28's decision 6 asks whether the rich schema is viable at all. The precise open question is narrower than the epic states:
Can a model hosted on koala or iguana clear the bar that berget/mistral-medium already set, on the same SRT, with the same prompt?
Fix these three things before the first run, or the result is worthless
1. max_tokens: 6000 (analyze_srt.py:194) truncates. The 4-minute reference analysis is 14,262 bytes ≈ 4,700 output tokens — 78% of budget for four minutes. Anything past ~6 minutes truncates for every model including the reference. Truncated JSON scored as "malformed" reads as a local-model failure and points straight at #22. Raise it, and record finish_reason per run so truncated and malformed are counted separately.
2. There is no control arm. The original banned berget/* on sovereignty grounds. But berget produced the reference — excluding it removes the only calibration point, not a temptation. Exclude cloud from the recommendation, not from the measurement. It is one --model flag on a script that already takes one.
3. Score citations in three tiers, not as a substring test. Measured against the reference output (75 items, 61 citations):
Check
berget/mistral-medium scores
timestamp is an exact SRT cue start
74/75 (99%)
citation is a verbatim substring
54/61 (89%)
— of which alternativ citations
2/7 (29%)
alternativ citations stitch non-contiguous passages by design — that is what makes the card readable. A strict substring test fails the model that made the page, and a local model behaving identically would be scored as ungrounded. analyze_srt.py:375 already grades a third tier (citat matchar bara delvis). Use it.
Note also what the 99% means: the timestamp check is near-solved, because the model copies cue starts it was handed. It is not the interesting metric.
Use the two checks that already exist and already caught real errors
The original issue proposed building the two weakest of four checks and omitted both that matter:
TACKNINGSLUCKA (:439) — clip spans no section covers. This is the recall term. Without it, {"avsnitt":[],"beslut":[],"matt":[]} scores zero fabrications, zero bad citations, parses 10/10, and wins. It fired on four of five prior runs.
TALET … FINNS INTE I CITATET (:407) — converts Swedish number-words to digits, then checks the claimed number against its own citation. It caught 45×230 where the transcript said 220, carried by a verbatim citation and a real timestamp. Both proposed checks pass; the document is wrong about a structural dimension. The script's own comment names this the most dangerous error class in a construction document, and it is right.
Method
Schema is analyze_srt.py's USER_TMPL — the real spec, not the HTML. Drop the fields nothing renders before running: projekt.sammanfattning, projekt.skede, avsnitt.bildtid, avsnitt.bildmotiv, material[], oklart[]. They are leftovers from a still-frame markdown output that build_page.py replaced, and they are paid for in output tokens on every run — directly against the ceiling above.
Candidates: iguana/gemma4-26b (the ADR-022 fallback, and the actual candidate — its near-twin berget/gemma4-31b was tested but it was not), whatever koala's llama-swap serves, and berget/mistral-mediumas control. Record the model list at run time; do not trust models.yml.
N=3 per model across 2 recordings, not N=10 on one. At temperature=0 repeated runs on one input measure near-zero sampling variance. The variance that matters is across recordings.
State the bar before the first run.
The contention experiment — keep, it is cheap and it settles #28 decision 3
Run an iguana-hosted analysis while a transcription is in flight, once, deliberately, with launchctl kickstart ready. Per infra#273 the expected result is not slowness: the chat model's footprint starves whisper of its ~3 GB and latches the backend permanently. If it reproduces, decision 3 is settled by evidence — analysis and transcription cannot share iguana.
Non-goals
Cloud in the recommendation. It stays in the measurement as the control.
Any database write or tapir code path. SRT in, JSON out.
Prompt-tuning to a shine, or shrinking the schema to whatever the model finds easy. A schema change is #28's ADR to make, with this evidence attached.
"Swedish quality by eye on the best run" — cut. Best-of selection is not a measurement.
Acceptance
caveman: me turn spoken words into timestamped decisions and numbers, using only machines me already own
max_tokens raised and justified; output tokens reported per run
Bar stated in this issue before the first run. Proposed: within 10 pp of the control on citation-grounding, zero number violations, <15 s uncovered.
Per model, four parse counts: parsed_first_try / parsed_tolerant / truncated (finish_reason=length) / malformed
Per model, four quality numbers: timestamps not matching any cue start; citations graded exact / partial / absent; coverage-gap seconds (TACKNINGSLUCKA); number violations (TALET)
All of the above alongside berget/mistral-medium on the same SRT and prompt
Red-first tests on the validator's four checks, using fixtures/fake.sv.srt + fixtures/fake.analys.json — no model call. The fixture has planted numbers precisely so the number check can be tested red first.
Contention experiment run once, result recorded either way
Recommendation to #28 decision 6 with the numbers attached
Writeup in docs/research/
What changed in this rewrite
The original: assumed the analysis was hand-made by a frontier model (it was scripted, by mistral-medium, at temperature 0); banned the control arm; proposed two precision-only checks that an empty answer passes perfectly and that score the reference itself at 89%/29%; missed max_tokens; missed the two existing checks that caught real errors; and specified N=10 on a single input at temperature 0.
Refs
scripts/spike-media/ (committed 2026-08-13 — the prompt, schema, validator, fixtures, and their documented defects)
#32 (blocks this), #29 (independent — not a dependency), #28 decisions 3 and 6, #22
**Rewritten 2026-08-13** after review. The original version would have returned a false negative and recommended buying cloud models to fix a hardcoded constant. What changed is recorded at the bottom.
Blocked on #32 (demand pre-spike). Runs against `IMG_1233.sv.srt`, which exists — **no new file, no ffmpeg, no whisper, no dependency on #29.**
## Question to answer
`scripts/spike-media/analyze_srt.py` produces the `/bygge` analysis today using **`berget/mistral-medium`** — a mid-tier *cloud* model. #28's decision 6 asks whether the rich schema is viable at all. The precise open question is narrower than the epic states:
> Can a model hosted on koala or iguana clear the bar that `berget/mistral-medium` already set, on the same SRT, with the same prompt?
## Fix these three things before the first run, or the result is worthless
**1. `max_tokens: 6000` (`analyze_srt.py:194`) truncates.** The 4-minute reference analysis is 14,262 bytes ≈ **4,700 output tokens — 78% of budget for four minutes.** Anything past ~6 minutes truncates for *every* model including the reference. Truncated JSON scored as "malformed" reads as a local-model failure and points straight at #22. Raise it, and record `finish_reason` per run so `truncated` and `malformed` are counted separately.
**2. There is no control arm.** The original banned `berget/*` on sovereignty grounds. But berget produced the reference — excluding it removes the only calibration point, not a temptation. **Exclude cloud from the recommendation, not from the measurement.** It is one `--model` flag on a script that already takes one.
**3. Score citations in three tiers, not as a substring test.** Measured against the reference output (75 items, 61 citations):
| Check | `berget/mistral-medium` scores |
|---|---|
| timestamp is an exact SRT cue start | 74/75 (99%) |
| citation is a verbatim substring | **54/61 (89%)** |
| — of which `alternativ` citations | **2/7 (29%)** |
`alternativ` citations stitch non-contiguous passages by design — that is what makes the card readable. A strict substring test **fails the model that made the page**, and a local model behaving identically would be scored as ungrounded. `analyze_srt.py:375` already grades a third tier (`citat matchar bara delvis`). Use it.
Note also what the 99% means: the timestamp check is near-solved, because the model copies cue starts it was handed. It is not the interesting metric.
## Use the two checks that already exist and already caught real errors
The original issue proposed building the two weakest of four checks and omitted both that matter:
- **`TACKNINGSLUCKA`** (`:439`) — clip spans no section covers. **This is the recall term.** Without it, `{"avsnitt":[],"beslut":[],"matt":[]}` scores zero fabrications, zero bad citations, parses 10/10, and wins. It fired on four of five prior runs.
- **`TALET … FINNS INTE I CITATET`** (`:407`) — converts Swedish number-words to digits, then checks the claimed number against its own citation. It caught `45×230` where the transcript said `220`, carried by a **verbatim** citation and a **real** timestamp. Both proposed checks pass; the document is wrong about a structural dimension. The script's own comment names this the most dangerous error class in a construction document, and it is right.
## Method
- Schema is `analyze_srt.py`'s `USER_TMPL` — the real spec, not the HTML. **Drop the fields nothing renders** before running: `projekt.sammanfattning`, `projekt.skede`, `avsnitt.bildtid`, `avsnitt.bildmotiv`, `material[]`, `oklart[]`. They are leftovers from a still-frame markdown output that `build_page.py` replaced, and they are paid for in output tokens on every run — directly against the ceiling above.
- Candidates: `iguana/gemma4-26b` (the ADR-022 fallback, and the actual candidate — its near-twin `berget/gemma4-31b` was tested but it was not), whatever koala's llama-swap serves, and `berget/mistral-medium` **as control**. Record the model list at run time; do not trust `models.yml`.
- **N=3 per model across 2 recordings, not N=10 on one.** At `temperature=0` repeated runs on one input measure near-zero sampling variance. The variance that matters is across recordings.
- State the bar **before** the first run.
## The contention experiment — keep, it is cheap and it settles #28 decision 3
Run an iguana-hosted analysis while a transcription is in flight, once, deliberately, with `launchctl kickstart` ready. Per infra#273 the expected result is not slowness: the chat model's footprint starves whisper of its ~3 GB and **latches the backend permanently**. If it reproduces, decision 3 is settled by evidence — analysis and transcription cannot share iguana.
## Non-goals
- Cloud in the *recommendation*. It stays in the measurement as the control.
- Any database write or tapir code path. SRT in, JSON out.
- Prompt-tuning to a shine, or shrinking the schema to whatever the model finds easy. A schema change is #28's ADR to make, with this evidence attached.
- "Swedish quality by eye on the best run" — cut. Best-of selection is not a measurement.
## Acceptance
- [ ] `caveman: me turn spoken words into timestamped decisions and numbers, using only machines me already own`
- [ ] `max_tokens` raised and justified; **output tokens reported per run**
- [ ] Bar stated in this issue **before** the first run. Proposed: within 10 pp of the control on citation-grounding, **zero** number violations, **<15 s** uncovered.
- [ ] Per model, four parse counts: `parsed_first_try` / `parsed_tolerant` / `truncated (finish_reason=length)` / `malformed`
- [ ] Per model, four quality numbers: timestamps not matching any cue start; citations graded **exact / partial / absent**; coverage-gap seconds (`TACKNINGSLUCKA`); number violations (`TALET`)
- [ ] All of the above **alongside `berget/mistral-medium` on the same SRT and prompt**
- [ ] Red-first tests on the validator's four checks, using `fixtures/fake.sv.srt` + `fixtures/fake.analys.json` — **no model call**. The fixture has planted numbers precisely so the number check can be tested red first.
- [ ] Contention experiment run once, result recorded either way
- [ ] Recommendation to #28 decision 6 with the numbers attached
- [ ] Writeup in `docs/research/`
## What changed in this rewrite
The original: assumed the analysis was hand-made by a frontier model (it was scripted, by mistral-medium, at temperature 0); banned the control arm; proposed two precision-only checks that an empty answer passes perfectly and that score the reference itself at 89%/29%; missed `max_tokens`; missed the two existing checks that caught real errors; and specified N=10 on a single input at temperature 0.
## Refs
- `scripts/spike-media/` (committed 2026-08-13 — the prompt, schema, validator, fixtures, and their documented defects)
- #32 (blocks this), #29 (independent — not a dependency), #28 decisions 3 and 6, #22
- ADR-022 (tolerant parse §3), infra#273 (Metal OOM — the contention mechanism)
mathias
changed title from Spike 2/3: can a local model produce the /bygge analysis schema reliably enough to parse? to Spike A: does a locally-hosted model hold the /bygge schema? (runs today, no new file)2026-08-13 09:38:25 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Rewritten 2026-08-13 after review. The original version would have returned a false negative and recommended buying cloud models to fix a hardcoded constant. What changed is recorded at the bottom.
Blocked on #32 (demand pre-spike). Runs against
IMG_1233.sv.srt, which exists — no new file, no ffmpeg, no whisper, no dependency on #29.Question to answer
scripts/spike-media/analyze_srt.pyproduces the/byggeanalysis today usingberget/mistral-medium— a mid-tier cloud model. #28's decision 6 asks whether the rich schema is viable at all. The precise open question is narrower than the epic states:Fix these three things before the first run, or the result is worthless
1.
max_tokens: 6000(analyze_srt.py:194) truncates. The 4-minute reference analysis is 14,262 bytes ≈ 4,700 output tokens — 78% of budget for four minutes. Anything past ~6 minutes truncates for every model including the reference. Truncated JSON scored as "malformed" reads as a local-model failure and points straight at #22. Raise it, and recordfinish_reasonper run sotruncatedandmalformedare counted separately.2. There is no control arm. The original banned
berget/*on sovereignty grounds. But berget produced the reference — excluding it removes the only calibration point, not a temptation. Exclude cloud from the recommendation, not from the measurement. It is one--modelflag on a script that already takes one.3. Score citations in three tiers, not as a substring test. Measured against the reference output (75 items, 61 citations):
berget/mistral-mediumscoresalternativcitationsalternativcitations stitch non-contiguous passages by design — that is what makes the card readable. A strict substring test fails the model that made the page, and a local model behaving identically would be scored as ungrounded.analyze_srt.py:375already grades a third tier (citat matchar bara delvis). Use it.Note also what the 99% means: the timestamp check is near-solved, because the model copies cue starts it was handed. It is not the interesting metric.
Use the two checks that already exist and already caught real errors
The original issue proposed building the two weakest of four checks and omitted both that matter:
TACKNINGSLUCKA(:439) — clip spans no section covers. This is the recall term. Without it,{"avsnitt":[],"beslut":[],"matt":[]}scores zero fabrications, zero bad citations, parses 10/10, and wins. It fired on four of five prior runs.TALET … FINNS INTE I CITATET(:407) — converts Swedish number-words to digits, then checks the claimed number against its own citation. It caught45×230where the transcript said220, carried by a verbatim citation and a real timestamp. Both proposed checks pass; the document is wrong about a structural dimension. The script's own comment names this the most dangerous error class in a construction document, and it is right.Method
analyze_srt.py'sUSER_TMPL— the real spec, not the HTML. Drop the fields nothing renders before running:projekt.sammanfattning,projekt.skede,avsnitt.bildtid,avsnitt.bildmotiv,material[],oklart[]. They are leftovers from a still-frame markdown output thatbuild_page.pyreplaced, and they are paid for in output tokens on every run — directly against the ceiling above.iguana/gemma4-26b(the ADR-022 fallback, and the actual candidate — its near-twinberget/gemma4-31bwas tested but it was not), whatever koala's llama-swap serves, andberget/mistral-mediumas control. Record the model list at run time; do not trustmodels.yml.temperature=0repeated runs on one input measure near-zero sampling variance. The variance that matters is across recordings.The contention experiment — keep, it is cheap and it settles #28 decision 3
Run an iguana-hosted analysis while a transcription is in flight, once, deliberately, with
launchctl kickstartready. Per infra#273 the expected result is not slowness: the chat model's footprint starves whisper of its ~3 GB and latches the backend permanently. If it reproduces, decision 3 is settled by evidence — analysis and transcription cannot share iguana.Non-goals
Acceptance
caveman: me turn spoken words into timestamped decisions and numbers, using only machines me already ownmax_tokensraised and justified; output tokens reported per runparsed_first_try/parsed_tolerant/truncated (finish_reason=length)/malformedTACKNINGSLUCKA); number violations (TALET)berget/mistral-mediumon the same SRT and promptfixtures/fake.sv.srt+fixtures/fake.analys.json— no model call. The fixture has planted numbers precisely so the number check can be tested red first.docs/research/What changed in this rewrite
The original: assumed the analysis was hand-made by a frontier model (it was scripted, by mistral-medium, at temperature 0); banned the control arm; proposed two precision-only checks that an empty answer passes perfectly and that score the reference itself at 89%/29%; missed
max_tokens; missed the two existing checks that caught real errors; and specified N=10 on a single input at temperature 0.Refs
scripts/spike-media/(committed 2026-08-13 — the prompt, schema, validator, fixtures, and their documented defects)Spike 2/3: can a local model produce the /bygge analysis schema reliably enough to parse?to Spike A: does a locally-hosted model hold the /bygge schema? (runs today, no new file)