mathiasandClaude Sonnet 5.5
e182a6b07d
Eleven local models, 120 calls each, scored against the speaker's blind answers on 30 disagreements, with pass criteria fixed before the run. gemma4-31b (97%) and gemma4-26b (92%) pass; nothing else clears accuracy and order-consistency. Records why 31b is slow (MLX gives the same speed; Low Power Mode is the unproven suspect) and the caveats: 30 items, one recording, thinking models under-budgeted. Aggregate numbers only; this repo is public. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
4.8 KiB
Can a local model adjudicate two ASR transcripts? (tapir#33, 2026-10-01)
Question: when iguana/kb-whisper and iguana/pianissimo disagree about a word or phrase,
can a local model say which reading is right, well enough to merge the two transcripts?
Ground truth: the speaker's own blind answers on 30 disagreements of one real recording
(see asr-kb-whisper-vs-pianissimo-2026-10-01.md; no transcript text is recorded here,
this repo is public). Reference: Claude Opus 5.5 agreed with the speaker on all 30.
Method
- Items: 30 disagreement spans that Opus had picked as changing meaning, the speaker's verdict as truth (pianissimo right 17, kb-whisper right 13).
- Prompt: one fixed English instruction (two transcriptions of Swedish speech differ at
ONE marked spot; say which reading the speaker most likely said; reply
ANSWER: 1|2|UNSUREplus one sentence). Each item was asked in both reading orders and in two contexts: narrow (+-8 words) and wide (+-100 words). 120 calls per model, temperature 0. - Models: 11, all local (
koala/*,iguana/*): nothing left the homelab. Run one iguana model at a time, each unloaded afterwards, to avoid the memory pressure behind infra#273; whisper was confirmed healthy afterwards. - Pass criteria, fixed before the run: wide context, strict accuracy >= 80% (an unparseable or "unsure" answer counts as wrong), order-swap consistency >= 85%, unparseable <= 10%.
Results (wide context, 60 calls per model)
| Model | Correct | 95% CI | Same pick when order swapped | Unparseable | s/call | Pass |
|---|---|---|---|---|---|---|
iguana/gemma4-31b |
58 = 97% | 89-99% | 93% | 0 | 10.4 | yes |
iguana/gemma4-26b |
55 = 92% | 82-96% | 93% | 0 | 2.0 | yes |
iguana/ornith-35b |
51 = 85% | 74-92% | 77% | 0 | 2.5 | no (consistency) |
koala/phi4-14b |
50 = 83% | 72-91% | 80% | 0 | 1.0 | no (consistency) |
koala/qwen36-35b-a3b |
49 = 82% | 70-89% | 83% | 0 | 3.3 | no (consistency) |
iguana/granite4 |
47 = 78% | 66-87% | 83% | 0 | 4.6 | no |
iguana/qwen3-14b |
44 = 73% | 61-83% | 87% | 0 | 4.4 | no (accuracy) |
iguana/gpt-oss-20b |
40 = 67% | 54-77% | 100% | 17 | 11.2 | no (unparseable) |
koala/phi4-mini |
37 = 62% | 49-73% | 24% | 1 | 0.5 | no |
iguana/gemma4-26b-think |
23 = 38% | 27-51% | 100% | 37 | 15.3 | no (unparseable) |
koala/qwen35-9b-fast |
0 | n/a | n/a | 60 | 7.2 | no (no output) |
Narrow context only (+-8 words) costs 6-12 points for the models that pass: gemma4-31b 92%, gemma4-26b 83%, ornith-35b 80%. Context is not decoration: Opus also used evidence from elsewhere in the recording (an amount stated earlier, a term that recurs).
Position bias: the passing models pick "Reading 1" 47-51% of the time. phi4-mini picks it 15%
and flips its answer when the order is swapped 76% of the time: it carries almost no signal.
Reading it
- Two models pass: the gemma4 pair. Both clear the bar and neither shows position bias. 26b is ~5x faster; their intervals overlap, so 30 items cannot separate 92% from 97%.
- The thinking models were not tested fairly.
gpt-oss-20bandgemma4-26b-thinkspend the 600-token budget reasoning and often never reach the answer line. When gemma4-26b-think did answer, it was right 23 of 23; gpt-oss-20b 93% of the answered. A larger budget would make them eligible; at 15 s a call it would not make them practical.
What this does not show
- Thirty items, one recording, one speaker, one domain. The 30 were chosen by Opus as decidable, so they overstate how easy the typical disagreement is; ~350 others are mostly spelling, compounds and fillers, which no ground truth covers.
- Opus agreeing 30/30 is not a measure of Opus over all 380 disagreements.
- Accuracy was measured at temperature 0 on one prompt; no prompt tuning was tried.
- The speaker's answers are the only truth; they are one person's recollection of what they said.
Why gemma4-31b is slow, and what was ruled out
On iguana (M2 Ultra, 64 GB) the dense 31B runs at 103 tok/s prefill, 7.9 tok/s decode; the
26B MoE at 641 and 39.6. Ruled out as the cause: memory pressure (no swap, 45% free, model 100%
on GPU), context length (32k vs 4k made no difference), and the inference engine — MLX 4-bit
of the same model measured 76 tok/s prefill and 8.6 decode, the same within noise. (Terse
output saves only ~3.6 s of ~9.3 s because prefill dominates.) Two unrelated engines at the same
speed points at the machine: ~140 GB/s effective memory bandwidth on a part rated for 800, and
iguana has macOS Low Power Mode on (pmset lowpowermode 1). That is the leading suspect
and is not proven: turning it off needs root and was not done. A first MLX benchmark was
wrong because mlx_lm.server caches prompts; the figures above are from the corrected run.