tapir/docs/research/local-model-adjudicator-sweep-2026-10-01.md

Can a local model adjudicate two ASR transcripts? (tapir#33, 2026-10-01)

Question: when iguana/kb-whisper and iguana/pianissimo disagree about a word or phrase, can a local model say which reading is right, well enough to merge the two transcripts? Ground truth: the speaker's own blind answers on 30 disagreements of one real recording (see asr-kb-whisper-vs-pianissimo-2026-10-01.md; no transcript text is recorded here, this repo is public). Reference: Claude Opus 5.5 agreed with the speaker on all 30.

Method

• Items: 30 disagreement spans that Opus had picked as changing meaning, the speaker's verdict as truth (pianissimo right 17, kb-whisper right 13).
• Prompt: one fixed English instruction (two transcriptions of Swedish speech differ at ONE marked spot; say which reading the speaker most likely said; reply ANSWER: 1|2|UNSURE plus one sentence). Each item was asked in both reading orders and in two contexts: narrow (+-8 words) and wide (+-100 words). 120 calls per model, temperature 0.
• Models: 11, all local (koala/*, iguana/*): nothing left the homelab. Run one iguana model at a time, each unloaded afterwards, to avoid the memory pressure behind infra#273; whisper was confirmed healthy afterwards.
• Pass criteria, fixed before the run: wide context, strict accuracy >= 80% (an unparseable or "unsure" answer counts as wrong), order-swap consistency >= 85%, unparseable <= 10%.

Results (wide context, 60 calls per model)

Model	Correct	95% CI	Same pick when order swapped	Unparseable	s/call	Pass
iguana/gemma4-31b	58 = 97%	89-99%	93%	0	10.4	yes
iguana/gemma4-26b	55 = 92%	82-96%	93%	0	2.0	yes
iguana/ornith-35b	51 = 85%	74-92%	77%	0	2.5	no (consistency)
koala/phi4-14b	50 = 83%	72-91%	80%	0	1.0	no (consistency)
koala/qwen36-35b-a3b	49 = 82%	70-89%	83%	0	3.3	no (consistency)
iguana/granite4	47 = 78%	66-87%	83%	0	4.6	no
iguana/qwen3-14b	44 = 73%	61-83%	87%	0	4.4	no (accuracy)
iguana/gpt-oss-20b	40 = 67%	54-77%	100%	17	11.2	no (unparseable)
koala/phi4-mini	37 = 62%	49-73%	24%	1	0.5	no
iguana/gemma4-26b-think	23 = 38%	27-51%	100%	37	15.3	no (unparseable)
koala/qwen35-9b-fast	0	n/a	n/a	60	7.2	no (no output)

Narrow context only (+-8 words) costs 6-12 points for the models that pass: gemma4-31b 92%, gemma4-26b 83%, ornith-35b 80%. Context is not decoration: Opus also used evidence from elsewhere in the recording (an amount stated earlier, a term that recurs).

Position bias: the passing models pick "Reading 1" 47-51% of the time. phi4-mini picks it 15% and flips its answer when the order is swapped 76% of the time: it carries almost no signal.

Reading it

• Two models pass: the gemma4 pair. Both clear the bar and neither shows position bias. 26b is ~5x faster; their intervals overlap, so 30 items cannot separate 92% from 97%.
• The thinking models were not tested fairly. gpt-oss-20b and gemma4-26b-think spend the 600-token budget reasoning and often never reach the answer line. When gemma4-26b-think did answer, it was right 23 of 23; gpt-oss-20b 93% of the answered. A larger budget would make them eligible; at 15 s a call it would not make them practical.

What this does not show

• Thirty items, one recording, one speaker, one domain. The 30 were chosen by Opus as decidable, so they overstate how easy the typical disagreement is; ~350 others are mostly spelling, compounds and fillers, which no ground truth covers.
• Opus agreeing 30/30 is not a measure of Opus over all 380 disagreements.
• Accuracy was measured at temperature 0 on one prompt; no prompt tuning was tried.
• The speaker's answers are the only truth; they are one person's recollection of what they said.

Why gemma4-31b is slow, and what was ruled out

On iguana (M2 Ultra, 64 GB) the dense 31B runs at 103 tok/s prefill, 7.9 tok/s decode; the 26B MoE at 641 and 39.6. Ruled out as the cause: memory pressure (no swap, 45% free, model 100% on GPU), context length (32k vs 4k made no difference), and the inference engine — MLX 4-bit of the same model measured 76 tok/s prefill and 8.6 decode, the same within noise. (Terse output saves only ~3.6 s of ~9.3 s because prefill dominates.) Two unrelated engines at the same speed points at the machine: ~140 GB/s effective memory bandwidth on a part rated for 800, and iguana had macOS Low Power Mode on (pmset lowpowermode 1). A first MLX benchmark was wrong because mlx_lm.server caches prompts; the figures above are from the corrected run.

Confirmed 2026-10-02. The owner turned Low Power Mode off (lowpowermode 0, read back over ssh) and gemma4:31b was re-measured with nothing else resident: 217 tok/s prefill, 22.3 tok/s decode (4 runs, spread 22.2-22.6), against 103 and 7.9 with it on, so 2.1x and 2.8x. The slowness was the power mode, not the engine or the model. Caveats on the comparison: the new run used a different, synthetic Swedish prompt (~700 words, the original items held real transcript text and were deleted) and the model answered in 26-43 tokens, not the ~200 of the first run, so it is a same-machine before/after, not a controlled A/B. gemma4:26b and the MLX figures were not re-measured. The 31B's 97% judge accuracy is therefore usable at roughly a third of the latency the sweep implied; the 26B-vs-31B choice in ADR-034 was made on the slow numbers and is reopened by this.