Upload a recording (audio, or video with the camera off), get a private Swedish transcript, and chat about it. - ADR-033: RLS-scoped private transcript store; ffprobe/ffmpeg intake; gateway transcription (iguana kb-whisper, berget fallback); in-process single-replica worker; chat over the transcript (local models only). - ADR-034: optional two-engine mode (pianissimo + kb-whisper) reconciled by a LOCAL model; off unless TAPIR_RECONCILE_BASE_MODEL is set. - ADR-035: real progress (phases, running/quiet/lost), phase logging, every judged spot stored, merged segments. - Hardening from four adversarial reviews: errors logged and stored as kinds never text; per-engine deadlines; converted-length check; bounded queue; client hang-ups are not errors; strict local-alias check. - Dockerfile: static ffmpeg/ffprobe 7.0.2 pinned by sha256 (#34). Built and inspected by hand; CI does not build the image on a PR, so its first CI build is this merge. Uploads stay OFF until TAPIR_UPLOADS_ENABLED is set in the deployment. Closes #33, #34, #42, #43, #44. Co-authored-by: Mathias <mthbqv@gmail.com>
This commit was merged in pull request #39.
This commit is contained in:
@@ -0,0 +1,85 @@
|
||||
# Can a local model adjudicate two ASR transcripts? (tapir#33, 2026-10-01)
|
||||
|
||||
Question: when `iguana/kb-whisper` and `iguana/pianissimo` disagree about a word or phrase,
|
||||
can a **local** model say which reading is right, well enough to merge the two transcripts?
|
||||
Ground truth: the speaker's own blind answers on 30 disagreements of one real recording
|
||||
(see `asr-kb-whisper-vs-pianissimo-2026-10-01.md`; no transcript text is recorded here,
|
||||
this repo is public). Reference: Claude Opus 5.5 agreed with the speaker on all 30.
|
||||
|
||||
## Method
|
||||
|
||||
- **Items:** 30 disagreement spans that Opus had picked as changing meaning, the speaker's
|
||||
verdict as truth (pianissimo right 17, kb-whisper right 13).
|
||||
- **Prompt:** one fixed English instruction (two transcriptions of Swedish speech differ at
|
||||
ONE marked spot; say which reading the speaker most likely said; reply `ANSWER: 1|2|UNSURE`
|
||||
plus one sentence). Each item was asked in **both reading orders** and in **two contexts**:
|
||||
narrow (+-8 words) and wide (+-100 words). 120 calls per model, temperature 0.
|
||||
- **Models:** 11, all local (`koala/*`, `iguana/*`): nothing left the homelab. Run one iguana
|
||||
model at a time, each unloaded afterwards, to avoid the memory pressure behind infra#273;
|
||||
whisper was confirmed healthy afterwards.
|
||||
- **Pass criteria, fixed before the run:** wide context, strict accuracy >= 80% (an
|
||||
unparseable or "unsure" answer counts as wrong), order-swap consistency >= 85%,
|
||||
unparseable <= 10%.
|
||||
|
||||
## Results (wide context, 60 calls per model)
|
||||
|
||||
| Model | Correct | 95% CI | Same pick when order swapped | Unparseable | s/call | Pass |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `iguana/gemma4-31b` | 58 = 97% | 89-99% | 93% | 0 | 10.4 | **yes** |
|
||||
| `iguana/gemma4-26b` | 55 = 92% | 82-96% | 93% | 0 | 2.0 | **yes** |
|
||||
| `iguana/ornith-35b` | 51 = 85% | 74-92% | 77% | 0 | 2.5 | no (consistency) |
|
||||
| `koala/phi4-14b` | 50 = 83% | 72-91% | 80% | 0 | 1.0 | no (consistency) |
|
||||
| `koala/qwen36-35b-a3b` | 49 = 82% | 70-89% | 83% | 0 | 3.3 | no (consistency) |
|
||||
| `iguana/granite4` | 47 = 78% | 66-87% | 83% | 0 | 4.6 | no |
|
||||
| `iguana/qwen3-14b` | 44 = 73% | 61-83% | 87% | 0 | 4.4 | no (accuracy) |
|
||||
| `iguana/gpt-oss-20b` | 40 = 67% | 54-77% | 100% | 17 | 11.2 | no (unparseable) |
|
||||
| `koala/phi4-mini` | 37 = 62% | 49-73% | 24% | 1 | 0.5 | no |
|
||||
| `iguana/gemma4-26b-think` | 23 = 38% | 27-51% | 100% | 37 | 15.3 | no (unparseable) |
|
||||
| `koala/qwen35-9b-fast` | 0 | n/a | n/a | 60 | 7.2 | no (no output) |
|
||||
|
||||
Narrow context only (+-8 words) costs 6-12 points for the models that pass: gemma4-31b 92%,
|
||||
gemma4-26b 83%, ornith-35b 80%. Context is not decoration: Opus also used evidence from
|
||||
elsewhere in the recording (an amount stated earlier, a term that recurs).
|
||||
|
||||
Position bias: the passing models pick "Reading 1" 47-51% of the time. `phi4-mini` picks it 15%
|
||||
and flips its answer when the order is swapped 76% of the time: it carries almost no signal.
|
||||
|
||||
## Reading it
|
||||
|
||||
- **Two models pass:** the gemma4 pair. Both clear the bar and neither shows position bias.
|
||||
26b is ~5x faster; their intervals overlap, so 30 items cannot separate 92% from 97%.
|
||||
- **The thinking models were not tested fairly.** `gpt-oss-20b` and `gemma4-26b-think` spend
|
||||
the 600-token budget reasoning and often never reach the answer line. When gemma4-26b-think
|
||||
did answer, it was right 23 of 23; gpt-oss-20b 93% of the answered. A larger budget would
|
||||
make them eligible; at 15 s a call it would not make them practical.
|
||||
|
||||
## What this does not show
|
||||
|
||||
- **Thirty items, one recording, one speaker, one domain.** The 30 were chosen by Opus as
|
||||
decidable, so they overstate how easy the typical disagreement is; ~350 others are mostly
|
||||
spelling, compounds and fillers, which no ground truth covers.
|
||||
- Opus agreeing 30/30 is not a measure of Opus over all 380 disagreements.
|
||||
- Accuracy was measured at temperature 0 on one prompt; no prompt tuning was tried.
|
||||
- The speaker's answers are the only truth; they are one person's recollection of what they said.
|
||||
|
||||
## Why gemma4-31b is slow, and what was ruled out
|
||||
|
||||
On iguana (M2 Ultra, 64 GB) the dense 31B runs at **103 tok/s prefill, 7.9 tok/s decode**; the
|
||||
26B MoE at 641 and 39.6. Ruled out as the cause: memory pressure (no swap, 45% free, model 100%
|
||||
on GPU), context length (32k vs 4k made no difference), and the **inference engine** — MLX 4-bit
|
||||
of the same model measured 76 tok/s prefill and 8.6 decode, the same within noise. (Terse
|
||||
output saves only ~3.6 s of ~9.3 s because prefill dominates.) Two unrelated engines at the same
|
||||
speed points at the machine: ~140 GB/s effective memory bandwidth on a part rated for 800, and
|
||||
**iguana had macOS Low Power Mode on** (`pmset` `lowpowermode 1`). A first MLX benchmark was
|
||||
wrong because `mlx_lm.server` caches prompts; the figures above are from the corrected run.
|
||||
|
||||
**Confirmed 2026-10-02.** The owner turned Low Power Mode off (`lowpowermode 0`, read back over
|
||||
ssh) and `gemma4:31b` was re-measured with nothing else resident: **217 tok/s prefill, 22.3
|
||||
tok/s decode** (4 runs, spread 22.2-22.6), against 103 and 7.9 with it on, so 2.1x and 2.8x. The
|
||||
slowness was the power mode, not the engine or the model. Caveats on the comparison: the new run
|
||||
used a different, synthetic Swedish prompt (~700 words, the original items held real transcript
|
||||
text and were deleted) and the model answered in 26-43 tokens, not the ~200 of the first run, so
|
||||
it is a same-machine before/after, not a controlled A/B. `gemma4:26b` and the MLX figures were
|
||||
not re-measured. The 31B's 97% judge accuracy is therefore usable at roughly a third of the
|
||||
latency the sweep implied; the 26B-vs-31B choice in ADR-034 was made on the slow numbers and is
|
||||
reopened by this.
|
||||
Reference in New Issue
Block a user