tapir/docs/research/reconcile-blind-check-2026-10-02.md

Two-engine reconcile, first check on a real recording (tapir#33, ADR-034, 2026-10-02)

Aggregate numbers only. This repo is public and the recording is private (#31 custody is open), so no transcript text, no disputed words and no per-item reading appears here.

What was run

One real 63.6-minute Teams recording of one speaker dictating project notes (the video track, camera off, was discarded). Build: PR #39 head 7f8e367. Everything local: iguana/pianissimo as base, iguana/kb-whisper as the second engine, iguana/gemma4-31b as reconciler (iguana Low Power Mode off), cloud fallbacks off, a throwaway Postgres.

Measure	Value
Job time, upload to saved	21 min 49 s for 63.6 min of audio (20.6 min per hour of audio)
Differences between the engines	362
Settled automatically (compound splits, up to two filler words)	80
Decided by the model	282
Of those: alternative reading used / base kept / uncertain	130 / 151 / 1
Model call errors	0

20.6 min per hour is inside the 25-minute resolution trigger in ADR-034, so no smaller adjudicator is needed on speed grounds. The earlier estimate was about 10 min per hour on gemma4-26b; it was not measured and this run used the 31B.

Blind check

30 spots drawn at random (fixed seed) from the 89 replacement spots that could be placed in the base text, across the whole recording (0:44 to 60:57). The two readings were shown as x and y in random order, with the key kept apart from the judge. Judge: Claude Opus, text only, no audio, at the speaker's request ("I trust Opus"). The speaker did not judge these items themselves.

Verdict	Items
The reconciler's reading (the alternative) is right	21
The base reading (pianissimo) is right, so the change made it worse	1
Neither reading is right	3
Cannot tell from text (compound, hyphen and number-format differences)	5

• Right of the items with a verdict: 21 of 22, 95% (95% interval 78% to 99%).
• Right of all 30, counting "neither" and "cannot tell" as not right: 21 of 30, 70% (52% to 83%).
• Right of the 25 that were not "cannot tell": 21 of 25, 84% (65% to 94%).
• Net effect against the base on these 30: 21 better, 1 worse, 8 unchanged in quality.

Reading the result

ADR-034's bar was "right at least 85% of the time". It did not say how ties count, and that was a flaw in the bar, not in the data. The literal reading (all 30) gives 70% and does not meet the bar; counting only decided items gives 95% and meets it. Neither is a hidden pass or a hidden failure; both are reported. What is not in doubt: on a random sample the reconciler improved the text far more often than it harmed it (21 to 1), and the 8 unchanged cases are mostly spellings the text cannot settle.

What this did not measure

• Better single engine. The second ADR-034 condition, "no worse than the better single engine", is not answered. Only changed spots are stored, not the 151 where the model kept pianissimo, so the reconciler cannot be compared with kb-whisper alone. The 21-to-1 result is against pianissimo.
• The speaker as judge. Opus judged from text. It cannot hear the audio, so it can be wrong where the right word is audible but not inferable (items such as a mis-heard number or a name). Earlier, Opus and the speaker agreed 30 of 30, but on items chosen as decidable; that does not transfer to a random sample.
• Insertions and deletions. 42 of the 131 stored spots were not sampled: 39 are an added or removed word, which cannot be placed in the base text, and 3 are the uncertain spot or could not be located. The 39 are mostly small words ("och", "det"), so this sample says little about them.
• One recording, one speaker, 30 items. Wide intervals, and neighbouring spots (three pairs within seconds of each other) are not independent.
• Segments keep the base text. The saved timed segments hold pianissimo's wording, not the merged text; only the stored full text is merged. Found while building this sample; not yet an issue.