From 2e9d76dad07e06e26ffc0d8af840b6e2edc1786d Mon Sep 17 00:00:00 2001 From: Mathias Date: Fri, 2 Oct 2026 08:01:31 +0200 Subject: [PATCH] docs(research): first blind check of two-engine reconcile on a real recording (aggregates only) Co-Authored-By: Claude Sonnet 5.5 --- .../reconcile-blind-check-2026-10-02.md | 68 +++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 docs/research/reconcile-blind-check-2026-10-02.md diff --git a/docs/research/reconcile-blind-check-2026-10-02.md b/docs/research/reconcile-blind-check-2026-10-02.md new file mode 100644 index 0000000..e0b6e22 --- /dev/null +++ b/docs/research/reconcile-blind-check-2026-10-02.md @@ -0,0 +1,68 @@ +# Two-engine reconcile, first check on a real recording (tapir#33, ADR-034, 2026-10-02) + +Aggregate numbers only. This repo is public and the recording is private (#31 custody is open), so no +transcript text, no disputed words and no per-item reading appears here. + +## What was run + +One real 63.6-minute Teams recording of one speaker dictating project notes (the video track, camera +off, was discarded). Build: PR #39 head `7f8e367`. Everything local: `iguana/pianissimo` as base, +`iguana/kb-whisper` as the second engine, `iguana/gemma4-31b` as reconciler (iguana Low Power Mode +off), cloud fallbacks off, a throwaway Postgres. + +| Measure | Value | +|---|---| +| Job time, upload to saved | 21 min 49 s for 63.6 min of audio (20.6 min per hour of audio) | +| Differences between the engines | 362 | +| Settled automatically (compound splits, up to two filler words) | 80 | +| Decided by the model | 282 | +| Of those: alternative reading used / base kept / uncertain | 130 / 151 / 1 | +| Model call errors | 0 | + +20.6 min per hour is inside the 25-minute resolution trigger in ADR-034, so no smaller adjudicator is +needed on speed grounds. The earlier estimate was about 10 min per hour on gemma4-26b; it was not +measured and this run used the 31B. + +## Blind check + +30 spots drawn at random (fixed seed) from the 89 replacement spots that could be placed in the base +text, across the whole recording (0:44 to 60:57). The two readings were shown as x and y in random +order, with the key kept apart from the judge. Judge: Claude Opus, text only, no audio, at the +speaker's request ("I trust Opus"). The speaker did not judge these items themselves. + +| Verdict | Items | +|---|---| +| The reconciler's reading (the alternative) is right | 21 | +| The base reading (pianissimo) is right, so the change made it worse | 1 | +| Neither reading is right | 3 | +| Cannot tell from text (compound, hyphen and number-format differences) | 5 | + +- Right of the items with a verdict: 21 of 22, 95% (95% interval 78% to 99%). +- Right of all 30, counting "neither" and "cannot tell" as not right: 21 of 30, 70% (52% to 83%). +- Right of the 25 that were not "cannot tell": 21 of 25, 84% (65% to 94%). +- Net effect against the base on these 30: 21 better, 1 worse, 8 unchanged in quality. + +## Reading the result + +ADR-034's bar was "right at least 85% of the time". It did not say how ties count, and that was a +flaw in the bar, not in the data. The literal reading (all 30) gives 70% and **does not meet** the bar; +counting only decided items gives 95% and meets it. Neither is a hidden pass or a hidden failure; both +are reported. What is not in doubt: on a random sample the reconciler improved the text far more often +than it harmed it (21 to 1), and the 8 unchanged cases are mostly spellings the text cannot settle. + +## What this did not measure + +- **Better single engine.** The second ADR-034 condition, "no worse than the better single engine", is + **not answered**. Only changed spots are stored, not the 151 where the model kept pianissimo, so the + reconciler cannot be compared with kb-whisper alone. The 21-to-1 result is against pianissimo. +- **The speaker as judge.** Opus judged from text. It cannot hear the audio, so it can be wrong where + the right word is audible but not inferable (items such as a mis-heard number or a name). Earlier, + Opus and the speaker agreed 30 of 30, but on items chosen as decidable; that does not transfer to a + random sample. +- **Insertions and deletions.** 42 of the 131 stored spots were not sampled: 39 are an added or + removed word, which cannot be placed in the base text, and 3 are the uncertain spot or could not be + located. The 39 are mostly small words ("och", "det"), so this sample says little about them. +- **One recording, one speaker, 30 items.** Wide intervals, and neighbouring spots (three pairs within + seconds of each other) are not independent. +- **Segments keep the base text.** The saved timed segments hold pianissimo's wording, not the merged + text; only the stored full text is merged. Found while building this sample; not yet an issue.