2026-10-03 14:13:37 +00:00
1 changed files with 68 additions and 0 deletions
@@ -0,0 +1,68 @@
+# Two-engine reconcile, first check on a real recording (tapir#33, ADR-034, 2026-10-02)
+
+Aggregate numbers only. This repo is public and the recording is private (#31 custody is open), so no
+transcript text, no disputed words and no per-item reading appears here.
+
+## What was run
+
+One real 63.6-minute Teams recording of one speaker dictating project notes (the video track, camera
+off, was discarded). Build: PR #39 head `7f8e367`. Everything local: `iguana/pianissimo` as base,
+`iguana/kb-whisper` as the second engine, `iguana/gemma4-31b` as reconciler (iguana Low Power Mode
+off), cloud fallbacks off, a throwaway Postgres.
+
+| Measure | Value |
+|---|---|
+| Job time, upload to saved | 21 min 49 s for 63.6 min of audio (20.6 min per hour of audio) |
+| Differences between the engines | 362 |
+| Settled automatically (compound splits, up to two filler words) | 80 |
+| Decided by the model | 282 |
+| Of those: alternative reading used / base kept / uncertain | 130 / 151 / 1 |
+| Model call errors | 0 |
+
+20.6 min per hour is inside the 25-minute resolution trigger in ADR-034, so no smaller adjudicator is
+needed on speed grounds. The earlier estimate was about 10 min per hour on gemma4-26b; it was not
+measured and this run used the 31B.
+
+## Blind check
+
+30 spots drawn at random (fixed seed) from the 89 replacement spots that could be placed in the base
+text, across the whole recording (0:44 to 60:57). The two readings were shown as x and y in random
+order, with the key kept apart from the judge. Judge: Claude Opus, text only, no audio, at the
+speaker's request ("I trust Opus"). The speaker did not judge these items themselves.
+
+| Verdict | Items |
+|---|---|
+| The reconciler's reading (the alternative) is right | 21 |
+| The base reading (pianissimo) is right, so the change made it worse | 1 |
+| Neither reading is right | 3 |
+| Cannot tell from text (compound, hyphen and number-format differences) | 5 |
+
+- Right of the items with a verdict: 21 of 22, 95% (95% interval 78% to 99%).
+- Right of all 30, counting "neither" and "cannot tell" as not right: 21 of 30, 70% (52% to 83%).
+- Right of the 25 that were not "cannot tell": 21 of 25, 84% (65% to 94%).
+- Net effect against the base on these 30: 21 better, 1 worse, 8 unchanged in quality.
+
+## Reading the result
+
+ADR-034's bar was "right at least 85% of the time". It did not say how ties count, and that was a
+flaw in the bar, not in the data. The literal reading (all 30) gives 70% and **does not meet** the bar;
+counting only decided items gives 95% and meets it. Neither is a hidden pass or a hidden failure; both
+are reported. What is not in doubt: on a random sample the reconciler improved the text far more often
+than it harmed it (21 to 1), and the 8 unchanged cases are mostly spellings the text cannot settle.
+
+## What this did not measure
+
+- **Better single engine.** The second ADR-034 condition, "no worse than the better single engine", is
+  **not answered**. Only changed spots are stored, not the 151 where the model kept pianissimo, so the
+  reconciler cannot be compared with kb-whisper alone. The 21-to-1 result is against pianissimo.
+- **The speaker as judge.** Opus judged from text. It cannot hear the audio, so it can be wrong where
+  the right word is audible but not inferable (items such as a mis-heard number or a name). Earlier,
+  Opus and the speaker agreed 30 of 30, but on items chosen as decidable; that does not transfer to a
+  random sample.
+- **Insertions and deletions.** 42 of the 131 stored spots were not sampled: 39 are an added or
+  removed word, which cannot be placed in the base text, and 3 are the uncertain spot or could not be
+  located. The 39 are mostly small words ("och", "det"), so this sample says little about them.
+- **One recording, one speaker, 30 items.** Wide intervals, and neighbouring spots (three pairs within
+  seconds of each other) are not independent.
+- **Segments keep the base text.** The saved timed segments hold pianissimo's wording, not the merged
+  text; only the stored full text is merged. Found while building this sample; not yet an issue.