docs: ADR-034 second blind check protocol; D8 decided (Berget allowed, per-recording local-only opt-out first, #60); #37 verified

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs/research/reconcile-second-check-protocol.md

@@ -0,0 +1,69 @@
+# ADR-034 second blind check: protocol (ready to run; tapir#33, D9, D10)
+
+This repo is public and the recordings are private, so this file holds the method only. The
+sheet, the key and the filled-in verdicts stay outside the repo, and the trial data is deleted
+after the check. Results go back here as aggregate numbers, as in
+`reconcile-blind-check-2026-10-02.md`.
+
+## What the check decides
+
+ADR-034 (two engines, a local model reconciles their disagreements) is `Proposed` and off in
+production. Two conditions were set for keeping it. The first run answered the first one on one
+reading only and did not answer the second:
+
+1. **The reconciled reading is right at least 85% of the time.** Open flaw in the bar: it did not
+   say how ties count. The first run gave 95% (decided items only) or 70% (all 30), so it was met on
+   one reading and missed on the other.
+2. **It is no worse than the better single engine.** Not answered: before ADR-035 the spots the
+   model *kept* as the base were not stored, so kb-whisper alone could not be compared.
+
+ADR-035 now stores every judged spot (`reconciliation.contested`: both readings, which one was
+used, unsure or not, a few words of left context, a time). That makes condition 2 measurable.
+
+## Rules to fix BEFORE looking at any result (the owner confirms or changes them)
+
+| Rule | Proposed |
+|---|---|
+| Who judges | The speaker, hearing the audio where the text cannot settle it (D9). Not Opus: it cannot hear, and its agreement with the speaker was only shown on items picked as decidable. |
+| Sample | 30 spots drawn at random with a fixed seed, from ALL judged spots (changed and kept-base alike), not only the changed ones. Insertions and deletions that cannot be placed are listed separately and not sampled. |
+| Verdict per spot | `x`, `y` (the two readings, shown in random order, key kept apart), `neither`, or `cannot tell`. |
+| Condition 1 | Reconciled is right on at least 85% of the spots where the judge could tell (`cannot tell` excluded, reported, and `neither` counts as not right). Also report all-30. Pass needs the decided-only figure at 85% or more AND the all-30 figure above 70%. |
+| Condition 2 | Over the same spots, the reconciled text is right at least as often as the better single engine (base engine alone is right where the base reading is right; the other engine alone where the alt reading is right). Ties favour the single engine. |
+| If it fails | Switch reconcile off by configuration; the single-engine path is unchanged (ADR-034 reversibility). |
+
+Why a fixed all-30 floor: the literal 70% last time sat well under the bar, and "decided only" can
+hide a model that mostly lands on spelling differences.
+
+## What a run needs (D10: the owner is present)
+
+- The same real recording as before (about 60 minutes). Only the owner has it.
+- A **throwaway** tapir instance with its own Postgres, not the production pod, so production's
+  key pin and deployment are untouched. Settings: `TAPIR_RECONCILE_BASE_MODEL=iguana/pianissimo`,
+  `TAPIR_TRANSCRIBE_MODEL=iguana/kb-whisper`, `TAPIR_RECONCILE_MODEL=iguana/gemma4-31b`, cloud
+  fallbacks empty. Job time was 21 min 49 s per 63.6 min last run.
+- iguana with Low Power Mode off and gemma4-31b able to load (the production-iguana model load
+  that needs a quiet window; D6).
+
+## Steps
+
+1. Start the throwaway instance, upload the recording, wait for `done` (about 22 minutes).
+2. Read `reconciliation.contested` from the throwaway database. From it build, **outside the repo**:
+   `sheet` (spot number, time, left context, reading x, reading y, random order) and `key` (which
+   of x/y is the base, which is the other engine, which one the reconciler used).
+3. The speaker fills in the sheet (about 20 to 30 minutes for 30 spots, audio open at the times).
+   The agent does not see the key until the sheet is returned.
+4. Score with the rules above. Write the aggregate numbers into a dated note next to this one,
+   and amend ADR-034's validation section (kept, or switched off, and the tie rule used).
+5. Delete the throwaway instance, its database and the sheet, as last time.
+6. Only if the check passes and the owner wants it in production: enabling reconcile there is
+   the separate D10 step (it needs `iguana/pianissimo` and the reconciler added to the tapir key
+   pin in the infra repo plus the env vars on the deployment, `risk/medium`, owner present).
+
+## Open and owner-only
+
+- Confirm or change the rules table above before the run.
+- The **tool that builds the sheet and key** is the one piece not in the repo (last time it was a
+  one-off script). It can be built test-first on request; until then the agent builds it ad hoc
+  outside the repo.
+- A Berget adjudicator (D8, tapir#45) is a different question with its own measurement and is
+  not part of this check: this check judges the local setup that ADR-034 describes.