tapir/docs/research/reconcile-second-check-protocol.md

ADR-034 second blind check: protocol (ready to run; tapir#33, D9, D10)

This repo is public and the recordings are private, so this file holds the method only. The sheet, the key and the filled-in verdicts stay outside the repo, and the trial data is deleted after the check. Results go back here as aggregate numbers, as in reconcile-blind-check-2026-10-02.md.

What the check decides

ADR-034 (two engines, a local model reconciles their disagreements) is Proposed and off in production. Two conditions were set for keeping it. The first run answered the first one on one reading only and did not answer the second:

1. The reconciled reading is right at least 85% of the time. Open flaw in the bar: it did not say how ties count. The first run gave 95% (decided items only) or 70% (all 30), so it was met on one reading and missed on the other.
2. It is no worse than the better single engine. Not answered: before ADR-035 the spots the model kept as the base were not stored, so kb-whisper alone could not be compared.

ADR-035 now stores every judged spot (reconciliation.contested: both readings, which one was used, unsure or not, a few words of left context, a time). That makes condition 2 measurable.

Rules to fix BEFORE looking at any result (the owner confirms or changes them)

Rule	Proposed
Who judges	The speaker, hearing the audio where the text cannot settle it (D9). Not Opus: it cannot hear, and its agreement with the speaker was only shown on items picked as decidable.
Sample	30 spots drawn at random with a fixed seed, from ALL judged spots (changed and kept-base alike), not only the changed ones. Insertions and deletions that cannot be placed are listed separately and not sampled.
Verdict per spot	x, y (the two readings, shown in random order, key kept apart), neither, or cannot tell.
Condition 1	Reconciled is right on at least 85% of the spots where the judge could tell (cannot tell excluded, reported, and neither counts as not right). Also report all-30. Pass needs the decided-only figure at 85% or more AND the all-30 figure above 70%.
Condition 2	Over the same spots, the reconciled text is right at least as often as the better single engine (base engine alone is right where the base reading is right; the other engine alone where the alt reading is right). Ties favour the single engine.
If it fails	Switch reconcile off by configuration; the single-engine path is unchanged (ADR-034 reversibility).

Why a fixed all-30 floor: the literal 70% last time sat well under the bar, and "decided only" can hide a model that mostly lands on spelling differences.

What a run needs (D10: the owner is present)

• The same real recording as before (about 60 minutes). Only the owner has it.
• A throwaway tapir instance with its own Postgres, not the production pod, so production's key pin and deployment are untouched. Settings: TAPIR_RECONCILE_BASE_MODEL=iguana/pianissimo, TAPIR_TRANSCRIBE_MODEL=iguana/kb-whisper, TAPIR_RECONCILE_MODEL=iguana/gemma4-31b, cloud fallbacks empty. Job time was 21 min 49 s per 63.6 min last run.
• iguana with Low Power Mode off and gemma4-31b able to load (the production-iguana model load that needs a quiet window; D6).

Steps

1. Start the throwaway instance, upload the recording, wait for done (about 22 minutes).
2. Read reconciliation.contested from the throwaway database. From it build, outside the repo: sheet (spot number, time, left context, reading x, reading y, random order) and key (which of x/y is the base, which is the other engine, which one the reconciler used).
3. The speaker fills in the sheet (about 20 to 30 minutes for 30 spots, audio open at the times). The agent does not see the key until the sheet is returned.
4. Score with the rules above. Write the aggregate numbers into a dated note next to this one, and amend ADR-034's validation section (kept, or switched off, and the tie rule used).
5. Delete the throwaway instance, its database and the sheet, as last time.
6. Only if the check passes and the owner wants it in production: enabling reconcile there is the separate D10 step (it needs iguana/pianissimo and the reconciler added to the tapir key pin in the infra repo plus the env vars on the deployment, risk/medium, owner present).

Open and owner-only

• Confirm or change the rules table above before the run.
• The tool that builds the sheet and key is the one piece not in the repo (last time it was a one-off script). It can be built test-first on request; until then the agent builds it ad hoc outside the repo.
• A Berget adjudicator (D8, tapir#45) is a different question with its own measurement and is not part of this check: this check judges the local setup that ADR-034 describes.