70 lines
4.8 KiB
Markdown
70 lines
4.8 KiB
Markdown
# ADR-034 second blind check: protocol (ready to run; tapir#33, D9, D10)
|
|
|
|
This repo is public and the recordings are private, so this file holds the method only. The
|
|
sheet, the key and the filled-in verdicts stay outside the repo, and the trial data is deleted
|
|
after the check. Results go back here as aggregate numbers, as in
|
|
`reconcile-blind-check-2026-10-02.md`.
|
|
|
|
## What the check decides
|
|
|
|
ADR-034 (two engines, a local model reconciles their disagreements) is `Proposed` and off in
|
|
production. Two conditions were set for keeping it. The first run answered the first one on one
|
|
reading only and did not answer the second:
|
|
|
|
1. **The reconciled reading is right at least 85% of the time.** Open flaw in the bar: it did not
|
|
say how ties count. The first run gave 95% (decided items only) or 70% (all 30), so it was met on
|
|
one reading and missed on the other.
|
|
2. **It is no worse than the better single engine.** Not answered: before ADR-035 the spots the
|
|
model *kept* as the base were not stored, so kb-whisper alone could not be compared.
|
|
|
|
ADR-035 now stores every judged spot (`reconciliation.contested`: both readings, which one was
|
|
used, unsure or not, a few words of left context, a time). That makes condition 2 measurable.
|
|
|
|
## Rules to fix BEFORE looking at any result (the owner confirms or changes them)
|
|
|
|
| Rule | Proposed |
|
|
|---|---|
|
|
| Who judges | The speaker, hearing the audio where the text cannot settle it (D9). Not Opus: it cannot hear, and its agreement with the speaker was only shown on items picked as decidable. |
|
|
| Sample | 30 spots drawn at random with a fixed seed, from ALL judged spots (changed and kept-base alike), not only the changed ones. Insertions and deletions that cannot be placed are listed separately and not sampled. |
|
|
| Verdict per spot | `x`, `y` (the two readings, shown in random order, key kept apart), `neither`, or `cannot tell`. |
|
|
| Condition 1 | Reconciled is right on at least 85% of the spots where the judge could tell (`cannot tell` excluded, reported, and `neither` counts as not right). Also report all-30. Pass needs the decided-only figure at 85% or more AND the all-30 figure above 70%. |
|
|
| Condition 2 | Over the same spots, the reconciled text is right at least as often as the better single engine (base engine alone is right where the base reading is right; the other engine alone where the alt reading is right). Ties favour the single engine. |
|
|
| If it fails | Switch reconcile off by configuration; the single-engine path is unchanged (ADR-034 reversibility). |
|
|
|
|
Why a fixed all-30 floor: the literal 70% last time sat well under the bar, and "decided only" can
|
|
hide a model that mostly lands on spelling differences.
|
|
|
|
## What a run needs (D10: the owner is present)
|
|
|
|
- The same real recording as before (about 60 minutes). Only the owner has it.
|
|
- A **throwaway** tapir instance with its own Postgres, not the production pod, so production's
|
|
key pin and deployment are untouched. Settings: `TAPIR_RECONCILE_BASE_MODEL=iguana/pianissimo`,
|
|
`TAPIR_TRANSCRIBE_MODEL=iguana/kb-whisper`, `TAPIR_RECONCILE_MODEL=iguana/gemma4-31b`, cloud
|
|
fallbacks empty. Job time was 21 min 49 s per 63.6 min last run.
|
|
- iguana with Low Power Mode off and gemma4-31b able to load (the production-iguana model load
|
|
that needs a quiet window; D6).
|
|
|
|
## Steps
|
|
|
|
1. Start the throwaway instance, upload the recording, wait for `done` (about 22 minutes).
|
|
2. Read `reconciliation.contested` from the throwaway database. From it build, **outside the repo**:
|
|
`sheet` (spot number, time, left context, reading x, reading y, random order) and `key` (which
|
|
of x/y is the base, which is the other engine, which one the reconciler used).
|
|
3. The speaker fills in the sheet (about 20 to 30 minutes for 30 spots, audio open at the times).
|
|
The agent does not see the key until the sheet is returned.
|
|
4. Score with the rules above. Write the aggregate numbers into a dated note next to this one,
|
|
and amend ADR-034's validation section (kept, or switched off, and the tie rule used).
|
|
5. Delete the throwaway instance, its database and the sheet, as last time.
|
|
6. Only if the check passes and the owner wants it in production: enabling reconcile there is
|
|
the separate D10 step (it needs `iguana/pianissimo` and the reconciler added to the tapir key
|
|
pin in the infra repo plus the env vars on the deployment, `risk/medium`, owner present).
|
|
|
|
## Open and owner-only
|
|
|
|
- Confirm or change the rules table above before the run.
|
|
- The **tool that builds the sheet and key** is the one piece not in the repo (last time it was a
|
|
one-off script). It can be built test-first on request; until then the agent builds it ad hoc
|
|
outside the repo.
|
|
- A Berget adjudicator (D8, tapir#45) is a different question with its own measurement and is
|
|
not part of this check: this check judges the local setup that ADR-034 describes.
|