docs: ADR-034 second blind check protocol; D8 decided (Berget allowed, per-recording local-only opt-out first, #60); #37 verified

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs/research/reconcile-second-check-protocol.md

@@ -0,0 +1,69 @@
+# ADR-034 second blind check: protocol (ready to run; tapir#33, D9, D10)
+
+This repo is public and the recordings are private, so this file holds the method only. The
+sheet, the key and the filled-in verdicts stay outside the repo, and the trial data is deleted
+after the check. Results go back here as aggregate numbers, as in
+`reconcile-blind-check-2026-10-02.md`.
+
+## What the check decides
+
+ADR-034 (two engines, a local model reconciles their disagreements) is `Proposed` and off in
+production. Two conditions were set for keeping it. The first run answered the first one on one
+reading only and did not answer the second:
+
+1. **The reconciled reading is right at least 85% of the time.** Open flaw in the bar: it did not
+   say how ties count. The first run gave 95% (decided items only) or 70% (all 30), so it was met on
+   one reading and missed on the other.
+2. **It is no worse than the better single engine.** Not answered: before ADR-035 the spots the
+   model *kept* as the base were not stored, so kb-whisper alone could not be compared.
+
+ADR-035 now stores every judged spot (`reconciliation.contested`: both readings, which one was
+used, unsure or not, a few words of left context, a time). That makes condition 2 measurable.
+
+## Rules to fix BEFORE looking at any result (the owner confirms or changes them)
+
+| Rule | Proposed |
+|---|---|
+| Who judges | The speaker, hearing the audio where the text cannot settle it (D9). Not Opus: it cannot hear, and its agreement with the speaker was only shown on items picked as decidable. |
+| Sample | 30 spots drawn at random with a fixed seed, from ALL judged spots (changed and kept-base alike), not only the changed ones. Insertions and deletions that cannot be placed are listed separately and not sampled. |
+| Verdict per spot | `x`, `y` (the two readings, shown in random order, key kept apart), `neither`, or `cannot tell`. |
+| Condition 1 | Reconciled is right on at least 85% of the spots where the judge could tell (`cannot tell` excluded, reported, and `neither` counts as not right). Also report all-30. Pass needs the decided-only figure at 85% or more AND the all-30 figure above 70%. |
+| Condition 2 | Over the same spots, the reconciled text is right at least as often as the better single engine (base engine alone is right where the base reading is right; the other engine alone where the alt reading is right). Ties favour the single engine. |
+| If it fails | Switch reconcile off by configuration; the single-engine path is unchanged (ADR-034 reversibility). |
+
+Why a fixed all-30 floor: the literal 70% last time sat well under the bar, and "decided only" can
+hide a model that mostly lands on spelling differences.
+
+## What a run needs (D10: the owner is present)
+
+- The same real recording as before (about 60 minutes). Only the owner has it.
+- A **throwaway** tapir instance with its own Postgres, not the production pod, so production's
+  key pin and deployment are untouched. Settings: `TAPIR_RECONCILE_BASE_MODEL=iguana/pianissimo`,
+  `TAPIR_TRANSCRIBE_MODEL=iguana/kb-whisper`, `TAPIR_RECONCILE_MODEL=iguana/gemma4-31b`, cloud
+  fallbacks empty. Job time was 21 min 49 s per 63.6 min last run.
+- iguana with Low Power Mode off and gemma4-31b able to load (the production-iguana model load
+  that needs a quiet window; D6).
+
+## Steps
+
+1. Start the throwaway instance, upload the recording, wait for `done` (about 22 minutes).
+2. Read `reconciliation.contested` from the throwaway database. From it build, **outside the repo**:
+   `sheet` (spot number, time, left context, reading x, reading y, random order) and `key` (which
+   of x/y is the base, which is the other engine, which one the reconciler used).
+3. The speaker fills in the sheet (about 20 to 30 minutes for 30 spots, audio open at the times).
+   The agent does not see the key until the sheet is returned.
+4. Score with the rules above. Write the aggregate numbers into a dated note next to this one,
+   and amend ADR-034's validation section (kept, or switched off, and the tie rule used).
+5. Delete the throwaway instance, its database and the sheet, as last time.
+6. Only if the check passes and the owner wants it in production: enabling reconcile there is
+   the separate D10 step (it needs `iguana/pianissimo` and the reconciler added to the tapir key
+   pin in the infra repo plus the env vars on the deployment, `risk/medium`, owner present).
+
+## Open and owner-only
+
+- Confirm or change the rules table above before the run.
+- The **tool that builds the sheet and key** is the one piece not in the repo (last time it was a
+  one-off script). It can be built test-first on request; until then the agent builds it ad hoc
+  outside the repo.
+- A Berget adjudicator (D8, tapir#45) is a different question with its own measurement and is
+  not part of this check: this check judges the local setup that ADR-034 describes.

docs/sprints/audio-upload-followups-2026-10.md

@@ -64,8 +64,8 @@ Release v0.31.1: one adversarial review round over 2-5, then tag, push, observe.
 | D5 | #38: Berget length measurement needs a throwaway key | Yes, throwaway key | #38 |
 | D6 | #38: load large models on production iguana | Yes, in a quiet window with the owner present | #38 |
 | D7 | #46: where the ffmpeg tarball lives | Do nothing now; file an issue | none |
-| D8 | #45: evaluate Berget as the reconciler | No; stay local-only | #45 |
-| D9 | Who judges ADR-034's second blind check | The owner; do not reuse the one-off Opus authorisation | ADR-034 |
+| D8 | #45: evaluate Berget as the reconciler | **Decided 2026-10-04 by the owner: yes, Berget may be used, and each user must be able to keep a recording on local models only** (reverses the recommendation). Opt-out first: #60; #45 stays as the model measurement | #60, #45 |
+| D9 | Who judges ADR-034's second blind check | The owner; do not reuse the one-off Opus authorisation. Protocol: `docs/research/reconcile-second-check-protocol.md` | ADR-034 |
 | D10 | When to enable reconcile in production | Needs `iguana/pianissimo` in the key pin and env vars; owner present | ADR-034 |
 
 ## Dropped or reduced, with reasons
@@ -100,17 +100,16 @@ refused (403) on production, a same-origin one reaches the login redirect (302).
 
 **Not observed live (needs an authenticated session, so it is the owner's look; closed on word or screenshot):**
 
-- #37: the home page shows a "Recordings" link; a finished recording offers a confirm-then-delete
-  control that removes it; a queued or transcribing one offers none.
-- #52: a transcript served by the cloud fallback shows the "audio left the homelab" note; a local
-  one does not. A reconciled (two-engine) all-local transcript must NOT show it.
+- #52, cloud side: a transcript served by the Berget fallback names `berget`; a reconciled all-local
+  one shows no note. The local side was observed on 2026-10-04 (a new clip served by iguana: chip
+  `iguana/…`, no cloud note). The cloud side is carried as an acceptance item in #60.
 
-**Observed live since:** #50: at the 2026-10-04 rollout the old pod went `Terminating` then `Completed`
+**Observed live since:** #37 (owner, 2026-10-04: link, confirm-then-delete, no control while transcribing; closed). #50: at the 2026-10-04 rollout the old pod went `Terminating` then `Completed`
 (not `Error`), 0 restarts; #50 closed. No job was in flight, so a drained job is still unobserved
 (a recorded limitation, D2).
 
-**Still open (owner's call, milestone 11):** #36, #38, #45, #46 remainder, #56 (slow-drip bodies hold
-a receive slot), #57 (shutdown logging, second signal). Decisions D4-D6, D8-D10 above are unchanged.
+**Still open (owner's call, milestone 11):** #36, #38, #45 (now the Berget adjudicator measurement), #60 (per-recording local-only choice, 4 sub-decisions), #46 remainder, #56 (slow-drip bodies hold
+a receive slot), #57 (shutdown logging, second signal). Decisions D4-D6, D9, D10 above are unchanged; D8 is decided (above).
 
 **Deferred and filed from the reviews:** #56, #57, #59 (start-up exits when the identity provider's OIDC
 discovery returns 503 (seen once at 19:40 on 2026-10-03; the pod restarted itself and recovered)).