From 4affeb12e868248772716d13fb5ba062d6fe4af2 Mon Sep 17 00:00:00 2001 From: Mathias Date: Sun, 4 Oct 2026 08:47:10 +0200 Subject: [PATCH] docs: ADR-034 second blind check protocol; D8 decided (Berget allowed, per-recording local-only opt-out first, #60); #37 verified Co-Authored-By: Claude Sonnet 5.5 --- .../reconcile-second-check-protocol.md | 69 +++++++++++++++++++ .../sprints/audio-upload-followups-2026-10.md | 17 +++-- 2 files changed, 77 insertions(+), 9 deletions(-) create mode 100644 docs/research/reconcile-second-check-protocol.md diff --git a/docs/research/reconcile-second-check-protocol.md b/docs/research/reconcile-second-check-protocol.md new file mode 100644 index 0000000..386fb55 --- /dev/null +++ b/docs/research/reconcile-second-check-protocol.md @@ -0,0 +1,69 @@ +# ADR-034 second blind check: protocol (ready to run; tapir#33, D9, D10) + +This repo is public and the recordings are private, so this file holds the method only. The +sheet, the key and the filled-in verdicts stay outside the repo, and the trial data is deleted +after the check. Results go back here as aggregate numbers, as in +`reconcile-blind-check-2026-10-02.md`. + +## What the check decides + +ADR-034 (two engines, a local model reconciles their disagreements) is `Proposed` and off in +production. Two conditions were set for keeping it. The first run answered the first one on one +reading only and did not answer the second: + +1. **The reconciled reading is right at least 85% of the time.** Open flaw in the bar: it did not + say how ties count. The first run gave 95% (decided items only) or 70% (all 30), so it was met on + one reading and missed on the other. +2. **It is no worse than the better single engine.** Not answered: before ADR-035 the spots the + model *kept* as the base were not stored, so kb-whisper alone could not be compared. + +ADR-035 now stores every judged spot (`reconciliation.contested`: both readings, which one was +used, unsure or not, a few words of left context, a time). That makes condition 2 measurable. + +## Rules to fix BEFORE looking at any result (the owner confirms or changes them) + +| Rule | Proposed | +|---|---| +| Who judges | The speaker, hearing the audio where the text cannot settle it (D9). Not Opus: it cannot hear, and its agreement with the speaker was only shown on items picked as decidable. | +| Sample | 30 spots drawn at random with a fixed seed, from ALL judged spots (changed and kept-base alike), not only the changed ones. Insertions and deletions that cannot be placed are listed separately and not sampled. | +| Verdict per spot | `x`, `y` (the two readings, shown in random order, key kept apart), `neither`, or `cannot tell`. | +| Condition 1 | Reconciled is right on at least 85% of the spots where the judge could tell (`cannot tell` excluded, reported, and `neither` counts as not right). Also report all-30. Pass needs the decided-only figure at 85% or more AND the all-30 figure above 70%. | +| Condition 2 | Over the same spots, the reconciled text is right at least as often as the better single engine (base engine alone is right where the base reading is right; the other engine alone where the alt reading is right). Ties favour the single engine. | +| If it fails | Switch reconcile off by configuration; the single-engine path is unchanged (ADR-034 reversibility). | + +Why a fixed all-30 floor: the literal 70% last time sat well under the bar, and "decided only" can +hide a model that mostly lands on spelling differences. + +## What a run needs (D10: the owner is present) + +- The same real recording as before (about 60 minutes). Only the owner has it. +- A **throwaway** tapir instance with its own Postgres, not the production pod, so production's + key pin and deployment are untouched. Settings: `TAPIR_RECONCILE_BASE_MODEL=iguana/pianissimo`, + `TAPIR_TRANSCRIBE_MODEL=iguana/kb-whisper`, `TAPIR_RECONCILE_MODEL=iguana/gemma4-31b`, cloud + fallbacks empty. Job time was 21 min 49 s per 63.6 min last run. +- iguana with Low Power Mode off and gemma4-31b able to load (the production-iguana model load + that needs a quiet window; D6). + +## Steps + +1. Start the throwaway instance, upload the recording, wait for `done` (about 22 minutes). +2. Read `reconciliation.contested` from the throwaway database. From it build, **outside the repo**: + `sheet` (spot number, time, left context, reading x, reading y, random order) and `key` (which + of x/y is the base, which is the other engine, which one the reconciler used). +3. The speaker fills in the sheet (about 20 to 30 minutes for 30 spots, audio open at the times). + The agent does not see the key until the sheet is returned. +4. Score with the rules above. Write the aggregate numbers into a dated note next to this one, + and amend ADR-034's validation section (kept, or switched off, and the tie rule used). +5. Delete the throwaway instance, its database and the sheet, as last time. +6. Only if the check passes and the owner wants it in production: enabling reconcile there is + the separate D10 step (it needs `iguana/pianissimo` and the reconciler added to the tapir key + pin in the infra repo plus the env vars on the deployment, `risk/medium`, owner present). + +## Open and owner-only + +- Confirm or change the rules table above before the run. +- The **tool that builds the sheet and key** is the one piece not in the repo (last time it was a + one-off script). It can be built test-first on request; until then the agent builds it ad hoc + outside the repo. +- A Berget adjudicator (D8, tapir#45) is a different question with its own measurement and is + not part of this check: this check judges the local setup that ADR-034 describes. diff --git a/docs/sprints/audio-upload-followups-2026-10.md b/docs/sprints/audio-upload-followups-2026-10.md index cc89b55..49a230d 100644 --- a/docs/sprints/audio-upload-followups-2026-10.md +++ b/docs/sprints/audio-upload-followups-2026-10.md @@ -64,8 +64,8 @@ Release v0.31.1: one adversarial review round over 2-5, then tag, push, observe. | D5 | #38: Berget length measurement needs a throwaway key | Yes, throwaway key | #38 | | D6 | #38: load large models on production iguana | Yes, in a quiet window with the owner present | #38 | | D7 | #46: where the ffmpeg tarball lives | Do nothing now; file an issue | none | -| D8 | #45: evaluate Berget as the reconciler | No; stay local-only | #45 | -| D9 | Who judges ADR-034's second blind check | The owner; do not reuse the one-off Opus authorisation | ADR-034 | +| D8 | #45: evaluate Berget as the reconciler | **Decided 2026-10-04 by the owner: yes, Berget may be used, and each user must be able to keep a recording on local models only** (reverses the recommendation). Opt-out first: #60; #45 stays as the model measurement | #60, #45 | +| D9 | Who judges ADR-034's second blind check | The owner; do not reuse the one-off Opus authorisation. Protocol: `docs/research/reconcile-second-check-protocol.md` | ADR-034 | | D10 | When to enable reconcile in production | Needs `iguana/pianissimo` in the key pin and env vars; owner present | ADR-034 | ## Dropped or reduced, with reasons @@ -100,17 +100,16 @@ refused (403) on production, a same-origin one reaches the login redirect (302). **Not observed live (needs an authenticated session, so it is the owner's look; closed on word or screenshot):** -- #37: the home page shows a "Recordings" link; a finished recording offers a confirm-then-delete - control that removes it; a queued or transcribing one offers none. -- #52: a transcript served by the cloud fallback shows the "audio left the homelab" note; a local - one does not. A reconciled (two-engine) all-local transcript must NOT show it. +- #52, cloud side: a transcript served by the Berget fallback names `berget`; a reconciled all-local + one shows no note. The local side was observed on 2026-10-04 (a new clip served by iguana: chip + `iguana/…`, no cloud note). The cloud side is carried as an acceptance item in #60. -**Observed live since:** #50: at the 2026-10-04 rollout the old pod went `Terminating` then `Completed` +**Observed live since:** #37 (owner, 2026-10-04: link, confirm-then-delete, no control while transcribing; closed). #50: at the 2026-10-04 rollout the old pod went `Terminating` then `Completed` (not `Error`), 0 restarts; #50 closed. No job was in flight, so a drained job is still unobserved (a recorded limitation, D2). -**Still open (owner's call, milestone 11):** #36, #38, #45, #46 remainder, #56 (slow-drip bodies hold -a receive slot), #57 (shutdown logging, second signal). Decisions D4-D6, D8-D10 above are unchanged. +**Still open (owner's call, milestone 11):** #36, #38, #45 (now the Berget adjudicator measurement), #60 (per-recording local-only choice, 4 sub-decisions), #46 remainder, #56 (slow-drip bodies hold +a receive slot), #57 (shutdown logging, second signal). Decisions D4-D6, D9, D10 above are unchanged; D8 is decided (above). **Deferred and filed from the reviews:** #56, #57, #59 (start-up exits when the identity provider's OIDC discovery returns 503 (seen once at 19:40 on 2026-10-03; the pod restarted itself and recovered)).