From de9871c9c3de9849c33b69afd061a930031ddcb2 Mon Sep 17 00:00:00 2001 From: Mathias Date: Thu, 1 Oct 2026 23:15:12 +0200 Subject: [PATCH] docs(adr): ADR-034 transcribe with two engines, reconcile with a local model Proposed, decider pending. Two local engines on one WAV; word-level diff with pianissimo as the base (readability and timing, NOT accuracy -- that was a tie); trivial disputes resolved without a model; the rest adjudicated by a LOCAL model (alias must start koala/ or iguana/) whose only freedom is choosing between the two readings. Reconciliation never fails a job; the page says what was done and lists the uncertain spots. Off by default. Validation: re-upload the real recording and keep the feature only if the reconciled reading is right >= 85% of the time and no worse than the better single engine. Co-Authored-By: Claude Sonnet 5.5 --- DECISIONS.md | 109 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) diff --git a/DECISIONS.md b/DECISIONS.md index 3aafc8c..97bf6d3 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -1533,6 +1533,113 @@ and ADR-021 are untouched. --- +## ADR-034 — Transcribe with two engines and let a local model reconcile their disagreements + +**Status:** Proposed (2026-10-01). **Decider: Mathias — pending.** Issue #40 (decision B) and #33. +Builds on ADR-033 (private upload transcripts), ADR-022 (endpoint chain, a pattern reused), ADR-018 +(single replica), ADR-025 (honest status). Evidence: +`docs/research/asr-kb-whisper-vs-pianissimo-2026-10-01.md` and +`docs/research/local-model-adjudicator-sweep-2026-10-01.md`. + +**Context.** On the first real one-hour recording `iguana/kb-whisper` and `iguana/pianissimo` were +a near tie on accuracy (the speaker's blind verdicts on the 30 meaning-changing disagreements: +pianissimo 17, kb-whisper 13) and they **err in different places**: pianissimo is better on terms and +numbers, kb-whisper on ordinary words; kb-whisper's errors garble, pianissimo's read as plausible. +Pianissimo is ~6x faster (40 s vs 4m08s for 63.7 min) and runs through the same adapter. Two engines +that disagree in ~380 places per hour, with a ~92% word agreement, is information a single-engine +pipeline throws away. Separately, eleven local models were scored on adjudicating those disagreements +against the speaker's answers: **`gemma4-31b` 97% and `gemma4-26b` 92%** passed the pre-set bar +(>= 80% accuracy, >= 85% order-swap consistency, <= 10% unparseable), nothing else did. + +**Unknowns, stated.** The adjudicator evidence is 30 items from one speaker and one domain, chosen by +Opus as decidable; the other ~350 disagreements (fillers, compounds, small words) have no ground +truth. Whether a *merged* transcript is better than the better single engine has not been measured. +Reconcile time per hour is estimated, not measured. `gemma4-31b`'s slowness is unexplained (Low Power +Mode on iguana is the unproven suspect). + +**Decision.** + +1. **Two engines, one WAV, both local, one after the other.** Each upload is transcribed by + `iguana/pianissimo` and `iguana/kb-whisper` from the same converted WAV, sequentially (concurrency + stays 1, ADR-033). Each engine fails independently; the berget fallback (ADR-033) applies only when + **both** fail. Off by default: with no second engine configured nothing changes. + +2. **Align by word, with pianissimo as the base text.** Both texts are tokenised (lower case, letters + and digits only) and diffed word by word (Myers); neighbouring differences separated by a single + equal word are merged into one **dispute**. The base is pianissimo: sentence-level prose with + punctuation and millisecond timestamps, where kb-whisper's output is wrapped fragments. **This + choice is about readability and timing, not accuracy — accuracy was a tie — so it is the cheaper + mistake, not a measured winner.** A replaced span takes the alternative's own words. + +3. **Resolve the trivial ones without a model.** A dispute that is identical once spaces are removed + (a compound split or joined) keeps the base; so does a dispute that is only a short filler word + on one side. Everything else is adjudicated. (Opus's classification of 380 disputes found ~46 + compound and ~26 punctuation-only residue, ~65 filler.) + +4. **A local model adjudicates the rest, bounded.** Each dispute goes to a local chat model (default + `iguana/gemma4-26b`; `gemma4-31b` is configurable and scored higher but is ~5x slower) with ~100 + words of context either side and the two readings, in the wording that was measured. The reading + order alternates by dispute so position bias averages out. The verdict is *base*, *alternative* or + *unsure*; **unsure keeps the base and is flagged**. An optional order-check mode asks twice with the + order swapped and treats disagreement as unsure; it is **off by default** because it doubles the time + and the measured consistency of the passing models is 93%. + - **The model must be local.** Its alias must begin `koala/` or `iguana/`; a cloud alias is a + start-up configuration error. Private meeting content never goes to a cloud model for this step. + - **Bounded impact.** The reconciler can only choose between two readings the engines already + produced; it cannot write new text. Speech that says "answer 2" can at worst flip one dispute. + +5. **Reconciliation never fails a job.** One engine fails → its text is used, recorded as + single-engine. The adjudicator is down (three consecutive errors) → the base text is used + unreconciled, recorded as skipped with the reason, and the remaining disputes are not retried one + timeout at a time. Both engines fail → ADR-033's fallback path. + +6. **What is stored and shown.** `content` is the merged text. `segments` stay the base engine's timed + segments, untouched: `content` is authoritative for chat, `segments` only for time. `endpoint` + lists every contributor and the reconciler. A new `reconciliation` column (migration 019, same + RLS-scoped table) records the status, counts, and the **uncertain and changed spots with a minute + mark and both readings**, so the user can check them against the audio. The transcript page says in + plain words what was done — "reconciled from two transcripts: N differences, K uncertain" — and + lists the uncertain spots. A reconciled transcript is never presented as simply "the" transcript. + +7. **Configuration.** `TAPIR_TRANSCRIBE_SECOND_MODEL` (empty = off), `TAPIR_RECONCILE_MODEL` + (required when a second engine is set), `TAPIR_RECONCILE_CHECK_ORDER` (off). The default job + timeout rises from 30 to 60 minutes: ~4 min of kb-whisper and ~40 s of pianissimo per hour of audio, + plus the adjudication. + +8. **Not decided here.** A cloud adjudicator (rejected: private content); tuning the prompt; making + `gemma4-31b` the default; a "reconciling" progress state (the page says "transcribing" for the whole + job); feeding the uncertain spots to chat; whether pianissimo should also be a middle tier in the + fallback chain (#40, option A). + +**Consequences.** +- **Positive.** Uses the finding that the engines err differently; stays local end to end; degrades + to today's behaviour at every failure; and is honest on the page about what it did. +- **Negative / watch.** + - Job time grows: roughly 300 adjudicated disputes x ~2 s on `gemma4-26b` ~ 10 minutes per hour of + audio — an estimate, not a measurement; double with order-check on, ~5x with `gemma4-31b`. + - Two gemma-class calls sit on iguana beside whisper. During the sweep a 17-20 GB model was resident + beside whisper-server without wedging it, far below the 52 GB that did (infra#273); the hazard is + reduced, not removed. + - A merged transcript can be wrong where both engines are wrong, and where the adjudicator picks + the wrong one (it was right 92% on a hand-picked set, ~97% for 31b). + - Reading the page takes more words: the reconciliation summary is new surface to keep honest. + +**Validation / resolution triggers.** +- **The test that decides it:** re-upload the same real recording after merge; the speaker judges, blind, + ~30 disputes where the reconciled text differs from pianissimo alone. Keep the feature only if the + reconciled reading is right at least 85% of the time **and** no worse than the better single engine. + If not, switch it off (config) and keep the single-engine path. +- Reconcile time above 25 minutes per hour of audio: drop to a smaller adjudicator or skip more + disputes automatically. +- A different recording or speaker in which the engines agree far less (or far more): re-measure before + trusting the 92%/97% figures. + +**Reversibility.** Off by default and additive: the second engine, the reconciler and the column are +unused when the settings are empty; the down-migration drops the column; the single-engine path is +unchanged. + +--- + ## Rejected alternatives Approaches considered during the 2026-06-02 planning + grill session and **deliberately not @@ -1560,6 +1667,8 @@ maps to the ADR that settles it. | Uploaded transcripts as a `user_id` column + partial RLS policy on `transcripts` | Turns ADR-021's single exception into an exception with a private lane inside it; the isolation test asserts `transcripts` has no `user_id`. A separate RLS table keeps both boundaries independently assertable | ADR-033 | | whisper-server `--convert` for server-side transcode | Its `--tmp-dir` is created at process start and rots under macOS periodic cleanup | ADR-033 | | Calling the public `llm-api.d-ma.be` from in-cluster Tapir | Its ingress returns 413 above ~8–10 MB (~5 min of WAV); the NodePort has no such cap | ADR-033 | +| A cloud model (e.g. Opus) as the adjudicator | It scored 30/30 but would send private meeting content to a cloud API; local gemma4-26b/31b passed the same bar | ADR-034 | +| Pick one engine globally (pianissimo only, or kb-whisper only) | A near tie on accuracy with different error profiles; choosing one discards the information in their disagreement | ADR-034 | If a future case genuinely reopens one of these, that's a new ADR superseding the relevant one — not a silent reversal.