Raised 2026-10-02 by Mathias while setting the reconciler for deployment ("we could also see if berget.ai has a better performing option. could be ok"). Recorded as an evaluation question, not as a policy change: nothing here relaxes ADR-034 until a decision says so.
Today. Reconciliation sends words from the transcript (about 100 words of context either side of each disagreement) to a language model. Config refuses to start unless that model is a koala/ or iguana/ alias (ADR-034 decision 7), so private meeting text cannot reach a cloud model. The deployment value chosen on 2026-10-02 is iguana/gemma4-31b: 97% against the speaker's blind answers in the earlier sweep, and about 22 minutes for the 63.6-minute recording with iguana's Low Power Mode off.
The question. Is there a berget.ai model that is better (more accurate, or much faster) at this job, and is it acceptable to use it? Using it means the transcript text of private recordings leaves the homelab. That is a different and larger egress than the audio fallback of #35 (which only fires when both local engines fail); here it would happen on every upload with reconciliation on. The repo-wide rule is that client data never goes to cloud APIs unless explicitly OK'd.
What an evaluation needs, and does not have
Items with ground truth. The earlier set (30 disagreements with the speaker's blind answers) and the later blind check were deleted after use, because they held real transcript text and this repo is public. A fresh set is needed: from a recording the owner is willing to send to berget for the test, or synthetic items (weaker evidence: the earlier failure modes were real Swedish speech with English payment terms).
The same pre-set bar as the local sweep: at least 80% strict accuracy, at least 85% order-swap consistency, at most 10% unparseable. Plus wall time per hour of audio, and cost.
An explicit OK for the test data itself. The evaluation sends text to the cloud even if the answer turns out to be "do not adopt".
Options
A. Do not evaluate; stay local-only. The 31B meets the bar and the local-only guarantee is the simpler thing to defend.
B. Evaluate on synthetic or already-public text only, then decide. Cheap, weak evidence, no private data leaves.
C. Evaluate on a real recording the owner approves for this, then decide. Needs the explicit OK above.
If a cloud reconciler is ever adopted: a new ADR superseding ADR-034 decision 7, a per-alias allow-list (never a general "cloud is fine"), the serving model shown on the page like the transcription endpoint, and a way to turn it off per recording.
Recommendation: A unless a concrete problem with the local reconciler appears (speed, accuracy on a new recording). The local 31B already meets the bar; the cost of leaving it is a few minutes per upload, and the cost of the alternative is a standing egress of private text.
Raised 2026-10-02 by Mathias while setting the reconciler for deployment ("we could also see if berget.ai has a better performing option. could be ok"). Recorded as an evaluation question, **not** as a policy change: nothing here relaxes ADR-034 until a decision says so.
**Today.** Reconciliation sends words from the transcript (about 100 words of context either side of each disagreement) to a language model. Config refuses to start unless that model is a `koala/` or `iguana/` alias (ADR-034 decision 7), so private meeting text cannot reach a cloud model. The deployment value chosen on 2026-10-02 is `iguana/gemma4-31b`: 97% against the speaker's blind answers in the earlier sweep, and about 22 minutes for the 63.6-minute recording with iguana's Low Power Mode off.
**The question.** Is there a berget.ai model that is better (more accurate, or much faster) at this job, and is it acceptable to use it? Using it means the **transcript text of private recordings leaves the homelab**. That is a different and larger egress than the audio fallback of #35 (which only fires when both local engines fail); here it would happen on every upload with reconciliation on. The repo-wide rule is that client data never goes to cloud APIs unless explicitly OK'd.
**What an evaluation needs, and does not have**
- Items with ground truth. The earlier set (30 disagreements with the speaker's blind answers) and the later blind check were **deleted** after use, because they held real transcript text and this repo is public. A fresh set is needed: from a recording the owner is willing to send to berget for the test, or synthetic items (weaker evidence: the earlier failure modes were real Swedish speech with English payment terms).
- The same pre-set bar as the local sweep: at least 80% strict accuracy, at least 85% order-swap consistency, at most 10% unparseable. Plus wall time per hour of audio, and cost.
- An explicit OK for the test data itself. The evaluation sends text to the cloud even if the answer turns out to be "do not adopt".
**Options**
- A. Do not evaluate; stay local-only. The 31B meets the bar and the local-only guarantee is the simpler thing to defend.
- B. Evaluate on synthetic or already-public text only, then decide. Cheap, weak evidence, no private data leaves.
- C. Evaluate on a real recording the owner approves for this, then decide. Needs the explicit OK above.
- If a cloud reconciler is ever adopted: a new ADR superseding ADR-034 decision 7, a per-alias allow-list (never a general "cloud is fine"), the serving model shown on the page like the transcription endpoint, and a way to turn it off per recording.
**Recommendation:** A unless a concrete problem with the local reconciler appears (speed, accuracy on a new recording). The local 31B already meets the bar; the cost of leaving it is a few minutes per upload, and the cost of the alternative is a standing egress of private text.
Refs: ADR-034, #35, `docs/research/local-model-adjudicator-sweep-2026-10-01.md`.
---
_Created via git-mcp on behalf of @mathias_
Owner decision 2026-10-04 (D8), recorded: "I trust Berget; we should allow the user to select local model if they don't want to share." This reverses my earlier recommendation (stay local-only). The question "may a cloud model see private transcript text for this step" is answered yes, unless the user opts the recording out.
What changes and what does not:
The opt-out is a separate prerequisite: #60 (a per-recording "keep on local models only" choice; control code, risk/high, with four sub-decisions for you in that issue). A Berget reconciler does not ship before it.
This issue is now the measurement: which Berget models, if any, pass ADR-034's adjudicator bar (>= 80% accuracy, >= 85% order-swap consistency, <= 10% unparseable) on the same 30 speaker-checked disputes the local sweep used (docs/research/local-model-adjudicator-sweep-2026-10-01.md; local result: gemma4-31b 97%, gemma4-26b 92%). It needs a Berget key allowed for those models: the throwaway key from D5 (#38), which is yours to provide, and no private text beyond the 30 already-judged items.
ADR-034 decision 4 and 7 ("the reconciler must be local; a cloud alias is a start-up error") are amended only when this and #60 are built, in a new ADR.
Left open as the evaluation task; D8 in the plan doc is updated.
**Owner decision 2026-10-04 (D8), recorded:** "I trust Berget; we should allow the user to select local model if they don't want to share." This reverses my earlier recommendation (stay local-only). The question "may a cloud model see private transcript text for this step" is answered **yes, unless the user opts the recording out**.
What changes and what does not:
- **The opt-out is a separate prerequisite: #60** (a per-recording "keep on local models only" choice; control code, `risk/high`, with four sub-decisions for you in that issue). A Berget reconciler does not ship before it.
- **This issue is now the measurement:** which Berget models, if any, pass ADR-034's adjudicator bar (>= 80% accuracy, >= 85% order-swap consistency, <= 10% unparseable) on the same 30 speaker-checked disputes the local sweep used (`docs/research/local-model-adjudicator-sweep-2026-10-01.md`; local result: gemma4-31b 97%, gemma4-26b 92%). It needs a Berget key allowed for those models: the throwaway key from D5 (#38), which is yours to provide, and no private text beyond the 30 already-judged items.
- ADR-034 decision 4 and 7 ("the reconciler must be local; a cloud alias is a start-up error") are amended only when this and #60 are built, in a new ADR.
Left open as the evaluation task; D8 in the plan doc is updated.
---
_Created via git-mcp on behalf of @mathias_
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Raised 2026-10-02 by Mathias while setting the reconciler for deployment ("we could also see if berget.ai has a better performing option. could be ok"). Recorded as an evaluation question, not as a policy change: nothing here relaxes ADR-034 until a decision says so.
Today. Reconciliation sends words from the transcript (about 100 words of context either side of each disagreement) to a language model. Config refuses to start unless that model is a
koala/origuana/alias (ADR-034 decision 7), so private meeting text cannot reach a cloud model. The deployment value chosen on 2026-10-02 isiguana/gemma4-31b: 97% against the speaker's blind answers in the earlier sweep, and about 22 minutes for the 63.6-minute recording with iguana's Low Power Mode off.The question. Is there a berget.ai model that is better (more accurate, or much faster) at this job, and is it acceptable to use it? Using it means the transcript text of private recordings leaves the homelab. That is a different and larger egress than the audio fallback of #35 (which only fires when both local engines fail); here it would happen on every upload with reconciliation on. The repo-wide rule is that client data never goes to cloud APIs unless explicitly OK'd.
What an evaluation needs, and does not have
Options
Recommendation: A unless a concrete problem with the local reconciler appears (speed, accuracy on a new recording). The local 31B already meets the bar; the cost of leaving it is a few minutes per upload, and the cost of the alternative is a standing egress of private text.
Refs: ADR-034, #35,
docs/research/local-model-adjudicator-sweep-2026-10-01.md.Created via git-mcp on behalf of @mathias
Owner decision 2026-10-04 (D8), recorded: "I trust Berget; we should allow the user to select local model if they don't want to share." This reverses my earlier recommendation (stay local-only). The question "may a cloud model see private transcript text for this step" is answered yes, unless the user opts the recording out.
What changes and what does not:
risk/high, with four sub-decisions for you in that issue). A Berget reconciler does not ship before it.docs/research/local-model-adjudicator-sweep-2026-10-01.md; local result: gemma4-31b 97%, gemma4-26b 92%). It needs a Berget key allowed for those models: the throwaway key from D5 (#38), which is yours to provide, and no private text beyond the 30 already-judged items.Left open as the evaluation task; D8 in the plan doc is updated.
Created via git-mcp on behalf of @mathias