Raised by the owner 2026-10-04: let the user pick the transcription language or auto-detect; Swedish goes to a Swedish transcriber, other languages to a general one; auto-detect might sample with the general one. Question put: is there a better way to detect?
What is true today (checked in code and infra, 2026-10-04)
internal/adapters/transcribe/gateway.go:76 sends language=sv on every request. A non-Swedish upload is decoded as Swedish: expect confident nonsense, not an error, and 20 minutes of job time spent on it. This is a defect now, independent of the feature.
Local engines (LiteLLM catalogue, k3s/apps/ai-stack/litellm-config.yaml): iguana/kb-whisper (whisper.cpp serving KB's Swedish fine-tune of whisper-large) and iguana/pianissimo (Swedish-only Parakeet fine-tune). There is no local general multilingual transcriber in the catalogue. The only general one is berget/whisper-large-v3 (cloud, billed per second, NOT in the tapir key's allowlist, llm-consumers.yaml).
The audio fallback is berget/kb-whisper, also Swedish-tuned. The two-engine reconcile (ADR-034) is Swedish-only by construction.
Nothing records the language of an upload.
Detection: options, and a recommendation
Whisper's own language identification on short samples (recommended). Whisper detects the language from the first 30 s window with one decoder step (about a second). Send 3 windows (say at 10%, 50%, 90% of the recording, cut with ffmpeg from the converted WAV, skipping silence) with language unset to a whisper that does LID, and vote. No new dependency; it uses what the stack already runs. Needs verifying: that the iguana whisper-server returns the detected language (and ideally a probability) in verbose_json through LiteLLM when language is omitted, and whether the KB fine-tune keeps a usable language head (if not, the LID has to run on a general model, which brings back the "no local general model" gap).
A dedicated LID model (SpeechBrain VoxLingua107, MMS-LID, Silero): better on close languages (Swedish vs Norwegian vs Danish is whisper's usual weak spot) but a new service to host and maintain. Only if (1) measures badly on sv/no/da.
Transcribe in Swedish and judge the output: rejected, a forced-Swedish decode of foreign audio looks fluent, so it is not a signal.
Ask the user only: the baseline, always offered.
Policy on uncertainty: ask, do not guess. A wrong route costs a whole job and returns plausible garbage. If the three windows disagree or the top language is under a threshold (e.g. 0.8), the page says what it heard ("mostly English, not sure") and asks the user to pick; it does not start the job.
Mixed language (a meeting that switches between Swedish and English) is not solved by a per-recording choice. First version: route by majority and say so on the page. Per-segment routing (VAD, LID per chunk, engine per chunk) is a later issue if the owner needs it. Swedish speech with English loanwords (the measured recording) is Swedish.
DECISION NEEDED (owner; recommendation first; blocks the work)
What is the "general" transcriber, and where does it run? (a) A second whisper.cpp server on iguana with large-v3 (local, free; costs memory beside kb-whisper and the co-bind and Metal-OOM hazards recorded in the brain, so owner present for the first load, like D6); (b) berget/whisper-large-v3 (cloud, no iguana change, needs the infra allowlist change and is subject to the local-only opt-out in #60). Recommendation: (a) for the default, (b) only when the user has not kept the recording local. Measure whether KB-whisper alone is acceptable on English first: if it is, no general model is needed for English.
Default selection on the upload form: Auto, Swedish, English, Other. Recommend Swedish preselected (today's behaviour, the owner's main use) with Auto one click away; make Auto the default only once detection has been measured good.
Reconcile for non-Swedish: off (single general engine). Recommend yes, and say so on the page.
Fallback per language: a non-Swedish job that fails locally falls back to the general cloud model only if cloud is permitted for that recording (#60), else it fails with a typed reason. Recommend yes.
Work (test-first, one commit per step, after the decisions)
Interim fix, can go first, small: stop sending language=sv for a recording that is not Swedish: at minimum a form field with Swedish/Other, Other omitting language (whisper auto), and record the language on the row. Red test: the request body carries the chosen language, and omits it for auto.
Migration: language (chosen) and language_detected columns on uploaded_transcripts (RLS unchanged); the form field; the value travels with the job.
Router in the transcriber chain: language to engine, per decision 1; Swedish keeps today's pianissimo/kb-whisper path unchanged.
Detection: ffmpeg cuts the sample windows (bounded, same hardening as the existing ffmpeg calls); a LID call per window; the vote and the ask-on-uncertainty rule; the page states what was detected and used.
Page: shows the language used and how it was chosen (chosen, or detected with confidence).
ADR: amend ADR-033 (the language is a per-upload input) and ADR-034 (reconcile is Swedish-only), Proposed, owner decides.
Measurement before building detection (spike, owner present for any model load)
Clips of known language, each cut to 30 s windows: Swedish, English, Norwegian, Danish, Finnish, German, and a Swedish speaker with heavy English loanwords. Record per window the detected language and probability from the whisper path in decision 1. Pass bar to propose: at least 95% correct on sv/en/de/fi, and Swedish never routed away as no/da; report sv/no/da confusion separately. The clips are synthetic or public-domain, never the owner's real recordings in the repo. Related measurements: #38 (limits), #45 (Berget models).
Acceptance
Red tests first; task check and golangci-lint run ./... exit 0; adversarial review focused on routing (can a non-permitted engine receive audio for a language, does a crafted language field reach the request unvalidated: allowlist the values), verdict recorded here; live: an English clip transcribed by the general engine and labelled so, a Swedish clip unchanged.
Source: owner request 2026-10-04; internal/adapters/transcribe/gateway.go; infra litellm-config.yaml; sprint plan docs/sprints/audio-upload-followups-2026-10.md. Interacts with #60 (local-only choice) and #38.
**Raised by the owner 2026-10-04:** let the user pick the transcription language or auto-detect; Swedish goes to a Swedish transcriber, other languages to a general one; auto-detect might sample with the general one. Question put: is there a better way to detect?
## What is true today (checked in code and infra, 2026-10-04)
- `internal/adapters/transcribe/gateway.go:76` sends `language=sv` on **every** request. A non-Swedish upload is decoded as Swedish: expect confident nonsense, not an error, and 20 minutes of job time spent on it. This is a defect now, independent of the feature.
- Local engines (LiteLLM catalogue, `k3s/apps/ai-stack/litellm-config.yaml`): `iguana/kb-whisper` (whisper.cpp serving KB's Swedish fine-tune of whisper-large) and `iguana/pianissimo` (Swedish-only Parakeet fine-tune). **There is no local general multilingual transcriber in the catalogue.** The only general one is `berget/whisper-large-v3` (cloud, billed per second, NOT in the tapir key's allowlist, `llm-consumers.yaml`).
- The audio fallback is `berget/kb-whisper`, also Swedish-tuned. The two-engine reconcile (ADR-034) is Swedish-only by construction.
- Nothing records the language of an upload.
## Detection: options, and a recommendation
1. **Whisper's own language identification on short samples (recommended).** Whisper detects the language from the first 30 s window with one decoder step (about a second). Send 3 windows (say at 10%, 50%, 90% of the recording, cut with ffmpeg from the converted WAV, skipping silence) with `language` unset to a whisper that does LID, and vote. No new dependency; it uses what the stack already runs. Needs **verifying**: that the iguana whisper-server returns the detected language (and ideally a probability) in `verbose_json` through LiteLLM when `language` is omitted, and whether the KB fine-tune keeps a usable language head (if not, the LID has to run on a general model, which brings back the "no local general model" gap).
2. A dedicated LID model (SpeechBrain VoxLingua107, MMS-LID, Silero): better on close languages (Swedish vs Norwegian vs Danish is whisper's usual weak spot) but a new service to host and maintain. Only if (1) measures badly on sv/no/da.
3. Transcribe in Swedish and judge the output: rejected, a forced-Swedish decode of foreign audio looks fluent, so it is not a signal.
4. Ask the user only: the baseline, always offered.
**Policy on uncertainty: ask, do not guess.** A wrong route costs a whole job and returns plausible garbage. If the three windows disagree or the top language is under a threshold (e.g. 0.8), the page says what it heard ("mostly English, not sure") and asks the user to pick; it does not start the job.
**Mixed language** (a meeting that switches between Swedish and English) is not solved by a per-recording choice. First version: route by majority and say so on the page. Per-segment routing (VAD, LID per chunk, engine per chunk) is a later issue if the owner needs it. Swedish speech with English loanwords (the measured recording) is Swedish.
## DECISION NEEDED (owner; recommendation first; blocks the work)
1. **What is the "general" transcriber, and where does it run?** (a) A second whisper.cpp server on iguana with `large-v3` (local, free; costs memory beside kb-whisper and the co-bind and Metal-OOM hazards recorded in the brain, so owner present for the first load, like D6); (b) `berget/whisper-large-v3` (cloud, no iguana change, needs the infra allowlist change and is subject to the local-only opt-out in #60). Recommendation: (a) for the default, (b) only when the user has not kept the recording local. Measure whether KB-whisper alone is acceptable on English first: if it is, no general model is needed for English.
2. **Default selection on the upload form:** Auto, Swedish, English, Other. Recommend **Swedish preselected** (today's behaviour, the owner's main use) with Auto one click away; make Auto the default only once detection has been measured good.
3. **Reconcile for non-Swedish:** off (single general engine). Recommend yes, and say so on the page.
4. **Fallback per language:** a non-Swedish job that fails locally falls back to the general cloud model only if cloud is permitted for that recording (#60), else it fails with a typed reason. Recommend yes.
## Work (test-first, one commit per step, after the decisions)
0. **Interim fix, can go first, small:** stop sending `language=sv` for a recording that is not Swedish: at minimum a form field with Swedish/Other, Other omitting `language` (whisper auto), and record the language on the row. Red test: the request body carries the chosen language, and omits it for auto.
1. Migration: `language` (chosen) and `language_detected` columns on `uploaded_transcripts` (RLS unchanged); the form field; the value travels with the job.
2. Router in the transcriber chain: language to engine, per decision 1; Swedish keeps today's pianissimo/kb-whisper path unchanged.
3. Detection: ffmpeg cuts the sample windows (bounded, same hardening as the existing ffmpeg calls); a LID call per window; the vote and the ask-on-uncertainty rule; the page states what was detected and used.
4. Page: shows the language used and how it was chosen (chosen, or detected with confidence).
5. ADR: amend ADR-033 (the language is a per-upload input) and ADR-034 (reconcile is Swedish-only), `Proposed`, owner decides.
## Measurement before building detection (spike, owner present for any model load)
Clips of known language, each cut to 30 s windows: Swedish, English, Norwegian, Danish, Finnish, German, and a Swedish speaker with heavy English loanwords. Record per window the detected language and probability from the whisper path in decision 1. Pass bar to propose: at least 95% correct on sv/en/de/fi, and Swedish never routed away as no/da; report sv/no/da confusion separately. The clips are synthetic or public-domain, **never the owner's real recordings in the repo**. Related measurements: #38 (limits), #45 (Berget models).
## Acceptance
Red tests first; `task check` and `golangci-lint run ./...` exit 0; adversarial review focused on routing (can a non-permitted engine receive audio for a language, does a crafted `language` field reach the request unvalidated: allowlist the values), verdict recorded here; live: an English clip transcribed by the general engine and labelled so, a Swedish clip unchanged.
Source: owner request 2026-10-04; `internal/adapters/transcribe/gateway.go`; infra `litellm-config.yaml`; sprint plan `docs/sprints/audio-upload-followups-2026-10.md`. Interacts with #60 (local-only choice) and #38.
---
_Created via git-mcp on behalf of @mathias_
mathias
added this to the v0.32.0 - audio upload follow-ups milestone 2026-10-04 07:08:09 +00:00
Finding, 2026-10-04 (answers the owner's "could we run whisper large locally too?"): it already runs. iguana has com.mathias.mlx-whisper as a LaunchAgent (KeepAlive), serving mlx-community/whisper-large-v3-mlx through mlx-openai-server on :8100 (scripts/iguana/com.mathias.mlx-whisper.plist; runbook table row mlx-whisper | 8100). Checked live just now from koala: GET http://iguana:8100/v1/models lists that model. Open WebUI uses it for speech-to-text (k3s/apps/open-webui/deployment.yaml, AUDIO_STT_OPENAI_API_BASE_URL).
So decision 1 above changes from "stand up a second whisper-server" to a smaller piece of wiring:
A LiteLLM alias for it (e.g. iguana/whisper-large-v3, same shape as iguana/kb-whisper: api_base http://iguana:8100/v1, sovereignty: strict), added to the tapir key's allowlist in llm-consumers.yaml. Infra change, owner present, its own review.
Set the cost fields on the alias.iguana/kb-whisper accrues phantom spend because its catalogue entry sets no cost and LiteLLM prices it at the stock whisper-1 rate (see the comment in llm-consumers.yaml; infra#524 is the open ticket). A new alias must not repeat that.
It is shared with Open WebUI's STT, so a long tapir job occupies the same service; concurrency in tapir is 1, but a chat dictation during a 60-minute job would queue. Acceptable at current volume; note it.
Memory: large-v3 MLX is already resident, so adding the alias adds no model load on iguana. The first-load-with-owner-present condition (D6) does not apply to this model.
Still unverified (put in the spike): whether mlx-openai-server returns the detected language (and a probability) in verbose_json when language is omitted. mlx-whisper's own transcribe() returns a language field, so it is likely, but the server shell may drop it. If it does not, the detection step can still work by sending short samples and reading the language from the transcript's script/lexicon only as a last resort, or the shell can be patched on iguana (owner present).
What this does not change: for Swedish, KB-whisper stays the accurate engine (Klang's own table: KB-Whisper large beats the general one on Swedish); the general model is for non-Swedish and for language detection. Decision 2-4 stand unchanged. The cloud berget/whisper-large-v3 becomes the fallback only, subject to #60.
**Finding, 2026-10-04 (answers the owner's "could we run whisper large locally too?"): it already runs.** iguana has `com.mathias.mlx-whisper` as a LaunchAgent (`KeepAlive`), serving `mlx-community/whisper-large-v3-mlx` through `mlx-openai-server` on `:8100` (`scripts/iguana/com.mathias.mlx-whisper.plist`; runbook table row `mlx-whisper | 8100`). Checked live just now from koala: `GET http://iguana:8100/v1/models` lists that model. Open WebUI uses it for speech-to-text (`k3s/apps/open-webui/deployment.yaml`, `AUDIO_STT_OPENAI_API_BASE_URL`).
So decision 1 above changes from "stand up a second whisper-server" to a smaller piece of wiring:
1. **A LiteLLM alias for it** (e.g. `iguana/whisper-large-v3`, same shape as `iguana/kb-whisper`: `api_base http://iguana:8100/v1`, `sovereignty: strict`), added to the tapir key's allowlist in `llm-consumers.yaml`. Infra change, owner present, its own review.
2. **Set the cost fields on the alias.** `iguana/kb-whisper` accrues phantom spend because its catalogue entry sets no cost and LiteLLM prices it at the stock whisper-1 rate (see the comment in `llm-consumers.yaml`; infra#524 is the open ticket). A new alias must not repeat that.
3. **It is shared** with Open WebUI's STT, so a long tapir job occupies the same service; concurrency in tapir is 1, but a chat dictation during a 60-minute job would queue. Acceptable at current volume; note it.
4. Memory: large-v3 MLX is already resident, so adding the alias adds no model load on iguana. The first-load-with-owner-present condition (D6) does not apply to this model.
**Still unverified (put in the spike):** whether `mlx-openai-server` returns the detected language (and a probability) in `verbose_json` when `language` is omitted. mlx-whisper's own `transcribe()` returns a `language` field, so it is likely, but the server shell may drop it. If it does not, the detection step can still work by sending short samples and reading the language from the transcript's script/lexicon only as a last resort, or the shell can be patched on iguana (owner present).
**What this does not change:** for Swedish, KB-whisper stays the accurate engine (Klang's own table: KB-Whisper large beats the general one on Swedish); the general model is for non-Swedish and for language detection. Decision 2-4 stand unchanged. The cloud `berget/whisper-large-v3` becomes the fallback only, subject to #60.
---
_Created via git-mcp on behalf of @mathias_
Correction to my previous comment. I wrote that "Klang's own table: KB-Whisper large beats the general one on Swedish". That table (brain note on pianissimo, infra) compares KB-Whisper large with Pianissimo, not with general whisper-large-v3, so it is not evidence for that claim. What KB Lab publishes for KB-Whisper is a Swedish WER improvement over OpenAI's whisper-large-v3, but I have not checked those figures here and nothing in this repo measures it. Treat "KB-whisper is more accurate than whisper-large-v3 on Swedish" as unmeasured on our recordings; the spike should include one Swedish clip through both so the Swedish routing is chosen by measurement. The decision-1 wiring (alias for the existing :8100 service) is unaffected.
**Correction to my previous comment.** I wrote that "Klang's own table: KB-Whisper large beats the general one on Swedish". That table (brain note on pianissimo, infra) compares KB-Whisper large with **Pianissimo**, not with general whisper-large-v3, so it is not evidence for that claim. What KB Lab publishes for KB-Whisper is a Swedish WER improvement over OpenAI's whisper-large-v3, but I have not checked those figures here and nothing in this repo measures it. Treat "KB-whisper is more accurate than whisper-large-v3 on Swedish" as **unmeasured on our recordings**; the spike should include one Swedish clip through both so the Swedish routing is chosen by measurement. The decision-1 wiring (alias for the existing `:8100` service) is unaffected.
---
_Created via git-mcp on behalf of @mathias_
Interim fix built: PR #62 (branch sprint/language; merge, tag and live check follow once CI on its head is green).
What it does: the upload form has a Language field, Swedish (preselected) or Other (detected by the engine); an upload that names none is Swedish; a value outside the set is a 400 before anything is spooled or stored. The language is stored on the row (migration 021, CHECK constraint, history rows sv), carried on the job and the worker's context, shown on the recording's page in every state. The gateway sends language=sv for Swedish, omits the field for Other, and refuses anything else. For Other the Swedish-only base engine never receives the audio and nothing is reconciled; the page plan lists only the phases that can run. ADR-033/034 amended.
Verification: red tests first, mutation checks on every piece, two adversarial review rounds by agent (round 1 BLOCK, narrow: nothing pinned that WithTimeout, in front of both local engines in production, hands the engine the job's language and cancellation; a vacuous assertion; a misleading progress plan; wording. Round 2 APPROVE). Verdicts on the PR.
What this does NOT do (remaining #61 work): automatic detection, a general-purpose engine for other languages, per-language fallback. In production, Other goes to the same kb-whisper (a Swedish fine-tune, not tuned for other languages) with the language left for it to detect, and the fallback berget/kb-whisper is also Swedish-tuned; both are said plainly in ADR-033. The spike (does the iguana whisper path return the detected language; Swedish vs Norwegian/Danish confusion; KB-whisper vs whisper-large-v3 on a Swedish clip) and the infra alias for the existing :8100 whisper-large-v3 are still to do. A live check needs a logged-in upload with Other selected.
**Interim fix built: PR #62** (branch `sprint/language`; merge, tag and live check follow once CI on its head is green).
What it does: the upload form has a Language field, Swedish (preselected) or Other (detected by the engine); an upload that names none is Swedish; a value outside the set is a 400 before anything is spooled or stored. The language is stored on the row (migration 021, CHECK constraint, history rows `sv`), carried on the job and the worker's context, shown on the recording's page in every state. The gateway sends `language=sv` for Swedish, omits the field for Other, and refuses anything else. For Other the Swedish-only base engine never receives the audio and nothing is reconciled; the page plan lists only the phases that can run. ADR-033/034 amended.
Verification: red tests first, mutation checks on every piece, two adversarial review rounds by agent (round 1 BLOCK, narrow: nothing pinned that `WithTimeout`, in front of both local engines in production, hands the engine the job's language and cancellation; a vacuous assertion; a misleading progress plan; wording. Round 2 APPROVE). Verdicts on the PR.
**What this does NOT do (remaining #61 work):** automatic detection, a general-purpose engine for other languages, per-language fallback. In production, Other goes to the same kb-whisper (a Swedish fine-tune, not tuned for other languages) with the language left for it to detect, and the fallback `berget/kb-whisper` is also Swedish-tuned; both are said plainly in ADR-033. The spike (does the iguana whisper path return the detected language; Swedish vs Norwegian/Danish confusion; KB-whisper vs whisper-large-v3 on a Swedish clip) and the infra alias for the existing `:8100` whisper-large-v3 are still to do. A live check needs a logged-in upload with Other selected.
---
_Created via git-mcp on behalf of @mathias_
Interim fix shipped and deployed: v0.33.0 (PR #62, merge b9643f5, CI green; production now runs image 813e679, which contains it plus the parallel v0.32.1 (#59, x/text) and #63 (#56) from another session). Pod 1/1, 0 restarts; a foreign-origin POST /uploads is refused (403) and a same-origin one reaches the login redirect (302). Migration 021 runs at start-up inside the store open, the pod serves, and no migration error was logged; the column itself has not been read directly (that needs a prod database session).
To verify live (needs a logged-in upload, the owner's look): (1) upload a Swedish clip with the default selection: it transcribes as before and its page header says "Language: Swedish"; (2) upload a clip with Other selected: it transcribes, the header says "Language: Other (detected by the engine)", and the transcript is whatever kb-whisper makes of it with the language left to detect (not tuned for other languages). A non-Swedish clip is the interesting one; if it comes back as nonsense, that is the evidence for building the general engine (decision 1 above: the existing iguana whisper-large-v3 on :8100).
Still open in this issue: the detection spike, the general engine alias (infra), per-language fallback, and the four decisions above.
**Interim fix shipped and deployed: v0.33.0** (PR #62, merge b9643f5, CI green; production now runs image 813e679, which contains it plus the parallel `v0.32.1` (#59, x/text) and #63 (#56) from another session). Pod 1/1, 0 restarts; a foreign-origin `POST /uploads` is refused (403) and a same-origin one reaches the login redirect (302). Migration 021 runs at start-up inside the store open, the pod serves, and no migration error was logged; the column itself has not been read directly (that needs a prod database session).
**To verify live (needs a logged-in upload, the owner's look):** (1) upload a Swedish clip with the default selection: it transcribes as before and its page header says "Language: Swedish"; (2) upload a clip with **Other** selected: it transcribes, the header says "Language: Other (detected by the engine)", and the transcript is whatever kb-whisper makes of it with the language left to detect (not tuned for other languages). A non-Swedish clip is the interesting one; if it comes back as nonsense, that is the evidence for building the general engine (decision 1 above: the existing iguana whisper-large-v3 on `:8100`).
Still open in this issue: the detection spike, the general engine alias (infra), per-language fallback, and the four decisions above.
---
_Created via git-mcp on behalf of @mathias_
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Raised by the owner 2026-10-04: let the user pick the transcription language or auto-detect; Swedish goes to a Swedish transcriber, other languages to a general one; auto-detect might sample with the general one. Question put: is there a better way to detect?
What is true today (checked in code and infra, 2026-10-04)
internal/adapters/transcribe/gateway.go:76sendslanguage=svon every request. A non-Swedish upload is decoded as Swedish: expect confident nonsense, not an error, and 20 minutes of job time spent on it. This is a defect now, independent of the feature.k3s/apps/ai-stack/litellm-config.yaml):iguana/kb-whisper(whisper.cpp serving KB's Swedish fine-tune of whisper-large) andiguana/pianissimo(Swedish-only Parakeet fine-tune). There is no local general multilingual transcriber in the catalogue. The only general one isberget/whisper-large-v3(cloud, billed per second, NOT in the tapir key's allowlist,llm-consumers.yaml).berget/kb-whisper, also Swedish-tuned. The two-engine reconcile (ADR-034) is Swedish-only by construction.Detection: options, and a recommendation
languageunset to a whisper that does LID, and vote. No new dependency; it uses what the stack already runs. Needs verifying: that the iguana whisper-server returns the detected language (and ideally a probability) inverbose_jsonthrough LiteLLM whenlanguageis omitted, and whether the KB fine-tune keeps a usable language head (if not, the LID has to run on a general model, which brings back the "no local general model" gap).Policy on uncertainty: ask, do not guess. A wrong route costs a whole job and returns plausible garbage. If the three windows disagree or the top language is under a threshold (e.g. 0.8), the page says what it heard ("mostly English, not sure") and asks the user to pick; it does not start the job.
Mixed language (a meeting that switches between Swedish and English) is not solved by a per-recording choice. First version: route by majority and say so on the page. Per-segment routing (VAD, LID per chunk, engine per chunk) is a later issue if the owner needs it. Swedish speech with English loanwords (the measured recording) is Swedish.
DECISION NEEDED (owner; recommendation first; blocks the work)
large-v3(local, free; costs memory beside kb-whisper and the co-bind and Metal-OOM hazards recorded in the brain, so owner present for the first load, like D6); (b)berget/whisper-large-v3(cloud, no iguana change, needs the infra allowlist change and is subject to the local-only opt-out in #60). Recommendation: (a) for the default, (b) only when the user has not kept the recording local. Measure whether KB-whisper alone is acceptable on English first: if it is, no general model is needed for English.Work (test-first, one commit per step, after the decisions)
language=svfor a recording that is not Swedish: at minimum a form field with Swedish/Other, Other omittinglanguage(whisper auto), and record the language on the row. Red test: the request body carries the chosen language, and omits it for auto.language(chosen) andlanguage_detectedcolumns onuploaded_transcripts(RLS unchanged); the form field; the value travels with the job.Proposed, owner decides.Measurement before building detection (spike, owner present for any model load)
Clips of known language, each cut to 30 s windows: Swedish, English, Norwegian, Danish, Finnish, German, and a Swedish speaker with heavy English loanwords. Record per window the detected language and probability from the whisper path in decision 1. Pass bar to propose: at least 95% correct on sv/en/de/fi, and Swedish never routed away as no/da; report sv/no/da confusion separately. The clips are synthetic or public-domain, never the owner's real recordings in the repo. Related measurements: #38 (limits), #45 (Berget models).
Acceptance
Red tests first;
task checkandgolangci-lint run ./...exit 0; adversarial review focused on routing (can a non-permitted engine receive audio for a language, does a craftedlanguagefield reach the request unvalidated: allowlist the values), verdict recorded here; live: an English clip transcribed by the general engine and labelled so, a Swedish clip unchanged.Source: owner request 2026-10-04;
internal/adapters/transcribe/gateway.go; infralitellm-config.yaml; sprint plandocs/sprints/audio-upload-followups-2026-10.md. Interacts with #60 (local-only choice) and #38.Created via git-mcp on behalf of @mathias
Finding, 2026-10-04 (answers the owner's "could we run whisper large locally too?"): it already runs. iguana has
com.mathias.mlx-whisperas a LaunchAgent (KeepAlive), servingmlx-community/whisper-large-v3-mlxthroughmlx-openai-serveron:8100(scripts/iguana/com.mathias.mlx-whisper.plist; runbook table rowmlx-whisper | 8100). Checked live just now from koala:GET http://iguana:8100/v1/modelslists that model. Open WebUI uses it for speech-to-text (k3s/apps/open-webui/deployment.yaml,AUDIO_STT_OPENAI_API_BASE_URL).So decision 1 above changes from "stand up a second whisper-server" to a smaller piece of wiring:
iguana/whisper-large-v3, same shape asiguana/kb-whisper:api_base http://iguana:8100/v1,sovereignty: strict), added to the tapir key's allowlist inllm-consumers.yaml. Infra change, owner present, its own review.iguana/kb-whisperaccrues phantom spend because its catalogue entry sets no cost and LiteLLM prices it at the stock whisper-1 rate (see the comment inllm-consumers.yaml; infra#524 is the open ticket). A new alias must not repeat that.Still unverified (put in the spike): whether
mlx-openai-serverreturns the detected language (and a probability) inverbose_jsonwhenlanguageis omitted. mlx-whisper's owntranscribe()returns alanguagefield, so it is likely, but the server shell may drop it. If it does not, the detection step can still work by sending short samples and reading the language from the transcript's script/lexicon only as a last resort, or the shell can be patched on iguana (owner present).What this does not change: for Swedish, KB-whisper stays the accurate engine (Klang's own table: KB-Whisper large beats the general one on Swedish); the general model is for non-Swedish and for language detection. Decision 2-4 stand unchanged. The cloud
berget/whisper-large-v3becomes the fallback only, subject to #60.Created via git-mcp on behalf of @mathias
Correction to my previous comment. I wrote that "Klang's own table: KB-Whisper large beats the general one on Swedish". That table (brain note on pianissimo, infra) compares KB-Whisper large with Pianissimo, not with general whisper-large-v3, so it is not evidence for that claim. What KB Lab publishes for KB-Whisper is a Swedish WER improvement over OpenAI's whisper-large-v3, but I have not checked those figures here and nothing in this repo measures it. Treat "KB-whisper is more accurate than whisper-large-v3 on Swedish" as unmeasured on our recordings; the spike should include one Swedish clip through both so the Swedish routing is chosen by measurement. The decision-1 wiring (alias for the existing
:8100service) is unaffected.Created via git-mcp on behalf of @mathias
Interim fix built: PR #62 (branch
sprint/language; merge, tag and live check follow once CI on its head is green).What it does: the upload form has a Language field, Swedish (preselected) or Other (detected by the engine); an upload that names none is Swedish; a value outside the set is a 400 before anything is spooled or stored. The language is stored on the row (migration 021, CHECK constraint, history rows
sv), carried on the job and the worker's context, shown on the recording's page in every state. The gateway sendslanguage=svfor Swedish, omits the field for Other, and refuses anything else. For Other the Swedish-only base engine never receives the audio and nothing is reconciled; the page plan lists only the phases that can run. ADR-033/034 amended.Verification: red tests first, mutation checks on every piece, two adversarial review rounds by agent (round 1 BLOCK, narrow: nothing pinned that
WithTimeout, in front of both local engines in production, hands the engine the job's language and cancellation; a vacuous assertion; a misleading progress plan; wording. Round 2 APPROVE). Verdicts on the PR.What this does NOT do (remaining #61 work): automatic detection, a general-purpose engine for other languages, per-language fallback. In production, Other goes to the same kb-whisper (a Swedish fine-tune, not tuned for other languages) with the language left for it to detect, and the fallback
berget/kb-whisperis also Swedish-tuned; both are said plainly in ADR-033. The spike (does the iguana whisper path return the detected language; Swedish vs Norwegian/Danish confusion; KB-whisper vs whisper-large-v3 on a Swedish clip) and the infra alias for the existing:8100whisper-large-v3 are still to do. A live check needs a logged-in upload with Other selected.Created via git-mcp on behalf of @mathias
Interim fix shipped and deployed: v0.33.0 (PR #62, merge
b9643f5, CI green; production now runs image813e679, which contains it plus the parallelv0.32.1(#59, x/text) and #63 (#56) from another session). Pod 1/1, 0 restarts; a foreign-originPOST /uploadsis refused (403) and a same-origin one reaches the login redirect (302). Migration 021 runs at start-up inside the store open, the pod serves, and no migration error was logged; the column itself has not been read directly (that needs a prod database session).To verify live (needs a logged-in upload, the owner's look): (1) upload a Swedish clip with the default selection: it transcribes as before and its page header says "Language: Swedish"; (2) upload a clip with Other selected: it transcribes, the header says "Language: Other (detected by the engine)", and the transcript is whatever kb-whisper makes of it with the language left to detect (not tuned for other languages). A non-Swedish clip is the interesting one; if it comes back as nonsense, that is the evidence for building the general engine (decision 1 above: the existing iguana whisper-large-v3 on
:8100).Still open in this issue: the detection spike, the general engine alias (infra), per-language fallback, and the four decisions above.
Created via git-mcp on behalf of @mathias