diff --git a/docs/research/asr-kb-whisper-vs-pianissimo-2026-10-01.md b/docs/research/asr-kb-whisper-vs-pianissimo-2026-10-01.md new file mode 100644 index 0000000..6a51e23 --- /dev/null +++ b/docs/research/asr-kb-whisper-vs-pianissimo-2026-10-01.md @@ -0,0 +1,81 @@ +# kb-whisper vs pianissimo on one real hour of Swedish speech (tapir#33, 2026-10-01) + +One recording, one speaker: a 63.7-minute Teams session of the maintainer dictating project +notes (domain-heavy, English loanwords and product names mixed into Swedish). **No transcript +text and no domain terms are recorded here**: this repo is public and the content is private. +Only aggregate numbers. + +Backends, both through the LiteLLM gateway and Tapir's own adapter, unchanged: +`iguana/kb-whisper` (whisper.cpp, KB's Swedish fine-tune) and `iguana/pianissimo` (KlangAI +Swedish Parakeet/FastConformer-TDT, MLX 8-bit, served by a small FastAPI shell on iguana:8102). + +## Result: a near tie on accuracy, a large difference in speed + +| | kb-whisper | pianissimo | +|---|---|---| +| Wall time for 63.7 min | 4m08s (~15x realtime) | **40 s (~95x realtime)** | +| Segments | 1,302 (timed pieces, wrapped at ~60 chars) | 434 (sentences) | +| Words (normalised) | 6,606 | 6,510 | +| Punctuation | light, every segment ends in a full stop | heavier (138 vs 82 commas) | +| Repetition loops | none | none | + +Word-level agreement between them: **92.4%** as first computed, **93.1%** after the segment +rejoin fix below. 380 aligned disagreement spans, median one word each. + +**Blind A/B by the speaker** (the only ground truth available), on the 30 spans a model +(Claude Opus 5.5) had picked as changing meaning, shuffled and unlabelled: **pianissimo 17, +kb-whisper 13**, no "neither", no "can't tell". + +| Category of the 30 | pianissimo | kb-whisper | +|---|---|---| +| Technical term / English loanword | 8 | 4 | +| Number / date / unit | 4 | 1 | +| Name / proper noun | 3 | 3 | +| Function word / small word | 1 | 2 | +| Other (everyday words) | 1 | 3 | + +The reading: pianissimo is better on terminology and numbers, kb-whisper on ordinary Swedish +words; names are even. Their errors differ in kind: kb-whisper's tend to be garbled words, a +couple were wrong numbers; pianissimo's tend to be fluent substitutions that read as plausible +(a wrong institution, an invented person, a dropped sign), which a reader or an LLM is less +likely to catch. + +Opus's own lean agreed with the speaker's blind answer on **30 of 30**. That is not a measure +of Opus over all 380 spans: it chose spans it could decide, and every one of the 30 carried a +lean. Its full classification of the 380 (counts only): function/small word 81, filler or +false start 65, technical term 58, compound split/join 46, other 39, name 28, casing or +punctuation residue 26, inflection 23, number/date/unit 10, content omission or insertion +(>= 3 words) 4. + +## A bug this comparison found in Tapir (fixed) + +kb-whisper's output is wrapped at ~60 characters and **a wrap can fall inside a word**; the +continuation segment has no leading space, the response's `text` field joins segments with +newlines, and a segment can be a lone `.`. Tapir stored that joined text, so chat received +broken words. Measured on real gateway output: 7% of segments (96/1,332) in a synthetic hour, +42/1,302 lines in the real one. berget's segments are whole sentences with no leading space, so +the two need different joins. `transcribe.joinSegments` now rebuilds the text from the segments. +Checked against the real transcript with pianissimo's vocabulary as an independent yardstick: +36 spans changed, 31 now entirely known words, words unknown to pianissimo 224 -> 177. +Five rejoined spans still contain an unknown word (a name or an unlucky glue): small residual +risk. Earlier tests used short sentences and could not see this. + +## What this does not show + +- **One recording, one speaker, one domain.** No claim about other voices, accents, crosstalk, + or room noise. 17 vs 13 is not a significant difference. +- **The other ~350 spans** were not judged by the speaker. They are mostly spelling, compounds, + fillers and small words; Opus's counts there are a model's judgement, not ground truth. +- Errors **both** models share in text they agree on are invisible to this method. +- Speed was measured once each, on an otherwise idle iguana; pianissimo's 40 s includes the + gateway hop and a 122 MB WAV upload. +- Whether pianissimo survives the infra#273 Metal-OOM contention the way whisper does not was + not tested; both run on iguana. + +## What it suggests for Tapir (not decided, filed as an issue) + +Pianissimo is a **second local backend on the same host, through the same adapter**. That +makes it a natural middle tier ahead of the cloud fallback (kb-whisper -> pianissimo -> +berget), so a wedged whisper no longer sends private audio off the homelab. At 40 s an hour it +is also cheap enough to run **both** on every upload and give chat both texts, or flag where +they disagree; the maintainer's blind answers show they err in different places.