mathiasandClaude Sonnet 5.5
ebc6cc3ff8
A near tie on accuracy (blind A/B by the speaker: 17 vs 13 on the 30 spans that change meaning) and a 6x difference in speed (40 s vs 4m08s for 63.7 min). Different error profiles: pianissimo better on terms and numbers, kb-whisper on everyday words. Aggregate numbers only -- no transcript text or domain terms, this repo is public. Also records the mid-word wrapping bug the comparison found and its fix. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
4.7 KiB
kb-whisper vs pianissimo on one real hour of Swedish speech (tapir#33, 2026-10-01)
One recording, one speaker: a 63.7-minute Teams session of the maintainer dictating project notes (domain-heavy, English loanwords and product names mixed into Swedish). No transcript text and no domain terms are recorded here: this repo is public and the content is private. Only aggregate numbers.
Backends, both through the LiteLLM gateway and Tapir's own adapter, unchanged:
iguana/kb-whisper (whisper.cpp, KB's Swedish fine-tune) and iguana/pianissimo (KlangAI
Swedish Parakeet/FastConformer-TDT, MLX 8-bit, served by a small FastAPI shell on iguana:8102).
Result: a near tie on accuracy, a large difference in speed
| kb-whisper | pianissimo | |
|---|---|---|
| Wall time for 63.7 min | 4m08s (~15x realtime) | 40 s (~95x realtime) |
| Segments | 1,302 (timed pieces, wrapped at ~60 chars) | 434 (sentences) |
| Words (normalised) | 6,606 | 6,510 |
| Punctuation | light, every segment ends in a full stop | heavier (138 vs 82 commas) |
| Repetition loops | none | none |
Word-level agreement between them: 92.4% as first computed, 93.1% after the segment rejoin fix below. 380 aligned disagreement spans, median one word each.
Blind A/B by the speaker (the only ground truth available), on the 30 spans a model (Claude Opus 5.5) had picked as changing meaning, shuffled and unlabelled: pianissimo 17, kb-whisper 13, no "neither", no "can't tell".
| Category of the 30 | pianissimo | kb-whisper |
|---|---|---|
| Technical term / English loanword | 8 | 4 |
| Number / date / unit | 4 | 1 |
| Name / proper noun | 3 | 3 |
| Function word / small word | 1 | 2 |
| Other (everyday words) | 1 | 3 |
The reading: pianissimo is better on terminology and numbers, kb-whisper on ordinary Swedish words; names are even. Their errors differ in kind: kb-whisper's tend to be garbled words, a couple were wrong numbers; pianissimo's tend to be fluent substitutions that read as plausible (a wrong institution, an invented person, a dropped sign), which a reader or an LLM is less likely to catch.
Opus's own lean agreed with the speaker's blind answer on 30 of 30. That is not a measure of Opus over all 380 spans: it chose spans it could decide, and every one of the 30 carried a lean. Its full classification of the 380 (counts only): function/small word 81, filler or false start 65, technical term 58, compound split/join 46, other 39, name 28, casing or punctuation residue 26, inflection 23, number/date/unit 10, content omission or insertion (>= 3 words) 4.
A bug this comparison found in Tapir (fixed)
kb-whisper's output is wrapped at ~60 characters and a wrap can fall inside a word; the
continuation segment has no leading space, the response's text field joins segments with
newlines, and a segment can be a lone .. Tapir stored that joined text, so chat received
broken words. Measured on real gateway output: 7% of segments (96/1,332) in a synthetic hour,
42/1,302 lines in the real one. berget's segments are whole sentences with no leading space, so
the two need different joins. transcribe.joinSegments now rebuilds the text from the segments.
Checked against the real transcript with pianissimo's vocabulary as an independent yardstick:
36 spans changed, 31 now entirely known words, words unknown to pianissimo 224 -> 177.
Five rejoined spans still contain an unknown word (a name or an unlucky glue): small residual
risk. Earlier tests used short sentences and could not see this.
What this does not show
- One recording, one speaker, one domain. No claim about other voices, accents, crosstalk, or room noise. 17 vs 13 is not a significant difference.
- The other ~350 spans were not judged by the speaker. They are mostly spelling, compounds, fillers and small words; Opus's counts there are a model's judgement, not ground truth.
- Errors both models share in text they agree on are invisible to this method.
- Speed was measured once each, on an otherwise idle iguana; pianissimo's 40 s includes the gateway hop and a 122 MB WAV upload.
- Whether pianissimo survives the infra#273 Metal-OOM contention the way whisper does not was not tested; both run on iguana.
What it suggests for Tapir (not decided, filed as an issue)
Pianissimo is a second local backend on the same host, through the same adapter. That makes it a natural middle tier ahead of the cloud fallback (kb-whisper -> pianissimo -> berget), so a wedged whisper no longer sends private audio off the homelab. At 40 s an hour it is also cheap enough to run both on every upload and give chat both texts, or flag where they disagree; the maintainer's blind answers show they err in different places.