2026-10-03 14:13:37 +00:00
1 changed files with 81 additions and 0 deletions
@@ -0,0 +1,81 @@
 # kb-whisper vs pianissimo on one real hour of Swedish speech (tapir#33, 2026-10-01)
 One recording, one speaker: a 63.7-minute Teams session of the maintainer dictating project
 notes (domain-heavy, English loanwords and product names mixed into Swedish). **No transcript
 text and no domain terms are recorded here**: this repo is public and the content is private.
 Only aggregate numbers.
 Backends, both through the LiteLLM gateway and Tapir's own adapter, unchanged:
 `iguana/kb-whisper` (whisper.cpp, KB's Swedish fine-tune) and `iguana/pianissimo` (KlangAI
 Swedish Parakeet/FastConformer-TDT, MLX 8-bit, served by a small FastAPI shell on iguana:8102).
 ## Result: a near tie on accuracy, a large difference in speed
 | | kb-whisper | pianissimo |
 |---|---|---|
 | Wall time for 63.7 min | 4m08s (~15x realtime) | **40 s (~95x realtime)** |
 | Segments | 1,302 (timed pieces, wrapped at ~60 chars) | 434 (sentences) |
 | Words (normalised) | 6,606 | 6,510 |
 | Punctuation | light, every segment ends in a full stop | heavier (138 vs 82 commas) |
 | Repetition loops | none | none |
 Word-level agreement between them: **92.4%** as first computed, **93.1%** after the segment
 rejoin fix below. 380 aligned disagreement spans, median one word each.
 **Blind A/B by the speaker** (the only ground truth available), on the 30 spans a model
 (Claude Opus 5.5) had picked as changing meaning, shuffled and unlabelled: **pianissimo 17,
 kb-whisper 13**, no "neither", no "can't tell".
 | Category of the 30 | pianissimo | kb-whisper |
 |---|---|---|
 | Technical term / English loanword | 8 | 4 |
 | Number / date / unit | 4 | 1 |
 | Name / proper noun | 3 | 3 |
 | Function word / small word | 1 | 2 |
 | Other (everyday words) | 1 | 3 |
 The reading: pianissimo is better on terminology and numbers, kb-whisper on ordinary Swedish
 words; names are even. Their errors differ in kind: kb-whisper's tend to be garbled words, a
 couple were wrong numbers; pianissimo's tend to be fluent substitutions that read as plausible
 (a wrong institution, an invented person, a dropped sign), which a reader or an LLM is less
 likely to catch.
 Opus's own lean agreed with the speaker's blind answer on **30 of 30**. That is not a measure
 of Opus over all 380 spans: it chose spans it could decide, and every one of the 30 carried a
 lean. Its full classification of the 380 (counts only): function/small word 81, filler or
 false start 65, technical term 58, compound split/join 46, other 39, name 28, casing or
 punctuation residue 26, inflection 23, number/date/unit 10, content omission or insertion
 (>= 3 words) 4.
 ## A bug this comparison found in Tapir (fixed)
 kb-whisper's output is wrapped at ~60 characters and **a wrap can fall inside a word**; the
 continuation segment has no leading space, the response's `text` field joins segments with
 newlines, and a segment can be a lone `.`. Tapir stored that joined text, so chat received
 broken words. Measured on real gateway output: 7% of segments (96/1,332) in a synthetic hour,
 42/1,302 lines in the real one. berget's segments are whole sentences with no leading space, so
 the two need different joins. `transcribe.joinSegments` now rebuilds the text from the segments.
 Checked against the real transcript with pianissimo's vocabulary as an independent yardstick:
 36 spans changed, 31 now entirely known words, words unknown to pianissimo 224 -> 177.
 Five rejoined spans still contain an unknown word (a name or an unlucky glue): small residual
 risk. Earlier tests used short sentences and could not see this.
 ## What this does not show
 - **One recording, one speaker, one domain.** No claim about other voices, accents, crosstalk,
  or room noise. 17 vs 13 is not a significant difference.
 - **The other ~350 spans** were not judged by the speaker. They are mostly spelling, compounds,
  fillers and small words; Opus's counts there are a model's judgement, not ground truth.
 - Errors **both** models share in text they agree on are invisible to this method.
 - Speed was measured once each, on an otherwise idle iguana; pianissimo's 40 s includes the
  gateway hop and a 122 MB WAV upload.
 - Whether pianissimo survives the infra#273 Metal-OOM contention the way whisper does not was
  not tested; both run on iguana.
 ## What it suggests for Tapir (not decided, filed as an issue)
 Pianissimo is a **second local backend on the same host, through the same adapter**. That
 makes it a natural middle tier ahead of the cloud fallback (kb-whisper -> pianissimo ->
 berget), so a wedged whisper no longer sends private audio off the homelab. At 40 s an hour it
 is also cheap enough to run **both** on every upload and give chat both texts, or flag where
 they disagree; the maintainer's blind answers show they err in different places.