mathias/tapir
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
2fe4833434deda840055da53d1722b9169f46824
tapir/docs/research/whisper-stt-feasibility.md
T
mathiasandClaude Opus 4.8 92da25d01e
CI / Lint / Test / Vet (push) Successful in 6s
Details
CI / Build & Import (push) Failing after 1s
Details
CI / Mirror to GitHub (push) Has been skipped
Details
docs(research): Vimeo + Whisper transcript feasibility findings 
Vimeo texttracks API is owner-only (worse than YouTube, no public timedtext) → defer.
Whisper viable as a no-caption fallback; berget/whisper-large-v3 already on the gateway
(cloud), local options on iguana/koala carry setup+GPU-contention cost (ADR-007). Both
investigated inline (session sub-agents are network-sandboxed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-02 23:14:03 +02:00

3.2 KiB
Raw Blame History

Whisper track — self-transcription feasibility for tapir (recon, 2026-06-02)

Investigated inline by the Conductor (sub-agents are network-sandboxed this session). Context: ADR-007 deferred audio-download + STT for (a) yt-dlp fragility, (b) ToS, (c) GPU contention with the JEPA PoC on koala. With ADR-010 (timedtext captions) now landed and working, Whisper is a fallback for videos with NO captions, not the primary path.

What exists on the infra (verified from koala)

  • koala host has NONE of the tooling: no ffmpeg, no yt-dlp, no whisper / whisper.cpp / faster-whisper / mlx_whisper. Only python3. Audio extraction would need a container/pod that bundles yt-dlp + ffmpeg.
  • Host GPU is currently unusable: nvidia-smi → "Driver/library version mismatch" (NVML 610.43). The RTX 5070 is reachable only via k3s pods with runtimeClassName: nvidia (matches the known koala-GPU brain note), not from the host.
  • A Whisper model is ALREADY on the gateway: berget/whisper-large-v3 is served via LiteLLM (http://koala:30401/v1). BUT berget/ = berget.ai = cloud (EU). Using it ships audio off-box — acceptable under the homelab's berget-as-trusted-fallback posture, but NOT "local-first" in the strict VISION sense.
  • iguana has mlx-whisper (per infra docs) — a local Mac-GPU option, no contention with koala's JEPA PoC — but it is not exposed on the gateway and its availability is uncertain (prior notes had iguana offline at times). Needs-verification.

The pipeline Whisper would require

  1. Audio extraction: yt-dlp -f bestaudio + ffmpeg to wav/16k. yt-dlp is exactly the ToS-grey + breakage risk ADR-007 named. Volume matters: full-audio download for many candidates is far heavier (bandwidth + time) than fetching caption text.
  2. Transcription compute — three options:
    • A. berget/whisper-large-v3 (cloud, ready now) — zero new infra, fast, but external; gate as opt-in per the local-first principle.
    • B. Local GPU whisper pod on koala (nvidia runtime) — true local, but: deploy work + GPU contention with JEPA (ADR-007's stated reason) + the driver-mismatch to sort.
    • C. iguana mlx-whisper — local, no koala-GPU contention, but needs exposing + uptime.

Verdict

  • Whisper is viable but remains a deferred fallback, and it reverses a settled ADR — so it needs a new ADR (extends/supersedes ADR-007's deferral) before building.
  • Lowest-effort path if/when pursued: route no-caption videos' audio to the existing berget/whisper-large-v3 on the gateway, behind an explicit opt-in flag (it's cloud). Local purity → iguana mlx-whisper or a koala nvidia whisper pod, both more setup.
  • Priority: LOW right now. ADR-010 timedtext captions already cover captioned videos (the bulk of the maintainer's subs). Whisper only buys the no-caption tail — measure how big that tail actually is (the run's skipped_no_text count) before investing.

Biggest risks: (1) yt-dlp fragility + ToS (the original ADR-007 objection, unchanged); (2) audio-download volume/latency vs. caption-text; (3) "local-first" is only truly honored by options B/C, both of which carry real setup/contention cost.

Reference in New Issue View Git Blame Copy Permalink