docs(research): Vimeo + Whisper transcript feasibility findings
Vimeo texttracks API is owner-only (worse than YouTube, no public timedtext) → defer. Whisper viable as a no-caption fallback; berget/whisper-large-v3 already on the gateway (cloud), local options on iguana/koala carry setup+GPU-contention cost (ADR-007). Both investigated inline (session sub-agents are network-sandboxed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,47 @@
|
||||
# Whisper track — self-transcription feasibility for tapir (recon, 2026-06-02)
|
||||
|
||||
Investigated inline by the Conductor (sub-agents are network-sandboxed this session).
|
||||
Context: ADR-007 deferred audio-download + STT for (a) yt-dlp fragility, (b) ToS, (c) GPU
|
||||
contention with the JEPA PoC on koala. With ADR-010 (timedtext captions) now landed and
|
||||
working, Whisper is a **fallback for videos with NO captions**, not the primary path.
|
||||
|
||||
## What exists on the infra (verified from koala)
|
||||
|
||||
- **koala host has NONE of the tooling**: no `ffmpeg`, no `yt-dlp`, no `whisper` /
|
||||
`whisper.cpp` / `faster-whisper` / `mlx_whisper`. Only `python3`. Audio extraction would
|
||||
need a container/pod that bundles yt-dlp + ffmpeg.
|
||||
- **Host GPU is currently unusable**: `nvidia-smi` → "Driver/library version mismatch"
|
||||
(NVML 610.43). The RTX 5070 is reachable only via k3s pods with `runtimeClassName: nvidia`
|
||||
(matches the known koala-GPU brain note), not from the host.
|
||||
- **A Whisper model is ALREADY on the gateway**: `berget/whisper-large-v3` is served via
|
||||
LiteLLM (`http://koala:30401/v1`). BUT `berget/` = berget.ai = **cloud (EU)**. Using it
|
||||
ships audio off-box — acceptable under the homelab's berget-as-trusted-fallback posture,
|
||||
but NOT "local-first" in the strict VISION sense.
|
||||
- **iguana** has `mlx-whisper` (per infra docs) — a local Mac-GPU option, no contention with
|
||||
koala's JEPA PoC — but it is **not exposed on the gateway** and its availability is
|
||||
uncertain (prior notes had iguana offline at times). Needs-verification.
|
||||
|
||||
## The pipeline Whisper would require
|
||||
1. **Audio extraction**: `yt-dlp -f bestaudio` + `ffmpeg` to wav/16k. yt-dlp is exactly the
|
||||
ToS-grey + breakage risk ADR-007 named. Volume matters: full-audio download for many
|
||||
candidates is far heavier (bandwidth + time) than fetching caption text.
|
||||
2. **Transcription compute** — three options:
|
||||
- **A. berget/whisper-large-v3 (cloud, ready now)** — zero new infra, fast, but external;
|
||||
gate as opt-in per the local-first principle.
|
||||
- **B. Local GPU whisper pod on koala** (nvidia runtime) — true local, but: deploy work +
|
||||
GPU contention with JEPA (ADR-007's stated reason) + the driver-mismatch to sort.
|
||||
- **C. iguana mlx-whisper** — local, no koala-GPU contention, but needs exposing + uptime.
|
||||
|
||||
## Verdict
|
||||
- **Whisper is viable but remains a deferred fallback, and it reverses a settled ADR** — so
|
||||
it needs a new ADR (extends/supersedes ADR-007's deferral) before building.
|
||||
- **Lowest-effort path if/when pursued**: route no-caption videos' audio to the existing
|
||||
`berget/whisper-large-v3` on the gateway, behind an explicit opt-in flag (it's cloud).
|
||||
Local purity → iguana mlx-whisper or a koala nvidia whisper pod, both more setup.
|
||||
- **Priority**: LOW right now. ADR-010 timedtext captions already cover captioned videos
|
||||
(the bulk of the maintainer's subs). Whisper only buys the no-caption tail — measure how
|
||||
big that tail actually is (the run's `skipped_no_text` count) before investing.
|
||||
|
||||
**Biggest risks:** (1) yt-dlp fragility + ToS (the original ADR-007 objection, unchanged);
|
||||
(2) audio-download volume/latency vs. caption-text; (3) "local-first" is only truly honored
|
||||
by options B/C, both of which carry real setup/contention cost.
|
||||
Reference in New Issue
Block a user