docs(research): Vimeo + Whisper transcript feasibility findings
Vimeo texttracks API is owner-only (worse than YouTube, no public timedtext) → defer. Whisper viable as a no-caption fallback; berget/whisper-large-v3 already on the gateway (cloud), local options on iguana/koala carry setup+GPU-contention cost (ADR-007). Both investigated inline (session sub-agents are network-sandboxed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,52 @@
|
|||||||
|
# Vimeo — captions/transcripts feasibility for tapir (research, 2026-06-02)
|
||||||
|
|
||||||
|
Researched by the Conductor (sub-agents were network-sandboxed). Mapped to tapir's
|
||||||
|
`VideoSource` port (`ListSubscriptions`, `NewVideos`, `FetchTranscript`).
|
||||||
|
|
||||||
|
## Verdict: SAME owner-only wall as YouTube, and WORSE fallback options.
|
||||||
|
|
||||||
|
### Q3 (the gating question) — text tracks for videos you DON'T own
|
||||||
|
- **Official API is owner-only.** Vimeo Help Center: *"You must use a personal token
|
||||||
|
generated as the account owner of the video in order to access your video's transcripts
|
||||||
|
via API."* So `GET /videos/{id}/texttracks` returns tracks only for videos the
|
||||||
|
authenticated account owns — identical to YouTube `captions.download`.
|
||||||
|
Source: https://help.vimeo.com/hc/en-us/articles/17480150130833
|
||||||
|
- The text-track `link` points to `captions.cloud.vimeo.com/....vtt?token=...` — a
|
||||||
|
**token-gated** URL, not a clean public file.
|
||||||
|
Source: https://developer.vimeo.com/api/reference/response/text-track
|
||||||
|
- The API is also widely reported flaky (vimeo/openapi#16 "Texttracks API doesn't work as
|
||||||
|
advertised"). Source: https://github.com/vimeo/openapi/issues/16
|
||||||
|
|
||||||
|
### Unofficial fallback — weaker than YouTube's timedtext
|
||||||
|
- Captions are loadable via the embed player, yielding `[n].vtt?token=...` requests, and
|
||||||
|
yt-dlp can pull Vimeo subs with cookies. But every documented method needs a
|
||||||
|
**registered/logged-in viewer** ("you need to be registered in Vimeo to retrieve
|
||||||
|
subtitles"). There is **no clean unauthenticated public timedtext equivalent** like
|
||||||
|
YouTube's `captionTracks[].baseUrl`.
|
||||||
|
Sources: https://gist.github.com/pjmartorell/384aa1d019f919af6b84894038bf29aa ,
|
||||||
|
https://github.com/vimeo/openapi/blob/master/api.yaml
|
||||||
|
- **Needs live verification:** whether `player.vimeo.com/video/{id}/config` returns a
|
||||||
|
`text_tracks[]` with usable token'd URLs for arbitrary PUBLIC videos without owner auth.
|
||||||
|
Likely partially works for public/embeddable videos but is undocumented and brittle.
|
||||||
|
|
||||||
|
### Subscriptions / new videos (the easy part)
|
||||||
|
- "Following" exists: `/me/following` (users), plus channels/groups feeds; per-user video
|
||||||
|
lists via `/users/{id}/videos` sortable by date. Maps to `ListSubscriptions`/`NewVideos`.
|
||||||
|
Source: https://developer.vimeo.com/api/reference
|
||||||
|
- Auth: OAuth2; many endpoints/scopes and higher rate limits require **app review**, and
|
||||||
|
some metadata is gated behind paid plans. (Tier/plan gating: needs-verification.)
|
||||||
|
|
||||||
|
## Feasibility for tapir
|
||||||
|
- A Vimeo adapter could do subscriptions + new-video detection fine.
|
||||||
|
- **Transcripts are the blocker, same as YouTube but harder**: official path owner-only,
|
||||||
|
unofficial path requires a logged-in viewer (cookie/token), no public timedtext.
|
||||||
|
- Realistic Stage-0 stance: **defer Vimeo.** YouTube via the timedtext path (ADR-010)
|
||||||
|
covers the maintainer's actual subscriptions; Vimeo adds a second brittle, auth-gated
|
||||||
|
caption path for a smaller share of content. If pursued: the cookie'd player-config
|
||||||
|
method or yt-dlp-with-cookies, behind the same "ToS-grey, opt-in" framing — a future ADR.
|
||||||
|
- This is consistent with the port design: Vimeo is just another adapter; nothing in the
|
||||||
|
engine changes when/if it's added.
|
||||||
|
|
||||||
|
**Biggest risk:** there is no public, unauthenticated caption path on Vimeo — any
|
||||||
|
third-party transcript fetch needs a logged-in session/cookie, which is more fragile and
|
||||||
|
more ToS-exposed than YouTube timedtext.
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
# Whisper track — self-transcription feasibility for tapir (recon, 2026-06-02)
|
||||||
|
|
||||||
|
Investigated inline by the Conductor (sub-agents are network-sandboxed this session).
|
||||||
|
Context: ADR-007 deferred audio-download + STT for (a) yt-dlp fragility, (b) ToS, (c) GPU
|
||||||
|
contention with the JEPA PoC on koala. With ADR-010 (timedtext captions) now landed and
|
||||||
|
working, Whisper is a **fallback for videos with NO captions**, not the primary path.
|
||||||
|
|
||||||
|
## What exists on the infra (verified from koala)
|
||||||
|
|
||||||
|
- **koala host has NONE of the tooling**: no `ffmpeg`, no `yt-dlp`, no `whisper` /
|
||||||
|
`whisper.cpp` / `faster-whisper` / `mlx_whisper`. Only `python3`. Audio extraction would
|
||||||
|
need a container/pod that bundles yt-dlp + ffmpeg.
|
||||||
|
- **Host GPU is currently unusable**: `nvidia-smi` → "Driver/library version mismatch"
|
||||||
|
(NVML 610.43). The RTX 5070 is reachable only via k3s pods with `runtimeClassName: nvidia`
|
||||||
|
(matches the known koala-GPU brain note), not from the host.
|
||||||
|
- **A Whisper model is ALREADY on the gateway**: `berget/whisper-large-v3` is served via
|
||||||
|
LiteLLM (`http://koala:30401/v1`). BUT `berget/` = berget.ai = **cloud (EU)**. Using it
|
||||||
|
ships audio off-box — acceptable under the homelab's berget-as-trusted-fallback posture,
|
||||||
|
but NOT "local-first" in the strict VISION sense.
|
||||||
|
- **iguana** has `mlx-whisper` (per infra docs) — a local Mac-GPU option, no contention with
|
||||||
|
koala's JEPA PoC — but it is **not exposed on the gateway** and its availability is
|
||||||
|
uncertain (prior notes had iguana offline at times). Needs-verification.
|
||||||
|
|
||||||
|
## The pipeline Whisper would require
|
||||||
|
1. **Audio extraction**: `yt-dlp -f bestaudio` + `ffmpeg` to wav/16k. yt-dlp is exactly the
|
||||||
|
ToS-grey + breakage risk ADR-007 named. Volume matters: full-audio download for many
|
||||||
|
candidates is far heavier (bandwidth + time) than fetching caption text.
|
||||||
|
2. **Transcription compute** — three options:
|
||||||
|
- **A. berget/whisper-large-v3 (cloud, ready now)** — zero new infra, fast, but external;
|
||||||
|
gate as opt-in per the local-first principle.
|
||||||
|
- **B. Local GPU whisper pod on koala** (nvidia runtime) — true local, but: deploy work +
|
||||||
|
GPU contention with JEPA (ADR-007's stated reason) + the driver-mismatch to sort.
|
||||||
|
- **C. iguana mlx-whisper** — local, no koala-GPU contention, but needs exposing + uptime.
|
||||||
|
|
||||||
|
## Verdict
|
||||||
|
- **Whisper is viable but remains a deferred fallback, and it reverses a settled ADR** — so
|
||||||
|
it needs a new ADR (extends/supersedes ADR-007's deferral) before building.
|
||||||
|
- **Lowest-effort path if/when pursued**: route no-caption videos' audio to the existing
|
||||||
|
`berget/whisper-large-v3` on the gateway, behind an explicit opt-in flag (it's cloud).
|
||||||
|
Local purity → iguana mlx-whisper or a koala nvidia whisper pod, both more setup.
|
||||||
|
- **Priority**: LOW right now. ADR-010 timedtext captions already cover captioned videos
|
||||||
|
(the bulk of the maintainer's subs). Whisper only buys the no-caption tail — measure how
|
||||||
|
big that tail actually is (the run's `skipped_no_text` count) before investing.
|
||||||
|
|
||||||
|
**Biggest risks:** (1) yt-dlp fragility + ToS (the original ADR-007 objection, unchanged);
|
||||||
|
(2) audio-download volume/latency vs. caption-text; (3) "local-first" is only truly honored
|
||||||
|
by options B/C, both of which carry real setup/contention cost.
|
||||||
Reference in New Issue
Block a user