diff --git a/docs/research/vimeo-captions-feasibility.md b/docs/research/vimeo-captions-feasibility.md new file mode 100644 index 0000000..a5f0cbf --- /dev/null +++ b/docs/research/vimeo-captions-feasibility.md @@ -0,0 +1,52 @@ +# Vimeo — captions/transcripts feasibility for tapir (research, 2026-06-02) + +Researched by the Conductor (sub-agents were network-sandboxed). Mapped to tapir's +`VideoSource` port (`ListSubscriptions`, `NewVideos`, `FetchTranscript`). + +## Verdict: SAME owner-only wall as YouTube, and WORSE fallback options. + +### Q3 (the gating question) — text tracks for videos you DON'T own +- **Official API is owner-only.** Vimeo Help Center: *"You must use a personal token + generated as the account owner of the video in order to access your video's transcripts + via API."* So `GET /videos/{id}/texttracks` returns tracks only for videos the + authenticated account owns — identical to YouTube `captions.download`. + Source: https://help.vimeo.com/hc/en-us/articles/17480150130833 +- The text-track `link` points to `captions.cloud.vimeo.com/....vtt?token=...` — a + **token-gated** URL, not a clean public file. + Source: https://developer.vimeo.com/api/reference/response/text-track +- The API is also widely reported flaky (vimeo/openapi#16 "Texttracks API doesn't work as + advertised"). Source: https://github.com/vimeo/openapi/issues/16 + +### Unofficial fallback — weaker than YouTube's timedtext +- Captions are loadable via the embed player, yielding `[n].vtt?token=...` requests, and + yt-dlp can pull Vimeo subs with cookies. But every documented method needs a + **registered/logged-in viewer** ("you need to be registered in Vimeo to retrieve + subtitles"). There is **no clean unauthenticated public timedtext equivalent** like + YouTube's `captionTracks[].baseUrl`. + Sources: https://gist.github.com/pjmartorell/384aa1d019f919af6b84894038bf29aa , + https://github.com/vimeo/openapi/blob/master/api.yaml +- **Needs live verification:** whether `player.vimeo.com/video/{id}/config` returns a + `text_tracks[]` with usable token'd URLs for arbitrary PUBLIC videos without owner auth. + Likely partially works for public/embeddable videos but is undocumented and brittle. + +### Subscriptions / new videos (the easy part) +- "Following" exists: `/me/following` (users), plus channels/groups feeds; per-user video + lists via `/users/{id}/videos` sortable by date. Maps to `ListSubscriptions`/`NewVideos`. + Source: https://developer.vimeo.com/api/reference +- Auth: OAuth2; many endpoints/scopes and higher rate limits require **app review**, and + some metadata is gated behind paid plans. (Tier/plan gating: needs-verification.) + +## Feasibility for tapir +- A Vimeo adapter could do subscriptions + new-video detection fine. +- **Transcripts are the blocker, same as YouTube but harder**: official path owner-only, + unofficial path requires a logged-in viewer (cookie/token), no public timedtext. +- Realistic Stage-0 stance: **defer Vimeo.** YouTube via the timedtext path (ADR-010) + covers the maintainer's actual subscriptions; Vimeo adds a second brittle, auth-gated + caption path for a smaller share of content. If pursued: the cookie'd player-config + method or yt-dlp-with-cookies, behind the same "ToS-grey, opt-in" framing — a future ADR. +- This is consistent with the port design: Vimeo is just another adapter; nothing in the + engine changes when/if it's added. + +**Biggest risk:** there is no public, unauthenticated caption path on Vimeo — any +third-party transcript fetch needs a logged-in session/cookie, which is more fragile and +more ToS-exposed than YouTube timedtext. diff --git a/docs/research/whisper-stt-feasibility.md b/docs/research/whisper-stt-feasibility.md new file mode 100644 index 0000000..8f1359b --- /dev/null +++ b/docs/research/whisper-stt-feasibility.md @@ -0,0 +1,47 @@ +# Whisper track — self-transcription feasibility for tapir (recon, 2026-06-02) + +Investigated inline by the Conductor (sub-agents are network-sandboxed this session). +Context: ADR-007 deferred audio-download + STT for (a) yt-dlp fragility, (b) ToS, (c) GPU +contention with the JEPA PoC on koala. With ADR-010 (timedtext captions) now landed and +working, Whisper is a **fallback for videos with NO captions**, not the primary path. + +## What exists on the infra (verified from koala) + +- **koala host has NONE of the tooling**: no `ffmpeg`, no `yt-dlp`, no `whisper` / + `whisper.cpp` / `faster-whisper` / `mlx_whisper`. Only `python3`. Audio extraction would + need a container/pod that bundles yt-dlp + ffmpeg. +- **Host GPU is currently unusable**: `nvidia-smi` → "Driver/library version mismatch" + (NVML 610.43). The RTX 5070 is reachable only via k3s pods with `runtimeClassName: nvidia` + (matches the known koala-GPU brain note), not from the host. +- **A Whisper model is ALREADY on the gateway**: `berget/whisper-large-v3` is served via + LiteLLM (`http://koala:30401/v1`). BUT `berget/` = berget.ai = **cloud (EU)**. Using it + ships audio off-box — acceptable under the homelab's berget-as-trusted-fallback posture, + but NOT "local-first" in the strict VISION sense. +- **iguana** has `mlx-whisper` (per infra docs) — a local Mac-GPU option, no contention with + koala's JEPA PoC — but it is **not exposed on the gateway** and its availability is + uncertain (prior notes had iguana offline at times). Needs-verification. + +## The pipeline Whisper would require +1. **Audio extraction**: `yt-dlp -f bestaudio` + `ffmpeg` to wav/16k. yt-dlp is exactly the + ToS-grey + breakage risk ADR-007 named. Volume matters: full-audio download for many + candidates is far heavier (bandwidth + time) than fetching caption text. +2. **Transcription compute** — three options: + - **A. berget/whisper-large-v3 (cloud, ready now)** — zero new infra, fast, but external; + gate as opt-in per the local-first principle. + - **B. Local GPU whisper pod on koala** (nvidia runtime) — true local, but: deploy work + + GPU contention with JEPA (ADR-007's stated reason) + the driver-mismatch to sort. + - **C. iguana mlx-whisper** — local, no koala-GPU contention, but needs exposing + uptime. + +## Verdict +- **Whisper is viable but remains a deferred fallback, and it reverses a settled ADR** — so + it needs a new ADR (extends/supersedes ADR-007's deferral) before building. +- **Lowest-effort path if/when pursued**: route no-caption videos' audio to the existing + `berget/whisper-large-v3` on the gateway, behind an explicit opt-in flag (it's cloud). + Local purity → iguana mlx-whisper or a koala nvidia whisper pod, both more setup. +- **Priority**: LOW right now. ADR-010 timedtext captions already cover captioned videos + (the bulk of the maintainer's subs). Whisper only buys the no-caption tail — measure how + big that tail actually is (the run's `skipped_no_text` count) before investing. + +**Biggest risks:** (1) yt-dlp fragility + ToS (the original ADR-007 objection, unchanged); +(2) audio-download volume/latency vs. caption-text; (3) "local-first" is only truly honored +by options B/C, both of which carry real setup/contention cost.