feat(youtube): acquire captions via player/timedtext baseUrl (ADR-010)
The Data API captions.download endpoint is owner-only: every subscription video the user does not own returned HTTP 403, producing 0 summaries and a ~150-line error spew in the first live Stage-0 run. Captions-first (ADR-007) is sound; only the acquisition mechanism was wrong. FetchTranscript now resolves caption tracks from the InnerTube player response (ANDROID client, unauthenticated) and GETs the chosen track's timedtext baseUrl with a plain http.Client — no OAuth token, which can break the endpoint. The srv3 XML, json3, and legacy <transcript> formats all parse; non-asr tracks in a preferred language win. Watch-page ytInitialPlayerResponse scrape is the fallback when InnerTube returns no tracks. Degrade, don't error (explicit quick-fix): no captionTracks, empty baseUrl, a non-200 fetch, or an unparseable body yield Source=none, not an error. Only genuine transport faults error — this kills the spew. OAuth stays on ListSubscriptions/NewVideos (Data API); only transcript fetch goes unauthed. Validated live from koala: the ANDROID client returned working baseUrls and real transcript text for public videos the run identity does not own. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
+51
-1
@@ -120,7 +120,8 @@ secret surface (see VISION Stage 2); it gets the existing, vetted secret path.
|
||||
|
||||
## ADR-007 — Captions-first; audio-download + speech-to-text deferred
|
||||
|
||||
**Status:** Accepted (2026-06-02)
|
||||
**Status:** Accepted (2026-06-02). Acquisition mechanism superseded by ADR-010 (Data API
|
||||
`captions.download` → player/timedtext baseUrl); captions-first stance and STT deferral stand.
|
||||
|
||||
**Context.** YouTube's Data API does not expose transcripts. Options: official captions
|
||||
(clean, limited coverage) vs. audio download + local Whisper (broad coverage, ToS-grey,
|
||||
@@ -153,6 +154,55 @@ governs advancement. Reversible: if demand appears, a new ADR opens the Future C
|
||||
|
||||
---
|
||||
|
||||
## ADR-010 — Third-party caption acquisition via the timedtext/player baseUrl
|
||||
|
||||
**Status:** Accepted (2026-06-02)
|
||||
|
||||
**Context.** ADR-007 settled *captions-first*. The first build used the YouTube Data API
|
||||
`captions.list` + `captions.download` endpoints to acquire them. A live Stage-0 run proved that
|
||||
`captions.download` is **owner-only**: it requires the OAuth identity to own the video, so every
|
||||
subscription video the user does *not* own returns HTTP 403. Result: 0 summaries produced and a
|
||||
~150-line error spew. The captions-first decision is sound; only the *acquisition mechanism* was
|
||||
wrong.
|
||||
|
||||
**Decision.** Acquire captions from the **player response + timedtext baseUrl**, not the Data API
|
||||
`captions` endpoints:
|
||||
|
||||
1. `POST https://www.youtube.com/youtubei/v1/player` with an **InnerTube `ANDROID` client
|
||||
context** (no API key, no OAuth). Read
|
||||
`captions.playerCaptionsTracklistRenderer.captionTracks[]`. Each track carries `baseUrl`,
|
||||
`languageCode`, and `kind` (`"asr"` = auto-generated).
|
||||
2. Select by `PreferredLanguages`, preferring non-`asr` when both exist.
|
||||
3. **GET the track's `baseUrl` unauthenticated** (plain `http.Client`, no OAuth token attached —
|
||||
the token can break the timedtext endpoint). The ANDROID `baseUrl` is pinned to `fmt=srv3`
|
||||
(timedtext XML); the parser also accepts `json3` and the legacy `<transcript>` XML.
|
||||
|
||||
A **watch-page scrape** of `ytInitialPlayerResponse` is the documented fallback if InnerTube
|
||||
returns no `captionTracks`.
|
||||
|
||||
**Live validation (from koala, 2026-06-02):** the `ANDROID` InnerTube client returned 6
|
||||
`captionTracks` with working `baseUrl`s for a public video the run identity does not own, and the
|
||||
unauthenticated `baseUrl` GET returned real transcript text. `ANDROID` is the client of record
|
||||
(historically returns baseUrls without a PoToken). The `WEB` client and watch-page scrape are
|
||||
fallbacks.
|
||||
|
||||
**Consequences.**
|
||||
|
||||
- Works for **any public captioned video**, not just owned ones — this is the fix for the 403 wall.
|
||||
- **ToS-grey:** `youtubei`/`timedtext` are unofficial endpoints. They can break when Google shifts
|
||||
InnerTube client requirements or introduces PoToken gating. Mitigation: degrade, never error — a
|
||||
missing/empty/403/unparseable caption yields `domain.Transcript{Source: SourceNone}`, so a future
|
||||
breakage produces "no transcript" rather than a crash or error spew. Only genuine transport
|
||||
(network) faults error.
|
||||
- **No OAuth** is needed for the transcript fetch. OAuth is still required for `ListSubscriptions`
|
||||
and `NewVideos` (Data API) — only the transcript path goes unauthenticated.
|
||||
- **Still no Whisper.** Speech-to-text stays deferred (ADR-007 unchanged).
|
||||
|
||||
**Supersedes:** the *acquisition mechanism* of ADR-007 (Data API `captions.download` →
|
||||
player/timedtext baseUrl) only. ADR-007's captions-first stance and the STT deferral stand.
|
||||
|
||||
---
|
||||
|
||||
## ADR-009 — Trunk-Based Development
|
||||
|
||||
**Status:** Accepted (2026-06-02)
|
||||
|
||||
Reference in New Issue
Block a user