mathias/tapir
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(youtube): use a browser UA for the watch-page caption scrape
CI / Lint / Test / Vet (push) Successful in 9s
Details
CI / Build & Import (push) Successful in 9s
Details
CI / Mirror to GitHub (push) Failing after 3s
Details

Browse Source 
The ANDROID InnerTube player now returns 0 captionTracks (PoToken-gated), so the
watch-page scrape is the real path — but it sent an Android *app* User-Agent,
which makes YouTube serve a page WITHOUT ytInitialPlayerResponse, so captions
came back empty. Result: 87% of a live run skipped as 'no transcript' despite
the videos having captions. A desktop-browser UA returns the player JSON with
captionTracks. Confirmed live: app-UA=0 tracks, browser-UA=1.
This commit is contained in:
mathias
2026-06-03 09:35:36 +02:00
parent c9863e9633
commit 7589c09201
1 changed files with 6 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/adapters/youtube/captions.go
+6 -1
View File
@@ -21,6 +21,11 @@ const defaultPlayerBaseURL = "https://www.youtube.com"
// returns caption baseUrls without a PoToken requirement (ADR-010). // returns caption baseUrls without a PoToken requirement (ADR-010).
const androidUserAgent = "com.google.android.youtube/20.10.38 (Linux; U; Android 11) gzip" const androidUserAgent = "com.google.android.youtube/20.10.38 (Linux; U; Android 11) gzip"
// browserUserAgent is required for the watch-page scrape: with an app UA YouTube
// serves a page WITHOUT the embedded ytInitialPlayerResponse, so captionTracks
// come back empty. A desktop-browser UA returns the player JSON with captions.
const browserUserAgent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
// maxCaptionBytes bounds a single timedtext/player body read. // maxCaptionBytes bounds a single timedtext/player body read.
const maxCaptionBytes = 16 << 20 // 16 MiB const maxCaptionBytes = 16 << 20 // 16 MiB
@@ -134,7 +139,7 @@ const ytInitialMarker = "ytInitialPlayerResponse"
// embedded ytInitialPlayerResponse JSON. Any failure degrades to no tracks. // embedded ytInitialPlayerResponse JSON. Any failure degrades to no tracks.
func (a *Adapter) scrapeCaptionTracks(ctx context.Context, client *http.Client, videoID string) ([]captionTrack, error) { func (a *Adapter) scrapeCaptionTracks(ctx context.Context, client *http.Client, videoID string) ([]captionTrack, error) {
body, status, err := a.httpGet(ctx, client, a.playerBaseURL+"/watch?v="+videoID, map[string]string{ body, status, err := a.httpGet(ctx, client, a.playerBaseURL+"/watch?v="+videoID, map[string]string{
"User-Agent": androidUserAgent, "User-Agent": browserUserAgent,
}) })
if err != nil { if err != nil {
return nil, err return nil, err