feat(discovery): drop Shorts and livestreams before the caption fetch (ADR-023)
The scarce resource is the per-IP timedtext caption fetch (ADR-014); the pilot's candidate set was mostly Shorts/clips/livestreams, each burning a fetch (a "none" result is a completed fetch — it costs budget even when it yields nothing). NewVideos now enriches candidates with one cheap Data API videos.list call (contentDetails.duration + snippet.liveBroadcastContent — the quota API, a DIFFERENT limit from the timedtext 429) and drops, before returning: videos shorter than TAPIR_MIN_VIDEO_SECONDS (default 60) and any live/upcoming broadcast. Dropped videos are never persisted, so the list declutters too. Degrade-open: MinVideoSeconds=0 disables it (no quota call); a videos.list error returns candidates unfiltered so discovery never breaks on a metadata hiccup. The paste-a-URL path (VideoByID) is not filtered — an explicit request is honoured. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -855,6 +855,42 @@ reply still parses; a transcript within budget is unchanged).
|
||||
|
||||
---
|
||||
|
||||
## ADR-023 — Drop Shorts/livestreams at discovery to protect the caption budget
|
||||
|
||||
**Status:** Accepted (2026-06-10). **Builds on ADR-014** (per-IP caption rate limit is the
|
||||
binding constraint) and the ADR-022 live-run findings.
|
||||
|
||||
**Context.** The scarce resource is the unofficial timedtext caption fetch (per-egress-IP
|
||||
429, ~3 successful/pass). The first multi-user run showed the candidate set was mostly noise —
|
||||
Shorts, sub-minute clips, and live broadcasts — each of which still consumes a caption-fetch
|
||||
attempt (and a "none" result is a *completed* fetch, so it costs budget even when it yields
|
||||
nothing). Spending the rate-limited budget on content the user will not read is the waste to
|
||||
cut first; it is cheaper and lower-risk than raising the ceiling (multi-IP, Whisper).
|
||||
|
||||
Duration and live status are NOT in the playlistItems discovery response, but they ARE in the
|
||||
Data API `videos.list` (contentDetails.duration + snippet.liveBroadcastContent) — the official
|
||||
**quota-based** API (1 unit/call, 50 ids/call), which is a *different* limit from the timedtext
|
||||
429. So one cheap quota call buys a filter that saves many expensive throttled fetches.
|
||||
|
||||
**Decision.**
|
||||
1. `NewVideos` enriches its candidates with a single `videos.list` call and drops, before
|
||||
returning: videos shorter than `TAPIR_MIN_VIDEO_SECONDS` (default 60) and any `live`/
|
||||
`upcoming` broadcast. Dropped videos are never persisted, so they also declutter the list.
|
||||
2. The filter is **degrade-open**: `MinVideoSeconds=0` disables it (no quota call), and a
|
||||
`videos.list` error returns the candidates unfiltered — discovery must never break because a
|
||||
metadata call hiccuped (worst case = pre-ADR-023 behaviour).
|
||||
3. The paste-a-URL path (`VideoByID`) is **not** filtered — an explicit user request for a
|
||||
specific video (even a Short) is honoured.
|
||||
|
||||
**Reversibility.** Pure discovery-time filter + config. `TAPIR_MIN_VIDEO_SECONDS=0` restores
|
||||
the old behaviour. No schema change, no effect on already-stored videos.
|
||||
|
||||
**Quota note.** Per-channel enrichment adds ~1 unit/channel/pass. At pilot scale (≤3 users)
|
||||
this is well under the 10k/day cap; at larger scale, batch `videos.list` across channels
|
||||
(50 ids/call) by collecting all discovered ids per pass before enriching.
|
||||
|
||||
---
|
||||
|
||||
## Rejected alternatives
|
||||
|
||||
Approaches considered during the 2026-06-02 planning + grill session and **deliberately not
|
||||
|
||||
Reference in New Issue
Block a user