feat(youtube): process-wide caption-fetch rate gate (ADR-014 item 2)
ADR-014 item 2 — a single per-egress-IP rate gate shared by every caption fetch — was specced but only per-video backoff (rate_limited_at) shipped. Build the real gate now: it is load-bearing once ADR-018 puts auto-summarize on an in-process schedule across multiple users (all fetches leave one pod's egress IP, concurrently with live "Summarize" clicks — without a shared gate that self-inflicts 429s every cycle). globalFetchGate (golang.org/x/time/rate, default 2s/req burst 1) is consulted in httpDo before every live outbound fetch — player, watch-page, timedtext — so the scheduler runners and the web click-path serialise through one limiter regardless of how many users/goroutines are upstream. The test seam (a.transport != nil) skips the gate so fakes are not throttled. TAPIR_FETCH_RATE (Go duration, default 2s, 0 = unlimited) wires SetFetchRate in cmdRun; the existing per-video backoff stays as the complementary 429 handler. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -65,6 +65,17 @@ type Config struct {
|
||||
// expires the video is retried. Zero means "always retry" (no backoff).
|
||||
FetchBackoff time.Duration
|
||||
|
||||
// FetchRate is the minimum interval between outbound caption fetches across the
|
||||
// whole process — the shared per-egress-IP rate gate (ADR-014 item 2). It is
|
||||
// the gate that makes auto-summarize-on-a-schedule safe: scheduler runners and
|
||||
// the web click-path serialise through it. Zero = unlimited (dev/tests).
|
||||
FetchRate time.Duration
|
||||
|
||||
// DiscoveryInterval, when > 0, makes `serve` run in-process scheduled discovery
|
||||
// for ALL users on that cadence (ADR-018). Zero/unset = disabled, so dev and
|
||||
// tests never auto-fetch. Single-replica assumption — see cmdServe.
|
||||
DiscoveryInterval time.Duration
|
||||
|
||||
// HTTPAddr is the listen address for `tapir serve` (the Stage-0 web UI).
|
||||
HTTPAddr string
|
||||
|
||||
@@ -97,6 +108,7 @@ const (
|
||||
defaultOAuthRedirectAddr = "localhost:8080"
|
||||
defaultHTTPAddr = ":8080"
|
||||
defaultFetchBackoff = time.Hour
|
||||
defaultFetchRate = 2 * time.Second
|
||||
defaultPublicURL = "https://tapir.d-ma.be"
|
||||
)
|
||||
|
||||
@@ -144,6 +156,18 @@ func Load() (Config, error) {
|
||||
}
|
||||
c.FetchBackoff = backoff
|
||||
|
||||
fetchRate, err := durationOr("TAPIR_FETCH_RATE", defaultFetchRate)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
c.FetchRate = fetchRate
|
||||
|
||||
discovery, err := durationOr("TAPIR_DISCOVERY_INTERVAL", 0)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
c.DiscoveryInterval = discovery
|
||||
|
||||
return c, nil
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user