mathiasandClaude Opus 4.8
f5021a8436
ADR-014 item 2 — a single per-egress-IP rate gate shared by every caption fetch — was specced but only per-video backoff (rate_limited_at) shipped. Build the real gate now: it is load-bearing once ADR-018 puts auto-summarize on an in-process schedule across multiple users (all fetches leave one pod's egress IP, concurrently with live "Summarize" clicks — without a shared gate that self-inflicts 429s every cycle). globalFetchGate (golang.org/x/time/rate, default 2s/req burst 1) is consulted in httpDo before every live outbound fetch — player, watch-page, timedtext — so the scheduler runners and the web click-path serialise through one limiter regardless of how many users/goroutines are upstream. The test seam (a.transport != nil) skips the gate so fakes are not throttled. TAPIR_FETCH_RATE (Go duration, default 2s, 0 = unlimited) wires SetFetchRate in cmdRun; the existing per-video backoff stays as the complementary 429 handler. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
38 lines
1.5 KiB
Go
38 lines
1.5 KiB
Go
package youtube
|
|
|
|
import (
|
|
"context"
|
|
"time"
|
|
|
|
"golang.org/x/time/rate"
|
|
)
|
|
|
|
// globalFetchGate is the process-wide rate limiter for outbound timedtext/caption
|
|
// fetches. A single instance is shared by ALL Adapter instances (the scheduler's
|
|
// per-user runners + the web click-path) so they cannot collectively exceed the
|
|
// per-egress-IP cap. ADR-014 item 2: the gate serialises/limits concurrent caption
|
|
// fetches regardless of how many users or goroutines are upstream. The 429 is per
|
|
// IP, not per user — so the gate is process-wide, not per-withUser, not per-video.
|
|
//
|
|
// Default 2s/req (burst 1): the first fetch passes immediately, subsequent fetches
|
|
// are spaced at least 2s apart. Production overrides via SetFetchRate from config.
|
|
var globalFetchGate = rate.NewLimiter(rate.Every(2*time.Second), 1)
|
|
|
|
// SetFetchRate replaces the process-wide gate's rate with one token per interval.
|
|
// Call once at startup from config (TAPIR_FETCH_RATE). A non-positive interval
|
|
// installs an unlimited gate (rate.Inf) — used in dev/tests so nothing throttles.
|
|
func SetFetchRate(interval time.Duration) {
|
|
if interval <= 0 {
|
|
globalFetchGate = rate.NewLimiter(rate.Inf, 1)
|
|
return
|
|
}
|
|
globalFetchGate = rate.NewLimiter(rate.Every(interval), 1)
|
|
}
|
|
|
|
// WaitFetchGate blocks until the process-wide gate allows one timedtext fetch,
|
|
// respecting ctx cancellation. Called from httpDo before every live outbound
|
|
// caption fetch so the scheduler and the click-path share the same egress budget.
|
|
func WaitFetchGate(ctx context.Context) error {
|
|
return globalFetchGate.Wait(ctx)
|
|
}
|