mathias/tapir
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(youtube): process-wide caption-fetch rate gate (ADR-014 item 2)

Browse Source 
ADR-014 item 2 — a single per-egress-IP rate gate shared by every caption
fetch — was specced but only per-video backoff (rate_limited_at) shipped.
Build the real gate now: it is load-bearing once ADR-018 puts auto-summarize
on an in-process schedule across multiple users (all fetches leave one pod's
egress IP, concurrently with live "Summarize" clicks — without a shared gate
that self-inflicts 429s every cycle).

globalFetchGate (golang.org/x/time/rate, default 2s/req burst 1) is consulted
in httpDo before every live outbound fetch — player, watch-page, timedtext —
so the scheduler runners and the web click-path serialise through one limiter
regardless of how many users/goroutines are upstream. The test seam
(a.transport != nil) skips the gate so fakes are not throttled.

TAPIR_FETCH_RATE (Go duration, default 2s, 0 = unlimited) wires SetFetchRate in
cmdRun; the existing per-video backoff stays as the complementary 429 handler.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
mathias
2026-06-05 23:36:17 +02:00
co-authored by Claude Opus 4.8
parent 88294d38bc
commit f5021a8436
8 changed files with 176 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cmd/tapir/main.go
+7 -1
View File
@@ -25,6 +25,7 @@ import (
"gitea.d-ma.be/mathias/tapir/internal/adapters/dex"
"gitea.d-ma.be/mathias/tapir/internal/adapters/secrets"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store"
"gitea.d-ma.be/mathias/tapir/internal/adapters/youtube"
"gitea.d-ma.be/mathias/tapir/internal/auth"
"gitea.d-ma.be/mathias/tapir/internal/config"
"gitea.d-ma.be/mathias/tapir/internal/runner"
@@ -132,10 +133,15 @@ func cmdRun(ctx context.Context, log *slog.Logger) error {
if engine == nil {
return fmt.Errorf("run: incomplete summarization config (gateway, youtube credentials, secrets file)")
}
// Process-wide caption-fetch rate gate (ADR-014 item 2): the batch path shares
// the same per-egress-IP limiter as the web click-path.
youtube.SetFetchRate(cfg.FetchRate)
r := runner.New(engine.Source, st, engine, cfg.UserID, log, runner.WithBackoff(cfg.FetchBackoff))
log.Info("starting run", "user", cfg.UserID, "model", cfg.SummarizerModel,
"gateway", cfg.GatewayURL, "poll_interval", cfg.PollInterval, "fetch_backoff", cfg.FetchBackoff)
"gateway", cfg.GatewayURL, "poll_interval", cfg.PollInterval, "fetch_backoff", cfg.FetchBackoff,
"fetch_rate", cfg.FetchRate)
return r.Loop(ctx, cfg.PollInterval)
}