mathias/tapir
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(serve): in-process scheduled discovery for all users (ADR-018)

Browse Source 
Stage 0's "returns and reads in >=2 weeks" gate can't be met while discovery
is host-side manual (`tapir run`): a newly onboarded user sees an empty list
and never comes back. Make Tapir watch on its own.

cmdServe launches a background goroutine (when TAPIR_DISCOVERY_INTERVAL > 0)
that runs a discovery pass for ALL users on that cadence: enumerate via the
un-RLS'd ListAllUsers, then run each user's pass through the EXISTING
runner.Runner — the only new code is the per-user loop, not a new scheduler.
Run-once-on-startup then ticked; ctx-cancelled on SIGTERM; per-user failures
(including buildUserRunner errors) are logged and skipped so one bad user
never aborts the rest. interval <= 0 disables it entirely (dev/tests).

buildUserRunner binds each runner to that user's own YouTube refresh token
(web.YouTubeTokenRef) — the Stage-1 per-tenant ref — reusing buildProcessor's
engine wiring. SetFetchRate is also wired in cmdServe so the click-path shares
the gate.

SINGLE-REPLICA is now load-bearing: the loop lives in the web process, so >1
replica double-runs discovery (429s + duplicate work). Documented in cmdServe
and warned at startup.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
mathias
2026-06-05 23:39:39 +02:00
co-authored by Claude Opus 4.8
parent 149ec2adae
commit f6623afd41
3 changed files with 312 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cmd/tapir/main.go
+28
View File
@@ -165,6 +165,11 @@ func cmdServe(ctx context.Context, log *slog.Logger) error {
}
defer st.Close()
// Process-wide caption-fetch rate gate (ADR-014 item 2): the web click-path
// and the scheduled-discovery runners share one per-egress-IP limiter so they
// cannot collectively trip 429s. Must be set before either path fetches.
youtube.SetFetchRate(cfg.FetchRate)
// Auth seam (handlers depend on web.Auth only). With Dex configured
// (TAPIR_OIDC_ISSUER set) serve uses real OIDC login — any Dex subject may
// authenticate, then registers a tapir user (ADR-012); otherwise it falls
@@ -234,6 +239,29 @@ func cmdServe(ctx context.Context, log *slog.Logger) error {
log.Info("web summarization is queue-only (incomplete engine config)")
}
// In-process scheduled discovery (ADR-018): when enabled, a background
// goroutine runs a discovery pass for ALL users on TAPIR_DISCOVERY_INTERVAL,
// reusing the per-user runner.Runner. Cancelled by the same ctx as the server.
//
// SINGLE-REPLICA ASSUMPTION (load-bearing): this loop lives in the web process.
// Running serve at >1 replica would make every replica fetch every user in
// parallel — duplicate work and self-inflicted 429s. replicas: 1 is required in
// the deployment manifest; scaling up needs a CronJob or leader election first.
if cfg.DiscoveryInterval > 0 {
log.Info("scheduled discovery enabled", "interval", cfg.DiscoveryInterval, "fetch_rate", cfg.FetchRate)
log.Warn("scheduled discovery assumes a SINGLE replica — running serve at >1 replica double-runs discovery (ADR-018)")
runUser := func(ctx context.Context, userID string) (runner.Stats, error) {
r, err := buildUserRunner(cfg, st, secretStore, userID, log)
if err != nil {
return runner.Stats{}, err
}
return r.RunOnce(ctx)
}
go runScheduler(ctx, cfg.DiscoveryInterval, st, runUser, log)
} else {
log.Info("scheduled discovery disabled (TAPIR_DISCOVERY_INTERVAL unset or 0)")
}
srv := &http.Server{
Addr: cfg.HTTPAddr,
Handler: app.Router(),