tapir/docs/specs/scheduled-discovery.md

Spec — In-process scheduled discovery + auto-summarize + rate-gate finish

Repo: tapir · Size: medium · Solo session (not a swarm).

Why this exists. The Stage-0 gate ("me or a friend returns and reads/acts in ≥2 separate weeks") cannot be met because the system is not usable unprompted: discovery (tapir run) is host-side manual, so a newly onboarded user sees an empty list and never comes back. This slice makes Tapir watch on its own — the thing that makes the gate experiment actually runnable.

Read CLAUDE.md + DECISIONS.md (esp. ADR-012, ADR-014, and the new ADR-018) first. TBD — commit directly to main, one logical change per commit, conventional commits, task check green before each commit. templ generate if any view changes.

Decisions already made (do not reopen)

• In-process scheduler, NOT a k8s CronJob (maintainer's call: simpler deploy, acceptable coupling at 3 users). The known cost — discovery shares the web process's lifetime and egress — is accepted and recorded in ADR-018.
• Auto-summarize ON for the maintainer + onboarded friends (zero-friction: the list fills and summarizes itself).
• Gate clock resets to when this ships (ADR-018) — until unprompted use is possible, the prior window measured nothing.

1. In-process scheduled discovery (core)

• In tapir serve startup, launch a background goroutine that runs discovery for ALL users on an interval: env TAPIR_DISCOVERY_INTERVAL (Go duration, e.g. 2h). Unset or 0 = disabled (so dev/tests never auto-fetch).
• Reuse the existing runner.Runner + Loop/RunOnce. Do NOT write a new scheduler. The per-user Runner already exists; the new work is iterating users and running one pass each per tick. Enumerate users from the un-RLS'd user_identities (the same enumerate-then-act pattern the login_events gate query established), then run each user's pass inside that user's RLS scope (withUser).
• Stateless timing: run-once-on-startup, then every interval — exactly the existing Loop shape. Do NOT persist schedule state; a pod restart just restarts the cycle. Acceptable at this scale. Do not build cron-in-Go.
• Graceful shutdown: the goroutine respects ctx cancellation so a pod term doesn't wedge.
• Failure isolation in the loop: one user's pass failing (or one channel/video) must not abort the other users or crash serve — log and continue. (RunOnce already collects per-item errors; preserve that at the per-user level too.)

2. Auto-summarize default ON for Future-B users

• New registrations default auto_summarize = true (so onboarded friends get zero-friction); keep the account-page toggle so a user can switch to manual. One-off update existing user rows to true as well (maintainer + any current users).
• Consequence (intended): scheduled discovery both discovers AND summarizes new videos — which is the point, and is why §3 is mandatory in the same slice.

3. Finish/confirm the ADR-014 shared per-egress-IP rate gate (NOW load-bearing)

• In-process scheduling + auto-summarize + multiple users = all caption fetches leave the one web pod's egress IP, concurrently with any live "Summarize" button clicks. The timedtext endpoint rate-limits per IP (ADR-010/014). Without a shared gate this self-inflicts 429s every cycle.
• Confirm in code whether ADR-014 item 2 (a single PROCESS-WIDE rate gate) exists. Reconciliation flagged it as possibly built only as per-video backoff. If it is not a process-wide gate, build it now: ONE shared limiter (token-bucket / min-interval) that every timedtext/caption fetch passes through — scheduler loop AND click-path alike. Per-process, not per-user, not per-video.
• Keep the existing per-video 429 backoff (rate_limited_at + retry window) — complementary: the gate prevents tripping 429; the backoff handles it if one still happens.
• Honest UX (ADR-014 item 3) still applies: a fetch waiting on the gate shows "queued/waiting", never a stuck spinner.

4. Gate-clock reset — already recorded in ADR-018; verify VISION reflects it

• ADR-018 (committed) resets the Stage-0 3–4 week window to start when this ships, and revises the check-in date. VISION Stage 0 carries a pointer to it. The build doesn't re-decide this; just ensure nothing in docs still implies the clock started earlier.

Tests

• Scheduler: fake clock + fake Runner → N users each get one pass per tick; one user's failure doesn't stop the others; ctx cancel stops the loop; interval=0 disables it entirely.
• Rate gate: concurrent fetches (scheduler + simulated click) are serialized/limited through the ONE gate — assert max-in-flight / min-interval honored regardless of caller.
• Auto-summarize default: new registration → auto_summarize = true; account-page toggle still flips it.

Out of scope / known constraints

• No CronJob / k8s objects (in-process chosen).
• SINGLE-REPLICA ASSUMPTION (load-bearing). In-process scheduling means if tapir serve ever runs >1 replica, every replica runs the discovery loop → every user fetched in parallel (429s + duplicate work). At 3 users this is single-replica, fine — but the build MUST note this constraint in ADR-018 / deploy docs so a future scale-up doesn't silently double-run.
• No persisted schedules, no multi-pod coordination.

Fallback if the session runs short

Ship the scheduler with auto-summarize OFF (discovery only; manual Summarize button) until the process-wide rate gate (§3) is confirmed/built. NEVER ship auto-summarize-on-a-schedule without the gate — that combination self-inflicts 429s for every user every cycle. Auto-summarize ON is gated on §3 being done.