tapir/docs/specs/scheduled-discovery.md

# Spec — In-process scheduled discovery + auto-summarize + rate-gate finish

**Repo:** tapir · **Size:** medium · **Solo session** (not a swarm).

**Why this exists.** The Stage-0 gate ("me or a friend returns and reads/acts in ≥2 separate
weeks") cannot be met because the system is not usable *unprompted*: discovery (`tapir run`) is
host-side manual, so a newly onboarded user sees an empty list and never comes back. This slice
makes Tapir watch on its own — the thing that makes the gate experiment actually runnable.

Read `CLAUDE.md` + `DECISIONS.md` (esp. ADR-012, ADR-014, and the new ADR-018) first. TBD —
commit directly to `main`, one logical change per commit, conventional commits, `task check`
green before each commit. `templ generate` if any view changes.

## Decisions already made (do not reopen)
- **In-process scheduler**, NOT a k8s CronJob (maintainer's call: simpler deploy, acceptable
  coupling at 3 users). The known cost — discovery shares the web process's lifetime and egress
  — is accepted and recorded in ADR-018.
- **Auto-summarize ON** for the maintainer + onboarded friends (zero-friction: the list fills
  and summarizes itself).
- **Gate clock resets** to when this ships (ADR-018) — until unprompted use is possible, the
  prior window measured nothing.

## 1. In-process scheduled discovery (core)
- In `tapir serve` startup, launch a background goroutine that runs discovery for ALL users on
  an interval: env `TAPIR_DISCOVERY_INTERVAL` (Go duration, e.g. `2h`). **Unset or 0 = disabled**
  (so dev/tests never auto-fetch).
- **Reuse the existing `runner.Runner` + `Loop`/`RunOnce`. Do NOT write a new scheduler.** The
  per-user Runner already exists; the new work is **iterating users** and running one pass each
  per tick. Enumerate users from the un-RLS'd `user_identities` (the same enumerate-then-act
  pattern the login_events gate query established), then run each user's pass **inside that
  user's RLS scope** (`withUser`).
- **Stateless timing:** run-once-on-startup, then every interval — exactly the existing `Loop`
  shape. Do NOT persist schedule state; a pod restart just restarts the cycle. Acceptable at this
  scale. Do not build cron-in-Go.
- **Graceful shutdown:** the goroutine respects `ctx` cancellation so a pod term doesn't wedge.
- **Failure isolation in the loop:** one user's pass failing (or one channel/video) must not
  abort the other users or crash `serve` — log and continue. (RunOnce already collects per-item
  errors; preserve that at the per-user level too.)

## 2. Auto-summarize default ON for Future-B users
- New registrations default `auto_summarize = true` (so onboarded friends get zero-friction);
  keep the account-page toggle so a user can switch to manual. One-off update existing user rows
  to `true` as well (maintainer + any current users).
- Consequence (intended): scheduled discovery both discovers AND summarizes new videos — which
  is the point, and is why §3 is mandatory in the same slice.

## 3. Finish/confirm the ADR-014 shared per-egress-IP rate gate (NOW load-bearing)
- In-process scheduling + auto-summarize + multiple users = all caption fetches leave the **one
  web pod's egress IP**, concurrently with any live "Summarize" button clicks. The timedtext
  endpoint rate-limits per IP (ADR-010/014). Without a shared gate this self-inflicts 429s every
  cycle.
- **Confirm in code whether ADR-014 item 2 (a single PROCESS-WIDE rate gate) exists.**
  Reconciliation flagged it as possibly built only as per-*video* backoff. If it is not a
  process-wide gate, **build it now**: ONE shared limiter (token-bucket / min-interval) that
  every timedtext/caption fetch passes through — scheduler loop AND click-path alike. Per-process,
  not per-user, not per-video.
- Keep the existing per-video 429 backoff (`rate_limited_at` + retry window) — complementary: the
  gate prevents tripping 429; the backoff handles it if one still happens.
- Honest UX (ADR-014 item 3) still applies: a fetch waiting on the gate shows "queued/waiting",
  never a stuck spinner.

## 4. Gate-clock reset — already recorded in ADR-018; verify VISION reflects it
- ADR-018 (committed) resets the Stage-0 3–4 week window to start when this ships, and revises the
  check-in date. VISION Stage 0 carries a pointer to it. The build doesn't re-decide this; just
  ensure nothing in docs still implies the clock started earlier.

## Tests
- **Scheduler:** fake clock + fake Runner → N users each get one pass per tick; one user's failure
  doesn't stop the others; `ctx` cancel stops the loop; interval=0 disables it entirely.
- **Rate gate:** concurrent fetches (scheduler + simulated click) are serialized/limited through
  the ONE gate — assert max-in-flight / min-interval honored regardless of caller.
- **Auto-summarize default:** new registration → `auto_summarize = true`; account-page toggle
  still flips it.

## Out of scope / known constraints
- No CronJob / k8s objects (in-process chosen).
- **SINGLE-REPLICA ASSUMPTION (load-bearing).** In-process scheduling means if `tapir serve` ever
  runs >1 replica, every replica runs the discovery loop → every user fetched in parallel (429s +
  duplicate work). At 3 users this is single-replica, fine — but the build MUST note this
  constraint in ADR-018 / deploy docs so a future scale-up doesn't silently double-run.
- No persisted schedules, no multi-pod coordination.

## Fallback if the session runs short
Ship the scheduler with **auto-summarize OFF** (discovery only; manual Summarize button) until the
process-wide rate gate (§3) is confirmed/built. NEVER ship auto-summarize-on-a-schedule without the
gate — that combination self-inflicts 429s for every user every cycle. Auto-summarize ON is gated
on §3 being done.