diff --git a/docs/specs/scheduled-discovery.md b/docs/specs/scheduled-discovery.md new file mode 100644 index 0000000..000c2eb --- /dev/null +++ b/docs/specs/scheduled-discovery.md @@ -0,0 +1,87 @@ +# Spec — In-process scheduled discovery + auto-summarize + rate-gate finish + +**Repo:** tapir · **Size:** medium · **Solo session** (not a swarm). + +**Why this exists.** The Stage-0 gate ("me or a friend returns and reads/acts in ≥2 separate +weeks") cannot be met because the system is not usable *unprompted*: discovery (`tapir run`) is +host-side manual, so a newly onboarded user sees an empty list and never comes back. This slice +makes Tapir watch on its own — the thing that makes the gate experiment actually runnable. + +Read `CLAUDE.md` + `DECISIONS.md` (esp. ADR-012, ADR-014, and the new ADR-018) first. TBD — +commit directly to `main`, one logical change per commit, conventional commits, `task check` +green before each commit. `templ generate` if any view changes. + +## Decisions already made (do not reopen) +- **In-process scheduler**, NOT a k8s CronJob (maintainer's call: simpler deploy, acceptable + coupling at 3 users). The known cost — discovery shares the web process's lifetime and egress + — is accepted and recorded in ADR-018. +- **Auto-summarize ON** for the maintainer + onboarded friends (zero-friction: the list fills + and summarizes itself). +- **Gate clock resets** to when this ships (ADR-018) — until unprompted use is possible, the + prior window measured nothing. + +## 1. In-process scheduled discovery (core) +- In `tapir serve` startup, launch a background goroutine that runs discovery for ALL users on + an interval: env `TAPIR_DISCOVERY_INTERVAL` (Go duration, e.g. `2h`). **Unset or 0 = disabled** + (so dev/tests never auto-fetch). +- **Reuse the existing `runner.Runner` + `Loop`/`RunOnce`. Do NOT write a new scheduler.** The + per-user Runner already exists; the new work is **iterating users** and running one pass each + per tick. Enumerate users from the un-RLS'd `user_identities` (the same enumerate-then-act + pattern the login_events gate query established), then run each user's pass **inside that + user's RLS scope** (`withUser`). +- **Stateless timing:** run-once-on-startup, then every interval — exactly the existing `Loop` + shape. Do NOT persist schedule state; a pod restart just restarts the cycle. Acceptable at this + scale. Do not build cron-in-Go. +- **Graceful shutdown:** the goroutine respects `ctx` cancellation so a pod term doesn't wedge. +- **Failure isolation in the loop:** one user's pass failing (or one channel/video) must not + abort the other users or crash `serve` — log and continue. (RunOnce already collects per-item + errors; preserve that at the per-user level too.) + +## 2. Auto-summarize default ON for Future-B users +- New registrations default `auto_summarize = true` (so onboarded friends get zero-friction); + keep the account-page toggle so a user can switch to manual. One-off update existing user rows + to `true` as well (maintainer + any current users). +- Consequence (intended): scheduled discovery both discovers AND summarizes new videos — which + is the point, and is why §3 is mandatory in the same slice. + +## 3. Finish/confirm the ADR-014 shared per-egress-IP rate gate (NOW load-bearing) +- In-process scheduling + auto-summarize + multiple users = all caption fetches leave the **one + web pod's egress IP**, concurrently with any live "Summarize" button clicks. The timedtext + endpoint rate-limits per IP (ADR-010/014). Without a shared gate this self-inflicts 429s every + cycle. +- **Confirm in code whether ADR-014 item 2 (a single PROCESS-WIDE rate gate) exists.** + Reconciliation flagged it as possibly built only as per-*video* backoff. If it is not a + process-wide gate, **build it now**: ONE shared limiter (token-bucket / min-interval) that + every timedtext/caption fetch passes through — scheduler loop AND click-path alike. Per-process, + not per-user, not per-video. +- Keep the existing per-video 429 backoff (`rate_limited_at` + retry window) — complementary: the + gate prevents tripping 429; the backoff handles it if one still happens. +- Honest UX (ADR-014 item 3) still applies: a fetch waiting on the gate shows "queued/waiting", + never a stuck spinner. + +## 4. Gate-clock reset — already recorded in ADR-018; verify VISION reflects it +- ADR-018 (committed) resets the Stage-0 3–4 week window to start when this ships, and revises the + check-in date. VISION Stage 0 carries a pointer to it. The build doesn't re-decide this; just + ensure nothing in docs still implies the clock started earlier. + +## Tests +- **Scheduler:** fake clock + fake Runner → N users each get one pass per tick; one user's failure + doesn't stop the others; `ctx` cancel stops the loop; interval=0 disables it entirely. +- **Rate gate:** concurrent fetches (scheduler + simulated click) are serialized/limited through + the ONE gate — assert max-in-flight / min-interval honored regardless of caller. +- **Auto-summarize default:** new registration → `auto_summarize = true`; account-page toggle + still flips it. + +## Out of scope / known constraints +- No CronJob / k8s objects (in-process chosen). +- **SINGLE-REPLICA ASSUMPTION (load-bearing).** In-process scheduling means if `tapir serve` ever + runs >1 replica, every replica runs the discovery loop → every user fetched in parallel (429s + + duplicate work). At 3 users this is single-replica, fine — but the build MUST note this + constraint in ADR-018 / deploy docs so a future scale-up doesn't silently double-run. +- No persisted schedules, no multi-pod coordination. + +## Fallback if the session runs short +Ship the scheduler with **auto-summarize OFF** (discovery only; manual Summarize button) until the +process-wide rate gate (§3) is confirmed/built. NEVER ship auto-summarize-on-a-schedule without the +gate — that combination self-inflicts 429s for every user every cycle. Auto-summarize ON is gated +on §3 being done.