mathiasandClaude Opus 4.8
27fd33c99c
Bring the living docs current with the recency-bounded auto-summarize + sparse
honesty + feed IA bundle (ADR-020):
- requirements (BDD): summarize_mode.feature — auto now summarizes RECENT new
videos; added a scenario for older videos (listed, on-demand), recency note.
- architecture.md: summarization-mode + new list-surface paragraph; scheduler
diagram + two-path table + three-phase pass now show the recency pre-filter;
dropped stale "Summarize now".
- data-model.md: auto_summarize is recent-only, older on-demand.
- README.md: one-line recency note on the serve scheduler.
- ui-spec.md: appended the as-built ADR-020 row (supersedes earlier copy/sort).
- specs/{video-card-states,newest-first-ordering,scheduled-discovery}.md:
superseded/extended banners pointing at ADR-020 (kept as design records).
Docs-only; task check green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
93 lines
6.0 KiB
Markdown
93 lines
6.0 KiB
Markdown
# Spec — In-process scheduled discovery + auto-summarize + rate-gate finish
|
||
|
||
> **Extended by ADR-020 (2026-06-08).** Auto-summarize is no longer "every unseen video": the
|
||
> scheduler now skips videos published before `TAPIR_AUTO_SUMMARIZE_WINDOW` (default ~7d) unless
|
||
> explicitly requested, so a back-catalogue does not re-drive the rate gate every cycle. See
|
||
> `DECISIONS.md` ADR-020.
|
||
|
||
**Repo:** tapir · **Size:** medium · **Solo session** (not a swarm).
|
||
|
||
**Why this exists.** The Stage-0 gate ("me or a friend returns and reads/acts in ≥2 separate
|
||
weeks") cannot be met because the system is not usable *unprompted*: discovery (`tapir run`) is
|
||
host-side manual, so a newly onboarded user sees an empty list and never comes back. This slice
|
||
makes Tapir watch on its own — the thing that makes the gate experiment actually runnable.
|
||
|
||
Read `CLAUDE.md` + `DECISIONS.md` (esp. ADR-012, ADR-014, and the new ADR-018) first. TBD —
|
||
commit directly to `main`, one logical change per commit, conventional commits, `task check`
|
||
green before each commit. `templ generate` if any view changes.
|
||
|
||
## Decisions already made (do not reopen)
|
||
- **In-process scheduler**, NOT a k8s CronJob (maintainer's call: simpler deploy, acceptable
|
||
coupling at 3 users). The known cost — discovery shares the web process's lifetime and egress
|
||
— is accepted and recorded in ADR-018.
|
||
- **Auto-summarize ON** for the maintainer + onboarded friends (zero-friction: the list fills
|
||
and summarizes itself).
|
||
- **Gate clock resets** to when this ships (ADR-018) — until unprompted use is possible, the
|
||
prior window measured nothing.
|
||
|
||
## 1. In-process scheduled discovery (core)
|
||
- In `tapir serve` startup, launch a background goroutine that runs discovery for ALL users on
|
||
an interval: env `TAPIR_DISCOVERY_INTERVAL` (Go duration, e.g. `2h`). **Unset or 0 = disabled**
|
||
(so dev/tests never auto-fetch).
|
||
- **Reuse the existing `runner.Runner` + `Loop`/`RunOnce`. Do NOT write a new scheduler.** The
|
||
per-user Runner already exists; the new work is **iterating users** and running one pass each
|
||
per tick. Enumerate users from the un-RLS'd `user_identities` (the same enumerate-then-act
|
||
pattern the login_events gate query established), then run each user's pass **inside that
|
||
user's RLS scope** (`withUser`).
|
||
- **Stateless timing:** run-once-on-startup, then every interval — exactly the existing `Loop`
|
||
shape. Do NOT persist schedule state; a pod restart just restarts the cycle. Acceptable at this
|
||
scale. Do not build cron-in-Go.
|
||
- **Graceful shutdown:** the goroutine respects `ctx` cancellation so a pod term doesn't wedge.
|
||
- **Failure isolation in the loop:** one user's pass failing (or one channel/video) must not
|
||
abort the other users or crash `serve` — log and continue. (RunOnce already collects per-item
|
||
errors; preserve that at the per-user level too.)
|
||
|
||
## 2. Auto-summarize default ON for Future-B users
|
||
- New registrations default `auto_summarize = true` (so onboarded friends get zero-friction);
|
||
keep the account-page toggle so a user can switch to manual. One-off update existing user rows
|
||
to `true` as well (maintainer + any current users).
|
||
- Consequence (intended): scheduled discovery both discovers AND summarizes new videos — which
|
||
is the point, and is why §3 is mandatory in the same slice.
|
||
|
||
## 3. Finish/confirm the ADR-014 shared per-egress-IP rate gate (NOW load-bearing)
|
||
- In-process scheduling + auto-summarize + multiple users = all caption fetches leave the **one
|
||
web pod's egress IP**, concurrently with any live "Summarize" button clicks. The timedtext
|
||
endpoint rate-limits per IP (ADR-010/014). Without a shared gate this self-inflicts 429s every
|
||
cycle.
|
||
- **Confirm in code whether ADR-014 item 2 (a single PROCESS-WIDE rate gate) exists.**
|
||
Reconciliation flagged it as possibly built only as per-*video* backoff. If it is not a
|
||
process-wide gate, **build it now**: ONE shared limiter (token-bucket / min-interval) that
|
||
every timedtext/caption fetch passes through — scheduler loop AND click-path alike. Per-process,
|
||
not per-user, not per-video.
|
||
- Keep the existing per-video 429 backoff (`rate_limited_at` + retry window) — complementary: the
|
||
gate prevents tripping 429; the backoff handles it if one still happens.
|
||
- Honest UX (ADR-014 item 3) still applies: a fetch waiting on the gate shows "queued/waiting",
|
||
never a stuck spinner.
|
||
|
||
## 4. Gate-clock reset — already recorded in ADR-018; verify VISION reflects it
|
||
- ADR-018 (committed) resets the Stage-0 3–4 week window to start when this ships, and revises the
|
||
check-in date. VISION Stage 0 carries a pointer to it. The build doesn't re-decide this; just
|
||
ensure nothing in docs still implies the clock started earlier.
|
||
|
||
## Tests
|
||
- **Scheduler:** fake clock + fake Runner → N users each get one pass per tick; one user's failure
|
||
doesn't stop the others; `ctx` cancel stops the loop; interval=0 disables it entirely.
|
||
- **Rate gate:** concurrent fetches (scheduler + simulated click) are serialized/limited through
|
||
the ONE gate — assert max-in-flight / min-interval honored regardless of caller.
|
||
- **Auto-summarize default:** new registration → `auto_summarize = true`; account-page toggle
|
||
still flips it.
|
||
|
||
## Out of scope / known constraints
|
||
- No CronJob / k8s objects (in-process chosen).
|
||
- **SINGLE-REPLICA ASSUMPTION (load-bearing).** In-process scheduling means if `tapir serve` ever
|
||
runs >1 replica, every replica runs the discovery loop → every user fetched in parallel (429s +
|
||
duplicate work). At 3 users this is single-replica, fine — but the build MUST note this
|
||
constraint in ADR-018 / deploy docs so a future scale-up doesn't silently double-run.
|
||
- No persisted schedules, no multi-pod coordination.
|
||
|
||
## Fallback if the session runs short
|
||
Ship the scheduler with **auto-summarize OFF** (discovery only; manual Summarize button) until the
|
||
process-wide rate gate (§3) is confirmed/built. NEVER ship auto-summarize-on-a-schedule without the
|
||
gate — that combination self-inflicts 429s for every user every cycle. Auto-summarize ON is gated
|
||
on §3 being done.
|