diff --git a/README.md b/README.md index 7c24434..017bebb 100644 --- a/README.md +++ b/README.md @@ -75,8 +75,10 @@ password or Google); a new Dex subject is routed to `/register` to create a Tapi Stage 1 is multi-user: each user connects their own YouTube account from the browser and manages their own summaries under DB-enforced RLS isolation. When `TAPIR_DISCOVERY_INTERVAL` is set (e.g. `2h`), the serve process runs a scheduled discovery pass for every registered -user automatically — no CronJob required. See `docs/homelab-integration.md` for the full -config reference. +user automatically — no CronJob required. In auto mode only videos published within +`TAPIR_AUTO_SUMMARIZE_WINDOW` (default ~7d, ADR-020) are summarised automatically; older videos +are listed and summarised on demand, so a large back-catalogue doesn't keep re-driving the +caption rate gate. See `docs/homelab-integration.md` for the full config reference. ### Headless on koala diff --git a/docs/architecture/architecture.md b/docs/architecture/architecture.md index 774cda0..4f4c9bd 100644 --- a/docs/architecture/architecture.md +++ b/docs/architecture/architecture.md @@ -147,9 +147,17 @@ graph TB rate-limiting — ADR-014). - **Summarization mode** — `users.auto_summarize` (migration 006). Default is **true** for new users (migration 011, ADR-018); all existing rows were back-filled via migration 012. Auto: - every new video is summarized. Manual: new videos appear unsummarized; the button sets - `videos.summarize_requested`, which the next `tapir run` processes and clears. Both the click - path and the batch `tapir run` drive the same unchanged engine. + new videos **published within the recency window** (`TAPIR_AUTO_SUMMARIZE_WINDOW`, default ~7d, + ADR-020) are summarized automatically; older videos are discovered and listed but wait for an + explicit "Summarize". Manual: new videos appear unsummarized; the button sets + `videos.summarize_requested`, which the next `tapir run` processes and clears. A manual request + bypasses the recency bound. Both the click path and the batch `tapir run` drive the same + unchanged engine. +- **List surface (ADR-020)** — the list reads `ListVideos` ordered summarized-first, then + `published_at DESC NULLS LAST`. The web layer collapses the noise so summaries are not buried: + un-summarized videos older than the recency window fold into one "Show N older videos" + disclosure, and caption-less videos collapse to a single count line. Copy surfaces scarcity + honestly (queue counts, gradual-fill note) — it never implies the feed is fuller than it is. The engine, ports, and sink adapters are **untouched** by all of the above — the web surface only reads the store and triggers the existing engine. Adding it changed wiring, not the core (ADR-003). @@ -174,6 +182,7 @@ sequenceDiagram loop per user S->>DB: GetAutoSummarize(userID) S->>YT: ListSubscriptions + NewVideos + Note over S,DB: auto: skip videos published before
TAPIR_AUTO_SUMMARIZE_WINDOW (ADR-020);
older ones listed, await manual request Note over S,YT: WaitFetchGate(ctx) throttles
all fetches to TAPIR_FETCH_RATE alt transcript available S->>LLM: Summarize @@ -214,19 +223,22 @@ Both paths share `globalFetchGate` — rate limiting is **respected in both**, n | Path | Trigger | Order | Rationale | |------|---------|-------|-----------| -| **Foreground** | User clicks "Summarize now" on any non-summarized card (`POST /v/{id}/retry-now` for rate-limited; `POST /v/{id}/summarize` for pending) | Single chosen video | On-demand value: user picks a specific video to read now | -| **Background batch** | Scheduled discovery pass every `TAPIR_DISCOVERY_INTERVAL` | **Newest-first across all channels** (see below) | Onboarding prioritisation: most recent, relevant videos surface first | +| **Foreground** | User clicks "Summarize" on any non-summarized card (`POST /v/{id}/retry-now` for rate-limited; `POST /v/{id}/summarize` for pending) | Single chosen video | On-demand value: user picks a specific video to read now — bypasses the recency bound | +| **Background batch** | Scheduled discovery pass every `TAPIR_DISCOVERY_INTERVAL` | **Newest-first across all channels** (see below), **bounded to the recency window** (ADR-020) | Onboarding prioritisation within bounded load: recent videos auto-fill; the older back-catalogue stays on-demand | -The rationale for both paths is **onboarding prioritisation** — a new user should get summaries -of their most recent, relevant videos quickly while the older back-catalogue fills in behind, -all within the honest shared rate limit. +The rationale for both paths is **onboarding prioritisation under an honest, bounded load** — a +new user gets summaries of their most recent videos automatically, while the older back-catalogue +is listed but summarised only on demand, so it never re-drives the shared rate gate every cycle. ### Newest-first batch ordering (ADR-018) Within each scheduled pass, `RunOnce` uses a three-phase structure: 1. **Discover + persist**: walk all channels, `UpsertVideo` every candidate (so it appears in - the list), apply pre-filters (seen/manual/backoff), collect surviving candidates. + the list), apply pre-filters (seen/manual/backoff/**recency**), collect surviving candidates. + The recency pre-filter (ADR-020) drops auto-mode videos published before + `now - TAPIR_AUTO_SUMMARIZE_WINDOW` unless they are explicitly requested; an undated video is + never aged out. They remain persisted/listed — only auto-summarisation is skipped. 2. **Sort**: order candidates `published_at DESC, NULLS LAST, discovery_pos ASC`. Videos with no publish date (schema 001: nullable) sort after all dated content. The sort is in-memory (`slices.SortStableFunc`) — at current scale this is fine. @@ -235,7 +247,8 @@ Within each scheduled pass, `RunOnce` uses a three-phase structure: Before (per-channel inline): `[chanA-old, chanA-mid, chanB-new, chanB-null]` After (newest-first): `[chanB-new, chanA-mid, chanA-old, chanB-null]` -The set of processed videos is identical; only the order within a pass changes. +The set of *processed* videos now also excludes auto-mode back-catalogue beyond the recency +window (those stay listed, summarised on demand); within the processed set, only order changes. --- diff --git a/docs/data-model.md b/docs/data-model.md index 261b61e..a0c21f8 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -147,9 +147,10 @@ mechanism. - **USER** — one row per registered user (Stage 1, ADR-012; no longer single-row). The Tapir-side profile; the Dex identity is held separately in `USER_IDENTITY`, not on this row. `auto_summarize` - (migration 006) is the per-user mode flag: `TRUE` = auto-summarize every new video. Default is - **true** for new users (migration 011, ADR-018); existing rows were back-filled via migration 012 - with RLS bypass. + (migration 006) is the per-user mode flag: `TRUE` = auto-summarize new videos **published within + the recency window** (`TAPIR_AUTO_SUMMARIZE_WINDOW`, default ~7d, ADR-020); older videos are + listed but summarised on demand. Default is **true** for new users (migration 011, ADR-018); + existing rows were back-filled via migration 012 with RLS bypass. - **USER_IDENTITY** (migration 004) — the `dex_subject → user_id` map. `dex_subject` is the PK, `user_id` a `UNIQUE` FK to `users` with `ON DELETE CASCADE`. This is the bridge resolved at login *before* a `user_id` is known, so it is **deliberately not RLS-enabled** (it holds no user data; diff --git a/docs/specs/newest-first-ordering.md b/docs/specs/newest-first-ordering.md index 2f028a4..ce98ff1 100644 --- a/docs/specs/newest-first-ordering.md +++ b/docs/specs/newest-first-ordering.md @@ -1,5 +1,11 @@ # Spec — Newest-first batch ordering + honest "Try now" / prioritisation docs +> **Extended by ADR-020 (2026-06-08).** This spec covers the *batch processing* order within a +> pass. ADR-020 adds (a) a recency pre-filter — auto mode skips videos published before +> `TAPIR_AUTO_SUMMARIZE_WINDOW`, listed but summarised on demand — and (b) the same +> `published_at DESC NULLS LAST` ordering on the **list read** (`ListVideos`), which previously +> sorted by `seen_at`. See `DECISIONS.md` ADR-020. + **Repo:** tapir · **Size:** small · **Solo session.** **Why.** Product intent (maintainer, 2026-06-06): a new user should get summaries of their diff --git a/docs/specs/scheduled-discovery.md b/docs/specs/scheduled-discovery.md index 000c2eb..86ea2a7 100644 --- a/docs/specs/scheduled-discovery.md +++ b/docs/specs/scheduled-discovery.md @@ -1,5 +1,10 @@ # Spec — In-process scheduled discovery + auto-summarize + rate-gate finish +> **Extended by ADR-020 (2026-06-08).** Auto-summarize is no longer "every unseen video": the +> scheduler now skips videos published before `TAPIR_AUTO_SUMMARIZE_WINDOW` (default ~7d) unless +> explicitly requested, so a back-catalogue does not re-drive the rate gate every cycle. See +> `DECISIONS.md` ADR-020. + **Repo:** tapir · **Size:** medium · **Solo session** (not a swarm). **Why this exists.** The Stage-0 gate ("me or a friend returns and reads/acts in ≥2 separate diff --git a/docs/specs/video-card-states.md b/docs/specs/video-card-states.md index d26c587..84eb62c 100644 --- a/docs/specs/video-card-states.md +++ b/docs/specs/video-card-states.md @@ -1,5 +1,12 @@ # Spec — Unify video-card states: one "Summarize now" verb, honest no-captions state +> **Superseded in part by ADR-020 (2026-06-08).** The five card states still hold, but the copy +> changed: the nudge verb is now **"Summarize"** (not "Summarize now"), the rate-limited state +> reads **"In queue"** (not "Fetching soon…"), and the queued state reads **"summarizing +> shortly"** (not "waiting for the next run"). The list also now collapses older un-summarized +> and caption-less videos. See `DECISIONS.md` ADR-020 and `views.templ` (`VideoCard`) for the +> current copy; this doc is kept as the original design record. + **Repo:** tapir · **Size:** small, **view-layer only** (`views.templ` + a little CSS in `view.go`; regenerate `views_templ.go`). No handler, store, or DB change. The two existing handlers (`/summarize`, `/retry-now`) stay exactly as they are — only what the card *shows* diff --git a/docs/ui-spec.md b/docs/ui-spec.md index 5a50969..96cda75 100644 --- a/docs/ui-spec.md +++ b/docs/ui-spec.md @@ -179,3 +179,4 @@ distinguishable. | **"Summarize now" foreground path** | Unified quiet nudge button on actionable non-summarized cards. Five explicit card states — (1) summarized: chip + no button; (2) no captions (`transcript_status = 'none'`): "No transcript available", no button; (3) queued: "Queued" chip, no button; (4) rate-limited: "Fetching soon…" + "Summarize now" → `POST /v/{id}/retry-now` (clears `rate_limited_at`, triggers engine); (5) pending: "Not summarized" + "Summarize now" → `POST /v/{id}/summarize` (queues + triggers engine). One verb, one style (`.btn-quiet`); backend difference invisible to user. Both handlers call `ProcessVideo` through `globalFetchGate`. Rate gate respected, not bypassed — this is onboarding prioritisation. | Fast onboarding value; honest dead-end for no-captions videos (no button that fails). | `internal/web/handlers.go` (`handleRetryNow`, `handleRequestSummarize`); `internal/web/views.templ` (`VideoCard`) | | **Pipeline stats bar** | A one-line status bar above the video list: `N summarized · M fetching soon · K no captions`. Computed from the unfiltered row set; hidden when all videos are summarized. Gives the user a clear read on pipeline state without any interaction. | Replaces the "why is nothing happening?" confusion when most videos are pending or rate-limited. | `internal/web/view.go` (`PipelineStats`, `pipelineStats`) | | **Unavailable channels (account page)** | The `/account` page shows a "Unavailable channels" section when any channels returned HTTP 404 on the last discovery pass. Lists channel name, an "unavailable" badge, and the first-seen date. Data sourced from the `channel_errors` table (migration 013). | Surfaces silent failures so users know why some subscribed channels produce no new videos. | migration 013; `internal/web/account.go`; `internal/adapters/youtube/youtube.go` (`domain.ErrChannelUnavailable`) | +| **Recency window + sparse-state honesty (ADR-020)** | Supersedes the copy/sort in the rows above. Auto-summarize is bounded to videos published within `TAPIR_AUTO_SUMMARIZE_WINDOW` (~7d); older un-summarized videos collapse behind a single "Show N older videos — summarize on demand" disclosure, and caption-less videos collapse to a one-line count (not N cards). List order is now `summarized-first, published_at DESC NULLS LAST`. Copy reframed for honest scarcity: pipeline bar reads "N ready · M in queue · K no captions" (no "fetching soon"); a gradual-fill note explains the rate limit; the nudge verb is "Summarize" (not "Summarize now"); the queued card says "summarizing shortly"; the empty-connected state drops the impossible `tapir run` instruction. Detail leads with Takeaways. Filters slimmed (no date pickers; hidden when empty); watched/skipped segmented; back link on detail; empty terms checkbox removed. | Make the sparse reality legible and honest instead of implying abundance/imminence; bound auto load so the back-catalogue doesn't re-drive the caption gate. Never fetch harder — scarcity is surfaced, not engineered around. | ADR-020; `2384c47`, `3df0459`, `40b703e`, `a1a5217`, `4a0a56e`, `9bf1c31`, `980638d`, `12fb031`, `f775441`, `51aa5d9` | diff --git a/docs/use-cases/summarize_mode.feature b/docs/use-cases/summarize_mode.feature index 0341506..535dd6a 100644 --- a/docs/use-cases/summarize_mode.feature +++ b/docs/use-cases/summarize_mode.feature @@ -6,12 +6,19 @@ Feature: Choose how new videos get summarized Background: Given I am a registered user with a connected video account - Scenario: Auto mode summarizes every new video + Scenario: Auto mode summarizes recent new videos automatically Given my summarization mode is "auto" - When a subscribed channel posts a new video with captions + When a subscribed channel posts a new video with captions within the recency window Then Tapir summarizes it without my asking And the summary appears in my list + Scenario: Auto mode lists older videos without summarizing them + Given my summarization mode is "auto" + When discovery finds a video published before the recency window + Then the video appears in my list with no summary + And it is not summarized automatically + And I can still summarize it on demand with "Summarize" + Scenario: Manual mode is the default and leaves new videos unsummarized Given I have not changed my summarization mode Then my mode is "manual" @@ -30,3 +37,8 @@ Feature: Choose how new videos get summarized # auto_summarize is a per-user setting and summarize_requested is a per-video queue # flag (migration 006). The web button sets the flag; `tapir run` processes both the # auto videos and the manually queued ones, then clears the flag. + # + # Recency bound (ADR-020): in auto mode the scheduler only summarizes videos published + # within TAPIR_AUTO_SUMMARIZE_WINDOW (default ~7d); older videos are discovered and + # listed but wait for an explicit "Summarize" — so a back-catalogue does not re-drive + # the per-IP caption gate (ADR-014) every cycle. A manual request bypasses the bound.