docs: reconcile requirements + architecture with ADR-020
CI / Lint / Test / Vet (push) Successful in 11s
CI / Build & Import (push) Successful in 10s

Bring the living docs current with the recency-bounded auto-summarize + sparse
honesty + feed IA bundle (ADR-020):

- requirements (BDD): summarize_mode.feature — auto now summarizes RECENT new
  videos; added a scenario for older videos (listed, on-demand), recency note.
- architecture.md: summarization-mode + new list-surface paragraph; scheduler
  diagram + two-path table + three-phase pass now show the recency pre-filter;
  dropped stale "Summarize now".
- data-model.md: auto_summarize is recent-only, older on-demand.
- README.md: one-line recency note on the serve scheduler.
- ui-spec.md: appended the as-built ADR-020 row (supersedes earlier copy/sort).
- specs/{video-card-states,newest-first-ordering,scheduled-discovery}.md:
  superseded/extended banners pointing at ADR-020 (kept as design records).

Docs-only; task check green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-08 20:21:14 +02:00
co-authored by Claude Opus 4.8
parent 2c96926ff7
commit 27fd33c99c
8 changed files with 64 additions and 17 deletions
+4 -2
View File
@@ -75,8 +75,10 @@ password or Google); a new Dex subject is routed to `/register` to create a Tapi
Stage 1 is multi-user: each user connects their own YouTube account from the browser and Stage 1 is multi-user: each user connects their own YouTube account from the browser and
manages their own summaries under DB-enforced RLS isolation. When `TAPIR_DISCOVERY_INTERVAL` manages their own summaries under DB-enforced RLS isolation. When `TAPIR_DISCOVERY_INTERVAL`
is set (e.g. `2h`), the serve process runs a scheduled discovery pass for every registered is set (e.g. `2h`), the serve process runs a scheduled discovery pass for every registered
user automatically — no CronJob required. See `docs/homelab-integration.md` for the full user automatically — no CronJob required. In auto mode only videos published within
config reference. `TAPIR_AUTO_SUMMARIZE_WINDOW` (default ~7d, ADR-020) are summarised automatically; older videos
are listed and summarised on demand, so a large back-catalogue doesn't keep re-driving the
caption rate gate. See `docs/homelab-integration.md` for the full config reference.
### Headless on koala ### Headless on koala
+23 -10
View File
@@ -147,9 +147,17 @@ graph TB
rate-limiting — ADR-014). rate-limiting — ADR-014).
- **Summarization mode** — `users.auto_summarize` (migration 006). Default is **true** for new - **Summarization mode** — `users.auto_summarize` (migration 006). Default is **true** for new
users (migration 011, ADR-018); all existing rows were back-filled via migration 012. Auto: users (migration 011, ADR-018); all existing rows were back-filled via migration 012. Auto:
every new video is summarized. Manual: new videos appear unsummarized; the button sets new videos **published within the recency window** (`TAPIR_AUTO_SUMMARIZE_WINDOW`, default ~7d,
`videos.summarize_requested`, which the next `tapir run` processes and clears. Both the click ADR-020) are summarized automatically; older videos are discovered and listed but wait for an
path and the batch `tapir run` drive the same unchanged engine. explicit "Summarize". Manual: new videos appear unsummarized; the button sets
`videos.summarize_requested`, which the next `tapir run` processes and clears. A manual request
bypasses the recency bound. Both the click path and the batch `tapir run` drive the same
unchanged engine.
- **List surface (ADR-020)** — the list reads `ListVideos` ordered summarized-first, then
`published_at DESC NULLS LAST`. The web layer collapses the noise so summaries are not buried:
un-summarized videos older than the recency window fold into one "Show N older videos"
disclosure, and caption-less videos collapse to a single count line. Copy surfaces scarcity
honestly (queue counts, gradual-fill note) — it never implies the feed is fuller than it is.
The engine, ports, and sink adapters are **untouched** by all of the above — the web surface only The engine, ports, and sink adapters are **untouched** by all of the above — the web surface only
reads the store and triggers the existing engine. Adding it changed wiring, not the core (ADR-003). reads the store and triggers the existing engine. Adding it changed wiring, not the core (ADR-003).
@@ -174,6 +182,7 @@ sequenceDiagram
loop per user loop per user
S->>DB: GetAutoSummarize(userID) S->>DB: GetAutoSummarize(userID)
S->>YT: ListSubscriptions + NewVideos S->>YT: ListSubscriptions + NewVideos
Note over S,DB: auto: skip videos published before<br/>TAPIR_AUTO_SUMMARIZE_WINDOW (ADR-020);<br/>older ones listed, await manual request
Note over S,YT: WaitFetchGate(ctx) throttles<br/>all fetches to TAPIR_FETCH_RATE Note over S,YT: WaitFetchGate(ctx) throttles<br/>all fetches to TAPIR_FETCH_RATE
alt transcript available alt transcript available
S->>LLM: Summarize S->>LLM: Summarize
@@ -214,19 +223,22 @@ Both paths share `globalFetchGate` — rate limiting is **respected in both**, n
| Path | Trigger | Order | Rationale | | Path | Trigger | Order | Rationale |
|------|---------|-------|-----------| |------|---------|-------|-----------|
| **Foreground** | User clicks "Summarize now" on any non-summarized card (`POST /v/{id}/retry-now` for rate-limited; `POST /v/{id}/summarize` for pending) | Single chosen video | On-demand value: user picks a specific video to read now | | **Foreground** | User clicks "Summarize" on any non-summarized card (`POST /v/{id}/retry-now` for rate-limited; `POST /v/{id}/summarize` for pending) | Single chosen video | On-demand value: user picks a specific video to read now — bypasses the recency bound |
| **Background batch** | Scheduled discovery pass every `TAPIR_DISCOVERY_INTERVAL` | **Newest-first across all channels** (see below) | Onboarding prioritisation: most recent, relevant videos surface first | | **Background batch** | Scheduled discovery pass every `TAPIR_DISCOVERY_INTERVAL` | **Newest-first across all channels** (see below), **bounded to the recency window** (ADR-020) | Onboarding prioritisation within bounded load: recent videos auto-fill; the older back-catalogue stays on-demand |
The rationale for both paths is **onboarding prioritisation** — a new user should get summaries The rationale for both paths is **onboarding prioritisation under an honest, bounded load** — a
of their most recent, relevant videos quickly while the older back-catalogue fills in behind, new user gets summaries of their most recent videos automatically, while the older back-catalogue
all within the honest shared rate limit. is listed but summarised only on demand, so it never re-drives the shared rate gate every cycle.
### Newest-first batch ordering (ADR-018) ### Newest-first batch ordering (ADR-018)
Within each scheduled pass, `RunOnce` uses a three-phase structure: Within each scheduled pass, `RunOnce` uses a three-phase structure:
1. **Discover + persist**: walk all channels, `UpsertVideo` every candidate (so it appears in 1. **Discover + persist**: walk all channels, `UpsertVideo` every candidate (so it appears in
the list), apply pre-filters (seen/manual/backoff), collect surviving candidates. the list), apply pre-filters (seen/manual/backoff/**recency**), collect surviving candidates.
The recency pre-filter (ADR-020) drops auto-mode videos published before
`now - TAPIR_AUTO_SUMMARIZE_WINDOW` unless they are explicitly requested; an undated video is
never aged out. They remain persisted/listed — only auto-summarisation is skipped.
2. **Sort**: order candidates `published_at DESC, NULLS LAST, discovery_pos ASC`. Videos with 2. **Sort**: order candidates `published_at DESC, NULLS LAST, discovery_pos ASC`. Videos with
no publish date (schema 001: nullable) sort after all dated content. The sort is in-memory no publish date (schema 001: nullable) sort after all dated content. The sort is in-memory
(`slices.SortStableFunc`) — at current scale this is fine. (`slices.SortStableFunc`) — at current scale this is fine.
@@ -235,7 +247,8 @@ Within each scheduled pass, `RunOnce` uses a three-phase structure:
Before (per-channel inline): `[chanA-old, chanA-mid, chanB-new, chanB-null]` Before (per-channel inline): `[chanA-old, chanA-mid, chanB-new, chanB-null]`
After (newest-first): `[chanB-new, chanA-mid, chanA-old, chanB-null]` After (newest-first): `[chanB-new, chanA-mid, chanA-old, chanB-null]`
The set of processed videos is identical; only the order within a pass changes. The set of *processed* videos now also excludes auto-mode back-catalogue beyond the recency
window (those stay listed, summarised on demand); within the processed set, only order changes.
--- ---
+4 -3
View File
@@ -147,9 +147,10 @@ mechanism.
- **USER** — one row per registered user (Stage 1, ADR-012; no longer single-row). The Tapir-side - **USER** — one row per registered user (Stage 1, ADR-012; no longer single-row). The Tapir-side
profile; the Dex identity is held separately in `USER_IDENTITY`, not on this row. `auto_summarize` profile; the Dex identity is held separately in `USER_IDENTITY`, not on this row. `auto_summarize`
(migration 006) is the per-user mode flag: `TRUE` = auto-summarize every new video. Default is (migration 006) is the per-user mode flag: `TRUE` = auto-summarize new videos **published within
**true** for new users (migration 011, ADR-018); existing rows were back-filled via migration 012 the recency window** (`TAPIR_AUTO_SUMMARIZE_WINDOW`, default ~7d, ADR-020); older videos are
with RLS bypass. listed but summarised on demand. Default is **true** for new users (migration 011, ADR-018);
existing rows were back-filled via migration 012 with RLS bypass.
- **USER_IDENTITY** (migration 004) — the `dex_subject → user_id` map. `dex_subject` is the PK, - **USER_IDENTITY** (migration 004) — the `dex_subject → user_id` map. `dex_subject` is the PK,
`user_id` a `UNIQUE` FK to `users` with `ON DELETE CASCADE`. This is the bridge resolved at login `user_id` a `UNIQUE` FK to `users` with `ON DELETE CASCADE`. This is the bridge resolved at login
*before* a `user_id` is known, so it is **deliberately not RLS-enabled** (it holds no user data; *before* a `user_id` is known, so it is **deliberately not RLS-enabled** (it holds no user data;
+6
View File
@@ -1,5 +1,11 @@
# Spec — Newest-first batch ordering + honest "Try now" / prioritisation docs # Spec — Newest-first batch ordering + honest "Try now" / prioritisation docs
> **Extended by ADR-020 (2026-06-08).** This spec covers the *batch processing* order within a
> pass. ADR-020 adds (a) a recency pre-filter — auto mode skips videos published before
> `TAPIR_AUTO_SUMMARIZE_WINDOW`, listed but summarised on demand — and (b) the same
> `published_at DESC NULLS LAST` ordering on the **list read** (`ListVideos`), which previously
> sorted by `seen_at`. See `DECISIONS.md` ADR-020.
**Repo:** tapir · **Size:** small · **Solo session.** **Repo:** tapir · **Size:** small · **Solo session.**
**Why.** Product intent (maintainer, 2026-06-06): a new user should get summaries of their **Why.** Product intent (maintainer, 2026-06-06): a new user should get summaries of their
+5
View File
@@ -1,5 +1,10 @@
# Spec — In-process scheduled discovery + auto-summarize + rate-gate finish # Spec — In-process scheduled discovery + auto-summarize + rate-gate finish
> **Extended by ADR-020 (2026-06-08).** Auto-summarize is no longer "every unseen video": the
> scheduler now skips videos published before `TAPIR_AUTO_SUMMARIZE_WINDOW` (default ~7d) unless
> explicitly requested, so a back-catalogue does not re-drive the rate gate every cycle. See
> `DECISIONS.md` ADR-020.
**Repo:** tapir · **Size:** medium · **Solo session** (not a swarm). **Repo:** tapir · **Size:** medium · **Solo session** (not a swarm).
**Why this exists.** The Stage-0 gate ("me or a friend returns and reads/acts in ≥2 separate **Why this exists.** The Stage-0 gate ("me or a friend returns and reads/acts in ≥2 separate
+7
View File
@@ -1,5 +1,12 @@
# Spec — Unify video-card states: one "Summarize now" verb, honest no-captions state # Spec — Unify video-card states: one "Summarize now" verb, honest no-captions state
> **Superseded in part by ADR-020 (2026-06-08).** The five card states still hold, but the copy
> changed: the nudge verb is now **"Summarize"** (not "Summarize now"), the rate-limited state
> reads **"In queue"** (not "Fetching soon…"), and the queued state reads **"summarizing
> shortly"** (not "waiting for the next run"). The list also now collapses older un-summarized
> and caption-less videos. See `DECISIONS.md` ADR-020 and `views.templ` (`VideoCard`) for the
> current copy; this doc is kept as the original design record.
**Repo:** tapir · **Size:** small, **view-layer only** (`views.templ` + a little CSS in **Repo:** tapir · **Size:** small, **view-layer only** (`views.templ` + a little CSS in
`view.go`; regenerate `views_templ.go`). No handler, store, or DB change. The two existing `view.go`; regenerate `views_templ.go`). No handler, store, or DB change. The two existing
handlers (`/summarize`, `/retry-now`) stay exactly as they are — only what the card *shows* handlers (`/summarize`, `/retry-now`) stay exactly as they are — only what the card *shows*
+1
View File
@@ -179,3 +179,4 @@ distinguishable.
| **"Summarize now" foreground path** | Unified quiet nudge button on actionable non-summarized cards. Five explicit card states — (1) summarized: chip + no button; (2) no captions (`transcript_status = 'none'`): "No transcript available", no button; (3) queued: "Queued" chip, no button; (4) rate-limited: "Fetching soon…" + "Summarize now" → `POST /v/{id}/retry-now` (clears `rate_limited_at`, triggers engine); (5) pending: "Not summarized" + "Summarize now" → `POST /v/{id}/summarize` (queues + triggers engine). One verb, one style (`.btn-quiet`); backend difference invisible to user. Both handlers call `ProcessVideo` through `globalFetchGate`. Rate gate respected, not bypassed — this is onboarding prioritisation. | Fast onboarding value; honest dead-end for no-captions videos (no button that fails). | `internal/web/handlers.go` (`handleRetryNow`, `handleRequestSummarize`); `internal/web/views.templ` (`VideoCard`) | | **"Summarize now" foreground path** | Unified quiet nudge button on actionable non-summarized cards. Five explicit card states — (1) summarized: chip + no button; (2) no captions (`transcript_status = 'none'`): "No transcript available", no button; (3) queued: "Queued" chip, no button; (4) rate-limited: "Fetching soon…" + "Summarize now" → `POST /v/{id}/retry-now` (clears `rate_limited_at`, triggers engine); (5) pending: "Not summarized" + "Summarize now" → `POST /v/{id}/summarize` (queues + triggers engine). One verb, one style (`.btn-quiet`); backend difference invisible to user. Both handlers call `ProcessVideo` through `globalFetchGate`. Rate gate respected, not bypassed — this is onboarding prioritisation. | Fast onboarding value; honest dead-end for no-captions videos (no button that fails). | `internal/web/handlers.go` (`handleRetryNow`, `handleRequestSummarize`); `internal/web/views.templ` (`VideoCard`) |
| **Pipeline stats bar** | A one-line status bar above the video list: `N summarized · M fetching soon · K no captions`. Computed from the unfiltered row set; hidden when all videos are summarized. Gives the user a clear read on pipeline state without any interaction. | Replaces the "why is nothing happening?" confusion when most videos are pending or rate-limited. | `internal/web/view.go` (`PipelineStats`, `pipelineStats`) | | **Pipeline stats bar** | A one-line status bar above the video list: `N summarized · M fetching soon · K no captions`. Computed from the unfiltered row set; hidden when all videos are summarized. Gives the user a clear read on pipeline state without any interaction. | Replaces the "why is nothing happening?" confusion when most videos are pending or rate-limited. | `internal/web/view.go` (`PipelineStats`, `pipelineStats`) |
| **Unavailable channels (account page)** | The `/account` page shows a "Unavailable channels" section when any channels returned HTTP 404 on the last discovery pass. Lists channel name, an "unavailable" badge, and the first-seen date. Data sourced from the `channel_errors` table (migration 013). | Surfaces silent failures so users know why some subscribed channels produce no new videos. | migration 013; `internal/web/account.go`; `internal/adapters/youtube/youtube.go` (`domain.ErrChannelUnavailable`) | | **Unavailable channels (account page)** | The `/account` page shows a "Unavailable channels" section when any channels returned HTTP 404 on the last discovery pass. Lists channel name, an "unavailable" badge, and the first-seen date. Data sourced from the `channel_errors` table (migration 013). | Surfaces silent failures so users know why some subscribed channels produce no new videos. | migration 013; `internal/web/account.go`; `internal/adapters/youtube/youtube.go` (`domain.ErrChannelUnavailable`) |
| **Recency window + sparse-state honesty (ADR-020)** | Supersedes the copy/sort in the rows above. Auto-summarize is bounded to videos published within `TAPIR_AUTO_SUMMARIZE_WINDOW` (~7d); older un-summarized videos collapse behind a single "Show N older videos — summarize on demand" disclosure, and caption-less videos collapse to a one-line count (not N cards). List order is now `summarized-first, published_at DESC NULLS LAST`. Copy reframed for honest scarcity: pipeline bar reads "N ready · M in queue · K no captions" (no "fetching soon"); a gradual-fill note explains the rate limit; the nudge verb is "Summarize" (not "Summarize now"); the queued card says "summarizing shortly"; the empty-connected state drops the impossible `tapir run` instruction. Detail leads with Takeaways. Filters slimmed (no date pickers; hidden when empty); watched/skipped segmented; back link on detail; empty terms checkbox removed. | Make the sparse reality legible and honest instead of implying abundance/imminence; bound auto load so the back-catalogue doesn't re-drive the caption gate. Never fetch harder — scarcity is surfaced, not engineered around. | ADR-020; `2384c47`, `3df0459`, `40b703e`, `a1a5217`, `4a0a56e`, `9bf1c31`, `980638d`, `12fb031`, `f775441`, `51aa5d9` |
+14 -2
View File
@@ -6,12 +6,19 @@ Feature: Choose how new videos get summarized
Background: Background:
Given I am a registered user with a connected video account Given I am a registered user with a connected video account
Scenario: Auto mode summarizes every new video Scenario: Auto mode summarizes recent new videos automatically
Given my summarization mode is "auto" Given my summarization mode is "auto"
When a subscribed channel posts a new video with captions When a subscribed channel posts a new video with captions within the recency window
Then Tapir summarizes it without my asking Then Tapir summarizes it without my asking
And the summary appears in my list And the summary appears in my list
Scenario: Auto mode lists older videos without summarizing them
Given my summarization mode is "auto"
When discovery finds a video published before the recency window
Then the video appears in my list with no summary
And it is not summarized automatically
And I can still summarize it on demand with "Summarize"
Scenario: Manual mode is the default and leaves new videos unsummarized Scenario: Manual mode is the default and leaves new videos unsummarized
Given I have not changed my summarization mode Given I have not changed my summarization mode
Then my mode is "manual" Then my mode is "manual"
@@ -30,3 +37,8 @@ Feature: Choose how new videos get summarized
# auto_summarize is a per-user setting and summarize_requested is a per-video queue # auto_summarize is a per-user setting and summarize_requested is a per-video queue
# flag (migration 006). The web button sets the flag; `tapir run` processes both the # flag (migration 006). The web button sets the flag; `tapir run` processes both the
# auto videos and the manually queued ones, then clears the flag. # auto videos and the manually queued ones, then clears the flag.
#
# Recency bound (ADR-020): in auto mode the scheduler only summarizes videos published
# within TAPIR_AUTO_SUMMARIZE_WINDOW (default ~7d); older videos are discovered and
# listed but wait for an explicit "Summarize" — so a back-catalogue does not re-drive
# the per-IP caption gate (ADR-014) every cycle. A manual request bypasses the bound.