docs: reconcile requirements + architecture with ADR-020
Bring the living docs current with the recency-bounded auto-summarize + sparse
honesty + feed IA bundle (ADR-020):
- requirements (BDD): summarize_mode.feature — auto now summarizes RECENT new
videos; added a scenario for older videos (listed, on-demand), recency note.
- architecture.md: summarization-mode + new list-surface paragraph; scheduler
diagram + two-path table + three-phase pass now show the recency pre-filter;
dropped stale "Summarize now".
- data-model.md: auto_summarize is recent-only, older on-demand.
- README.md: one-line recency note on the serve scheduler.
- ui-spec.md: appended the as-built ADR-020 row (supersedes earlier copy/sort).
- specs/{video-card-states,newest-first-ordering,scheduled-discovery}.md:
superseded/extended banners pointing at ADR-020 (kept as design records).
Docs-only; task check green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.8
@@ -75,8 +75,10 @@ password or Google); a new Dex subject is routed to `/register` to create a Tapi
|
||||
Stage 1 is multi-user: each user connects their own YouTube account from the browser and
|
||||
manages their own summaries under DB-enforced RLS isolation. When `TAPIR_DISCOVERY_INTERVAL`
|
||||
is set (e.g. `2h`), the serve process runs a scheduled discovery pass for every registered
|
||||
user automatically — no CronJob required. See `docs/homelab-integration.md` for the full
|
||||
config reference.
|
||||
user automatically — no CronJob required. In auto mode only videos published within
|
||||
`TAPIR_AUTO_SUMMARIZE_WINDOW` (default ~7d, ADR-020) are summarised automatically; older videos
|
||||
are listed and summarised on demand, so a large back-catalogue doesn't keep re-driving the
|
||||
caption rate gate. See `docs/homelab-integration.md` for the full config reference.
|
||||
|
||||
### Headless on koala
|
||||
|
||||
|
||||
@@ -147,9 +147,17 @@ graph TB
|
||||
rate-limiting — ADR-014).
|
||||
- **Summarization mode** — `users.auto_summarize` (migration 006). Default is **true** for new
|
||||
users (migration 011, ADR-018); all existing rows were back-filled via migration 012. Auto:
|
||||
every new video is summarized. Manual: new videos appear unsummarized; the button sets
|
||||
`videos.summarize_requested`, which the next `tapir run` processes and clears. Both the click
|
||||
path and the batch `tapir run` drive the same unchanged engine.
|
||||
new videos **published within the recency window** (`TAPIR_AUTO_SUMMARIZE_WINDOW`, default ~7d,
|
||||
ADR-020) are summarized automatically; older videos are discovered and listed but wait for an
|
||||
explicit "Summarize". Manual: new videos appear unsummarized; the button sets
|
||||
`videos.summarize_requested`, which the next `tapir run` processes and clears. A manual request
|
||||
bypasses the recency bound. Both the click path and the batch `tapir run` drive the same
|
||||
unchanged engine.
|
||||
- **List surface (ADR-020)** — the list reads `ListVideos` ordered summarized-first, then
|
||||
`published_at DESC NULLS LAST`. The web layer collapses the noise so summaries are not buried:
|
||||
un-summarized videos older than the recency window fold into one "Show N older videos"
|
||||
disclosure, and caption-less videos collapse to a single count line. Copy surfaces scarcity
|
||||
honestly (queue counts, gradual-fill note) — it never implies the feed is fuller than it is.
|
||||
|
||||
The engine, ports, and sink adapters are **untouched** by all of the above — the web surface only
|
||||
reads the store and triggers the existing engine. Adding it changed wiring, not the core (ADR-003).
|
||||
@@ -174,6 +182,7 @@ sequenceDiagram
|
||||
loop per user
|
||||
S->>DB: GetAutoSummarize(userID)
|
||||
S->>YT: ListSubscriptions + NewVideos
|
||||
Note over S,DB: auto: skip videos published before<br/>TAPIR_AUTO_SUMMARIZE_WINDOW (ADR-020);<br/>older ones listed, await manual request
|
||||
Note over S,YT: WaitFetchGate(ctx) throttles<br/>all fetches to TAPIR_FETCH_RATE
|
||||
alt transcript available
|
||||
S->>LLM: Summarize
|
||||
@@ -214,19 +223,22 @@ Both paths share `globalFetchGate` — rate limiting is **respected in both**, n
|
||||
|
||||
| Path | Trigger | Order | Rationale |
|
||||
|------|---------|-------|-----------|
|
||||
| **Foreground** | User clicks "Summarize now" on any non-summarized card (`POST /v/{id}/retry-now` for rate-limited; `POST /v/{id}/summarize` for pending) | Single chosen video | On-demand value: user picks a specific video to read now |
|
||||
| **Background batch** | Scheduled discovery pass every `TAPIR_DISCOVERY_INTERVAL` | **Newest-first across all channels** (see below) | Onboarding prioritisation: most recent, relevant videos surface first |
|
||||
| **Foreground** | User clicks "Summarize" on any non-summarized card (`POST /v/{id}/retry-now` for rate-limited; `POST /v/{id}/summarize` for pending) | Single chosen video | On-demand value: user picks a specific video to read now — bypasses the recency bound |
|
||||
| **Background batch** | Scheduled discovery pass every `TAPIR_DISCOVERY_INTERVAL` | **Newest-first across all channels** (see below), **bounded to the recency window** (ADR-020) | Onboarding prioritisation within bounded load: recent videos auto-fill; the older back-catalogue stays on-demand |
|
||||
|
||||
The rationale for both paths is **onboarding prioritisation** — a new user should get summaries
|
||||
of their most recent, relevant videos quickly while the older back-catalogue fills in behind,
|
||||
all within the honest shared rate limit.
|
||||
The rationale for both paths is **onboarding prioritisation under an honest, bounded load** — a
|
||||
new user gets summaries of their most recent videos automatically, while the older back-catalogue
|
||||
is listed but summarised only on demand, so it never re-drives the shared rate gate every cycle.
|
||||
|
||||
### Newest-first batch ordering (ADR-018)
|
||||
|
||||
Within each scheduled pass, `RunOnce` uses a three-phase structure:
|
||||
|
||||
1. **Discover + persist**: walk all channels, `UpsertVideo` every candidate (so it appears in
|
||||
the list), apply pre-filters (seen/manual/backoff), collect surviving candidates.
|
||||
the list), apply pre-filters (seen/manual/backoff/**recency**), collect surviving candidates.
|
||||
The recency pre-filter (ADR-020) drops auto-mode videos published before
|
||||
`now - TAPIR_AUTO_SUMMARIZE_WINDOW` unless they are explicitly requested; an undated video is
|
||||
never aged out. They remain persisted/listed — only auto-summarisation is skipped.
|
||||
2. **Sort**: order candidates `published_at DESC, NULLS LAST, discovery_pos ASC`. Videos with
|
||||
no publish date (schema 001: nullable) sort after all dated content. The sort is in-memory
|
||||
(`slices.SortStableFunc`) — at current scale this is fine.
|
||||
@@ -235,7 +247,8 @@ Within each scheduled pass, `RunOnce` uses a three-phase structure:
|
||||
Before (per-channel inline): `[chanA-old, chanA-mid, chanB-new, chanB-null]`
|
||||
After (newest-first): `[chanB-new, chanA-mid, chanA-old, chanB-null]`
|
||||
|
||||
The set of processed videos is identical; only the order within a pass changes.
|
||||
The set of *processed* videos now also excludes auto-mode back-catalogue beyond the recency
|
||||
window (those stay listed, summarised on demand); within the processed set, only order changes.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+4
-3
@@ -147,9 +147,10 @@ mechanism.
|
||||
|
||||
- **USER** — one row per registered user (Stage 1, ADR-012; no longer single-row). The Tapir-side
|
||||
profile; the Dex identity is held separately in `USER_IDENTITY`, not on this row. `auto_summarize`
|
||||
(migration 006) is the per-user mode flag: `TRUE` = auto-summarize every new video. Default is
|
||||
**true** for new users (migration 011, ADR-018); existing rows were back-filled via migration 012
|
||||
with RLS bypass.
|
||||
(migration 006) is the per-user mode flag: `TRUE` = auto-summarize new videos **published within
|
||||
the recency window** (`TAPIR_AUTO_SUMMARIZE_WINDOW`, default ~7d, ADR-020); older videos are
|
||||
listed but summarised on demand. Default is **true** for new users (migration 011, ADR-018);
|
||||
existing rows were back-filled via migration 012 with RLS bypass.
|
||||
- **USER_IDENTITY** (migration 004) — the `dex_subject → user_id` map. `dex_subject` is the PK,
|
||||
`user_id` a `UNIQUE` FK to `users` with `ON DELETE CASCADE`. This is the bridge resolved at login
|
||||
*before* a `user_id` is known, so it is **deliberately not RLS-enabled** (it holds no user data;
|
||||
|
||||
@@ -1,5 +1,11 @@
|
||||
# Spec — Newest-first batch ordering + honest "Try now" / prioritisation docs
|
||||
|
||||
> **Extended by ADR-020 (2026-06-08).** This spec covers the *batch processing* order within a
|
||||
> pass. ADR-020 adds (a) a recency pre-filter — auto mode skips videos published before
|
||||
> `TAPIR_AUTO_SUMMARIZE_WINDOW`, listed but summarised on demand — and (b) the same
|
||||
> `published_at DESC NULLS LAST` ordering on the **list read** (`ListVideos`), which previously
|
||||
> sorted by `seen_at`. See `DECISIONS.md` ADR-020.
|
||||
|
||||
**Repo:** tapir · **Size:** small · **Solo session.**
|
||||
|
||||
**Why.** Product intent (maintainer, 2026-06-06): a new user should get summaries of their
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
# Spec — In-process scheduled discovery + auto-summarize + rate-gate finish
|
||||
|
||||
> **Extended by ADR-020 (2026-06-08).** Auto-summarize is no longer "every unseen video": the
|
||||
> scheduler now skips videos published before `TAPIR_AUTO_SUMMARIZE_WINDOW` (default ~7d) unless
|
||||
> explicitly requested, so a back-catalogue does not re-drive the rate gate every cycle. See
|
||||
> `DECISIONS.md` ADR-020.
|
||||
|
||||
**Repo:** tapir · **Size:** medium · **Solo session** (not a swarm).
|
||||
|
||||
**Why this exists.** The Stage-0 gate ("me or a friend returns and reads/acts in ≥2 separate
|
||||
|
||||
@@ -1,5 +1,12 @@
|
||||
# Spec — Unify video-card states: one "Summarize now" verb, honest no-captions state
|
||||
|
||||
> **Superseded in part by ADR-020 (2026-06-08).** The five card states still hold, but the copy
|
||||
> changed: the nudge verb is now **"Summarize"** (not "Summarize now"), the rate-limited state
|
||||
> reads **"In queue"** (not "Fetching soon…"), and the queued state reads **"summarizing
|
||||
> shortly"** (not "waiting for the next run"). The list also now collapses older un-summarized
|
||||
> and caption-less videos. See `DECISIONS.md` ADR-020 and `views.templ` (`VideoCard`) for the
|
||||
> current copy; this doc is kept as the original design record.
|
||||
|
||||
**Repo:** tapir · **Size:** small, **view-layer only** (`views.templ` + a little CSS in
|
||||
`view.go`; regenerate `views_templ.go`). No handler, store, or DB change. The two existing
|
||||
handlers (`/summarize`, `/retry-now`) stay exactly as they are — only what the card *shows*
|
||||
|
||||
@@ -179,3 +179,4 @@ distinguishable.
|
||||
| **"Summarize now" foreground path** | Unified quiet nudge button on actionable non-summarized cards. Five explicit card states — (1) summarized: chip + no button; (2) no captions (`transcript_status = 'none'`): "No transcript available", no button; (3) queued: "Queued" chip, no button; (4) rate-limited: "Fetching soon…" + "Summarize now" → `POST /v/{id}/retry-now` (clears `rate_limited_at`, triggers engine); (5) pending: "Not summarized" + "Summarize now" → `POST /v/{id}/summarize` (queues + triggers engine). One verb, one style (`.btn-quiet`); backend difference invisible to user. Both handlers call `ProcessVideo` through `globalFetchGate`. Rate gate respected, not bypassed — this is onboarding prioritisation. | Fast onboarding value; honest dead-end for no-captions videos (no button that fails). | `internal/web/handlers.go` (`handleRetryNow`, `handleRequestSummarize`); `internal/web/views.templ` (`VideoCard`) |
|
||||
| **Pipeline stats bar** | A one-line status bar above the video list: `N summarized · M fetching soon · K no captions`. Computed from the unfiltered row set; hidden when all videos are summarized. Gives the user a clear read on pipeline state without any interaction. | Replaces the "why is nothing happening?" confusion when most videos are pending or rate-limited. | `internal/web/view.go` (`PipelineStats`, `pipelineStats`) |
|
||||
| **Unavailable channels (account page)** | The `/account` page shows a "Unavailable channels" section when any channels returned HTTP 404 on the last discovery pass. Lists channel name, an "unavailable" badge, and the first-seen date. Data sourced from the `channel_errors` table (migration 013). | Surfaces silent failures so users know why some subscribed channels produce no new videos. | migration 013; `internal/web/account.go`; `internal/adapters/youtube/youtube.go` (`domain.ErrChannelUnavailable`) |
|
||||
| **Recency window + sparse-state honesty (ADR-020)** | Supersedes the copy/sort in the rows above. Auto-summarize is bounded to videos published within `TAPIR_AUTO_SUMMARIZE_WINDOW` (~7d); older un-summarized videos collapse behind a single "Show N older videos — summarize on demand" disclosure, and caption-less videos collapse to a one-line count (not N cards). List order is now `summarized-first, published_at DESC NULLS LAST`. Copy reframed for honest scarcity: pipeline bar reads "N ready · M in queue · K no captions" (no "fetching soon"); a gradual-fill note explains the rate limit; the nudge verb is "Summarize" (not "Summarize now"); the queued card says "summarizing shortly"; the empty-connected state drops the impossible `tapir run` instruction. Detail leads with Takeaways. Filters slimmed (no date pickers; hidden when empty); watched/skipped segmented; back link on detail; empty terms checkbox removed. | Make the sparse reality legible and honest instead of implying abundance/imminence; bound auto load so the back-catalogue doesn't re-drive the caption gate. Never fetch harder — scarcity is surfaced, not engineered around. | ADR-020; `2384c47`, `3df0459`, `40b703e`, `a1a5217`, `4a0a56e`, `9bf1c31`, `980638d`, `12fb031`, `f775441`, `51aa5d9` |
|
||||
|
||||
@@ -6,12 +6,19 @@ Feature: Choose how new videos get summarized
|
||||
Background:
|
||||
Given I am a registered user with a connected video account
|
||||
|
||||
Scenario: Auto mode summarizes every new video
|
||||
Scenario: Auto mode summarizes recent new videos automatically
|
||||
Given my summarization mode is "auto"
|
||||
When a subscribed channel posts a new video with captions
|
||||
When a subscribed channel posts a new video with captions within the recency window
|
||||
Then Tapir summarizes it without my asking
|
||||
And the summary appears in my list
|
||||
|
||||
Scenario: Auto mode lists older videos without summarizing them
|
||||
Given my summarization mode is "auto"
|
||||
When discovery finds a video published before the recency window
|
||||
Then the video appears in my list with no summary
|
||||
And it is not summarized automatically
|
||||
And I can still summarize it on demand with "Summarize"
|
||||
|
||||
Scenario: Manual mode is the default and leaves new videos unsummarized
|
||||
Given I have not changed my summarization mode
|
||||
Then my mode is "manual"
|
||||
@@ -30,3 +37,8 @@ Feature: Choose how new videos get summarized
|
||||
# auto_summarize is a per-user setting and summarize_requested is a per-video queue
|
||||
# flag (migration 006). The web button sets the flag; `tapir run` processes both the
|
||||
# auto videos and the manually queued ones, then clears the flag.
|
||||
#
|
||||
# Recency bound (ADR-020): in auto mode the scheduler only summarizes videos published
|
||||
# within TAPIR_AUTO_SUMMARIZE_WINDOW (default ~7d); older videos are discovered and
|
||||
# listed but wait for an explicit "Summarize" — so a back-catalogue does not re-drive
|
||||
# the per-IP caption gate (ADR-014) every cycle. A manual request bypasses the bound.
|
||||
|
||||
Reference in New Issue
Block a user