From 8403e8e524a3eda37c5a8eab113b188f14789eda Mon Sep 17 00:00:00 2001 From: mathias Date: Sat, 6 Jun 2026 19:20:27 +0000 Subject: [PATCH] docs: spec newest-first batch ordering + honest Try-now/prioritisation docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Product intent: new users get summaries of their newest videos fast while the back-catalogue fills behind, within the shared rate gate. The background batch currently processes in subscription/channel order, not newest-first — this spec closes that gap (collect candidates, sort published_at DESC NULLS LAST, process in order, gate unchanged). Also corrects the docs to describe Try-now as onboarding prioritisation, explicitly removing the prior "looks organic to YouTube" traffic-disguising framing — rate limiting is respected, not evaded. --- docs/specs/newest-first-ordering.md | 71 +++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 docs/specs/newest-first-ordering.md diff --git a/docs/specs/newest-first-ordering.md b/docs/specs/newest-first-ordering.md new file mode 100644 index 0000000..2f028a4 --- /dev/null +++ b/docs/specs/newest-first-ordering.md @@ -0,0 +1,71 @@ +# Spec — Newest-first batch ordering + honest "Try now" / prioritisation docs + +**Repo:** tapir · **Size:** small · **Solo session.** + +**Why.** Product intent (maintainer, 2026-06-06): a new user should get summaries of their +**newest** videos quickly, while the older back-catalogue fills in behind — all within the one +shared rate gate. Today the foreground path ("Try now" button) lets a user hand-pick a video, +but the **background batch processes in subscription/channel order, not newest-first** — so a new +user with a large candidate set sees the batch summarise whatever channel is first in their +subscription list, not their newest videos. This slice makes the batch agree with the intent, and +fixes the docs to describe the real rationale (onboarding prioritisation), not the +traffic-disguising framing a prior session wrote. + +Read `CLAUDE.md` + `DECISIONS.md` (ADR-014, ADR-018) first. TBD, conventional commits, +`task check` green per commit, `templ generate` if views change. + +## 1. Newest-first batch ordering (the build) +In `internal/runner/runner.go` `RunOnce`: today the loop processes each video inline while +walking subscriptions channel-by-channel (`for sub → NewVideos → for v → process`). Change so +that, within a pass, **candidates are processed newest-first across ALL channels**: +- Collect the candidate videos across channels first (after dedup/seen/manual/rate-limit + filtering as today), then **sort by `published_at` descending before processing**, then process + in that order through the engine + shared `globalFetchGate`. +- **`published_at` is nullable** (schema 001). Sort **NULLS LAST** — videos with no publish date + must not jump ahead of dated newest videos. Decide a stable tiebreak (e.g. `seen_at DESC`) for + equal/again-null dates. +- Keep all existing behaviour: per-item failure isolation, the rate-limit backoff skip, manual + mode, channel-unavailable handling, stats. Ordering is the only change — not what gets + processed, just the order. +- At 868 candidates a collect-then-sort in memory is fine; do **not** build a streaming/external + sort. Keep it simple. +- The shared rate gate (`globalFetchGate`) is unchanged and still governs fetch pacing — ordering + does not bypass or weaken it. + +**Optional (only if cheap and clearly correct):** a soft cap so the *first* pass for a brand-new +user summarises the newest N (e.g. 20) quickly and defers the long tail to subsequent passes — so +onboarding value lands fast without waiting for the whole sorted set. If this adds real +complexity, SKIP it and just do the newest-first ordering; the ordering alone delivers the intent. + +## 2. Tests +- Given candidates across multiple channels with mixed `published_at` (incl. some NULL), assert + the processing order is newest-first, NULLS LAST, with the chosen tiebreak. Use the existing + fake VideoStore/Processor pattern in `runner_test.go`. +- Assert ordering does not change *which* videos are processed vs. today (same set, new order). +- Rate-gate / backoff / manual-mode behaviour unchanged (existing tests stay green). + +## 3. Docs — describe the REAL rationale (replace prior framing) +The "Try now" button and the discovery batch together implement **onboarding prioritisation**: +foreground (user-clicked "Try now") summarises a specific video on demand; background batch +summarises newest-first; both honour the shared rate gate. **Update the docs to state this intent +— and explicitly REMOVE/replace any framing that describes "Try now" as making traffic "look +organic to YouTube" or evading rate limits.** That is not the rationale. The rationale is: *get +the user a few summaries of their newest, most relevant videos fast; process the back-catalogue in +the background; always within the honest shared rate limit.* Rate limiting is **respected**, not +evaded. +- `docs/ui-spec.md`: "Try now" = on-demand foreground summarisation of a chosen (typically newer) + video; rationale = fast onboarding value, not traffic shaping. +- `docs/architecture/architecture.md`: document the two-path model — foreground on-demand vs. + background newest-first batch, both through `globalFetchGate` — and the newest-first ordering. +- Any requirements/use-case doc mentioning discovery order: state newest-first. +- If a brain note or `wiki` entry captured the "looks organic" rationale, correct it there too. + +## Boundaries +- Do NOT increase fetch rate or weaken the rate gate. Account-safety constraint stands: the + caption endpoint is unofficial (ADR-010) and must be treated with honest backoff, never evasion. +- Do NOT touch RLS, credentials, or the Dex surface. +- Ordering change is within a pass only — no persisted priority queue, no new table. + +## Out of scope +Per-user configurable ordering; priority weighting beyond newest-first; the soft-cap if it proves +non-trivial.