mathias
8403e8e524
Product intent: new users get summaries of their newest videos fast while the back-catalogue fills behind, within the shared rate gate. The background batch currently processes in subscription/channel order, not newest-first — this spec closes that gap (collect candidates, sort published_at DESC NULLS LAST, process in order, gate unchanged). Also corrects the docs to describe Try-now as onboarding prioritisation, explicitly removing the prior "looks organic to YouTube" traffic-disguising framing — rate limiting is respected, not evaded.
72 lines
4.7 KiB
Markdown
72 lines
4.7 KiB
Markdown
# Spec — Newest-first batch ordering + honest "Try now" / prioritisation docs
|
|
|
|
**Repo:** tapir · **Size:** small · **Solo session.**
|
|
|
|
**Why.** Product intent (maintainer, 2026-06-06): a new user should get summaries of their
|
|
**newest** videos quickly, while the older back-catalogue fills in behind — all within the one
|
|
shared rate gate. Today the foreground path ("Try now" button) lets a user hand-pick a video,
|
|
but the **background batch processes in subscription/channel order, not newest-first** — so a new
|
|
user with a large candidate set sees the batch summarise whatever channel is first in their
|
|
subscription list, not their newest videos. This slice makes the batch agree with the intent, and
|
|
fixes the docs to describe the real rationale (onboarding prioritisation), not the
|
|
traffic-disguising framing a prior session wrote.
|
|
|
|
Read `CLAUDE.md` + `DECISIONS.md` (ADR-014, ADR-018) first. TBD, conventional commits,
|
|
`task check` green per commit, `templ generate` if views change.
|
|
|
|
## 1. Newest-first batch ordering (the build)
|
|
In `internal/runner/runner.go` `RunOnce`: today the loop processes each video inline while
|
|
walking subscriptions channel-by-channel (`for sub → NewVideos → for v → process`). Change so
|
|
that, within a pass, **candidates are processed newest-first across ALL channels**:
|
|
- Collect the candidate videos across channels first (after dedup/seen/manual/rate-limit
|
|
filtering as today), then **sort by `published_at` descending before processing**, then process
|
|
in that order through the engine + shared `globalFetchGate`.
|
|
- **`published_at` is nullable** (schema 001). Sort **NULLS LAST** — videos with no publish date
|
|
must not jump ahead of dated newest videos. Decide a stable tiebreak (e.g. `seen_at DESC`) for
|
|
equal/again-null dates.
|
|
- Keep all existing behaviour: per-item failure isolation, the rate-limit backoff skip, manual
|
|
mode, channel-unavailable handling, stats. Ordering is the only change — not what gets
|
|
processed, just the order.
|
|
- At 868 candidates a collect-then-sort in memory is fine; do **not** build a streaming/external
|
|
sort. Keep it simple.
|
|
- The shared rate gate (`globalFetchGate`) is unchanged and still governs fetch pacing — ordering
|
|
does not bypass or weaken it.
|
|
|
|
**Optional (only if cheap and clearly correct):** a soft cap so the *first* pass for a brand-new
|
|
user summarises the newest N (e.g. 20) quickly and defers the long tail to subsequent passes — so
|
|
onboarding value lands fast without waiting for the whole sorted set. If this adds real
|
|
complexity, SKIP it and just do the newest-first ordering; the ordering alone delivers the intent.
|
|
|
|
## 2. Tests
|
|
- Given candidates across multiple channels with mixed `published_at` (incl. some NULL), assert
|
|
the processing order is newest-first, NULLS LAST, with the chosen tiebreak. Use the existing
|
|
fake VideoStore/Processor pattern in `runner_test.go`.
|
|
- Assert ordering does not change *which* videos are processed vs. today (same set, new order).
|
|
- Rate-gate / backoff / manual-mode behaviour unchanged (existing tests stay green).
|
|
|
|
## 3. Docs — describe the REAL rationale (replace prior framing)
|
|
The "Try now" button and the discovery batch together implement **onboarding prioritisation**:
|
|
foreground (user-clicked "Try now") summarises a specific video on demand; background batch
|
|
summarises newest-first; both honour the shared rate gate. **Update the docs to state this intent
|
|
— and explicitly REMOVE/replace any framing that describes "Try now" as making traffic "look
|
|
organic to YouTube" or evading rate limits.** That is not the rationale. The rationale is: *get
|
|
the user a few summaries of their newest, most relevant videos fast; process the back-catalogue in
|
|
the background; always within the honest shared rate limit.* Rate limiting is **respected**, not
|
|
evaded.
|
|
- `docs/ui-spec.md`: "Try now" = on-demand foreground summarisation of a chosen (typically newer)
|
|
video; rationale = fast onboarding value, not traffic shaping.
|
|
- `docs/architecture/architecture.md`: document the two-path model — foreground on-demand vs.
|
|
background newest-first batch, both through `globalFetchGate` — and the newest-first ordering.
|
|
- Any requirements/use-case doc mentioning discovery order: state newest-first.
|
|
- If a brain note or `wiki` entry captured the "looks organic" rationale, correct it there too.
|
|
|
|
## Boundaries
|
|
- Do NOT increase fetch rate or weaken the rate gate. Account-safety constraint stands: the
|
|
caption endpoint is unofficial (ADR-010) and must be treated with honest backoff, never evasion.
|
|
- Do NOT touch RLS, credentials, or the Dex surface.
|
|
- Ordering change is within a pass only — no persisted priority queue, no new table.
|
|
|
|
## Out of scope
|
|
Per-user configurable ordering; priority weighting beyond newest-first; the soft-cap if it proves
|
|
non-trivial.
|