tapir/docs/specs/newest-first-ordering.md

Spec — Newest-first batch ordering + honest "Try now" / prioritisation docs

Extended by ADR-020 (2026-06-08). This spec covers the batch processing order within a pass. ADR-020 adds (a) a recency pre-filter — auto mode skips videos published before TAPIR_AUTO_SUMMARIZE_WINDOW, listed but summarised on demand — and (b) the same published_at DESC NULLS LAST ordering on the list read (ListVideos), which previously sorted by seen_at. See DECISIONS.md ADR-020.

Repo: tapir · Size: small · Solo session.

Why. Product intent (maintainer, 2026-06-06): a new user should get summaries of their newest videos quickly, while the older back-catalogue fills in behind — all within the one shared rate gate. Today the foreground path ("Try now" button) lets a user hand-pick a video, but the background batch processes in subscription/channel order, not newest-first — so a new user with a large candidate set sees the batch summarise whatever channel is first in their subscription list, not their newest videos. This slice makes the batch agree with the intent, and fixes the docs to describe the real rationale (onboarding prioritisation), not the traffic-disguising framing a prior session wrote.

Read CLAUDE.md + DECISIONS.md (ADR-014, ADR-018) first. TBD, conventional commits, task check green per commit, templ generate if views change.

1. Newest-first batch ordering (the build)

In internal/runner/runner.go RunOnce: today the loop processes each video inline while walking subscriptions channel-by-channel (for sub → NewVideos → for v → process). Change so that, within a pass, candidates are processed newest-first across ALL channels:

• Collect the candidate videos across channels first (after dedup/seen/manual/rate-limit filtering as today), then sort by published_at descending before processing, then process in that order through the engine + shared globalFetchGate.
• published_at is nullable (schema 001). Sort NULLS LAST — videos with no publish date must not jump ahead of dated newest videos. Decide a stable tiebreak (e.g. seen_at DESC) for equal/again-null dates.
• Keep all existing behaviour: per-item failure isolation, the rate-limit backoff skip, manual mode, channel-unavailable handling, stats. Ordering is the only change — not what gets processed, just the order.
• At 868 candidates a collect-then-sort in memory is fine; do not build a streaming/external sort. Keep it simple.
• The shared rate gate (globalFetchGate) is unchanged and still governs fetch pacing — ordering does not bypass or weaken it.

Optional (only if cheap and clearly correct): a soft cap so the first pass for a brand-new user summarises the newest N (e.g. 20) quickly and defers the long tail to subsequent passes — so onboarding value lands fast without waiting for the whole sorted set. If this adds real complexity, SKIP it and just do the newest-first ordering; the ordering alone delivers the intent.

2. Tests

• Given candidates across multiple channels with mixed published_at (incl. some NULL), assert the processing order is newest-first, NULLS LAST, with the chosen tiebreak. Use the existing fake VideoStore/Processor pattern in runner_test.go.
• Assert ordering does not change which videos are processed vs. today (same set, new order).
• Rate-gate / backoff / manual-mode behaviour unchanged (existing tests stay green).

3. Docs — describe the REAL rationale (replace prior framing)

The "Try now" button and the discovery batch together implement onboarding prioritisation: foreground (user-clicked "Try now") summarises a specific video on demand; background batch summarises newest-first; both honour the shared rate gate. Update the docs to state this intent — and explicitly REMOVE/replace any framing that describes "Try now" as making traffic "look organic to YouTube" or evading rate limits. That is not the rationale. The rationale is: get the user a few summaries of their newest, most relevant videos fast; process the back-catalogue in the background; always within the honest shared rate limit. Rate limiting is respected, not evaded.

• docs/ui-spec.md: "Try now" = on-demand foreground summarisation of a chosen (typically newer) video; rationale = fast onboarding value, not traffic shaping.
• docs/architecture/architecture.md: document the two-path model — foreground on-demand vs. background newest-first batch, both through globalFetchGate — and the newest-first ordering.
• Any requirements/use-case doc mentioning discovery order: state newest-first.
• If a brain note or wiki entry captured the "looks organic" rationale, correct it there too.

Boundaries

• Do NOT increase fetch rate or weaken the rate gate. Account-safety constraint stands: the caption endpoint is unofficial (ADR-010) and must be treated with honest backoff, never evasion.
• Do NOT touch RLS, credentials, or the Dex surface.
• Ordering change is within a pass only — no persisted priority queue, no new table.

Out of scope

Per-user configurable ordering; priority weighting beyond newest-first; the soft-cap if it proves non-trivial.