docs: spec first-session wow — verify & improve onboarding summary burst (investigate-first)
CI / Lint / Test / Vet (push) Successful in 11s
CI / Build & Import (push) Successful in 10s

The concern is new-user first-contact: see some GOOD summaries fast or they
don't return. Reframed as curation/latency for ~3 videos, NOT a 429/throughput
problem (3 fetches is nowhere near the wall). Phase 1 (report-and-stop) verifies
whether the existing cap-3 onboarding burst even fires today, what it delivers,
and — critically — how much transcript-cache overlap exists between users (drives
the blend). Phase 2 levers: cached-transcript-first (instant, zero-fetch),
likely-good selection (has-captions/good-length, not just newest), and optionally
the stronger model for the burst's few summaries. Blend deferred to the
maintainer post-Phase-1. Explicitly NOT bulk-fetch, NOT credentials (ADR-010/026
dead end), NOT a client extension.
This commit is contained in:
mathias
2026-06-11 15:36:09 +00:00
parent 19ca4282a8
commit b8fbc5a805
+74
View File
@@ -0,0 +1,74 @@
# Spec — First-session "wow": verify & improve the onboarding summary burst
**Repo:** tapir · **Size:** small-medium · **Solo session, INVESTIGATE-FIRST.** Read CLAUDE.md,
DECISIONS.md (ADR-018 onboarding burst origins, ADR-020 recency, ADR-021 transcript store,
ADR-022 model chain, ADR-023 Shorts/metadata filter, ADR-024 caption memory), and the
connect/onboarding flow. TBD, conventional commits.
**The concern (maintainer):** a brand-new user needs to see *some* good summaries quickly on
connect — a first-session "wow" — or they don't come back (the Stage-0 gate is return usage).
**This is a curation/latency problem for ~3 videos, NOT a throughput/429 problem.** Fetching 3
captions is nowhere near the rate limit; the wall only bites at hundreds. So nothing here fetches
harder or touches the rate-gate budget meaningfully — it's about picking the *right* few videos
and making sure they actually land. Do NOT turn this into a bulk-fetch / rate-pressure change.
## PHASE 1 — verify what actually happens on connect today. REPORT and STOP before building.
The maintainer does not know if the existing onboarding burst is even firing. Establish ground
truth:
1. **Does the connect-time summarize burst still fire?** There was a cap-3 onboarding burst
(summarize newest ~3 on YouTube connect, through the gate). Trace the *current* connect flow
(OAuth callback → discovery → summarize). Is the burst still wired, or did a later refactor
(recency window ADR-020, discovery changes, the ADR-023/024 filters) bypass or break it?
2. **If it fires, what does it deliver?** For a realistic new connect: how many of the burst
videos actually reach *summarized* vs. land caption-less / queued / rate-limited / weak? Use
Jonas's actual onboarding history if traceable (his first summaries — fast, or trickled?).
3. **How much transcript-cache overlap exists between users?** Query the shared transcript store
(ADR-021): for the existing users, how many videos does a *new* user's subscription set
already have cached transcripts for? This decides whether "cached-first" (instant, zero-fetch
wow) is a real lever or nearly empty at pilot scale. **This number drives the Phase-2 blend.**
4. **What cheap "likely-good" signals are already available at connect?** ADR-023 already pulls
Data API `videos.list` metadata (duration, liveBroadcastContent) — is caption-availability or
anything quality-predictive already in hand, or what would it cost to know "this video has
captions / reasonable length" *before* spending a fetch?
**Report findings + a recommended Phase-2 shape, then STOP.** The build differs sharply by what
Phase 1 finds:
- burst not firing → fix is "make it fire" (and Phase 2 selection is gravy).
- fires but picks bad videos (caption-less/weak) → fix is the selection logic below.
- fires and works fine → the wow gap is *quality*, not selection → use the stronger model for the
burst (see Phase 2 option C), maybe nothing else.
- near-zero cache overlap → drop cached-first; Phase 2 is just "likely-good fresh".
## PHASE 2 — improve the burst (design AFTER Phase 1 findings; maintainer picks the blend)
Candidate levers (the maintainer chose 1+2; final blend decided by Phase 1's overlap number):
- **(1) Cached-transcript-first:** fill burst slots from videos whose transcripts are ALREADY in
the shared store (ADR-021) — instant, zero fetch, guaranteed-captioned. Strength depends on the
Phase-1 overlap number.
- **(2) Likely-good selection (not just newest):** for uncached slots, pick videos predicted to
summarize well — has-captions (use/extend the ADR-023 metadata path), reasonable length (not a
3h podcast that truncates to mush, not a <2min clip), from the user's more-engaged channels if
that signal exists. Skip newest-but-caption-less rather than burning a wow slot on a "no
transcript" card.
- **(C) Stronger model for the burst only:** run the ~3 onboarding summaries through
`gemma4-26b` (stronger) instead of `phi4-mini`, even though slower — first impressions matter
disproportionately and it's only 3 videos, so quality > speed here. Cheap to try, possibly the
highest-wow lever if Phase 1 shows the burst fires but summaries underwhelm.
- **Honest framing:** present the burst as "summaries of a few of your videos to get you started —
the rest fill in over the coming days," so "only a few" reads as intentional taster, not failure.
**The blend (cached-instant vs. relevant-fresh) is explicitly deferred to the maintainer after
Phase 1** reports the overlap number — do not hard-code it.
## Boundaries
- NOT a throughput/429 change — it's ~3 videos; do not bulk-fetch or pressure the rate gate.
- No credentials-based caption fetch (ADR-010/026 dead end — auth doesn't work on timedtext).
- No client-side fetcher / extension (out of scope; that's a throughput idea, not a wow idea).
- Respect ADR-021 (read cached transcripts, never re-fetch), ADR-023/024 (don't undo the filters).
- Investigate-first: no selection-logic changes until Phase 1 is reported.
## Tests (Phase 2, once shaped)
- Burst fires on connect and summarizes the selected set through the gate.
- Cached-transcript videos in the burst summarize with NO fetch (assert zero caption calls for
those).
- Caption-less newest videos are NOT chosen for the burst (no "no transcript" wow-killer cards).
- (If model lever) burst uses the configured stronger model; rest of pipeline unchanged.