diff --git a/docs/specs/onboarding-wow-burst.md b/docs/specs/onboarding-wow-burst.md new file mode 100644 index 0000000..ce1fec5 --- /dev/null +++ b/docs/specs/onboarding-wow-burst.md @@ -0,0 +1,74 @@ +# Spec — First-session "wow": verify & improve the onboarding summary burst + +**Repo:** tapir · **Size:** small-medium · **Solo session, INVESTIGATE-FIRST.** Read CLAUDE.md, +DECISIONS.md (ADR-018 onboarding burst origins, ADR-020 recency, ADR-021 transcript store, +ADR-022 model chain, ADR-023 Shorts/metadata filter, ADR-024 caption memory), and the +connect/onboarding flow. TBD, conventional commits. + +**The concern (maintainer):** a brand-new user needs to see *some* good summaries quickly on +connect — a first-session "wow" — or they don't come back (the Stage-0 gate is return usage). +**This is a curation/latency problem for ~3 videos, NOT a throughput/429 problem.** Fetching 3 +captions is nowhere near the rate limit; the wall only bites at hundreds. So nothing here fetches +harder or touches the rate-gate budget meaningfully — it's about picking the *right* few videos +and making sure they actually land. Do NOT turn this into a bulk-fetch / rate-pressure change. + +## PHASE 1 — verify what actually happens on connect today. REPORT and STOP before building. +The maintainer does not know if the existing onboarding burst is even firing. Establish ground +truth: +1. **Does the connect-time summarize burst still fire?** There was a cap-3 onboarding burst + (summarize newest ~3 on YouTube connect, through the gate). Trace the *current* connect flow + (OAuth callback → discovery → summarize). Is the burst still wired, or did a later refactor + (recency window ADR-020, discovery changes, the ADR-023/024 filters) bypass or break it? +2. **If it fires, what does it deliver?** For a realistic new connect: how many of the burst + videos actually reach *summarized* vs. land caption-less / queued / rate-limited / weak? Use + Jonas's actual onboarding history if traceable (his first summaries — fast, or trickled?). +3. **How much transcript-cache overlap exists between users?** Query the shared transcript store + (ADR-021): for the existing users, how many videos does a *new* user's subscription set + already have cached transcripts for? This decides whether "cached-first" (instant, zero-fetch + wow) is a real lever or nearly empty at pilot scale. **This number drives the Phase-2 blend.** +4. **What cheap "likely-good" signals are already available at connect?** ADR-023 already pulls + Data API `videos.list` metadata (duration, liveBroadcastContent) — is caption-availability or + anything quality-predictive already in hand, or what would it cost to know "this video has + captions / reasonable length" *before* spending a fetch? + +**Report findings + a recommended Phase-2 shape, then STOP.** The build differs sharply by what +Phase 1 finds: +- burst not firing → fix is "make it fire" (and Phase 2 selection is gravy). +- fires but picks bad videos (caption-less/weak) → fix is the selection logic below. +- fires and works fine → the wow gap is *quality*, not selection → use the stronger model for the + burst (see Phase 2 option C), maybe nothing else. +- near-zero cache overlap → drop cached-first; Phase 2 is just "likely-good fresh". + +## PHASE 2 — improve the burst (design AFTER Phase 1 findings; maintainer picks the blend) +Candidate levers (the maintainer chose 1+2; final blend decided by Phase 1's overlap number): +- **(1) Cached-transcript-first:** fill burst slots from videos whose transcripts are ALREADY in + the shared store (ADR-021) — instant, zero fetch, guaranteed-captioned. Strength depends on the + Phase-1 overlap number. +- **(2) Likely-good selection (not just newest):** for uncached slots, pick videos predicted to + summarize well — has-captions (use/extend the ADR-023 metadata path), reasonable length (not a + 3h podcast that truncates to mush, not a <2min clip), from the user's more-engaged channels if + that signal exists. Skip newest-but-caption-less rather than burning a wow slot on a "no + transcript" card. +- **(C) Stronger model for the burst only:** run the ~3 onboarding summaries through + `gemma4-26b` (stronger) instead of `phi4-mini`, even though slower — first impressions matter + disproportionately and it's only 3 videos, so quality > speed here. Cheap to try, possibly the + highest-wow lever if Phase 1 shows the burst fires but summaries underwhelm. +- **Honest framing:** present the burst as "summaries of a few of your videos to get you started — + the rest fill in over the coming days," so "only a few" reads as intentional taster, not failure. + +**The blend (cached-instant vs. relevant-fresh) is explicitly deferred to the maintainer after +Phase 1** reports the overlap number — do not hard-code it. + +## Boundaries +- NOT a throughput/429 change — it's ~3 videos; do not bulk-fetch or pressure the rate gate. +- No credentials-based caption fetch (ADR-010/026 dead end — auth doesn't work on timedtext). +- No client-side fetcher / extension (out of scope; that's a throughput idea, not a wow idea). +- Respect ADR-021 (read cached transcripts, never re-fetch), ADR-023/024 (don't undo the filters). +- Investigate-first: no selection-logic changes until Phase 1 is reported. + +## Tests (Phase 2, once shaped) +- Burst fires on connect and summarizes the selected set through the gate. +- Cached-transcript videos in the burst summarize with NO fetch (assert zero caption calls for + those). +- Caption-less newest videos are NOT chosen for the burst (no "no transcript" wow-killer cards). +- (If model lever) burst uses the configured stronger model; rest of pipeline unchanged.