docs: spec first-session wow — verify & improve onboarding summary burst (investigate-first)
The concern is new-user first-contact: see some GOOD summaries fast or they don't return. Reframed as curation/latency for ~3 videos, NOT a 429/throughput problem (3 fetches is nowhere near the wall). Phase 1 (report-and-stop) verifies whether the existing cap-3 onboarding burst even fires today, what it delivers, and — critically — how much transcript-cache overlap exists between users (drives the blend). Phase 2 levers: cached-transcript-first (instant, zero-fetch), likely-good selection (has-captions/good-length, not just newest), and optionally the stronger model for the burst's few summaries. Blend deferred to the maintainer post-Phase-1. Explicitly NOT bulk-fetch, NOT credentials (ADR-010/026 dead end), NOT a client extension.
This commit is contained in:
@@ -0,0 +1,74 @@
|
||||
# Spec — First-session "wow": verify & improve the onboarding summary burst
|
||||
|
||||
**Repo:** tapir · **Size:** small-medium · **Solo session, INVESTIGATE-FIRST.** Read CLAUDE.md,
|
||||
DECISIONS.md (ADR-018 onboarding burst origins, ADR-020 recency, ADR-021 transcript store,
|
||||
ADR-022 model chain, ADR-023 Shorts/metadata filter, ADR-024 caption memory), and the
|
||||
connect/onboarding flow. TBD, conventional commits.
|
||||
|
||||
**The concern (maintainer):** a brand-new user needs to see *some* good summaries quickly on
|
||||
connect — a first-session "wow" — or they don't come back (the Stage-0 gate is return usage).
|
||||
**This is a curation/latency problem for ~3 videos, NOT a throughput/429 problem.** Fetching 3
|
||||
captions is nowhere near the rate limit; the wall only bites at hundreds. So nothing here fetches
|
||||
harder or touches the rate-gate budget meaningfully — it's about picking the *right* few videos
|
||||
and making sure they actually land. Do NOT turn this into a bulk-fetch / rate-pressure change.
|
||||
|
||||
## PHASE 1 — verify what actually happens on connect today. REPORT and STOP before building.
|
||||
The maintainer does not know if the existing onboarding burst is even firing. Establish ground
|
||||
truth:
|
||||
1. **Does the connect-time summarize burst still fire?** There was a cap-3 onboarding burst
|
||||
(summarize newest ~3 on YouTube connect, through the gate). Trace the *current* connect flow
|
||||
(OAuth callback → discovery → summarize). Is the burst still wired, or did a later refactor
|
||||
(recency window ADR-020, discovery changes, the ADR-023/024 filters) bypass or break it?
|
||||
2. **If it fires, what does it deliver?** For a realistic new connect: how many of the burst
|
||||
videos actually reach *summarized* vs. land caption-less / queued / rate-limited / weak? Use
|
||||
Jonas's actual onboarding history if traceable (his first summaries — fast, or trickled?).
|
||||
3. **How much transcript-cache overlap exists between users?** Query the shared transcript store
|
||||
(ADR-021): for the existing users, how many videos does a *new* user's subscription set
|
||||
already have cached transcripts for? This decides whether "cached-first" (instant, zero-fetch
|
||||
wow) is a real lever or nearly empty at pilot scale. **This number drives the Phase-2 blend.**
|
||||
4. **What cheap "likely-good" signals are already available at connect?** ADR-023 already pulls
|
||||
Data API `videos.list` metadata (duration, liveBroadcastContent) — is caption-availability or
|
||||
anything quality-predictive already in hand, or what would it cost to know "this video has
|
||||
captions / reasonable length" *before* spending a fetch?
|
||||
|
||||
**Report findings + a recommended Phase-2 shape, then STOP.** The build differs sharply by what
|
||||
Phase 1 finds:
|
||||
- burst not firing → fix is "make it fire" (and Phase 2 selection is gravy).
|
||||
- fires but picks bad videos (caption-less/weak) → fix is the selection logic below.
|
||||
- fires and works fine → the wow gap is *quality*, not selection → use the stronger model for the
|
||||
burst (see Phase 2 option C), maybe nothing else.
|
||||
- near-zero cache overlap → drop cached-first; Phase 2 is just "likely-good fresh".
|
||||
|
||||
## PHASE 2 — improve the burst (design AFTER Phase 1 findings; maintainer picks the blend)
|
||||
Candidate levers (the maintainer chose 1+2; final blend decided by Phase 1's overlap number):
|
||||
- **(1) Cached-transcript-first:** fill burst slots from videos whose transcripts are ALREADY in
|
||||
the shared store (ADR-021) — instant, zero fetch, guaranteed-captioned. Strength depends on the
|
||||
Phase-1 overlap number.
|
||||
- **(2) Likely-good selection (not just newest):** for uncached slots, pick videos predicted to
|
||||
summarize well — has-captions (use/extend the ADR-023 metadata path), reasonable length (not a
|
||||
3h podcast that truncates to mush, not a <2min clip), from the user's more-engaged channels if
|
||||
that signal exists. Skip newest-but-caption-less rather than burning a wow slot on a "no
|
||||
transcript" card.
|
||||
- **(C) Stronger model for the burst only:** run the ~3 onboarding summaries through
|
||||
`gemma4-26b` (stronger) instead of `phi4-mini`, even though slower — first impressions matter
|
||||
disproportionately and it's only 3 videos, so quality > speed here. Cheap to try, possibly the
|
||||
highest-wow lever if Phase 1 shows the burst fires but summaries underwhelm.
|
||||
- **Honest framing:** present the burst as "summaries of a few of your videos to get you started —
|
||||
the rest fill in over the coming days," so "only a few" reads as intentional taster, not failure.
|
||||
|
||||
**The blend (cached-instant vs. relevant-fresh) is explicitly deferred to the maintainer after
|
||||
Phase 1** reports the overlap number — do not hard-code it.
|
||||
|
||||
## Boundaries
|
||||
- NOT a throughput/429 change — it's ~3 videos; do not bulk-fetch or pressure the rate gate.
|
||||
- No credentials-based caption fetch (ADR-010/026 dead end — auth doesn't work on timedtext).
|
||||
- No client-side fetcher / extension (out of scope; that's a throughput idea, not a wow idea).
|
||||
- Respect ADR-021 (read cached transcripts, never re-fetch), ADR-023/024 (don't undo the filters).
|
||||
- Investigate-first: no selection-logic changes until Phase 1 is reported.
|
||||
|
||||
## Tests (Phase 2, once shaped)
|
||||
- Burst fires on connect and summarizes the selected set through the gate.
|
||||
- Cached-transcript videos in the burst summarize with NO fetch (assert zero caption calls for
|
||||
those).
|
||||
- Caption-less newest videos are NOT chosen for the burst (no "no transcript" wow-killer cards).
|
||||
- (If model lever) burst uses the configured stronger model; rest of pipeline unchanged.
|
||||
Reference in New Issue
Block a user