feat(discovery): event-trigger discovery on YouTube connect (+ throttled login refresh) #6

Closed
opened 2026-06-09 17:42:20 +00:00 by mathias · 1 comment
Owner

Goal

Make a user's videos appear promptly after onboarding, instead of only on the 2h TAPIR_DISCOVERY_INTERVAL scheduler tick (ADR-018). Surfaced by onboarding a second user (Jonas): he connected YouTube but saw nothing until the next scheduled pass — and onboarding shouldn't feel broken.

Recommendation (opinionated)

1. Trigger on YouTube connect — YES, do this.
On the successful /oauth/youtube/callback exchange, enqueue a discovery pass for that one user. Event-driven, scoped, cheap, and it directly fixes the "connected but empty" onboarding gap. This is the high-value trigger.

2. Trigger on login — only as a THROTTLED staleness refresh, not unconditional.
Firing a full discovery on every login is a footgun:

  • YouTube timedtext/caption fetching is per-IP rate-limited (the known 429 we already back off from). Frequent logins → 429 storms.
  • Wasteful repeats; a user reloading the page shouldn't re-poll YouTube.

Acceptable form: trigger on login only if the user's last successful discovery is older than a staleness threshold (e.g. ≥ the discovery interval). Otherwise skip. Prefer a manual "Discover now" button for explicit user control over a chatty login hook.

3. Concurrency is the real constraint — must serialize.
The scheduler assumes a single replica and runs users serially (deployment warns replicas: 1 is load-bearing — >1 double-runs discovery, ADR-018). An event-triggered per-user pass running concurrently with the scheduler would double-fetch and self-inflict 429s. So event-triggered discovery must share one global discovery lock / work queue with the scheduler — enqueue, don't run inline.

Scope

  • Connect-callback → enqueue per-user discovery (in-process queue, deduped per user)
  • Single discovery worker honoring TAPIR_FETCH_RATE (no concurrent passes)
  • Optional: login staleness-gated refresh + a "Discover now" button
  • Metrics/log line per triggered pass (source: connect | login | schedule)

Out of scope

Multi-replica / leader election (separate, ADR-018 follow-up).

## Goal Make a user's videos appear promptly after onboarding, instead of only on the 2h `TAPIR_DISCOVERY_INTERVAL` scheduler tick (ADR-018). Surfaced by onboarding a second user (Jonas): he connected YouTube but saw nothing until the next scheduled pass — and onboarding shouldn't feel broken. ## Recommendation (opinionated) **1. Trigger on YouTube connect — YES, do this.** On the successful `/oauth/youtube/callback` exchange, enqueue a discovery pass **for that one user**. Event-driven, scoped, cheap, and it directly fixes the "connected but empty" onboarding gap. This is the high-value trigger. **2. Trigger on login — only as a THROTTLED staleness refresh, not unconditional.** Firing a full discovery on every login is a footgun: - YouTube `timedtext`/caption fetching is per-IP rate-limited (the known 429 we already back off from). Frequent logins → 429 storms. - Wasteful repeats; a user reloading the page shouldn't re-poll YouTube. Acceptable form: trigger on login **only if** the user's last successful discovery is older than a staleness threshold (e.g. ≥ the discovery interval). Otherwise skip. Prefer a manual **"Discover now"** button for explicit user control over a chatty login hook. **3. Concurrency is the real constraint — must serialize.** The scheduler assumes a **single replica** and runs users serially (deployment warns `replicas: 1` is load-bearing — >1 double-runs discovery, ADR-018). An event-triggered per-user pass running **concurrently** with the scheduler would double-fetch and self-inflict 429s. So event-triggered discovery must share one global discovery lock / work queue with the scheduler — enqueue, don't run inline. ## Scope - [ ] Connect-callback → enqueue per-user discovery (in-process queue, deduped per user) - [ ] Single discovery worker honoring `TAPIR_FETCH_RATE` (no concurrent passes) - [ ] Optional: login staleness-gated refresh + a "Discover now" button - [ ] Metrics/log line per triggered pass (source: connect | login | schedule) ## Out of scope Multi-replica / leader election (separate, ADR-018 follow-up).
Author
Owner

Shipped in v0.10.0 (deployed via infra c17c641, image tapir:e4c701c; verified live: "connect-triggered discovery enabled").

A YouTube connect now fires an immediate discovery pass for the connecting user, so videos appear promptly instead of after the next 2h scheduled pass — closing the onboarding gap that started this thread (a new user connected, saw nothing, read as broken).

Concurrency handled: scheduled and connect-triggered passes share one lock (serialize), so they never fetch concurrently (single-fetcher invariant, ADR-018); the trigger interleaves between the scheduler'''s per-user passes and is non-blocking for the request.

Scope note: connect-trigger only. The optional login-refresh / "Discover now" button were intentionally NOT built — an unconditional login hook risks 429 storms (per this ticket'''s own recommendation). File a follow-up if a manual refresh button is wanted. Closing.

Shipped in **v0.10.0** (deployed via infra c17c641, image tapir:e4c701c; verified live: "connect-triggered discovery enabled"). A YouTube connect now fires an immediate discovery pass for the connecting user, so videos appear promptly instead of after the next 2h scheduled pass — closing the onboarding gap that started this thread (a new user connected, saw nothing, read as broken). Concurrency handled: scheduled and connect-triggered passes share one lock (serialize), so they never fetch concurrently (single-fetcher invariant, ADR-018); the trigger interleaves between the scheduler'''s per-user passes and is non-blocking for the request. Scope note: connect-trigger only. The optional login-refresh / "Discover now" button were intentionally NOT built — an unconditional login hook risks 429 storms (per this ticket'''s own recommendation). File a follow-up if a manual refresh button is wanted. Closing.
Sign in to join this conversation.
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/tapir#6