Pure parser for watch?v=, youtu.be/, shorts/, embed/, and bare ids; rejects
non-YouTube hosts and malformed input. Foundation for paste-a-URL summarize
(Feature 2). No fetch, no gate interaction — parsing only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A newly connected account showed no videos until the next 2h scheduled pass —
the gap that made onboarding look broken (a second user connected, saw nothing,
read as failure). The connect callback now fires an out-of-band discovery pass
for the connecting user, so videos appear promptly.
Concurrency: scheduled and connect-triggered passes share one lock (serialize),
preserving the single-fetcher invariant (ADR-018). A trigger interleaves between
the scheduler's per-user passes rather than fetching concurrently or waiting for
a whole pass. The trigger runs on the server ctx (survives the redirect) and is
non-blocking for the request goroutine.
Scope: connect-trigger only. The optional login-refresh / "Discover now" button
from #6 are intentionally not built — an unconditional login hook risks 429
storms (per the ticket's own recommendation); defer until wanted.
TDD: TestCallbackTriggersDiscovery, TestSerializeRunsOneAtATime,
TestDiscoveryTriggerEnqueueRunsUser; new BDD scenario mapped.
Refs #6
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The scheduler enumerates every user_identities row (ListAllUsers) and ran a
discovery pass for each — including users who never connected a video source.
Their per-user runner then tried to resolve a YouTube refresh token that was
never minted, logging a spurious "secrets: ref not found:
youtube/<uid>/refresh_token" every tick (e.g. stale Dex-era orphan identities
left by the Authentik migration).
Skip users whose ConnectionsForUser is empty before running their pass. Removes
the recurring noise — which actively misled a debug session into thinking a
healthy onboarded user was broken — with no change to connected users.
Refs #7
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Close the gap where docs/use-cases/*.feature claimed to be the behavior spec
but nothing executed them — so scenarios drifted (the stale "auto summarizes
every new video" and "manual is the default" were proof).
Decision (per issue #5 BDD-runner fork): no godog — keep .feature as design
records, add a cheap name-coverage gate instead. TestScenarioCoverage parses
every scenario and asserts each non-@pending one maps to an existing Go test in
the scenarioCoverage manifest; it flags unmapped scenarios, missing/renamed
tests, and stale entries. It checks the link, not that the test exercises the
scenario (the deliberate trade for skipping godog).
Also:
- Fix the stale ADR-018 drift: "Manual is the default" -> auto is the default
for new users; added an explicit default scenario + a plain manual scenario.
- Tag 4 documented-but-unbuilt/untested scenarios @pending with reasons (Vimeo
connect, BYO config flow, logout->welcome, re-register-after-delete) so they
are tracked without a false coverage claim.
- CLAUDE.md BDD section now describes the real setup (design records + the gate
+ @pending convention) instead of claiming an executable spec.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bring the living docs current with the recency-bounded auto-summarize + sparse
honesty + feed IA bundle (ADR-020):
- requirements (BDD): summarize_mode.feature — auto now summarizes RECENT new
videos; added a scenario for older videos (listed, on-demand), recency note.
- architecture.md: summarization-mode + new list-surface paragraph; scheduler
diagram + two-path table + three-phase pass now show the recency pre-filter;
dropped stale "Summarize now".
- data-model.md: auto_summarize is recent-only, older on-demand.
- README.md: one-line recency note on the serve scheduler.
- ui-spec.md: appended the as-built ADR-020 row (supersedes earlier copy/sort).
- specs/{video-card-states,newest-first-ordering,scheduled-discovery}.md:
superseded/extended banners pointing at ADR-020 (kept as design records).
Docs-only; task check green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document the architecture decision behind this bundle: bound auto-summarize to
a recency window (refines ADR-018; bounds load against the ADR-014 gate without
fetching harder), surface scarcity honestly, and collapse the un-summarized
back-catalogue in a single feed. Records the return-nudge as a deliberate
non-goal (it would contaminate the Stage-0 unprompted-return signal, ADR-016).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend the card copy guard so "Try now", "Summarize now", "Fetching soon", and
"the next run" can't silently return to any card state (UX review honesty pass).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Watched and skipped are mutually exclusive (the store clears one when the other
is set), but rendered as three independent-looking buttons the exclusivity was
invisible. Group watched|skipped into a single segmented control and keep Saved
apart as an independent toggle (UX review C5). HTMX posting and the active/✓/
aria-pressed semantics are unchanged; extracted a shared actionButton component.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The register step asked the user to accept "the terms of use" with no terms
linked anywhere — ceremony accepting nothing on a friends-only tool (UX review
C4). Remove the checkbox and the server-side acceptance requirement; only a
display name is required now.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The summary detail page only returned to the list via the brand logo. Add an
explicit "← Summaries" link at the top (UX review C3).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
At current scale the date-range pickers are dead weight (UX review C1/C2):
- Drop the From/To date inputs from the filter bar; keep the Channel field and
the "Summarized only" toggle. (Filter still parses from/to for hand-built
URLs and apply() compatibility — only the UI is removed.)
- Hide the filter bar entirely on a genuinely empty account (no rows AND no
active filter) so the connect CTA stands alone; a filter that matches nothing
still shows the bar so it can be cleared.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stop the un-summarized back-catalogue from burying the readable summaries
(UX review B3/B4). One feed, with a noise-collapse — not sections:
- Summarized + recent un-summarized videos lead inline as cards.
- Un-summarized videos older than the recency window collapse into a single
"Show N older videos — summarize on demand" disclosure (they will not
auto-fill; they are manual-only). Window comes from App.RecencyWindow
(= cfg.AutoSummarizeWindow); 0 disables the collapse (all inline).
- Caption-less videos collapse into one honest line ("N videos have no
captions and can't be summarized") instead of N dead terminal cards.
bucketRows is a pure classifier (cutoff-driven; undated rows never age out);
App gains RecencyWindow + an injectable clock for the cutoff.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ListVideos now sorts summarized-first, then published_at DESC with undated
videos last, then seen_at DESC as a tiebreak (was seen_at only). Aligns the
list with the recency framing — newest content surfaces first — so the
recency-bounded feed reads coherently (UX review B2).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In automatic mode the scheduler now only summarizes videos published within
TAPIR_AUTO_SUMMARIZE_WINDOW (default ~7d). Older videos are still discovered
and listed — they keep the manual "Summarize" affordance — but are not
auto-processed, so a large back-catalogue (the maintainer's ~256-deep queue)
stops self-inflicting 429s against the per-IP caption gate each cycle (UX
review B1, recency design).
- runner.WithAutoWindow + Stats.SkippedTooOld; tooOld() treats a zero window
as disabled and an undated video as never-aged-out (processed, not stranded).
- An explicit manual request bypasses the bound even in auto mode (requested
videos are loaded in auto mode when a window is active).
- Wired through cmdRun, the scheduler's per-user runner, sumStats, and pass
logging. config: TAPIR_AUTO_SUMMARIZE_WINDOW (default 168h), .env.example.
- Account copy (A7) updated to match: "Automatic summarizes new videos from
about the last week; older videos stay browsable — summarize on demand."
The rate gate is untouched; the manual path still serialises through it. This
bounds auto LOAD, it does not fetch harder.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The product promise is "decide what's worth your time", but the detail page
buried Takeaways — the verdict that answers that — below the full Summary.
Reorder to Takeaways → Highlights → Summary so the attention-saving payload
leads (UX review A8). Data already existed; this is a section reorder only.
Conditional sections mean a video without takeaways/highlights still leads
with the Summary naturally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The landing page promised "if you have an invite link, it will set up your
account automatically" — but invites moved to Authentik (ADR-019); Tapir no
longer handles invite links and "Get Started" goes straight to OIDC. Replace
with honest "invite-only — if you've been invited, sign in" and set the
gradual-fill expectation before the login wall (UX review A5). Test pins the
stale phrase out.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the sparse reality legible instead of implying abundance or imminence
(UX review A1-A4, A6):
- Empty-connected state drops the impossible "Run `tapir run`" instruction
(no shell for web users; discovery is in-process since ADR-018) for a
passive "summaries appear gradually, check back later".
- Pipeline bar reframes counts by what the user can do: "N ready · M in queue
· K no captions" (was "summarized / fetching soon / pending").
- A one-line note explains captions are fetched slowly on purpose to respect
YouTube's limits — turning confusing emptiness into intentional design.
- Card state for throttled videos reads "In queue", not "Fetching soon…"
(256 items behind a per-IP gate are not all imminent — ADR-014).
- Quiet nudge button drops the over-promising "now": "Summarize", not
"Summarize now". On click the card still honestly becomes "Queued".
- Queued card says "summarizing shortly", not "waiting for the next run"
(no scheduler jargon).
Pure copy/label — no logic, DB, or fetch-rate change. The rate gate is
untouched; scarcity is surfaced, never engineered around.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prioritized UX findings for the product as it actually is — sparse feed,
respected caption rate limit, recency-bounded auto-summarize (incoming),
single-user. 15 findings, NOW/LATER tagged. P0s target the first-contact
return-cliff that the Stage-0 gate depends on.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Record the invite-provisioning removal; mark ADR-017 superseded; fix
ui-spec invite-onboarding + auth-delegation sections to reflect Authentik.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Authentik owns invites now (infra ADR-0001). Delete adapters/dex, the
/invite set-password UI, the tapir invite CLI, the InvitationStore/
DexPasswordCreator ports + App wiring, the invite Templ pages, and the
invite Taskfile target. New users are invited via Authentik, log in via
OIDC, and hit the existing /register gate. invitations table (mig 009)
left in place (append-only; harmless). task check green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tapir used human-readable substitutions ('@' -> '-at-', '.' -> '-dot-') when
deriving the Password CR name from an email. Dex's internal passwordID() maps
every non-[a-z0-9-] character to plain '-'. This caused a name mismatch:
Tapir wrote the CR as 'mathias-at-d-ma-dot-be', Dex looked it up as
'mathias-d-ma-be', got not-found, and returned 'Invalid credentials' on every
invite login — while static configmap passwords (a different code path) worked
fine. Diagnosed by adding the email to staticPasswords and confirming login
succeeded, proving the kubernetes CR lookup was the failure point.
The original NOTE claimed Dex's kubernetes storage types Hash as []byte,
requiring the bcrypt string to be base64-encoded before storage. This was
wrong: Dex v2.41 stores and compares the hash field as a plain string. The
base64-encoding caused every invite login to fail with 'Invalid credentials'
because Dex passed the base64 bytes (starting with 'J' not '$') directly to
bcrypt. Static passwords in the configmap always used raw bcrypt strings and
worked fine — confirming the dynamic CR encoding was the bug.
Five explicit footer states, status-primary:
1. Summarized — chip + actions, no button (unchanged)
2. No captions (TranscriptStatus=="none") — NEW: 'No transcript available' muted
text, no button, no POST URL. Removes the dead-end 'Summarize' button that
tried and failed when there were no captions to fetch.
3. Queued (SummarizeRequested) — chip + muted text, no button (unchanged)
4. Rate-limited — 'Fetching soon…' + quiet 'Summarize now' → /retry-now
5. Pending — 'Not summarized' + quiet 'Summarize now' → /summarize
One verb ('Summarize now'), one quiet style (.btn-quiet, renamed from .btn-retry
which was state-specific). User doesn't see the internal pipeline distinction;
both buttons post to their existing handlers unchanged. Form class renamed
card-nudge-form. Dropped engineer-facing tooltip; user-facing hint added.
'Try now' wording removed entirely.
The card shows two verbs ("Try now" for rate-limited, "Summarize" for pending)
for one user intent, and — the real bug — a no-captions video falls into the
pending branch and wrongly shows a Summarize button that can only fail. Spec
collapses to one quiet "Summarize now" verb wherever a nudge is possible (both
handlers unchanged underneath), adds an honest no-button "No transcript
available" state, and keeps the card status-first (buttons are exceptions in auto
mode). View-layer only.
'Try now' and the newest-first batch implement onboarding prioritisation:
foreground (user-clicked 'Try now') summarises a chosen video on demand;
background batch summarises newest-first; both honour the shared rate gate.
Remove any prior framing that described 'Try now' as making traffic 'look
organic to YouTube' or as rate-limit evasion — that was not the rationale
and contradicts ADR-014's explicit account-safety constraint.
Correct statement: rate limiting is respected, not evaded. TAPIR_FETCH_RATE
and TAPIR_FETCH_BACKOFF are honest rate controls; they govern how fast Tapir
fetches captions, not how the requests appear to YouTube.
Architecture: add two-path model table (foreground/background, both through
globalFetchGate) and newest-first batch ordering doc (three-phase RunOnce,
before/after example).
ui-spec: add 'Try now' row with correct rationale; add pipeline stats bar row;
update Summarized-only filter row to mention sort-to-top.
Restructures RunOnce from per-channel inline processing to collect-sort-process:
Phase 1 — discover, persist (UpsertVideo), apply pre-filters (seen/manual/backoff)
and collect surviving candidates with their discovery position.
Phase 2 — sort candidates by published_at DESC, NULLS LAST, pos ASC tiebreak
so videos with no publish date never jump ahead of dated content.
Phase 3 — process in sorted order through the unchanged globalFetchGate.
Before (per-channel): chanA=[v-old, v-mid], chanB=[v-new, v-null]
→ [v-old, v-mid, v-new, v-null]
After (newest-first): [v-new, v-mid, v-old, v-null]
Same set of videos processed; only the order changes within a pass. All existing
behaviour is preserved: failure isolation, backoff skip, manual mode,
channel-unavailable, stats. In-memory sort; no new table or persisted queue.
The ordering is onboarding prioritisation — new users get summaries of their most
recent, relevant videos first; the back-catalogue fills in behind across subsequent
passes. Both this background batch and the foreground 'Try now' button honour the
shared globalFetchGate: rate limiting is respected, not evaded.
Product intent: new users get summaries of their newest videos fast while the
back-catalogue fills behind, within the shared rate gate. The background batch
currently processes in subscription/channel order, not newest-first — this spec
closes that gap (collect candidates, sort published_at DESC NULLS LAST, process
in order, gate unchanged). Also corrects the docs to describe Try-now as
onboarding prioritisation, explicitly removing the prior "looks organic to
YouTube" traffic-disguising framing — rate limiting is respected, not evaded.
Three UX improvements for the pending-transcript state:
1. Summarized videos sort to top (ORDER BY (s.id IS NOT NULL) DESC, seen_at DESC)
so completed summaries are always immediately visible without filtering.
ListVideos default limit raised from 50 to 500 to show the full backlog.
2. Pipeline stats bar above the video list: '2 summarized · 256 fetching soon · 12
no captions' — computed from the unfiltered row set, hidden when everything is
summarized.
3. 'Try now' button on rate-limited cards replaces the passive 'Retrying later'
chip. POST /v/{id}/retry-now clears rate_limited_at then calls ProcessVideo
through the shared globalFetchGate — same rate limiting as the scheduler, safe
under concurrent use.
YouTube channels that 404 on playlist discovery (deleted/private) are now:
1. Wrapped in domain.ErrChannelUnavailable by the YouTube adapter (instead of
a generic error), so the runner can identify them without string-matching.
2. Stored per-user in channel_errors (migration 013, RLS-guarded) via runner's
new UpsertChannelError path — removed from the generic Errors counter,
counted separately as ChannelUnavailable.
3. Shown on the account page under "Unavailable channels" with name, chip-warn
badge, and first-seen date, so users know why some subscribed channels
produce no videos.
Migration 011's UPDATE ran without tapir.current_user_id set, so FORCE RLS
blocked all rows and 0 users were updated (skipped_manual=607 in scheduler).
Migration 012 temporarily drops FORCE so the table owner can run the UPDATE,
then restores it.
homelab-integration.md gains a "Scheduled discovery" section documenting
TAPIR_DISCOVERY_INTERVAL and TAPIR_FETCH_RATE and the load-bearing
single-replica constraint (in-process scheduler → replicas: 1 is required;
>1 double-runs discovery). VISION Stage 0 carries a pointer to ADR-018's
gate-clock reset so nothing in docs implies the window started before
unprompted use was possible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migration 011 flips the auto_summarize column default to TRUE and brings
existing rows (maintainer + current registrations) along. Onboarded friends
now get zero-friction discovery: scheduled discovery (ADR-018) both discovers
AND summarizes new videos, so a user's list fills and summarizes itself
instead of presenting an empty list of manual Summarize buttons.
Safe only because the process-wide caption-fetch rate gate (ADR-014 item 2,
prior commit) now exists — auto + scheduled + multi-user would otherwise
self-inflict 429s every cycle. The down migration reverts the default but
intentionally leaves existing rows as-is (no surprise manual regression on
rollback). RegisterUser already lets the column default drive the value, so
no app change is needed; the account-page manual toggle still works.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stage 0's "returns and reads in >=2 weeks" gate can't be met while discovery
is host-side manual (`tapir run`): a newly onboarded user sees an empty list
and never comes back. Make Tapir watch on its own.
cmdServe launches a background goroutine (when TAPIR_DISCOVERY_INTERVAL > 0)
that runs a discovery pass for ALL users on that cadence: enumerate via the
un-RLS'd ListAllUsers, then run each user's pass through the EXISTING
runner.Runner — the only new code is the per-user loop, not a new scheduler.
Run-once-on-startup then ticked; ctx-cancelled on SIGTERM; per-user failures
(including buildUserRunner errors) are logged and skipped so one bad user
never aborts the rest. interval <= 0 disables it entirely (dev/tests).
buildUserRunner binds each runner to that user's own YouTube refresh token
(web.YouTubeTokenRef) — the Stage-1 per-tenant ref — reusing buildProcessor's
engine wiring. SetFetchRate is also wired in cmdServe so the click-path shares
the gate.
SINGLE-REPLICA is now load-bearing: the loop lives in the web process, so >1
replica double-runs discovery (429s + duplicate work). Documented in cmdServe
and warned at startup.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The in-process scheduler (ADR-018) needs to enumerate every user to run a
discovery pass each. user_identities is the un-RLS'd map; add ListAllUsers as
a plain pool query (no withUser) — the same enumerate-then-act pattern
UserBySubject and the login_events gate query established. Scoping it to a
single user would defeat the point; user_identities carries no RLS by design.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ADR-014 item 2 — a single per-egress-IP rate gate shared by every caption
fetch — was specced but only per-video backoff (rate_limited_at) shipped.
Build the real gate now: it is load-bearing once ADR-018 puts auto-summarize
on an in-process schedule across multiple users (all fetches leave one pod's
egress IP, concurrently with live "Summarize" clicks — without a shared gate
that self-inflicts 429s every cycle).
globalFetchGate (golang.org/x/time/rate, default 2s/req burst 1) is consulted
in httpDo before every live outbound fetch — player, watch-page, timedtext —
so the scheduler runners and the web click-path serialise through one limiter
regardless of how many users/goroutines are upstream. The test seam
(a.transport != nil) skips the gate so fakes are not throttled.
TAPIR_FETCH_RATE (Go duration, default 2s, 0 = unlimited) wires SetFetchRate in
cmdRun; the existing per-video backoff stays as the complementary 429 handler.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Records the Stage-0 usability decision: tapir serve runs discovery for all users
on an interval (reusing runner.Loop), auto-summarize defaults ON so the list fills
itself, and the gate clock resets to when this ships (unprompted use was
impossible before, so the prior window measured nothing — framed as starting the
clock when the experiment can run, not dodging a failing gate). Records the
in-process-vs-CronJob tradeoff, the load-bearing single-replica constraint, and
that it makes ADR-014 item 2 (process-wide rate gate) a hard requirement folded
into the build. Cross-referenced ADR-014/016; added CronJob to rejected-alts.
The Stage-0 usability fix: tapir serve runs discovery for all users on an
interval (reusing the existing Runner.Loop, enumerate-users-then-withUser),
auto-summarize defaults ON for Future-B users so the list fills itself, and the
ADR-014 process-wide per-egress-IP rate gate is confirmed/finished in the same
slice because in-process + auto + multi-user makes it load-bearing. Records the
single-replica constraint as load-bearing, and a fallback (auto-summarize OFF
until the gate exists) so the dangerous combination never ships half-built.
Records the deliberate keep-or-reverse decision on the Dex-write invite flow.
Decision: KEEP. Deciding fact: not all intended Future-B users will use Google
accounts, so Google OIDC alone can't onboard them — the invite flow is
load-bearing, not redundant. Trust-surface cost accepted deliberately, explicitly
NOT as a precedent for widening further, and explicitly NOT by adding delete RBAC
to fix the orphan gap. Open items reframed as tracked follow-ups (verify RBAC
against the real manifest; accept orphan for Future B, revisit before Future C).
Added the reversal to rejected-alternatives.
mathiasandClaude Opus 4.8