YouTube channels that 404 on playlist discovery (deleted/private) are now:
1. Wrapped in domain.ErrChannelUnavailable by the YouTube adapter (instead of
a generic error), so the runner can identify them without string-matching.
2. Stored per-user in channel_errors (migration 013, RLS-guarded) via runner's
new UpsertChannelError path — removed from the generic Errors counter,
counted separately as ChannelUnavailable.
3. Shown on the account page under "Unavailable channels" with name, chip-warn
badge, and first-seen date, so users know why some subscribed channels
produce no videos.
Migration 011's UPDATE ran without tapir.current_user_id set, so FORCE RLS
blocked all rows and 0 users were updated (skipped_manual=607 in scheduler).
Migration 012 temporarily drops FORCE so the table owner can run the UPDATE,
then restores it.
homelab-integration.md gains a "Scheduled discovery" section documenting
TAPIR_DISCOVERY_INTERVAL and TAPIR_FETCH_RATE and the load-bearing
single-replica constraint (in-process scheduler → replicas: 1 is required;
>1 double-runs discovery). VISION Stage 0 carries a pointer to ADR-018's
gate-clock reset so nothing in docs implies the window started before
unprompted use was possible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migration 011 flips the auto_summarize column default to TRUE and brings
existing rows (maintainer + current registrations) along. Onboarded friends
now get zero-friction discovery: scheduled discovery (ADR-018) both discovers
AND summarizes new videos, so a user's list fills and summarizes itself
instead of presenting an empty list of manual Summarize buttons.
Safe only because the process-wide caption-fetch rate gate (ADR-014 item 2,
prior commit) now exists — auto + scheduled + multi-user would otherwise
self-inflict 429s every cycle. The down migration reverts the default but
intentionally leaves existing rows as-is (no surprise manual regression on
rollback). RegisterUser already lets the column default drive the value, so
no app change is needed; the account-page manual toggle still works.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stage 0's "returns and reads in >=2 weeks" gate can't be met while discovery
is host-side manual (`tapir run`): a newly onboarded user sees an empty list
and never comes back. Make Tapir watch on its own.
cmdServe launches a background goroutine (when TAPIR_DISCOVERY_INTERVAL > 0)
that runs a discovery pass for ALL users on that cadence: enumerate via the
un-RLS'd ListAllUsers, then run each user's pass through the EXISTING
runner.Runner — the only new code is the per-user loop, not a new scheduler.
Run-once-on-startup then ticked; ctx-cancelled on SIGTERM; per-user failures
(including buildUserRunner errors) are logged and skipped so one bad user
never aborts the rest. interval <= 0 disables it entirely (dev/tests).
buildUserRunner binds each runner to that user's own YouTube refresh token
(web.YouTubeTokenRef) — the Stage-1 per-tenant ref — reusing buildProcessor's
engine wiring. SetFetchRate is also wired in cmdServe so the click-path shares
the gate.
SINGLE-REPLICA is now load-bearing: the loop lives in the web process, so >1
replica double-runs discovery (429s + duplicate work). Documented in cmdServe
and warned at startup.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The in-process scheduler (ADR-018) needs to enumerate every user to run a
discovery pass each. user_identities is the un-RLS'd map; add ListAllUsers as
a plain pool query (no withUser) — the same enumerate-then-act pattern
UserBySubject and the login_events gate query established. Scoping it to a
single user would defeat the point; user_identities carries no RLS by design.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ADR-014 item 2 — a single per-egress-IP rate gate shared by every caption
fetch — was specced but only per-video backoff (rate_limited_at) shipped.
Build the real gate now: it is load-bearing once ADR-018 puts auto-summarize
on an in-process schedule across multiple users (all fetches leave one pod's
egress IP, concurrently with live "Summarize" clicks — without a shared gate
that self-inflicts 429s every cycle).
globalFetchGate (golang.org/x/time/rate, default 2s/req burst 1) is consulted
in httpDo before every live outbound fetch — player, watch-page, timedtext —
so the scheduler runners and the web click-path serialise through one limiter
regardless of how many users/goroutines are upstream. The test seam
(a.transport != nil) skips the gate so fakes are not throttled.
TAPIR_FETCH_RATE (Go duration, default 2s, 0 = unlimited) wires SetFetchRate in
cmdRun; the existing per-video backoff stays as the complementary 429 handler.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Records the Stage-0 usability decision: tapir serve runs discovery for all users
on an interval (reusing runner.Loop), auto-summarize defaults ON so the list fills
itself, and the gate clock resets to when this ships (unprompted use was
impossible before, so the prior window measured nothing — framed as starting the
clock when the experiment can run, not dodging a failing gate). Records the
in-process-vs-CronJob tradeoff, the load-bearing single-replica constraint, and
that it makes ADR-014 item 2 (process-wide rate gate) a hard requirement folded
into the build. Cross-referenced ADR-014/016; added CronJob to rejected-alts.
The Stage-0 usability fix: tapir serve runs discovery for all users on an
interval (reusing the existing Runner.Loop, enumerate-users-then-withUser),
auto-summarize defaults ON for Future-B users so the list fills itself, and the
ADR-014 process-wide per-egress-IP rate gate is confirmed/finished in the same
slice because in-process + auto + multi-user makes it load-bearing. Records the
single-replica constraint as load-bearing, and a fallback (auto-summarize OFF
until the gate exists) so the dangerous combination never ships half-built.
Records the deliberate keep-or-reverse decision on the Dex-write invite flow.
Decision: KEEP. Deciding fact: not all intended Future-B users will use Google
accounts, so Google OIDC alone can't onboard them — the invite flow is
load-bearing, not redundant. Trust-surface cost accepted deliberately, explicitly
NOT as a precedent for widening further, and explicitly NOT by adding delete RBAC
to fix the orphan gap. Open items reframed as tracked follow-ups (verify RBAC
against the real manifest; accept orphan for Future B, revisit before Future C).
Added the reversal to rejected-alternatives.
Reconciliation finding: the v0.6.0 report's "migration 008 videos.rate_limited_at"
was a mislabel. Both transcript_status and rate_limited_at shipped in migration
007; the sequence legitimately skips 008, nothing was lost, and the runner's
column reads are sound. Updated the ADR-014 implementation note to state this
(was flagged as an unresolved discrepancy). The item-2 (shared per-egress-IP rate
gate vs per-video backoff) flag stays open — still unconfirmed.
ActiveWeeks computes per-user distinct active weeks (reads login_events UNION
acts summary_actions) for the gate (VISION/ADR-016: usage in >=2 distinct
weeks). Because the user-owned tables are FORCE RLS under a non-superuser owner,
a single cross-user query is deny-all; instead it enumerates users from the
un-RLS'd identity map and counts each inside withUser — no privilege escalation,
no policy change.
`tapir report` prints the per-user table and the pass/fail verdict (needs only
TAPIR_DB_DSN). Pure formatter + store query are unit-tested, including the
cross-table shared-week dedup.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The registration gate, once it resolves the authenticated subject to a tapir
user_id, calls StampLogin (store-throttled to one row per user per day). Best-
effort: a stamp failure is logged and swallowed so it never breaks the request.
This is what makes the read-side Stage-0 usage signal actually accrue.
Tests cover the happy-path stamp, the same-day throttle, and that an
unregistered subject (redirected to /register) is never stamped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
StampLogin appends one login_events row per user per day via an atomic
INSERT ... SELECT ... WHERE NOT EXISTS, run through withUser so the throttle
probe is itself RLS-scoped to the caller. DeleteUser now deletes login_events
explicitly (no FK = no cascade — the summary_actions footgun, repeated).
Extends the two-user RLS isolation proof and the delete-account proof to cover
login_events, and adds throttle / new-day / user-scoping tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stage-0 usage measurement (VISION/ADR-016): summary_actions captures acts
(watch/skip/save) but not reads. A reader who logs in weekly and clicks
nothing is invisible — for a reading product that return is the signal the
gate ("usage in >=2 distinct weeks") is defined on. login_events records
THAT a user was active, append-only, one row per user per active day.
Per-user isolation via the same GUC-keyed FORCE RLS policy as migration 003.
No FK to users (mirrors summary_actions) — the cascade footgun is handled by
DeleteUser in a later commit. Adds an up/down reversibility test and registers
the table in both truncate helpers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reconciliation pass after parallel agent sessions shipped v0.6.0/v0.7.0.
ADR-017 documents the v0.7.0 invite flow, which gave Tapir scoped create+get on
passwords.dex.coreos.com in the auth namespace — Tapir now WRITES to the shared
identity provider. This shipped with no ADR; recorded retroactively with the
principle-reversal named (partially supersedes ADR-002/013), the security analysis
(bounded RBAC, but a real larger trust surface), and the open gaps (orphaned Dex
accounts on delete; plaintext invite tokens). Cross-referenced in ADR-002 and
ADR-013 status lines so a future reader isn't misled.
ADR-014 annotated: 429 handling shipped in v0.6.0 but decision item 2 (shared
per-egress-IP rate gate) appears realised as per-VIDEO backoff, not a process-wide
IP gate — flagged not-confirmed-done. Also flags the missing migration 008 /
rate_limited_at discrepancy (v0.6.0 report cited 008; tree jumps 007->009).
No code changed in this commit — audit trail only.
Logged-out visitors on /welcome and /invite should not see Account or
Log out. Split Layout into Layout (authenticated, full nav) and
PublicLayout (public, brand-only header). WelcomePage + InvitePage
variants now use PublicLayout.
The Stage-1 onboarding path: an invited user opens their emailed link,
sets a password, and Tapir creates their Dex local-password account so
they can log in. Mounted on root OUTSIDE Auth.Middleware — the visitor
has no Dex session yet; the token in the path is the capability.
handleInviteForm previews the token (no consume) and shows the form, or
a clear "expired / already used" page. handleInviteSubmit validates the
password BEFORE consuming the token (a typo is retryable), then claims
the invite exactly once, bcrypt-hashes (cost 12), and creates the Dex
account — mapping ErrPasswordExists -> "log in instead" and ErrForbidden
-> "contact the administrator". Off-cluster (App.Dex nil) it degrades to
a "deployed-only" message without burning the token. On success it sets
an account_created flash and redirects to /auth/login.
Welcome sub-text now states access is invite-only. Handlers depend on
narrow ports (InvitationStore, DexPasswordCreator) so tests use fakes;
cmdServe wires the store + an in-cluster dex.PasswordClient.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mints a single-use invitation and prints the absolute claim URL for the
operator to send. The URL base is TAPIR_PUBLIC_URL (default
https://tapir.d-ma.be). runInvite is factored from config/store wiring so
it's unit-tested against a fake inviter — no Postgres.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Writes passwords.dex.coreos.com CRs against the in-cluster Kubernetes
API using the pod's service-account token + cluster CA (no kubectl /
client-go dependency). NewPasswordClient returns ErrNotInCluster off
cluster so the web layer degrades gracefully in dev.
Load-bearing: Dex's kubernetes storage types Password.Hash as []byte,
which k8s JSON-marshals as base64 — so the `hash` field carries the
base64 of the bcrypt string, not the raw string. Storing the raw string
makes Dex's base64-decode-on-login produce garbage and every login fail.
409 -> ErrPasswordExists, 401/403 -> ErrForbidden (RBAC missing) so the
handler can give precise messages. Tested against an httptest TLS server.
bcrypt cost-12 hashing lives in the web handler; golang.org/x/crypto was
already a transitive dep (now promoted in go.sum).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stage-1 email onboarding: Mathias mints an invite, the recipient claims
it to set a Dex password. Invitations exist before their user, so the
table carries no user_id FK and is deliberately outside RLS — the
32-byte crypto-random token is the capability (single-use, time-boxed).
ClaimInvitation consumes atomically (UPDATE ... WHERE used_at IS NULL
... RETURNING) so concurrent claims of one token can't both succeed.
PeekInvitation validates the link for the form without consuming it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Small tapir slice to make the gate measurable as written: append-only
login_events (RLS, per-user-per-day throttle) + a union query over reads
(login_events) and acts (summary_actions) for distinct-active-weeks. Carries the
honesty caveats (unprompted not measurable; data accrues from deploy; week-bucket
noise at low N) and the delete-cascade footgun (no FK, needs explicit delete +
test) from the prior delete work. Out of scope: analytics, prompt-tracking,
dashboards.
Reframes "unprompted" from an enforced criterion to a named measurement
limitation: organic-vs-prompted returns aren't distinguishable from any data
Tapir holds, so in practice all returns are counted and the result read with that
caveat (a nudged return is a weaker signal). Adds a "how it's measured" note
pointing at summary_actions (acts) + a new append-only login-events table
(read-returns), which accrue from deploy onward. Honest about the gap rather than
silently dropping the word.
A discovered-but-unsummarized video whose caption fetch was rate-limited now
shows a passive ⏳ "Retrying later" chip (dim CharmDim styling, not the accent)
instead of the Summarize button — the user cannot fix a 429, the runner retries
automatically once the backoff window expires. Regenerated views_templ.go.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After ProcessNewVideo the runner records transcript_status per outcome:
rate_limited (stamps the backoff clock), none, or fetched. Before fetching, a
video inside the TAPIR_FETCH_BACKOFF window is skipped (SkippedRateLimited) so a
just-429'd caption endpoint is not re-hit; once the window expires it retries.
Backoff/clock injected via variadic Options (WithBackoff, WithClock) so existing
New call sites and the fake-driven loop tests stay valid. Backoff 0 = always retry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds FetchBackoff (Go duration, default 1h) controlling how long the run loop
waits before re-fetching a transcript that returned HTTP 429. Zero means always
retry. Not required by ValidateForRun — a zero/unset value is a valid policy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The engine already distinguishes SourceNone from SourceRateLimited internally
but collapsed both into Skipped. Expose the source string so the runner can
persist the right transcript_status and apply rate-limit backoff, without the
engine taking on any store/retry concern (dependencies still point inward).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surfaces videos.transcript_status (migration 007) on SummaryRow and adds
SetTranscriptStatus / GetTranscriptStatus / RateLimitedVideoIDs.
SetTranscriptStatus is the single choke point for the rate-limit lifecycle:
"rate_limited" stamps rate_limited_at = NOW(), every other status clears it,
so the runner's backoff window and the UI badge read one consistent source.
RateLimitedVideoIDs is the per-pass loader (mirrors SeenVideoIDs) the runner
uses to skip still-throttled videos without re-hitting the caption endpoint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Logout was only reachable from /welcome. Users who are logged in had no way
to sign out from any app page (list, detail, account). Added to the shared
nav alongside Account.
Records the gate change: Stage 0 now passes when either the maintainer or an
onboarded friend returns unprompted in >=2 separate weeks. Behavioural (return
usage), not feedback-based, to resist politeness bias. Includes an honest
self-scrutiny note that this is a guardrail edit made while the original gate was
unmet — examined on that basis and proceeding because it broadens who supplies the
signal without softening the kind of signal required. Adds feedback-based-gate to
rejected alternatives.
Adds videos.transcript_status (NULL|none|fetched|rate_limited) and
videos.rate_limited_at, so the runner can record a 429 and skip re-fetching a
still-throttled video until a backoff window elapses. Columns inherit the
existing videos RLS policy (migration 003); no policy change needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The baseUrl fetch mapped every non-200 to SourceNone, recording a 429 as a
permanent "no captions". 429 is the IP being rate-limited, not an absent
transcript. Return SourceRateLimited (still a graceful degrade, no error) so
the runner can retry after a backoff window. Other non-200s (403/404/5xx)
stay SourceNone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Videos rows are created by `tapir run`, not when a YouTube account is
connected, so a freshly-connected account correctly shows an empty list —
but the old empty state ("No videos yet") gave no clue why or what to do.
Split it on whether the user has any connection:
- connected, no videos: a distinct accent callout telling them to run
`tapir run` to discover subscriptions.
- not connected: a prompt with a Connect YouTube button.
handleList fetches connections only when the list is empty. Includes
web-shot captures of all three states under docs/ux-review/fixes/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
429 from the caption endpoint means the IP is rate-limited (retry later),
not that the video has no captions. Distinguishing it from SourceNone is the
prerequisite for the runner's backoff/retry logic.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "Get Started" button drops users straight into the shared Dex flow,
which has no separate "register" option — registration completes
automatically after first login. Users new to Tapir had no signal that
signing in is also how they sign up. Reword the sub-text to say so
explicitly, keeping the single Dex CTA (sign-in and sign-up are one flow).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The .btn class sets color:var(--accent-fg), but the generic a{} and
a:visited{} rules outrank it on <a> elements, so anchor buttons (the
landing "Get Started" CTA, "Connect YouTube") rendered their label
accent-on-accent — invisible. Add a.btn / a.btn:visited to restore the
button foreground.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the original "useful to me, specifically" gate with "me OR a friend
returns unprompted in >=2 separate weeks" — friendly-user signal counts, but the
test stays behavioural (return usage) not feedback-based, to resist politeness
bias. Folds the old Stage 1 ("a trusted user returns") into the new Stage 0 (they
were near-identical), renumbers hardening to Stage 1, and updates the drift
signals (the gate can be softened by mistaking polite feedback for evidence;
multi-user shipping ahead of the gate was a recorded exception per ADR-012, not a
precedent). Rationale recorded in ADR-016.
Records the infra#88 spike decision: per-user OAuth tokens are runtime app-state,
not config, so they live envelope-encrypted in PG18 under RLS (key from 1P via the
existing read-only SA) rather than in the vault. Infra creds stay ESO/1Password —
two mechanisms because they're two different things. Includes the falsification
conditions (frequent rotation; estate audit policy; key-rotation cost) so the
choice is earned not assumed. Adds the two rejected candidates (vault-write SA;
Supabase) to the rejected-alternatives table. Full reasoning in the #88 decision
doc; build + reboot-validation in #89.
Mirror to github.com was failing with 'unsupported in libcrypto' on every run
(OpenSSL 3.6 / OpenSSH 10 dropped support for the existing key format). Removed
rather than leave it polluting the CI signal. Re-add when the deploy key is
rotated to ed25519.
The Stage-0 ui-spec (ADR-011) predated multi-user and several UX
features. Appended a "Deviations and additions (as-built)" table —
without rewriting the spec — recording each feature shipped beyond it
(multi-user+RLS, registration gate, per-user YouTube connect, account
management, immediate web summarization, charmbracelet spinner,
auto/manual mode, public landing page) with the why and the
commit/ADR that covers each. Preserves the intent-vs-reality split.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The .feature spec lagged shipped behaviour. Added three files, no
duplication of existing scenarios:
- registration.feature: new Dex subject -> registration gate (users +
user_identities), returning subject straight through, account delete
is tapir-side only and leaves other users intact, clean
re-registration (ADR-012, ADR-013).
- summarize_mode.feature: auto summarizes every new video; manual
(default) leaves them unsummarized until queued; queued video is
processed and the flag cleared (migration 006).
- landing_page.feature: unauthenticated / -> /welcome, Get Started for
guests, summary link + logout for authed users, logout -> /welcome.
Scoped to built features only — no Vimeo/Whisper/billing scenarios.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The C4 view predated the web transport. Added a "Web surface" section
with a component diagram and prose covering: tapir serve (HTMX+Templ
over the unchanged store), oidc authenticate-only session, registration
gate (users + user_identities), per-user YouTube web connect callback,
account disconnect/delete (ADR-013 tapir-side only), immediate
summarization via background goroutine + HTMX status poll, and
auto/manual summarize mode (migration 006). Relabeled the L2 http node
to tapir serve. Corrected the out-of-scope isolation bullet: RLS is live
(ADR-012, migration 003), not deferred. ADR-003 stance preserved — the
engine/ports/sinks core is untouched; the web is a new transport.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mathias
