Compare commits

...
15 Commits
Author SHA1 Message Date
mathiasandClaude Opus 4.8 943554a96c feat(web): "Retrying later" badge for rate-limited videos
CI / Lint / Test / Vet (push) Successful in 10s
CI / Build & Import (push) Successful in 10s
A discovered-but-unsummarized video whose caption fetch was rate-limited now
shows a passive  "Retrying later" chip (dim CharmDim styling, not the accent)
instead of the Summarize button — the user cannot fix a 429, the runner retries
automatically once the backoff window expires. Regenerated views_templ.go.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 22:56:48 +02:00
mathiasandClaude Opus 4.8 40a614c8d4 feat(runner): 429 backoff — skip still-throttled videos, persist status
After ProcessNewVideo the runner records transcript_status per outcome:
rate_limited (stamps the backoff clock), none, or fetched. Before fetching, a
video inside the TAPIR_FETCH_BACKOFF window is skipped (SkippedRateLimited) so a
just-429'd caption endpoint is not re-hit; once the window expires it retries.

Backoff/clock injected via variadic Options (WithBackoff, WithClock) so existing
New call sites and the fake-driven loop tests stay valid. Backoff 0 = always retry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 22:56:07 +02:00
mathiasandClaude Opus 4.8 ce2fc62ef8 feat(config): TAPIR_FETCH_BACKOFF for rate-limit retry window
Adds FetchBackoff (Go duration, default 1h) controlling how long the run loop
waits before re-fetching a transcript that returned HTTP 429. Zero means always
retry. Not required by ValidateForRun — a zero/unset value is a valid policy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 22:54:24 +02:00
mathiasandClaude Opus 4.8 0ceacc8230 feat(usecase): surface TranscriptSource on ProcessResult
The engine already distinguishes SourceNone from SourceRateLimited internally
but collapsed both into Skipped. Expose the source string so the runner can
persist the right transcript_status and apply rate-limit backoff, without the
engine taking on any store/retry concern (dependencies still point inward).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 22:53:11 +02:00
mathiasandClaude Opus 4.8 1e81965519 feat(store): TranscriptStatus read field + status setter/loader
Surfaces videos.transcript_status (migration 007) on SummaryRow and adds
SetTranscriptStatus / GetTranscriptStatus / RateLimitedVideoIDs.

SetTranscriptStatus is the single choke point for the rate-limit lifecycle:
"rate_limited" stamps rate_limited_at = NOW(), every other status clears it,
so the runner's backoff window and the UI badge read one consistent source.
RateLimitedVideoIDs is the per-pass loader (mirrors SeenVideoIDs) the runner
uses to skip still-throttled videos without re-hitting the caption endpoint.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 22:52:48 +02:00
mathias f50c072d65 fix(web): add Log out to the persistent nav header
CI / Lint / Test / Vet (push) Successful in 12s
CI / Build & Import (push) Successful in 10s
Logout was only reachable from /welcome. Users who are logged in had no way
to sign out from any app page (list, detail, account). Added to the shared
nav alongside Account.
2026-06-03 22:49:14 +02:00
mathias e6f508824b docs: add ADR-016 — Stage 0 gate revised to "me or a friend", behavioural
CI / Lint / Test / Vet (push) Successful in 19s
CI / Build & Import (push) Successful in 11s
Records the gate change: Stage 0 now passes when either the maintainer or an
onboarded friend returns unprompted in >=2 separate weeks. Behavioural (return
usage), not feedback-based, to resist politeness bias. Includes an honest
self-scrutiny note that this is a guardrail edit made while the original gate was
unmet — examined on that basis and proceeding because it broadens who supplies the
signal without softening the kind of signal required. Adds feedback-based-gate to
rejected alternatives.
2026-06-03 20:46:40 +00:00
mathiasandClaude Opus 4.8 689500c85e feat(store): migration 007 — per-video transcript status
CI / Lint / Test / Vet (push) Successful in 20s
CI / Build & Import (push) Successful in 10s
Adds videos.transcript_status (NULL|none|fetched|rate_limited) and
videos.rate_limited_at, so the runner can record a 429 and skip re-fetching a
still-throttled video until a backoff window elapses. Columns inherit the
existing videos RLS policy (migration 003); no policy change needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 22:45:29 +02:00
mathiasandClaude Opus 4.8 4678d473b8 feat(youtube): map caption 429 to SourceRateLimited
The baseUrl fetch mapped every non-200 to SourceNone, recording a 429 as a
permanent "no captions". 429 is the IP being rate-limited, not an absent
transcript. Return SourceRateLimited (still a graceful degrade, no error) so
the runner can retry after a backoff window. Other non-200s (403/404/5xx)
stay SourceNone.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 22:45:29 +02:00
mathiasandClaude Opus 4.8 c63b2de66d fix(web): actionable empty state for the summary list
Videos rows are created by `tapir run`, not when a YouTube account is
connected, so a freshly-connected account correctly shows an empty list —
but the old empty state ("No videos yet") gave no clue why or what to do.
Split it on whether the user has any connection:

- connected, no videos: a distinct accent callout telling them to run
  `tapir run` to discover subscriptions.
- not connected: a prompt with a Connect YouTube button.

handleList fetches connections only when the list is empty. Includes
web-shot captures of all three states under docs/ux-review/fixes/.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 22:45:29 +02:00
mathiasandClaude Opus 4.8 27aa319f1d feat(domain): add SourceRateLimited transcript source
429 from the caption endpoint means the IP is rate-limited (retry later),
not that the video has no captions. Distinguishing it from SourceNone is the
prerequisite for the runner's backoff/retry logic.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 22:45:29 +02:00
mathiasandClaude Opus 4.8 61d4d5bc4a fix(web): clarify landing CTA copy for new users
The "Get Started" button drops users straight into the shared Dex flow,
which has no separate "register" option — registration completes
automatically after first login. Users new to Tapir had no signal that
signing in is also how they sign up. Reword the sub-text to say so
explicitly, keeping the single Dex CTA (sign-in and sign-up are one flow).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 22:45:29 +02:00
mathiasandClaude Opus 4.8 483730cd03 fix(web): make anchor-styled buttons readable
The .btn class sets color:var(--accent-fg), but the generic a{} and
a:visited{} rules outrank it on <a> elements, so anchor buttons (the
landing "Get Started" CTA, "Connect YouTube") rendered their label
accent-on-accent — invisible. Add a.btn / a.btn:visited to restore the
button foreground.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 22:45:29 +02:00
mathias 477701fea2 docs: revise Stage 0 gate to "useful to me or a friend" (behavioural)
CI / Lint / Test / Vet (push) Successful in 11s
CI / Build & Import (push) Successful in 10s
Replaces the original "useful to me, specifically" gate with "me OR a friend
returns unprompted in >=2 separate weeks" — friendly-user signal counts, but the
test stays behavioural (return usage) not feedback-based, to resist politeness
bias. Folds the old Stage 1 ("a trusted user returns") into the new Stage 0 (they
were near-identical), renumbers hardening to Stage 1, and updates the drift
signals (the gate can be softened by mistaking polite feedback for evidence;
multi-user shipping ahead of the gate was a recorded exception per ADR-012, not a
precedent). Rationale recorded in ADR-016.
2026-06-03 20:44:19 +00:00
mathias 17fad140a6 docs: add ADR-015 — per-user credentials envelope-encrypted in PG18
CI / Lint / Test / Vet (push) Successful in 10s
CI / Build & Import (push) Successful in 10s
Records the infra#88 spike decision: per-user OAuth tokens are runtime app-state,
not config, so they live envelope-encrypted in PG18 under RLS (key from 1P via the
existing read-only SA) rather than in the vault. Infra creds stay ESO/1Password —
two mechanisms because they're two different things. Includes the falsification
conditions (frequent rotation; estate audit policy; key-rotation cost) so the
choice is earned not assumed. Adds the two rejected candidates (vault-write SA;
Supabase) to the rejected-alternatives table. Full reasoning in the #88 decision
doc; build + reboot-validation in #89.
2026-06-03 20:29:50 +00:00
25 changed files with 855 additions and 217 deletions
+5
View File
@@ -46,3 +46,8 @@ TAPIR_SECRETS_FILE=
# --- run loop ------------------------------------------------------------- # --- run loop -------------------------------------------------------------
# Empty/0 = single pass. Set (e.g. 15m) to poll on that cadence. # Empty/0 = single pass. Set (e.g. 15m) to poll on that cadence.
TAPIR_POLL_INTERVAL= TAPIR_POLL_INTERVAL=
# How long to wait before re-fetching a transcript that returned HTTP 429
# (rate_limited). Inside the window the video is skipped without hitting the
# caption endpoint; after it expires the video is retried. 0 = always retry.
# Go duration; default 1h.
TAPIR_FETCH_BACKOFF=
+104
View File
@@ -388,6 +388,107 @@ the rate gate (the batch path moves into k3s per the deferred CronJob item).
--- ---
## ADR-015 — Per-user credentials: envelope-encrypted in PG18, not vault-stored
**Status:** Accepted (2026-06-03)
**Context.** The Stage-0/1 SecretStore (`internal/adapters/secrets/file.go`) holds per-user
YouTube OAuth refresh tokens as a flat key-value JSON map on a PVC — explicitly a stand-in for
"op/ESO later" (ADR-002, ADR-006). infra#86 proposed migrating it to an ESO-backed store. The
decision spike (infra#88) found that framing subtly wrong: **ESO syncs vault→cluster at
deploy/refresh time; it is not a runtime write API.** Per-user tokens are written *at runtime,
per end-user* (every YouTube connect; on token rotation) — they are application state, not
configuration. The homelab 1Password SA is also read-only, so a vault-write path would require
a new write-capable SA, widening Tapir's blast radius to shared estate infra to store what is
fundamentally Tapir's own row-data. Reading the actual SecretStore confirmed the shape: a
3-method port (`Get`/`Put`/`Delete`) over opaque refs, written interactively per user.
**Decision.** Per-user credentials are stored **envelope-encrypted in PG18**, not in the vault:
1. Tokens are encrypted with a **single app-level envelope key** and stored as ciphertext in
PG18, under the Row-Level Security already enforced and tested (ADR-012). Reads/writes go
through the existing `withUser` RLS-scoped seam.
2. The **envelope key** is the only secret in 1Password — fetched via the **existing read-only
SA** (confirmed working). No new write-capable SA; no per-user vault items.
3. The `ports.SecretStore` port is unchanged (`Get`/`Put`/`Delete`). The implementation swaps
`FileStore` (PVC JSON) for a `PGStore` (encrypted rows). Every consumer — connect,
disconnect, delete-account — is untouched (the port abstraction holds, ADR-003 spirit).
4. **Infra/operator credentials** (Dex client secret, MCP-auth tokens, service tokens) stay an
**ESO/1Password** concern. This ADR governs *per-user runtime* credentials only. The two
classes use two mechanisms deliberately — because they are two different things (runtime
app-state vs deploy-time config), not as a compromise. The "one mechanism" question
(maintainer's initial preference) was answered in #88 by correctly *classifying* the
secrets rather than unifying their storage.
**Consequences.**
- Runtime credential writes are normal RLS'd DB writes — no ESO sync latency, no indirection,
no write-SA blast radius. The interactive connect→store→use flow works without a vault
round-trip.
- Keeps PG18 and keeps ADR-002 intact (Supabase was considered and rejected again in #88
adding a datastore to hold a few encrypted strings PG18 already holds).
- Adds an encrypt/decrypt seam and an **envelope-key rotation** responsibility (re-encrypt the
per-user rows under a new key). infra#89 (build) must implement and test rotation, not assume
it — this is the real engineering cost of the choice.
- The vault's involvement shrinks to one static key via the SA already trusted for reads.
- **Supersedes** the "PVC stand-in for op/ESO" intent recorded in `secrets/file.go` and
`docs/homelab-integration.md` for the *per-user* secret path (the ESO/1Password reference in
ADR-006 stands for the *infra-cred* path).
**Reversibility / falsification (from infra#88).** Revisit if: per-user tokens need
high-frequency rotation writes (weak — PG18 handles it); an estate compliance policy requires
all credentials in 1P for a single audit surface (maintainer-knowable, not currently believed
to hold — would favour the vault-write path on policy grounds); or envelope-key rotation proves
operationally worse than per-secret vault rotation (the real cost #89 must prove). If none hold,
this stands. Full reasoning + rejected candidates (write-capable SA; Supabase): the infra#88
decision doc (`infra/docs/superpowers/handoffs/`). Build + reboot-validation: infra#89.
---
## ADR-016 — Stage 0 gate revised: "useful to me OR a friend", behavioural not feedback
**Status:** Accepted (2026-06-03). Revises the Stage 0 definition in VISION.md (supersedes the
original "useful to me, specifically" gate and folds in the old Stage 1 "a trusted user returns"
test).
**Context.** The original Stage 0 gate was "the maintainer reads summaries weekly for four weeks
and acts on one." The maintainer chose to change it to include friendly users, reasoning that
early signal from friendly users is valuable. Two sub-decisions shaped the final form:
- *Me OR a friend* (not AND): either the maintainer or an onboarded friend showing use clears it.
- *Behavioural, not feedback*: the test is **return usage**, not stated approval.
**Decision.** Stage 0 passes when, over a 34 week window, **either the maintainer or at least
one onboarded friend returns to Tapir unprompted and reads/acts on summaries in ≥2 separate
weeks.** Friend feedback is gathered and valued but is **not** the gate.
**Why behavioural, not feedback (the load-bearing part).** Asked-for feedback from friendly
users is the least reliable signal in product development — politeness bias means a friend you
onboarded will tend to say encouraging things regardless of real value. The thing actually worth
knowing is whether they *come back on their own*. So the gate measures returns, not nice words.
This deliberately resists the most common way a principled gate dies: being declared "passed" on
the strength of a polite reaction.
**Honest note on what this change does.** This is a *guardrail edit made while the original gate
was unmet* (Stage 0 had barely started; build had run well ahead of use-evidence). That is
precisely the pattern that warrants scrutiny — redrawing a gate around work already done. It was
examined on that basis and proceeds because: (a) the new gate is **not softer in kind** — it
stays behavioural and sustained, merely broadening *who* can supply the signal; (b) friendly-user
signal is genuinely valuable; (c) the politeness-bias guard keeps it from collapsing into
"someone said it's nice." It is *not* a licence to treat the already-shipped Stage-1 machinery as
evidence the gate passed — use-evidence remains open.
**Consequences.**
- VISION.md Stage 0 rewritten; old Stage 1 ("a trusted user returns") folded in (it was
near-identical to the new test); hardening renumbered to Stage 1.
- New drift signal added: declaring the gate passed on polite feedback rather than return-usage.
- The 2026-07-01 check-in now asks "is anyone (me or a friend) coming back unprompted?", not
"am I using it weekly?".
**Reversibility.** A superseding ADR could tighten it back to maintainer-only or raise it to
require multiple returning users. Recorded with the full rationale (including the self-scrutiny
about editing a gate while it's unmet) so the reasoning survives, not just the new wording.
---
## Rejected alternatives ## Rejected alternatives
Approaches considered during the 2026-06-02 planning + grill session and **deliberately not Approaches considered during the 2026-06-02 planning + grill session and **deliberately not
@@ -406,6 +507,9 @@ maps to the ADR that settles it.
| Audio-download + Whisper STT in the core path | ToS-grey, breakage-prone (yt-dlp), contends for koala GPU with the JEPA PoC; captions alone test the core hypothesis | ADR-007 | | Audio-download + Whisper STT in the core path | ToS-grey, breakage-prone (yt-dlp), contends for koala GPU with the JEPA PoC; captions alone test the core hypothesis | ADR-007 |
| Building multi-tenant SaaS / Google OAuth verification now | "Real users soon" was lowered to Future B; SaaS machinery before the Stage 0 self-use gate is the primary documented anti-goal | ADR-008, VISION | | Building multi-tenant SaaS / Google OAuth verification now | "Real users soon" was lowered to Future B; SaaS machinery before the Stage 0 self-use gate is the primary documented anti-goal | ADR-008, VISION |
| Delegating the S5 reuse spike to an agent swarm | A 1-hour sequential read-and-judge with a single coupled conclusion; orchestration overhead exceeds the work, and it's Diamond-1 judgment the maintainer wanted to own | (process note) | | Delegating the S5 reuse spike to an agent swarm | A 1-hour sequential read-and-judge with a single coupled conclusion; orchestration overhead exceeds the work, and it's Diamond-1 judgment the maintainer wanted to own | (process note) |
| Vault-write SA for per-user OAuth tokens (ESO as runtime write path) | ESO syncs vault→cluster at deploy time, not a runtime write API; a write-SA widens blast radius to shared infra to store app row-data | ADR-015, infra#88 |
| Supabase for per-user credential storage | Adds a second datastore for a few encrypted strings PG18 already holds; reopens ADR-002 | ADR-015, infra#88 |
| Feedback-based Stage 0 gate (friends saying it's useful) | Politeness bias makes asked-for feedback the least reliable signal; return-usage is the real test | ADR-016 |
If a future case genuinely reopens one of these, that's a new ADR superseding the relevant one — If a future case genuinely reopens one of these, that's a new ADR superseding the relevant one —
not a silent reversal. not a silent reversal.
+35 -27
View File
@@ -44,50 +44,54 @@ fallback — their key, their choice.
## Who it is for ## Who it is for
- **Now (the first customer):** the maintainer — one person, their own subscriptions, - **Now (the first customers):** the maintainer and a small number of known, trusted
summaries delivered to their own store and brain. friends — each with their own account, isolated data, optional BYO-AI. The maintainer is
- **Soon (Future B):** a small number of known, trusted users (friends / beta) — each with the first customer; friendly users provide the earliest real-world signal.
their own account, isolated data, optional BYO-AI.
- **Maybe (Future C, explicitly not built yet):** a public multi-tenant service. Deferred - **Maybe (Future C, explicitly not built yet):** a public multi-tenant service. Deferred
until there is evidence of sustained personal use **and** real demand. Building for C until there is evidence of sustained use **and** real demand. Building for C before that
before that evidence is a known anti-goal. evidence is a known anti-goal.
## Definition of Success ## Definition of Success
Success is staged. Each stage has a single, falsifiable headline test. We do not advance Success is staged. Each stage has a single, falsifiable headline test. We do not advance
to the next stage's ambition until the current stage's test passes. to the next stage's ambition until the current stage's test passes.
### Stage 0 — Useful to me (the gate) ### Stage 0 — Useful to me or a friend (the gate)
> **Headline test:** For four consecutive weeks, the maintainer reads Tapir-produced > **Headline test:** Over a 34 week window, *either* the maintainer *or* at least one
> summaries for their own subscriptions at least weekly, and at least once acts on a > onboarded friend returns to Tapir **unprompted** and reads/acts on summaries in **≥2
> summary (watches / skips / saves a video *because of* the summary). > separate weeks**. The test is *return usage* (behavioural), not stated approval.
- Captions-first summarization works end-to-end for the maintainer's real subscriptions. - Captions-first summarization works end-to-end for real subscriptions (the maintainer's
- Summaries land in the maintainer's store and (optionally) brain. and onboarded friends').
- Summaries land in each user's own store and (optionally) brain.
- Local-first AI produces summaries of acceptable quality without manual intervention - Local-first AI produces summaries of acceptable quality without manual intervention
most of the time. most of the time.
- **This is the gate.** Multi-user, BYO-AI-for-others, and any SaaS ambition stay deferred - **Why behavioural, not feedback.** Friend *feedback* is gathered and genuinely valuable —
until Stage 0 holds. (Ties to the 2026-07-01 self-use check-in.) but it is **not** the gate. Asked-for feedback from friendly users is the least reliable
signal in product development (politeness bias); whether they *come back on their own* is
the thing we actually care about. So the gate measures returns, not nice words.
- **Why "me OR a friend".** This replaces the original "useful to *me*, specifically" gate
(2026-06-03 decision, recorded in DECISIONS.md ADR-016). Getting signal from friendly
users is valuable enough to count — but the bar stays behavioural so it can't be cleared
by a polite reaction. (Ties to the 2026-07-01 check-in.)
- **This is the gate.** Hardening (Stage 1) and any SaaS ambition stay deferred until this
behavioural signal exists. Note: multi-user machinery was deliberately built *ahead* of
this gate (ADR-012) with isolation enforced — that was an explicit, recorded call, not a
sign the gate had passed. The gate is about *evidence of use*, which is still open.
### Stage 1 — Useful to a few (Future B) ### Stage 1 — Trustworthy at rest (hardening, Future B)
> **Headline test:** At least one trusted user other than the maintainer connects their
> own account and, within their first month, keeps using it (returns to read summaries in
> ≥2 separate weeks) without the maintainer hand-holding each summary.
- Multiple users, each with isolated accounts, credentials, and summaries.
- A new user can self-connect a YouTube/Vimeo account and get summaries with no code change.
- Optional BYO-AI works per-user.
- No cross-user data leakage — demonstrable, not assumed.
### Stage 2 — Trustworthy at rest (hardening, still Future B)
> **Headline test:** Credentials (OAuth tokens, BYO-AI keys) are encrypted at rest via the > **Headline test:** Credentials (OAuth tokens, BYO-AI keys) are encrypted at rest via the
> homelab's existing secrets convention; a documented, rehearsed recovery path exists; and > homelab's existing secrets convention; a documented, rehearsed recovery path exists; and
> a deliberate isolation test (user A cannot read user B's data) passes in CI or a > a deliberate isolation test (user A cannot read user B's data) passes in CI or a
> documented manual drill. > documented manual drill.
- Per-user data isolation is enforced and tested (delivered early via ADR-012 RLS).
- Per-user credentials are encrypted at rest (ADR-015 envelope encryption; build in infra#89).
- A new user can self-connect a YouTube/Vimeo account and get summaries with no code change.
- Optional BYO-AI works per-user.
### Non-goals (current) ### Non-goals (current)
- Public sign-up / billing / a marketing surface. - Public sign-up / billing / a marketing surface.
@@ -98,7 +102,11 @@ to the next stage's ambition until the current stage's test passes.
## How we will know we are drifting ## How we will know we are drifting
- We are building Stage 1+ machinery before the Stage 0 gate has passed. - We declare the Stage 0 gate "passed" on the strength of polite feedback rather than
behavioural return-usage (the politeness-bias trap the gate is designed to resist).
- We build Stage 1 hardening or Future C machinery while the Stage 0 use-evidence is still
absent. (Multi-user machinery already shipped ahead of the gate via ADR-012 — a recorded,
deliberate exception, not a precedent for more.)
- A user's content reaches a third-party model without that user's explicit, per-user opt-in. - A user's content reaches a third-party model without that user's explicit, per-user opt-in.
- "Brain ingestion" starts dictating the architecture instead of being one sink behind an - "Brain ingestion" starts dictating the architecture instead of being one sink behind an
interface. interface.
+2 -2
View File
@@ -125,10 +125,10 @@ func cmdRun(ctx context.Context, log *slog.Logger) error {
if engine == nil { if engine == nil {
return fmt.Errorf("run: incomplete summarization config (gateway, youtube credentials, secrets file)") return fmt.Errorf("run: incomplete summarization config (gateway, youtube credentials, secrets file)")
} }
r := runner.New(engine.Source, st, engine, cfg.UserID, log) r := runner.New(engine.Source, st, engine, cfg.UserID, log, runner.WithBackoff(cfg.FetchBackoff))
log.Info("starting run", "user", cfg.UserID, "model", cfg.SummarizerModel, log.Info("starting run", "user", cfg.UserID, "model", cfg.SummarizerModel,
"gateway", cfg.GatewayURL, "poll_interval", cfg.PollInterval) "gateway", cfg.GatewayURL, "poll_interval", cfg.PollInterval, "fetch_backoff", cfg.FetchBackoff)
return r.Loop(ctx, cfg.PollInterval) return r.Loop(ctx, cfg.PollInterval)
} }
Binary file not shown.

After

Width:  |  Height:  |  Size: 116 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 73 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 58 KiB

@@ -0,0 +1,2 @@
ALTER TABLE videos DROP COLUMN IF EXISTS rate_limited_at;
ALTER TABLE videos DROP COLUMN IF EXISTS transcript_status;
@@ -0,0 +1,17 @@
-- Migration 007: per-video transcript fetch status, for rate-limit backoff.
--
-- transcript_status records the outcome of the last transcript attempt:
-- NULL = not yet attempted
-- 'none' = checked, no usable transcript (permanent — SourceNone)
-- 'fetched' = transcript resolved and summarized (summary_id not null)
-- 'rate_limited'= the caption endpoint returned 429; retry after a backoff window
--
-- rate_limited_at stamps WHEN the 429 was seen, so the runner can skip re-fetching
-- a still-throttled video until NOW() - rate_limited_at exceeds TAPIR_FETCH_BACKOFF.
-- It is cleared (set NULL) whenever the status moves off 'rate_limited'.
--
-- No RLS policy changes needed: videos already has ENABLE + FORCE ROW LEVEL
-- SECURITY (migration 003) with the videos_isolation policy. New columns inherit
-- that protection automatically.
ALTER TABLE videos ADD COLUMN transcript_status TEXT;
ALTER TABLE videos ADD COLUMN rate_limited_at TIMESTAMPTZ;
+7 -1
View File
@@ -49,6 +49,10 @@ type SummaryRow struct {
// set by the web "Summarize" button and cleared by the next `tapir run`. Only // set by the web "Summarize" button and cleared by the next `tapir run`. Only
// populated by ListVideos/GetVideoRow (summary-only reads leave it false). // populated by ListVideos/GetVideoRow (summary-only reads leave it false).
SummarizeRequested bool SummarizeRequested bool
// TranscriptStatus mirrors videos.transcript_status (migration 007): "" (unset),
// "none", "rate_limited", or "fetched". Drives the "Retrying later" list badge.
// Only populated by ListVideos/GetVideoRow ("" on summary-only reads).
TranscriptStatus string
} }
// selectSummary is the shared projection for both reads. videos is LEFT JOINed // selectSummary is the shared projection for both reads. videos is LEFT JOINed
@@ -132,7 +136,8 @@ const selectVideo = `
COALESCE(s.fallback_used, FALSE), COALESCE(s.fallback_used, FALSE),
COALESCE(s.created_at, v.seen_at), COALESCE(s.created_at, v.seen_at),
(s.id IS NOT NULL) AS summarized, (s.id IS NOT NULL) AS summarized,
v.summarize_requested v.summarize_requested,
COALESCE(v.transcript_status, '')
FROM videos v FROM videos v
LEFT JOIN summaries s ON s.video_id = v.id AND s.user_id = v.user_id` LEFT JOIN summaries s ON s.video_id = v.id AND s.user_id = v.user_id`
@@ -245,6 +250,7 @@ func scanVideoRow(rows pgx.Row) (SummaryRow, error) {
&row.CreatedAt, &row.CreatedAt,
&row.Summarized, &row.Summarized,
&row.SummarizeRequested, &row.SummarizeRequested,
&row.TranscriptStatus,
); err != nil { ); err != nil {
return SummaryRow{}, fmt.Errorf("store: scan video: %w", err) return SummaryRow{}, fmt.Errorf("store: scan video: %w", err)
} }
@@ -0,0 +1,102 @@
package store
import (
"context"
"errors"
"fmt"
"time"
"github.com/jackc/pgx/v5"
)
// validTranscriptStatuses bounds SetTranscriptStatus input. "" clears the status
// (column NULL); the three named states mirror migration 007's documented values.
var validTranscriptStatuses = map[string]bool{
"": true,
"none": true,
"rate_limited": true,
"fetched": true,
}
// SetTranscriptStatus records the outcome of the last transcript attempt for a
// video (migration 007). When status is "rate_limited" it also stamps
// rate_limited_at = NOW() so the runner can back off; every other status clears
// that timestamp. "" unsets the status (column NULL). An unknown status is
// rejected. Scoped via withUser, so RLS confines the UPDATE to the caller's own
// video; ErrNotFound when the user has no such video.
func (s *Store) SetTranscriptStatus(ctx context.Context, userID, videoID, status string) error {
if !validTranscriptStatuses[status] {
return fmt.Errorf("store: invalid transcript status %q", status)
}
return s.withUser(ctx, userID, func(tx pgx.Tx) error {
ct, err := tx.Exec(ctx,
`UPDATE videos
SET transcript_status = NULLIF($1, ''),
rate_limited_at = CASE WHEN $1 = 'rate_limited' THEN NOW() ELSE NULL END
WHERE id = $2`,
status, videoID)
if err != nil {
return fmt.Errorf("store: set transcript status: %w", err)
}
if ct.RowsAffected() == 0 {
return ErrNotFound
}
return nil
})
}
// GetTranscriptStatus returns a video's transcript_status ("" when unset/NULL).
// Returns ErrNotFound when the user has no such video. Scoped via withUser.
func (s *Store) GetTranscriptStatus(ctx context.Context, userID, videoID string) (string, error) {
var status string
if err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
err := tx.QueryRow(ctx,
`SELECT COALESCE(transcript_status, '') FROM videos WHERE id = $1`, videoID).Scan(&status)
if errors.Is(err, pgx.ErrNoRows) {
return ErrNotFound
}
return err
}); err != nil {
if errors.Is(err, ErrNotFound) {
return "", ErrNotFound
}
return "", fmt.Errorf("store: get transcript status: %w", err)
}
return status, nil
}
// RateLimitedVideoIDs returns the user's videos currently in the "rate_limited"
// state, mapped to when the 429 was stamped (rate_limited_at). The run loop loads
// it once per pass (mirroring SeenVideoIDs) to skip re-fetching a video still
// inside the backoff window, saving caption requests. Scoped by user_id.
func (s *Store) RateLimitedVideoIDs(ctx context.Context, userID string) (map[string]time.Time, error) {
out := make(map[string]time.Time)
if err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
rows, err := tx.Query(ctx,
`SELECT id, rate_limited_at FROM videos
WHERE user_id = $1 AND transcript_status = 'rate_limited' AND rate_limited_at IS NOT NULL`,
userID)
if err != nil {
return fmt.Errorf("store: rate limited video ids: %w", err)
}
defer rows.Close()
for rows.Next() {
var (
id string
at time.Time
)
if err := rows.Scan(&id, &at); err != nil {
return fmt.Errorf("store: scan rate limited id: %w", err)
}
out[id] = at
}
if err := rows.Err(); err != nil {
return fmt.Errorf("store: iterate rate limited ids: %w", err)
}
return nil
}); err != nil {
return nil, err
}
return out, nil
}
@@ -0,0 +1,80 @@
package store_test
import (
"context"
"testing"
"github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store"
)
func TestSetTranscriptStatus_RoundTrip(t *testing.T) {
ctx := context.Background()
s := newStore(t)
resetDB(t, rawPool(t))
id, err := s.UpsertVideo(ctx, ytVideo(userA, "rt12345", "round trip"))
require.NoError(t, err)
// Unset by default.
got, err := s.GetTranscriptStatus(ctx, userA, id)
require.NoError(t, err)
require.Equal(t, "", got)
for _, status := range []string{"none", "fetched", "rate_limited", ""} {
require.NoError(t, s.SetTranscriptStatus(ctx, userA, id, status))
got, err := s.GetTranscriptStatus(ctx, userA, id)
require.NoError(t, err)
require.Equal(t, status, got)
}
}
func TestSetTranscriptStatus_RejectsInvalid(t *testing.T) {
ctx := context.Background()
s := newStore(t)
resetDB(t, rawPool(t))
id, err := s.UpsertVideo(ctx, ytVideo(userA, "bad12345", "bad status"))
require.NoError(t, err)
require.Error(t, s.SetTranscriptStatus(ctx, userA, id, "bogus"))
// The rejected write left the status untouched.
got, err := s.GetTranscriptStatus(ctx, userA, id)
require.NoError(t, err)
require.Equal(t, "", got)
}
func TestSetTranscriptStatus_NotFound(t *testing.T) {
ctx := context.Background()
s := newStore(t)
resetDB(t, rawPool(t))
require.ErrorIs(t, s.SetTranscriptStatus(ctx, userA, videoX, "fetched"), store.ErrNotFound)
_, err := s.GetTranscriptStatus(ctx, userA, videoX)
require.ErrorIs(t, err, store.ErrNotFound)
}
func TestRateLimitedVideoIDs_StampsAndClears(t *testing.T) {
ctx := context.Background()
s := newStore(t)
resetDB(t, rawPool(t))
id, err := s.UpsertVideo(ctx, ytVideo(userA, "rl12345", "rate limited"))
require.NoError(t, err)
// Marking rate_limited stamps rate_limited_at, so the video appears.
require.NoError(t, s.SetTranscriptStatus(ctx, userA, id, "rate_limited"))
rl, err := s.RateLimitedVideoIDs(ctx, userA)
require.NoError(t, err)
require.Contains(t, rl, id)
require.False(t, rl[id].IsZero(), "rate_limited_at must be stamped")
// Moving off rate_limited clears the timestamp, so it drops out.
require.NoError(t, s.SetTranscriptStatus(ctx, userA, id, "fetched"))
rl, err = s.RateLimitedVideoIDs(ctx, userA)
require.NoError(t, err)
require.NotContains(t, rl, id)
}
+6
View File
@@ -60,6 +60,12 @@ func (a *Adapter) FetchTranscript(ctx context.Context, v domain.Video) (domain.T
if err != nil { if err != nil {
return domain.Transcript{}, fmt.Errorf("download caption track for %q: %w", v.ProviderVideoID, err) return domain.Transcript{}, fmt.Errorf("download caption track for %q: %w", v.ProviderVideoID, err)
} }
if status == http.StatusTooManyRequests {
// 429 means the IP is rate-limited; record for retry, not a permanent
// absence. Degrade gracefully (no error, no text) like SourceNone, but
// flag it distinctly so the runner backs off and retries (ADR-007/010).
return domain.Transcript{VideoID: v.ID, UserID: v.UserID, Source: domain.SourceRateLimited}, nil
}
if status != http.StatusOK { if status != http.StatusOK {
// Owner-only 403, region/age gate, or transient unavailability: not an error. // Owner-only 403, region/age gate, or transient unavailability: not an error.
return noTranscript(v), nil return noTranscript(v), nil
+31
View File
@@ -388,6 +388,37 @@ func TestFetchTranscriptBaseURLForbiddenDegrades(t *testing.T) {
} }
} }
// A 429 on the baseUrl fetch is the IP being rate-limited, NOT a permanent
// absence of captions: it returns SourceRateLimited (no error, no text) so the
// runner can record it and retry after a backoff window rather than recording a
// false "no transcript".
func TestFetchTranscriptRateLimitedReturnsSourceRateLimited(t *testing.T) {
a, _ := newTestAdapter(t, func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/youtubei/v1/player":
base := "http://" + r.Host
_, _ = w.Write([]byte(`{"captions":{"playerCaptionsTracklistRenderer":{"captionTracks":[` +
`{"baseUrl":"` + base + `/api/timedtext?lang=en","languageCode":"en"}]}}}`))
case "/api/timedtext":
w.WriteHeader(http.StatusTooManyRequests)
}
})
tr, err := a.FetchTranscript(context.Background(), domain.Video{ID: "v1", UserID: "u1", ProviderVideoID: "vid1"})
if err != nil {
t.Fatalf("429 on baseUrl must degrade, not error: %v", err)
}
if tr.Source != domain.SourceRateLimited {
t.Fatalf("expected SourceRateLimited on 429, got %q", tr.Source)
}
if tr.HasText() {
t.Error("expected HasText() false for SourceRateLimited")
}
if tr.Content != "" {
t.Errorf("expected empty content on 429, got %q", tr.Content)
}
}
// An empty baseUrl on the selected track degrades to SourceNone, never an error. // An empty baseUrl on the selected track degrades to SourceNone, never an error.
func TestFetchTranscriptEmptyBaseURLDegrades(t *testing.T) { func TestFetchTranscriptEmptyBaseURLDegrades(t *testing.T) {
a, _ := newTestAdapter(t, func(w http.ResponseWriter, r *http.Request) { a, _ := newTestAdapter(t, func(w http.ResponseWriter, r *http.Request) {
+13
View File
@@ -59,6 +59,12 @@ type Config struct {
// PollInterval, when > 0, makes `run` loop on that cadence; 0 means run once. // PollInterval, when > 0, makes `run` loop on that cadence; 0 means run once.
PollInterval time.Duration PollInterval time.Duration
// FetchBackoff is how long the run loop waits before re-fetching a transcript
// that previously returned HTTP 429 (rate_limited). Inside the window the video
// is skipped without hitting the caption endpoint, saving requests; after it
// expires the video is retried. Zero means "always retry" (no backoff).
FetchBackoff time.Duration
// HTTPAddr is the listen address for `tapir serve` (the Stage-0 web UI). // HTTPAddr is the listen address for `tapir serve` (the Stage-0 web UI).
HTTPAddr string HTTPAddr string
@@ -85,6 +91,7 @@ const (
defaultYTConnectRedirectURL = "https://tapir.d-ma.be/oauth/youtube/callback" defaultYTConnectRedirectURL = "https://tapir.d-ma.be/oauth/youtube/callback"
defaultOAuthRedirectAddr = "localhost:8080" defaultOAuthRedirectAddr = "localhost:8080"
defaultHTTPAddr = ":8080" defaultHTTPAddr = ":8080"
defaultFetchBackoff = time.Hour
) )
// Load reads the environment into a Config, applying defaults. It does not // Load reads the environment into a Config, applying defaults. It does not
@@ -124,6 +131,12 @@ func Load() (Config, error) {
} }
c.PollInterval = interval c.PollInterval = interval
backoff, err := durationOr("TAPIR_FETCH_BACKOFF", defaultFetchBackoff)
if err != nil {
return Config{}, err
}
c.FetchBackoff = backoff
return c, nil return c, nil
} }
+7
View File
@@ -45,6 +45,9 @@ func TestLoad_AppliesDefaults(t *testing.T) {
if c.PollInterval != 0 { if c.PollInterval != 0 {
t.Errorf("PollInterval = %v, want 0 (run once)", c.PollInterval) t.Errorf("PollInterval = %v, want 0 (run once)", c.PollInterval)
} }
if c.FetchBackoff != defaultFetchBackoff {
t.Errorf("FetchBackoff = %v, want default %v", c.FetchBackoff, defaultFetchBackoff)
}
} }
func TestLoad_ParsesValues(t *testing.T) { func TestLoad_ParsesValues(t *testing.T) {
@@ -56,6 +59,7 @@ func TestLoad_ParsesValues(t *testing.T) {
"TAPIR_SUMMARIZER_TIMEOUT": "90s", "TAPIR_SUMMARIZER_TIMEOUT": "90s",
"TAPIR_DB_DSN": "postgres://x", "TAPIR_DB_DSN": "postgres://x",
"TAPIR_POLL_INTERVAL": "10m", "TAPIR_POLL_INTERVAL": "10m",
"TAPIR_FETCH_BACKOFF": "30m",
}) })
c, err := Load() c, err := Load()
@@ -77,6 +81,9 @@ func TestLoad_ParsesValues(t *testing.T) {
if c.PollInterval != 10*time.Minute { if c.PollInterval != 10*time.Minute {
t.Errorf("PollInterval = %v, want 10m", c.PollInterval) t.Errorf("PollInterval = %v, want 10m", c.PollInterval)
} }
if c.FetchBackoff != 30*time.Minute {
t.Errorf("FetchBackoff = %v, want 30m", c.FetchBackoff)
}
} }
func TestLoad_RejectsBadDuration(t *testing.T) { func TestLoad_RejectsBadDuration(t *testing.T) {
+6
View File
@@ -18,6 +18,12 @@ type TranscriptSource string
const ( const (
SourceCaptions TranscriptSource = "captions" SourceCaptions TranscriptSource = "captions"
SourceNone TranscriptSource = "none" SourceNone TranscriptSource = "none"
// SourceRateLimited records that the caption endpoint returned HTTP 429.
// Unlike SourceNone (a permanent absence), this is a transient "retry later":
// the IP is rate-limited, not the video caption-less. It carries no text
// (HasText is false), so the engine degrades the same as SourceNone, but the
// runner persists it distinctly to retry after a backoff window.
SourceRateLimited TranscriptSource = "rate_limited"
) )
// User is the Tapir-side profile. At Stage 0 there is exactly one. // User is the Tapir-side profile. At Stage 0 there is exactly one.
+70 -3
View File
@@ -32,6 +32,12 @@ type VideoStore interface {
GetAutoSummarize(ctx context.Context, userID string) (bool, error) GetAutoSummarize(ctx context.Context, userID string) (bool, error)
RequestedVideoIDs(ctx context.Context, userID string) (map[string]bool, error) RequestedVideoIDs(ctx context.Context, userID string) (map[string]bool, error)
ClearSummarizeRequested(ctx context.Context, userID, videoID string) error ClearSummarizeRequested(ctx context.Context, userID, videoID string) error
// RateLimitedVideoIDs maps the user's still-throttled videos to when they were
// rate-limited, so the loop can back off without re-hitting the caption endpoint.
RateLimitedVideoIDs(ctx context.Context, userID string) (map[string]time.Time, error)
// SetTranscriptStatus records the outcome of a transcript attempt: "none",
// "rate_limited" (stamps the backoff clock), or "fetched".
SetTranscriptStatus(ctx context.Context, userID, videoID, status string) error
} }
// Processor runs the core use case for a single video. *usecase.Engine // Processor runs the core use case for a single video. *usecase.Engine
@@ -48,14 +54,35 @@ type Runner struct {
engine Processor engine Processor
userID string userID string
log *slog.Logger log *slog.Logger
backoff time.Duration // rate-limit retry window; 0 = always retry
now func() time.Time // injectable clock (tests); defaults to time.Now
} }
// Option configures a Runner at construction. Variadic so existing call sites
// stay valid as new knobs (backoff, clock) are added.
type Option func(*Runner)
// WithBackoff sets the rate-limit retry window. A video that returned HTTP 429 is
// skipped (no caption fetch) until this much time has passed; 0 = always retry.
func WithBackoff(d time.Duration) Option { return func(r *Runner) { r.backoff = d } }
// WithClock overrides the clock used for backoff comparisons. Tests inject a
// fixed time; production leaves the time.Now default.
func WithClock(now func() time.Time) Option { return func(r *Runner) { r.now = now } }
// New builds a Runner. A nil logger falls back to slog.Default. // New builds a Runner. A nil logger falls back to slog.Default.
func New(src ports.VideoSource, store VideoStore, engine Processor, userID string, log *slog.Logger) *Runner { func New(src ports.VideoSource, store VideoStore, engine Processor, userID string, log *slog.Logger, opts ...Option) *Runner {
if log == nil { if log == nil {
log = slog.Default() log = slog.Default()
} }
return &Runner{src: src, store: store, engine: engine, userID: userID, log: log} r := &Runner{src: src, store: store, engine: engine, userID: userID, log: log, now: time.Now}
for _, opt := range opts {
opt(r)
}
if r.now == nil {
r.now = time.Now
}
return r
} }
// Stats summarizes one RunOnce pass. // Stats summarizes one RunOnce pass.
@@ -65,6 +92,7 @@ type Stats struct {
SkippedSeen int SkippedSeen int
SkippedNoText int SkippedNoText int
SkippedManual int // discovered but not queued, in manual mode SkippedManual int // discovered but not queued, in manual mode
SkippedRateLimited int // 429'd previously and still inside the backoff window
Errors int Errors int
} }
@@ -104,6 +132,18 @@ func (r *Runner) RunOnce(ctx context.Context) (Stats, error) {
} }
} }
// Rate-limit backoff: videos that 429'd on a prior pass, mapped to when. Inside
// the backoff window they are skipped before any caption fetch, so a throttled
// IP is not hammered. Loaded once per pass (like seen/requested). Disabled when
// backoff <= 0 ("always retry").
var rateLimited map[string]time.Time
if r.backoff > 0 {
rateLimited, err = r.store.RateLimitedVideoIDs(ctx, r.userID)
if err != nil {
return stats, fmt.Errorf("runner: load rate-limited videos: %w", err)
}
}
subs, err := r.src.ListSubscriptions(ctx, r.userID) subs, err := r.src.ListSubscriptions(ctx, r.userID)
if err != nil { if err != nil {
return stats, fmt.Errorf("runner: list subscriptions: %w", err) return stats, fmt.Errorf("runner: list subscriptions: %w", err)
@@ -142,6 +182,15 @@ func (r *Runner) RunOnce(ctx context.Context) (Stats, error) {
continue continue
} }
// Still inside the rate-limit backoff window: skip without fetching, so
// we don't re-hit a caption endpoint that just 429'd us. After the window
// expires the video falls through and is retried normally.
if at, ok := rateLimited[id]; ok && r.now().Sub(at) < r.backoff {
stats.SkippedRateLimited++
r.log.Info("skipped video (rate-limited, backing off)", "video", v.ProviderVideoID, "title", v.Title)
continue
}
if fetchDelay > 0 { if fetchDelay > 0 {
time.Sleep(fetchDelay) time.Sleep(fetchDelay)
} }
@@ -152,11 +201,28 @@ func (r *Runner) RunOnce(ctx context.Context) (Stats, error) {
continue continue
} }
switch { switch {
case res.Skipped && res.TranscriptSource == string(domain.SourceRateLimited):
// Fresh 429 this pass: persist rate_limited (stamps the backoff clock)
// so the next pass skips it until the window expires.
stats.SkippedRateLimited++
if err := r.store.SetTranscriptStatus(ctx, r.userID, id, "rate_limited"); err != nil {
errs = append(errs, fmt.Errorf("set rate_limited status %q: %w", v.ProviderVideoID, err))
stats.Errors++
}
r.log.Info("skipped video (rate-limited)", "video", v.ProviderVideoID, "title", v.Title)
case res.Skipped: case res.Skipped:
stats.SkippedNoText++ stats.SkippedNoText++
if err := r.store.SetTranscriptStatus(ctx, r.userID, id, "none"); err != nil {
errs = append(errs, fmt.Errorf("set none status %q: %w", v.ProviderVideoID, err))
stats.Errors++
}
r.log.Info("skipped video (no transcript)", "video", v.ProviderVideoID, "title", v.Title) r.log.Info("skipped video (no transcript)", "video", v.ProviderVideoID, "title", v.Title)
case res.Summary != nil: case res.Summary != nil:
stats.Summarized++ stats.Summarized++
if err := r.store.SetTranscriptStatus(ctx, r.userID, id, "fetched"); err != nil {
errs = append(errs, fmt.Errorf("set fetched status %q: %w", v.ProviderVideoID, err))
stats.Errors++
}
// In manual mode the video was processed because it was queued; // In manual mode the video was processed because it was queued;
// clear the flag so it is not re-summarized and the UI drops the // clear the flag so it is not re-summarized and the UI drops the
// "Queued" chip. (Auto mode never sets the flag.) // "Queued" chip. (Auto mode never sets the flag.)
@@ -184,7 +250,8 @@ func (r *Runner) Loop(ctx context.Context, interval time.Duration) error {
r.log.Info("run pass complete", r.log.Info("run pass complete",
"candidates", stats.Candidates, "summarized", stats.Summarized, "candidates", stats.Candidates, "summarized", stats.Summarized,
"skipped_seen", stats.SkippedSeen, "skipped_no_text", stats.SkippedNoText, "skipped_seen", stats.SkippedSeen, "skipped_no_text", stats.SkippedNoText,
"skipped_manual", stats.SkippedManual, "errors", stats.Errors) "skipped_manual", stats.SkippedManual, "skipped_rate_limited", stats.SkippedRateLimited,
"errors", stats.Errors)
if err != nil { if err != nil {
r.log.Warn("run pass had errors", "err", err) r.log.Warn("run pass had errors", "err", err)
} }
+75
View File
@@ -5,6 +5,7 @@ import (
"io" "io"
"log/slog" "log/slog"
"testing" "testing"
"time"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
@@ -48,6 +49,8 @@ type fakeStore struct {
auto bool auto bool
requested map[string]bool requested map[string]bool
cleared []string cleared []string
rateLimited map[string]time.Time // id -> when 429'd (seeds the backoff window)
statuses map[string]string // id -> last SetTranscriptStatus value
} }
func (f *fakeStore) UpsertVideo(_ context.Context, v domain.Video) (string, error) { func (f *fakeStore) UpsertVideo(_ context.Context, v domain.Video) (string, error) {
@@ -80,6 +83,22 @@ func (f *fakeStore) ClearSummarizeRequested(_ context.Context, _, videoID string
return nil return nil
} }
func (f *fakeStore) RateLimitedVideoIDs(_ context.Context, _ string) (map[string]time.Time, error) {
cp := make(map[string]time.Time, len(f.rateLimited))
for k, v := range f.rateLimited {
cp[k] = v
}
return cp, nil
}
func (f *fakeStore) SetTranscriptStatus(_ context.Context, _, videoID, status string) error {
if f.statuses == nil {
f.statuses = map[string]string{}
}
f.statuses[videoID] = status
return nil
}
type fakeSummarizer struct{} type fakeSummarizer struct{}
func (fakeSummarizer) Summarize(_ context.Context, v domain.Video, _ domain.Transcript) (domain.Summary, error) { func (fakeSummarizer) Summarize(_ context.Context, v domain.Video, _ domain.Transcript) (domain.Summary, error) {
@@ -207,6 +226,62 @@ func TestRunOnce_ManualMode_ProcessesRequested(t *testing.T) {
require.Equal(t, []string{"id-v1"}, st.cleared, "the queue flag is cleared after summarizing") require.Equal(t, []string{"id-v1"}, st.cleared, "the queue flag is cleared after summarizing")
} }
// noFetchSource fails the test if a transcript fetch happens — used to prove the
// runner skips a rate-limited video before touching the caption endpoint.
type noFetchSource struct{ *fakeSource }
func (noFetchSource) FetchTranscript(context.Context, domain.Video) (domain.Transcript, error) {
panic("FetchTranscript must not be called for a rate-limited video within the backoff window")
}
func TestRunOnce_SkipsRateLimitedWithinBackoff(t *testing.T) {
base := time.Date(2026, 6, 3, 12, 0, 0, 0, time.UTC)
src := &fakeSource{
subs: []domain.Subscription{sub("chan1", "Channel One")},
videos: map[string][]domain.Video{"chan1": {vid("v1", "Video 1")}},
}
// v1 was rate-limited 5m ago; backoff is 1h, so it is still inside the window.
st := &fakeStore{
seen: map[string]bool{},
auto: true,
rateLimited: map[string]time.Time{"id-v1": base.Add(-5 * time.Minute)},
}
eng := usecase.NewEngine(noFetchSource{src}, fakeSummarizer{}, &recordingSink{})
r := runner.New(noFetchSource{src}, st, eng, testUser, quietLogger(),
runner.WithBackoff(time.Hour), runner.WithClock(func() time.Time { return base }))
stats, err := r.RunOnce(context.Background())
require.NoError(t, err)
require.Equal(t, 1, stats.SkippedRateLimited, "still throttled -> skipped")
require.Equal(t, 0, stats.Summarized)
require.Empty(t, st.statuses, "no status write: the engine was never invoked")
}
func TestRunOnce_RetriesRateLimitedAfterBackoff(t *testing.T) {
base := time.Date(2026, 6, 3, 12, 0, 0, 0, time.UTC)
src := &fakeSource{
subs: []domain.Subscription{sub("chan1", "Channel One")},
videos: map[string][]domain.Video{"chan1": {vid("v1", "Video 1")}},
}
// v1 was rate-limited 2h ago; backoff is 1h, so the window has expired.
st := &fakeStore{
seen: map[string]bool{},
auto: true,
rateLimited: map[string]time.Time{"id-v1": base.Add(-2 * time.Hour)},
}
sink := &recordingSink{}
eng := usecase.NewEngine(src, fakeSummarizer{}, sink)
r := runner.New(src, st, eng, testUser, quietLogger(),
runner.WithBackoff(time.Hour), runner.WithClock(func() time.Time { return base }))
stats, err := r.RunOnce(context.Background())
require.NoError(t, err)
require.Equal(t, 0, stats.SkippedRateLimited, "window expired -> not skipped")
require.Equal(t, 1, stats.Summarized, "the video is retried and summarized")
require.Len(t, sink.delivered, 1)
require.Equal(t, "fetched", st.statuses["id-v1"], "status advances to fetched on success")
}
func TestRunOnce_UpsertsEveryCandidate(t *testing.T) { func TestRunOnce_UpsertsEveryCandidate(t *testing.T) {
src := &fakeSource{ src := &fakeSource{
subs: []domain.Subscription{sub("chan1", "Channel One")}, subs: []domain.Subscription{sub("chan1", "Channel One")},
+9 -2
View File
@@ -45,6 +45,11 @@ type ProcessResult struct {
Video domain.Video Video domain.Video
Skipped bool Skipped bool
Reason string // set when Skipped (e.g. "no transcript") Reason string // set when Skipped (e.g. "no transcript")
// TranscriptSource is how the transcript resolved (or that there was none):
// the domain.TranscriptSource value as a string. The runner reads it to tell a
// permanent absence (SourceNone) from a transient 429 (SourceRateLimited) and
// persist the right transcript_status. Empty when a fetch error short-circuits.
TranscriptSource string
Summary *domain.Summary // nil when Skipped Summary *domain.Summary // nil when Skipped
} }
@@ -59,7 +64,9 @@ func (e *Engine) ProcessNewVideo(ctx context.Context, v domain.Video) (ProcessRe
if !t.HasText() { if !t.HasText() {
// No usable transcript: record the skip, produce no summary, deliver nothing // No usable transcript: record the skip, produce no summary, deliver nothing
// (captions-first, ADR-007; the watcher uses this to avoid reprocessing). // (captions-first, ADR-007; the watcher uses this to avoid reprocessing).
return ProcessResult{Video: v, Skipped: true, Reason: "no transcript"}, nil // Surface the source so the runner separates SourceNone (permanent) from
// SourceRateLimited (retry after a backoff window).
return ProcessResult{Video: v, Skipped: true, Reason: "no transcript", TranscriptSource: string(t.Source)}, nil
} }
sum, err := e.AI.Summarize(ctx, v, t) sum, err := e.AI.Summarize(ctx, v, t)
@@ -76,7 +83,7 @@ func (e *Engine) ProcessNewVideo(ctx context.Context, v domain.Video) (ProcessRe
} }
} }
return ProcessResult{Video: v, Summary: &sum}, errors.Join(errs...) return ProcessResult{Video: v, Summary: &sum, TranscriptSource: string(t.Source)}, errors.Join(errs...)
} }
// ProcessNewVideos walks a user's subscriptions and processes each newly seen // ProcessNewVideos walks a user's subscriptions and processes each newly seen
+16 -3
View File
@@ -155,11 +155,24 @@ func (a *App) handleList(w http.ResponseWriter, r *http.Request) {
} }
rows = f.apply(rows) rows = f.apply(rows)
if isHTMX(r) { // hasConnected drives the empty state: a fresh account with a connection but
a.render(w, r, summaryList(rows)) // no `tapir run` yet has zero rows, and we want it to read "connected, run
// tapir" rather than "nothing here". Only needed when the list is empty.
hasConnected := false
if len(rows) == 0 {
conns, err := a.Store.ConnectionsForUser(r.Context(), userID)
if err != nil {
a.serverError(w, r, "connections for user", err)
return return
} }
a.render(w, r, ListPage(rows, f, takeFlash(w, r))) hasConnected = len(conns) > 0
}
if isHTMX(r) {
a.render(w, r, summaryList(rows, hasConnected))
return
}
a.render(w, r, ListPage(rows, f, takeFlash(w, r), hasConnected))
} }
// handleDetail renders one summary in full (highlights, takeaways, action group). // handleDetail renders one summary in full (highlights, takeaways, action group).
+55
View File
@@ -0,0 +1,55 @@
package web
import (
"context"
"strings"
"testing"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store"
)
func renderVideoCard(t *testing.T, r store.SummaryRow) string {
t.Helper()
var sb strings.Builder
if err := VideoCard(r).Render(context.Background(), &sb); err != nil {
t.Fatalf("render VideoCard: %v", err)
}
return sb.String()
}
// A rate-limited, unsummarized video shows the passive "Retrying later" badge and
// hides the Summarize button — the user can't fix it, retry is automatic.
func TestVideoCard_RateLimitedShowsRetryingBadge(t *testing.T) {
html := renderVideoCard(t, store.SummaryRow{
VideoID: "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa",
Title: "Throttled Video",
Summarized: false,
TranscriptStatus: "rate_limited",
})
if !strings.Contains(html, "Retrying later") {
t.Errorf("expected a 'Retrying later' badge, got:\n%s", html)
}
if !strings.Contains(html, "chip-retry") {
t.Errorf("expected the passive chip-retry styling, got:\n%s", html)
}
if strings.Contains(html, ">Summarize<") {
t.Errorf("the Summarize button must be hidden for a rate-limited video, got:\n%s", html)
}
}
// An ordinary unsummarized video still offers the Summarize button.
func TestVideoCard_UnsummarizedShowsSummarize(t *testing.T) {
html := renderVideoCard(t, store.SummaryRow{
VideoID: "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa",
Title: "Fresh Video",
Summarized: false,
})
if !strings.Contains(html, ">Summarize<") {
t.Errorf("expected a Summarize button, got:\n%s", html)
}
if strings.Contains(html, "Retrying later") {
t.Errorf("no retry badge for a non-rate-limited video, got:\n%s", html)
}
}
+12
View File
@@ -479,6 +479,10 @@ main { max-width: 60rem; margin: 0 auto; padding: var(--s4) var(--s3); }
.filters input { font: inherit; padding: .4rem .55rem; border: 1px solid var(--line); border-radius: var(--radius); background: var(--card); color: var(--fg); min-width: 9rem; } .filters input { font: inherit; padding: .4rem .55rem; border: 1px solid var(--line); border-radius: var(--radius); background: var(--card); color: var(--fg); min-width: 9rem; }
.filters input:focus-visible { outline: 2px solid var(--accent); outline-offset: 1px; border-color: var(--accent); } .filters input:focus-visible { outline: 2px solid var(--accent); outline-offset: 1px; border-color: var(--accent); }
.btn { font: inherit; font-weight: 600; padding: .45rem 1rem; border: 1px solid var(--accent); border-radius: var(--radius); background: var(--accent); color: var(--accent-fg); cursor: pointer; } .btn { font: inherit; font-weight: 600; padding: .45rem 1rem; border: 1px solid var(--accent); border-radius: var(--radius); background: var(--accent); color: var(--accent-fg); cursor: pointer; }
/* anchors styled as buttons: the generic a{} / a:visited{} colour rules outrank
.btn on <a>, painting the label accent-on-accent (invisible). Restore the
button foreground for anchor buttons, visited included. */
a.btn, a.btn:visited { color: var(--accent-fg); }
.btn:hover { filter: brightness(1.05); } .btn:hover { filter: brightness(1.05); }
.btn:active { transform: translateY(1px); } .btn:active { transform: translateY(1px); }
@@ -490,6 +494,9 @@ main { max-width: 60rem; margin: 0 auto; padding: var(--s4) var(--s3); }
.card-preview { color: var(--muted); font-size: .9rem; line-height: 1.5; display: -webkit-box; -webkit-line-clamp: 1; line-clamp: 1; -webkit-box-orient: vertical; overflow: hidden; } .card-preview { color: var(--muted); font-size: .9rem; line-height: 1.5; display: -webkit-box; -webkit-line-clamp: 1; line-clamp: 1; -webkit-box-orient: vertical; overflow: hidden; }
.card-foot { display: flex; gap: var(--s2); align-items: center; flex-wrap: wrap; margin-top: var(--s1); } .card-foot { display: flex; gap: var(--s2); align-items: center; flex-wrap: wrap; margin-top: var(--s1); }
.chip { display: inline-block; padding: .15rem .55rem; border-radius: 999px; background: var(--accent-weak); color: var(--accent); font-size: .72rem; font-weight: 600; } .chip { display: inline-block; padding: .15rem .55rem; border-radius: 999px; background: var(--accent-weak); color: var(--accent); font-size: .72rem; font-weight: 600; }
/* passive "retrying later" chip: dim/grey (CharmDim), not the accent — it is a
status, not an action the user can take. */
.chip-retry { background: rgba(108, 108, 108, .16); color: #6c6c6c; }
.card-state { color: var(--muted); font-size: .8rem; } .card-state { color: var(--muted); font-size: .8rem; }
.badge { display: inline-block; padding: .15rem .55rem; border-radius: 999px; background: var(--badge-bg); color: var(--badge-fg); font-size: .72rem; font-weight: 600; } .badge { display: inline-block; padding: .15rem .55rem; border-radius: 999px; background: var(--badge-bg); color: var(--badge-fg); font-size: .72rem; font-weight: 600; }
@@ -531,6 +538,11 @@ main { max-width: 60rem; margin: 0 auto; padding: var(--s4) var(--s3); }
.empty { text-align: center; color: var(--muted); padding: var(--s5) var(--s4); border: 1px dashed var(--line); border-radius: var(--radius); background: var(--card); } .empty { text-align: center; color: var(--muted); padding: var(--s5) var(--s4); border: 1px dashed var(--line); border-radius: var(--radius); background: var(--card); }
.empty strong { display: block; color: var(--fg); font-size: 1.05rem; margin-bottom: var(--s2); } .empty strong { display: block; color: var(--fg); font-size: 1.05rem; margin-bottom: var(--s2); }
.empty code { background: var(--accent-weak); color: var(--accent); padding: .1rem .35rem; border-radius: .3rem; } .empty code { background: var(--accent-weak); color: var(--accent); padding: .1rem .35rem; border-radius: .3rem; }
.empty p { margin: var(--s3) 0 0; }
/* connected-but-empty: a distinct accent callout, not a muted blank state, so a
fresh account knows the next step is to run tapir, not "something is broken". */
.empty-connected { border-style: solid; border-color: var(--accent); background: var(--accent-weak); color: var(--fg); }
.empty-connected strong { color: var(--accent); }
/* flash / notification banner */ /* flash / notification banner */
.flash { padding: var(--s2) var(--s3); border-radius: var(--radius); margin-bottom: var(--s4); font-size: .92rem; border: 1px solid var(--line); } .flash { padding: var(--s2) var(--s3); border-radius: var(--radius); margin-bottom: var(--s4); font-size: .92rem; border: 1px solid var(--line); }
+16 -6
View File
@@ -22,7 +22,7 @@ templ Layout(title string) {
<body> <body>
<header> <header>
<a href="/" class="brand">Tapir</a> <a href="/" class="brand">Tapir</a>
<nav class="nav"><a href="/account">Account</a></nav> <nav class="nav"><a href="/account">Account</a><a href="/auth/logout">Log out</a></nav>
</header> </header>
<main> <main>
{ children... } { children... }
@@ -59,7 +59,7 @@ templ WelcomePage(user User, loggedIn bool) {
<div class="welcome-cta"> <div class="welcome-cta">
<a class="btn btn-lg" href="/auth/login">Get Started</a> <a class="btn btn-lg" href="/auth/login">Get Started</a>
</div> </div>
<p class="welcome-sub">Already have an account? You'll go straight through.</p> <p class="welcome-sub">New to Tapir? Just sign in you'll complete a quick setup right after. Already have an account? You'll go straight through.</p>
} }
</section> </section>
} }
@@ -79,12 +79,12 @@ templ flashBanner(code string) {
// #summary-list region; a non-HTMX request renders the whole page. flash carries // #summary-list region; a non-HTMX request renders the whole page. flash carries
// a one-shot notification (e.g. "connected", "registered") surfaced on arrival // a one-shot notification (e.g. "connected", "registered") surfaced on arrival
// after a POST→redirect. // after a POST→redirect.
templ ListPage(rows []store.SummaryRow, f Filter, flash string) { templ ListPage(rows []store.SummaryRow, f Filter, flash string, hasConnected bool) {
@Layout("Tapir — Summaries") { @Layout("Tapir — Summaries") {
@flashBanner(flash) @flashBanner(flash)
@filterForm(f) @filterForm(f)
<div id="summary-list"> <div id="summary-list">
@summaryList(rows) @summaryList(rows, hasConnected)
</div> </div>
} }
} }
@@ -110,12 +110,20 @@ templ filterForm(f Filter) {
// summaryList is the swappable list fragment: one card per video (summarized or // summaryList is the swappable list fragment: one card per video (summarized or
// not). Cards reflow to a single column on mobile; an empty list shows a friendly // not). Cards reflow to a single column on mobile; an empty list shows a friendly
// first-run state instead of a blank table. // first-run state instead of a blank table.
templ summaryList(rows []store.SummaryRow) { templ summaryList(rows []store.SummaryRow, hasConnected bool) {
if len(rows) == 0 { if len(rows) == 0 {
if hasConnected {
<div class="empty empty-connected">
<strong>Your YouTube account is connected!</strong>
<span>Run <code>tapir run</code> to discover your subscriptions. Videos will appear here once discovered. In manual mode, each new video gets a Summarize button.</span>
</div>
} else {
<div class="empty"> <div class="empty">
<strong>No videos yet</strong> <strong>No videos yet</strong>
<span>Videos appear here as your subscriptions are processed run <code>tapir run</code> to fetch them. In manual mode, use the Summarize button to queue one.</span> <span>Connect your YouTube account to get started.</span>
<p><a class="btn" href="/oauth/youtube/connect">Connect YouTube</a></p>
</div> </div>
}
} else { } else {
<ul class="cards"> <ul class="cards">
for _, r := range rows { for _, r := range rows {
@@ -156,6 +164,8 @@ templ VideoCard(r store.SummaryRow) {
if len(r.Actions) > 0 { if len(r.Actions) > 0 {
<span class="card-state">{ strings.Join(r.Actions, ", ") }</span> <span class="card-state">{ strings.Join(r.Actions, ", ") }</span>
} }
} else if r.TranscriptStatus == "rate_limited" {
<span class="chip chip-retry" title="Caption fetch was rate-limited; tapir will retry automatically."> Retrying later</span>
} else if r.SummarizeRequested { } else if r.SummarizeRequested {
<span class="chip">Queued</span> <span class="chip">Queued</span>
<span class="card-state muted">waiting for the next run</span> <span class="card-state muted">waiting for the next run</span>
File diff suppressed because it is too large Load Diff