Wire the metrics package into the live paths and serve it:
- summarizer: per-endpoint latency by model/outcome(success|error|parse_error)/fallback + slog.
- youtube.FetchTranscript: latency by outcome (captions|none|rate_limited) + slog.
- chat: answer latency by model + slog.
- llm usage hook → token counts (prompt|completion) per model, wired in buildSummarizer/buildChat.
- oidc callback: login counter.
- cmdServe: wrap Router in metrics.HTTPMiddleware (request count + latency by bounded
route pattern) and serve /metrics on TAPIR_METRICS_ADDR (default :9090), a SEPARATE
port — never on the public app mux.
BDD: observability.feature scenarios un-pended + mapped. TDD: summarizer wiring tested
black-box via the /metrics scrape; metrics-not-on-public-mux asserted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two knobs for the onboarding-burst quality work:
- TAPIR_ONBOARD_SUMMARIZER_MODEL (default iguana/gemma4-26b): the stronger model
the burst leads its chain with; empty collapses the burst onto the shared
processor (lookupOr, so explicit-empty disables).
- TAPIR_ONBOARD_MAX_VIDEO_SECONDS (default 14400/4h): upper duration bound for
burst picks; 0 disables, negative clamps to 0.
Table-driven tests cover defaults, explicit, disable, and invalid input.
koala now carries other GPU loads, so the first fallback should not run there.
Change the default chain to koala/phi4-mini → iguana/gemma4-26b → berget/mistral-small:
the local fallback now runs on iguana (M2 Ultra headroom, different host = different
egress IP for the rare fallback fetch). gemma4-26b is the brain-validated homelab
general-purpose model (agentsquad H2/H3 executor) and returned valid summary JSON on
the real prompt in a smoke test (~37s incl. cold-load — fine for a fallback path).
Pure config default (TAPIR_FALLBACK_MODEL); chain mechanism (ADR-022) unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After transcript caching (ADR-021) and the Shorts filter (ADR-023), the remaining
caption waste is the first fetch on every new video of a channel that never has
English captions — each costs one rate-limited fetch to resolve to "none", and on
a throttled IP churns the backoff machinery first.
Remember, per (user, channel), a streak of consecutive no-caption outcomes
(channel_caption_state, migration 016, RLS-scoped). Once it reaches
TAPIR_CHANNEL_CAPTIONLESS_THRESHOLD (default 5) the channel is suppressed — videos
discovered/listed but not caption-fetched — for TAPIR_CHANNEL_CAPTIONLESS_WINDOW
(default 14d), then one is re-probed (auto-recovery). A successful fetch resets
the streak; a 429 does not count; an explicit manual request bypasses suppression.
threshold=0 disables.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The scarce resource is the per-IP timedtext caption fetch (ADR-014); the pilot's
candidate set was mostly Shorts/clips/livestreams, each burning a fetch (a "none"
result is a completed fetch — it costs budget even when it yields nothing).
NewVideos now enriches candidates with one cheap Data API videos.list call
(contentDetails.duration + snippet.liveBroadcastContent — the quota API, a
DIFFERENT limit from the timedtext 429) and drops, before returning: videos
shorter than TAPIR_MIN_VIDEO_SECONDS (default 60) and any live/upcoming
broadcast. Dropped videos are never persisted, so the list declutters too.
Degrade-open: MinVideoSeconds=0 disables it (no quota call); a videos.list error
returns candidates unfiltered so discovery never breaks on a metadata hiccup. The
paste-a-URL path (VideoByID) is not filtered — an explicit request is honoured.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first friendly-pilot live run produced zero summaries: koala/phi4-mini hit
three silent failure modes — 8k context overflow on long transcripts (HTTP 400),
intermittent malformed JSON (highlights as a bare string), and no fallback wired
at all (summarizer.New(primary, nil)).
Keep phi4-mini as the fast primary and add resilience around it:
- Ordered endpoint chain (summarizer.NewChain): phi4-mini → koala/phi4-14b
(local) → berget/mistral-small (worst-case external). All reached through the
one LiteLLM gateway by alias.
- A parse failure now advances the chain like a transport error — the old
Primary→Fallback shape returned the parse error without trying anyone else.
- Tolerant parse: highlights/takeaways coerce string→[]string, absorbing the
common small-model quirk without spending a fallback round-trip.
- Transcript truncation (TAPIR_MAX_TRANSCRIPT_CHARS=18000) prevents the overflow
rather than recovering from it; validated to fit phi4-mini's 8k window.
- Bounded completion budget (TAPIR_SUMMARY_MAX_TOKENS=1500) — the old 8192 budget
itself contributed to the overflow.
Local-first guarantee preserved by ordering: external endpoint is tried only
after every local one fails. TAPIR_CLOUD_FALLBACK_MODEL="" disables it entirely
for client/NDA deployments.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bounds the connect-time onboarding summary burst (Feature 1). Hard-capped at 5
and clamped (negative->0, >cap->cap) so onboarding can never bulk-fetch; 0
disables. The cap bounds COUNT only — every fetch still flows through the shared
caption rate gate (ADR-014).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In automatic mode the scheduler now only summarizes videos published within
TAPIR_AUTO_SUMMARIZE_WINDOW (default ~7d). Older videos are still discovered
and listed — they keep the manual "Summarize" affordance — but are not
auto-processed, so a large back-catalogue (the maintainer's ~256-deep queue)
stops self-inflicting 429s against the per-IP caption gate each cycle (UX
review B1, recency design).
- runner.WithAutoWindow + Stats.SkippedTooOld; tooOld() treats a zero window
as disabled and an undated video as never-aged-out (processed, not stranded).
- An explicit manual request bypasses the bound even in auto mode (requested
videos are loaded in auto mode when a window is active).
- Wired through cmdRun, the scheduler's per-user runner, sumStats, and pass
logging. config: TAPIR_AUTO_SUMMARIZE_WINDOW (default 168h), .env.example.
- Account copy (A7) updated to match: "Automatic summarizes new videos from
about the last week; older videos stay browsable — summarize on demand."
The rate gate is untouched; the manual path still serialises through it. This
bounds auto LOAD, it does not fetch harder.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ADR-014 item 2 — a single per-egress-IP rate gate shared by every caption
fetch — was specced but only per-video backoff (rate_limited_at) shipped.
Build the real gate now: it is load-bearing once ADR-018 puts auto-summarize
on an in-process schedule across multiple users (all fetches leave one pod's
egress IP, concurrently with live "Summarize" clicks — without a shared gate
that self-inflicts 429s every cycle).
globalFetchGate (golang.org/x/time/rate, default 2s/req burst 1) is consulted
in httpDo before every live outbound fetch — player, watch-page, timedtext —
so the scheduler runners and the web click-path serialise through one limiter
regardless of how many users/goroutines are upstream. The test seam
(a.transport != nil) skips the gate so fakes are not throttled.
TAPIR_FETCH_RATE (Go duration, default 2s, 0 = unlimited) wires SetFetchRate in
cmdRun; the existing per-video backoff stays as the complementary 429 handler.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mints a single-use invitation and prints the absolute claim URL for the
operator to send. The URL base is TAPIR_PUBLIC_URL (default
https://tapir.d-ma.be). runInvite is factored from config/store wiring so
it's unit-tested against a fake inviter — no Postgres.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds FetchBackoff (Go duration, default 1h) controlling how long the run loop
waits before re-fetching a transcript that returned HTTP 429. Zero means always
retry. Not required by ValidateForRun — a zero/unset value is a valid policy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add GET /oauth/youtube/connect and /oauth/youtube/callback, mounted inside
the login + registration guard so CurrentUserID is always set and every
connection binds to the authenticated tapir user.
- connect: generate a per-user CSRF state (single-use, short TTL, bound to
the user), redirect to Google consent with access_type=offline and
prompt=consent so a refresh token comes back.
- callback: verify the state belongs to this user, exchange the code via the
existing auth.Exchange, persist the refresh token under a PER-USER ref
(web.YouTubeTokenRef = "youtube/<userID>/refresh_token") so tenants never
collide, then UpsertConnection (provider=youtube, status=active). Any
failure renders a clean error page and leaves no half-written state.
Reuses auth.Exchange and adds auth.AuthCodeURL (offline + consent) rather
than the CLI's listener/terminal flow (ADR-006: web flow, not CLI). The
ConnectHandler depends on a narrow web.Connections port, not the concrete
store. Wired in cmdServe only when YT client credentials are present;
TAPIR_YT_CONNECT_REDIRECT_URL configures the callback URL. Per-user token-ref
scheme documented in docs/homelab-integration.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ADR-011's single-user authz (ID-token subject must equal AllowedSubject,
else 403) is replaced by ADR-012's model: Dex authentication is the only
gate — any Dex-authenticated subject may establish a session. Whether that
subject has a tapir user, and routing to registration if not, is decided
downstream in internal/web (next commit).
Removals (noted): oidc.Config.AllowedSubject + its required-field check + the
callback 403 branch; config.Config.AllowedSubject + TAPIR_ALLOWED_SUBJECT env
wiring; the AllowedSubject arg in cmdServe. ui-spec.md updated to reflect the
supersession. Sessions, cookie signing, login/callback/logout unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
serve now uses oidc.DexAuth (single-user allowlist authz, ADR-011) when
TAPIR_OIDC_ISSUER is set, falling back to allow-all StubAuth for local dev.
Adds the Dex config fields (TAPIR_OIDC_ISSUER/DEX_CLIENT_ID/SECRET/
OIDC_REDIRECT_URL/SESSION_SECRET/ALLOWED_SUBJECT) + Config.DexConfigured().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the lane-C reader surface: list, detail, and an action button-group
fragment over the lane-A store reads/actions, behind the web.Auth seam.
- Templ components (base layout, list+filters, detail, ActionButtons) with
committed *_templ.go so go build/task check work without the templ binary;
`task generate` regenerates. Filters and action toggles are HTMX-swapped and
degrade to plain form GET/POST (POST→303→GET) without JS.
- Handlers (internal/web): GET / (channel+date filters, in-memory),
GET /v/{videoId}, POST /v/{videoId}/action (re-click clears, else SetAction;
store enforces watched↔skipped exclusion), GET /healthz (no auth). Store ops
run as the configured UserID; Auth only gates.
- `tapir serve` wires store + StubAuth{Subject: cfg.UserID} + http.Server on
TAPIR_HTTP_ADDR (default :8080), graceful shutdown on signal. Handlers depend
only on web.Auth — Conductor swaps StubAuth → oidc.DexAuth at merge (one line
in cmdServe).
- Handler tests: real store (embedded-postgres) + StubAuth — list rows+state,
HTMX fragment vs full page, channel filter, detail highlights/takeaways,
404, action toggle+clear, no-JS redirect, bad-verb 400.
New dep: github.com/a-h/templ — the house default for typed server-rendered
HTML (CLAUDE.md stack, ui-spec.md §3). Generated code is committed so the
templ binary is build-time-optional.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Parse TAPIR_* env into a typed Config with homelab defaults (gateway URL,
summarizer model, token ref, redirect addr). Secrets (gateway key, OAuth
client secret) come from env only; the refresh token never lives here — it is
addressed by an opaque ref behind the SecretStore port. Per-command validation
(ValidateForAuth/ValidateForRun) so each command demands only what it needs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mathiasandClaude Opus 4.8