Diagnosis of a live run: caption tracks resolve fine (player + watch-page scrape),
but the timedtext baseUrl fetch returns 429 under back-to-back volume — YouTube
rate-limits the unauthenticated caption-download endpoint per IP. A per-video
delay spaces the fetches. (Follow-up: treat 429 distinctly from genuine
no-caption instead of silently degrading to SourceNone; consider Whisper if the
endpoint stays hostile at any sustainable rate.)
The ANDROID InnerTube player now returns 0 captionTracks (PoToken-gated), so the
watch-page scrape is the real path — but it sent an Android *app* User-Agent,
which makes YouTube serve a page WITHOUT ytInitialPlayerResponse, so captions
came back empty. Result: 87% of a live run skipped as 'no transcript' despite
the videos having captions. A desktop-browser UA returns the player JSON with
captionTracks. Confirmed live: app-UA=0 tracks, browser-UA=1.
First-login chicken-egg: TAPIR_ALLOWED_SUBJECT can't be known until the user
logs in once, but the allowlist gates login. Echo the (non-secret, opaque)
subject in the forbidden response so the maintainer can read it in the browser,
set the 1P item, and lock the allowlist.
Implements the Top 5 fixes from the Stage-0 UX review (docs/ux-review/UX-REVIEW.md):
1. Dark mode: full light+dark custom-property palette (bg/fg/muted/line/accent/
card) under :root + @media (prefers-color-scheme: dark), applied to body.
color-scheme: light dark is now actually honoured — summary text was invisible
on a dark canvas before.
2. Table -> responsive card list: one card per summary (title link, channel·date
meta, provider chip, fallback badge, action state). Single-column reflow at
375px, no horizontal crush.
3. Minimal design system: 4/8px spacing scale, one accent, styled accent links
(underline-on-hover), real buttons with active/pressed state, consistent
radius and dividers — applied across list + detail.
4. Detail page as a reader: prose capped at 38rem, title->meta->summary->
highlights->takeaways hierarchy with section rules, 1.7 line-height. Meta is
built from non-empty parts (detailMeta) so the no-video edge case no longer
renders a stray "· — ·".
5. Contrast + a11y: muted bumped to #595959 (~7:1, clears WCAG AA), fallback
badge gets vertical padding + aria-label/title, friendly first-run empty
state, hx-indicator on the filter form.
Tests updated for the card markup (table -> cards); missing date is now omitted
rather than em-dashed. task check green; HTMX action toggles verified working.
Stage-0 web UI needs a self-contained container image. Two changes:
- Dockerfile: multi-stage build (golang:1.25 builder, CGO off + static
link, -trimpath -s -w) into distroless static nonroot. The committed
templ output and vendored asset mean a plain `go build` suffices — no
codegen or CDN at build/run time. Existing .gitea CI already builds and
pushes localhost:5000/tapir:<sha> + mirrors to GitHub (deploy patch
intentionally omitted — cutover is held), so it only needed this file.
- Vendor htmx 1.9.12 locally (internal/web/static/, embed.FS, served at
/static/ outside the auth guard) and point Layout at /static/htmx.min.js
instead of unpkg. The deployed UI must not depend on an external CDN
being reachable from the cluster.
task check green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
serve now uses oidc.DexAuth (single-user allowlist authz, ADR-011) when
TAPIR_OIDC_ISSUER is set, falling back to allow-all StubAuth for local dev.
Adds the Dex config fields (TAPIR_OIDC_ISSUER/DEX_CLIENT_ID/SECRET/
OIDC_REDIRECT_URL/SESSION_SECRET/ALLOWED_SUBJECT) + Config.DexConfigured().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the lane-C reader surface: list, detail, and an action button-group
fragment over the lane-A store reads/actions, behind the web.Auth seam.
- Templ components (base layout, list+filters, detail, ActionButtons) with
committed *_templ.go so go build/task check work without the templ binary;
`task generate` regenerates. Filters and action toggles are HTMX-swapped and
degrade to plain form GET/POST (POST→303→GET) without JS.
- Handlers (internal/web): GET / (channel+date filters, in-memory),
GET /v/{videoId}, POST /v/{videoId}/action (re-click clears, else SetAction;
store enforces watched↔skipped exclusion), GET /healthz (no auth). Store ops
run as the configured UserID; Auth only gates.
- `tapir serve` wires store + StubAuth{Subject: cfg.UserID} + http.Server on
TAPIR_HTTP_ADDR (default :8080), graceful shutdown on signal. Handlers depend
only on web.Auth — Conductor swaps StubAuth → oidc.DexAuth at merge (one line
in cmdServe).
- Handler tests: real store (embedded-postgres) + StubAuth — list rows+state,
HTMX fragment vs full page, channel filter, detail highlights/takeaways,
404, action toggle+clear, no-JS redirect, bad-verb 400.
New dep: github.com/a-h/templ — the house default for typed server-rendered
HTML (CLAUDE.md stack, ui-spec.md §3). Generated code is committed so the
templ binary is build-time-optional.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds internal/web/oidc.DexAuth, the production web.Auth impl behind the seam
(ADR-011, docs/ui-spec.md §6). Standard Authorization Code flow against Dex:
- Routes() mounts /auth/login (state+nonce, redirect to authorize),
/auth/callback (code exchange, ID-token verify, nonce check, allowlist:
sub must equal Config.AllowedSubject else 403, set session, redirect /),
/auth/logout (clear session).
- Middleware redirects unauthenticated requests to /auth/login, slides the
session expiry on each authenticated request; /healthz and /auth/* bypass.
- CurrentUser resolves the principal from the session cookie.
- Sessions: server-side in-memory store (single Stage-0 replica) keyed by an
HMAC-SHA256 (HS256) signed, HttpOnly, Secure, SameSite=Lax cookie with a
short TTL + sliding refresh. State->nonce pending map is one-time + expiring
(replay/CSRF defense). Tokens are never logged.
Constructor New(ctx, Config, ...Option); the six-field Config (Issuer,
ClientID, ClientSecret, RedirectURL, SessionSecret, AllowedSubject) is what
cmd/tapir wires from TAPIR_OIDC_*/TAPIR_DEX_*/TAPIR_SESSION_SECRET/
TAPIR_ALLOWED_SUBJECT. Options (clock, TTL, insecure cookies) are test-only.
Tests use a fake OIDC issuer via httptest (discovery + JWKS + token endpoint
signing an RS256 ID token) — no live Dex: login 302s to authorize; callback
for the allowlisted sub sets a session and 302s to /; non-allowlisted sub 403;
middleware redirects unauthenticated and passes authenticated; logout clears;
plus expiry, tampered-cookie, and unknown-state cases.
Deps (per ADR-006 / ui-spec §6): adds github.com/coreos/go-oidc/v3 — the
homelab-standard OIDC lib, small, handles discovery + JWKS + ID-token
verification; pairs with the already-present golang.org/x/oauth2. go-jose/v4
(transitive via go-oidc) is used directly only in tests to sign the fake
issuer's tokens.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lets the Dex session layer (lane B) and page handlers (lane C) build independently:
handlers depend only on web.Auth; oidc.DexAuth (B) and StubAuth (dev) implement it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The summary_actions table records the maintainer's act on a summary, the
column that makes the Stage-0 headline test ("acts on >=1 summary") queryable
(ui-spec.md §5, ADR-011). This is the gate lanes B/C build on.
- Migration 002: summary_actions (id, user_id, video_id TEXT, action, acted_at)
with a CHECK on action IN ('watched','skipped','saved') and a UNIQUE
(user_id, video_id, action). Per-user isolation: every row carries user_id.
- New actions.go: SetAction (idempotent, atomic watched<->skipped mutual
exclusion in one tx; saved independent), ClearAction, ActionsFor for the list
view, plus Go-side action validation.
- reads.go: additive SummaryRow.Actions, populated by composing ActionsFor
(Go-side, not a SQL join — summaries.video_id is UUID, actions.video_id TEXT).
- embedded-postgres tests: set/clear, mutual exclusion, saved coexistence,
idempotency, invalid rejection, user scoping, read-view surfacing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
NewVideos called search.list at 100 quota units/call. With ~143 channels one
discovery pass = 14,300 units > the 10,000/day YouTube Data API cap, exhausting
the whole day in a single loop (live quotaExceeded).
Switch to playlistItems.list (1 unit/call) against the channel's uploads
playlist. For a standard channel id UCxxxx the uploads playlist is UUxxxx,
derived at zero API cost (uploadsPlaylistID). Non-standard ids fall back to
channels.list (1 unit) to read contentDetails.relatedPlaylists.uploads. Newest-
first ordering and MaxVideosPerSubscription cap preserved.
Side effect: removing search.list also removes the accountDelegationForbidden
error that endpoint threw for one channel — no separate hardening needed.
New per-pass quota: /subscriptions (1) + ~1/channel discovery (143) + any
channels.list fallbacks ≈ 145 units/day, well under 10k. Caption fetch (ADR-010
timedtext) uses no Data API quota.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Data API captions.download endpoint is owner-only: every subscription
video the user does not own returned HTTP 403, producing 0 summaries and a
~150-line error spew in the first live Stage-0 run. Captions-first (ADR-007)
is sound; only the acquisition mechanism was wrong.
FetchTranscript now resolves caption tracks from the InnerTube player
response (ANDROID client, unauthenticated) and GETs the chosen track's
timedtext baseUrl with a plain http.Client — no OAuth token, which can break
the endpoint. The srv3 XML, json3, and legacy <transcript> formats all parse;
non-asr tracks in a preferred language win. Watch-page ytInitialPlayerResponse
scrape is the fallback when InnerTube returns no tracks.
Degrade, don't error (explicit quick-fix): no captionTracks, empty baseUrl, a
non-200 fetch, or an unparseable body yield Source=none, not an error. Only
genuine transport faults error — this kills the spew. OAuth stays on
ListSubscriptions/NewVideos (Data API); only transcript fetch goes unauthed.
Validated live from koala: the ANDROID client returned working baseUrls and
real transcript text for public videos the run identity does not own.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
main.go dispatches `tapir auth` (interactive OAuth → persist refresh token via
SecretStore) and `tapir run` (wire YouTube source + local summarizer + store
sink, build engine, run the dedup-aware loop). Config-driven so live creds plug
in at demo time; SIGINT stops the loop cleanly. Block kept minimal so Worker E's
list/show cases union cleanly at merge.
Add .env.example documenting every TAPIR_* var and a README demo runbook. Pin
the summarizer alias-as-config decision and record the max_tokens fix in
docs/homelab-integration.md (clears two `confirm` items).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RunOnce walks the user's subscriptions, upserts each candidate video (assigning
its durable store id), skips videos already summarized via the store's
SeenVideoIDs (cross-restart dedup the engine's in-memory map can't provide),
and processes the rest through the engine. Loop adds an optional poll cadence;
per-item errors are collected, not fatal. Tested with fakes — no live deps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Sink port carries only a Summary, so video title/url/published_at would
never reach the store. UpsertVideo (new file, store.go untouched) persists them
and returns the durable videos.id UUID, idempotent on
(user_id, provider, provider_video_id). The run loop uses that id as v.ID, so
it equals summaries.video_id and SeenVideoIDs dedup survives restarts.
subscription_id stays NULL: the YouTube resource id is not a UUID (Stage 0).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`tapir auth` mints a refresh token for the single Stage-0 user: bind a local
redirect listener, print the consent URL (offline access + forced consent so
Google returns a refresh token), validate the state param, exchange the code,
and persist the refresh token through the SecretStore port. Written fresh on
x/oauth2 (ADR-006). Token is never logged or returned. Tests cover exchange,
missing-refresh-token rejection, and the full listener flow with httptest.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ListSummaries (recent-first, user-scoped, limit) and GetSummaryByVideo
LEFT JOIN videos for title/url/published_at, null-safe when no videos
row exists. Channel mirrors provider for now — channel_title lives on
the not-yet-migrated subscriptions table (data-model.md). New file so it
does not collide with Worker F's concurrent edits to store.go.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements ports.SecretStore over a 0600 JSON file as a stand-in for op/ESO so
the demo runs without live op. Put persists atomically (temp + rename) and
merges; Get returns ErrNotFound for unknown refs so a missing token fails loud.
Behind the port, so swapping to op/ESO later is wiring, not code (ADR-002).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Parse TAPIR_* env into a typed Config with homelab defaults (gateway URL,
summarizer model, token ref, redirect addr). Secrets (gateway key, OAuth
client secret) come from env only; the refresh token never lives here — it is
addressed by an opaque ref behind the SecretStore port. Per-command validation
(ValidateForAuth/ValidateForRun) so each command demands only what it needs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The copied OpenAI-compatible client sent no max_tokens. Thinking models
(qwen3, deepseek-r1) spend their budget on the reasoning trace and return
EMPTY content when max_tokens is unset, which the summarizer treats as an
error. ADR-004 says change Tapir's copy rather than the hyperguild upstream,
so set a generous default (8192) leaving room for both reasoning and output.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements ports.Sink over Postgres (pgx/v5 + pgxpool, DSN from env per
estate convention). This is the primary sink (ADR-003) and the source of
the engine's durable, cross-restart dedup — the in-engine processed map is
process-lifetime only.
- Migrations (golang-migrate, NNN_name.up/down.sql per estate convention,
applied from an embedded FS on New): users, videos, transcripts,
summaries, sink_deliveries. Every user-owned table carries user_id
(Stage-0 per-user isolation promise, data-model.md). summaries has
UNIQUE(user_id, video_id) — at most one summary per video; highlights /
takeaways are jsonb.
- Deliver upserts the summary idempotently on (user_id, video_id)
(ON CONFLICT DO UPDATE) inside one tx with its sink_delivery row. Re-
delivering the same summary updates in place, never duplicates or errors.
- Dedup reads (store methods, not a new port): HasSummary(ctx,userID,
videoID) and SeenVideoIDs(ctx,userID) — both user_id-scoped, so one
user never sees another's videos.
summaries.video_id is intentionally not FK-constrained to videos at Stage 0:
the sink receives only a Summary, so the dedup key stands alone; video-row
persistence is the engine/source's concern, deferred.
Tested against a real in-process Postgres via embedded-postgres (real SQL:
constraints, ON CONFLICT, jsonb, user_id scoping) — no docker, no live
cluster, no creds, fully offline.
Deps: golang-migrate/migrate/v4 and jackc/pgx/v5 (runtime),
fergusstrange/embedded-postgres + stretchr/testify (test-only). go mod tidy
raised the go directive to 1.25.0 (minimum required by the dep graph;
estate elsewhere already runs 1.26.1).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements ports.VideoSource against the YouTube Data API v3:
ListSubscriptions (paginated), NewVideos (recent per channel), and
captions-first FetchTranscript — an absent caption track yields
domain.SourceNone (not an error) per ADR-007, with no audio download
or speech-to-text.
OAuth is written fresh on golang.org/x/oauth2 (ADR-006, distinct from
ingestion's inbound MCP auth); the Google token endpoint is inlined to
avoid the heavy x/oauth2/google dep. The per-connection refresh token is
resolved through the SecretStore port from an opaque TokenSecretRef and
is never stored on the adapter or logged.
Unit-tested against an httptest server + fake SecretStore (no live
googleapis egress): subscriptions list/pagination, new-video detection,
captions present -> Source set, captions absent -> SourceNone no error,
and secret-ref resolution failure surfacing as an error.
oauth2 pinned to v0.30.0 to keep the go directive at 1.23.x (koala
runner), not the v0.36 line that requires a newer toolchain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implement ports.Summarizer in internal/adapters/summarizer. It routes through a
local Primary endpoint first and an optional BYO Fallback, owning the routing
itself (not delegating to llm.Router) so it can record AIProvider, AIModel, and
FallbackUsed on domain.Summary. Prompt asks for JSON {summary, highlights,
takeaways}; the parser tolerates thinking-model fences/reasoning and rejects an
empty summary.
The summarizer is the single egress point for content toward an AI model, so it
enforces the local-first guarantee from ai_routing.feature: with no BYO
configured (nil fallback) there is no external endpoint, so content reaches the
local stack and nowhere else. Tests assert all four scenarios via a fake client.
Model alias is config (TAPIR_SUMMARIZER_MODEL, host/name) — not hardcoded;
docs/homelab-integration.md notes it stays `confirm` and that thinking models
need an explicit max_tokens or they return empty content.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copy hyperguild/ingestion/internal/llm into internal/adapters/llm and own it.
Tapir owes that repo nothing at the dependency level — no module dep added.
Router gives the local-Primary -> BYO-Fallback path needed for ai_routing.feature.
Copied tests rewritten from testify to stdlib testing to keep go.mod
dependency-free (repo has zero deps; acceptance tests are stdlib too).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ProcessNewVideos walks a user's subscriptions and processes each newly
seen video. Two remaining .feature scenarios are now covered: a channel
the user is not subscribed to is never surfaced (so never processed), and
a video already processed in this engine's lifetime is not summarized
twice. Dedup is in-memory and per-user; durable cross-restart dedup stays
the store's concern (no new port), per docs/data-model.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve transcript -> summarize -> deliver, or skip when no usable
transcript. Sinks fail independently: a failing sink does not abort the
others and successful deliveries are kept; per-sink errors are returned
joined. Makes the two scaffolded acceptance scenarios GREEN.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Engine wires the ports and exposes ProcessNewVideo, the core use case. Returns
ErrNotImplemented on purpose so the acceptance suite fails RED — implementing it
to make those tests pass is the first build task. Depends only on ports + domain
(dependencies point inward).
The hexagonal interfaces the engine depends on. Keeps the engine provider- and
sink-agnostic: YouTube/Vimeo implement VideoSource, the AI router implements
Summarizer, store/brain implement Sink, ESO implements SecretStore. This is what
makes standalone-vs-homelab a wiring choice (ADR-003).
Pure domain types matching docs/data-model.md: User, Subscription, Video,
Transcript, Summary, plus Provider and TranscriptSource enums. Stdlib-only,
no outward dependencies — the innermost layer. Video carries user_id per the
per-user-isolation decision (no global dedup).
mathias
