The Data API captions.download endpoint is owner-only: every subscription
video the user does not own returned HTTP 403, producing 0 summaries and a
~150-line error spew in the first live Stage-0 run. Captions-first (ADR-007)
is sound; only the acquisition mechanism was wrong.
FetchTranscript now resolves caption tracks from the InnerTube player
response (ANDROID client, unauthenticated) and GETs the chosen track's
timedtext baseUrl with a plain http.Client — no OAuth token, which can break
the endpoint. The srv3 XML, json3, and legacy <transcript> formats all parse;
non-asr tracks in a preferred language win. Watch-page ytInitialPlayerResponse
scrape is the fallback when InnerTube returns no tracks.
Degrade, don't error (explicit quick-fix): no captionTracks, empty baseUrl, a
non-200 fetch, or an unparseable body yield Source=none, not an error. Only
genuine transport faults error — this kills the spew. OAuth stays on
ListSubscriptions/NewVideos (Data API); only transcript fetch goes unauthed.
Validated live from koala: the ANDROID client returned working baseUrls and
real transcript text for public videos the run identity does not own.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tapir auth binds localhost:8080 on koala and prints the consent URL (no browser
auto-open), so it works headless via 'ssh -L 8080:localhost:8080 koala'. run/list/
show are already non-interactive; document the 'op run --env-file' invocation with
a service-account token so secrets resolve without an interactive signin. Also
correct the stale 'Pre-code' status.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Worker F's README told users to 'cp .env.example .env' but a blanket .env.*
gitignore rule silently dropped it. Un-ignore .env.example (real .env stays
ignored) and generate the template from internal/config: every TAPIR_* var,
which command needs it, accurate defaults, op/port-forward notes for demo time.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
main.go dispatches `tapir auth` (interactive OAuth → persist refresh token via
SecretStore) and `tapir run` (wire YouTube source + local summarizer + store
sink, build engine, run the dedup-aware loop). Config-driven so live creds plug
in at demo time; SIGINT stops the loop cleanly. Block kept minimal so Worker E's
list/show cases union cleanly at merge.
Add .env.example documenting every TAPIR_* var and a README demo runbook. Pin
the summarizer alias-as-config decision and record the max_tokens fix in
docs/homelab-integration.md (clears two `confirm` items).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RunOnce walks the user's subscriptions, upserts each candidate video (assigning
its durable store id), skips videos already summarized via the store's
SeenVideoIDs (cross-restart dedup the engine's in-memory map can't provide),
and processes the rest through the engine. Loop adds an optional poll cadence;
per-item errors are collected, not fatal. Tested with fakes — no live deps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Sink port carries only a Summary, so video title/url/published_at would
never reach the store. UpsertVideo (new file, store.go untouched) persists them
and returns the durable videos.id UUID, idempotent on
(user_id, provider, provider_video_id). The run loop uses that id as v.ID, so
it equals summaries.video_id and SeenVideoIDs dedup survives restarts.
subscription_id stays NULL: the YouTube resource id is not a UUID (Stage 0).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`tapir auth` mints a refresh token for the single Stage-0 user: bind a local
redirect listener, print the consent URL (offline access + forced consent so
Google returns a refresh token), validate the state param, exchange the code,
and persist the refresh token through the SecretStore port. Written fresh on
x/oauth2 (ADR-006). Token is never logged or returned. Tests cover exchange,
missing-refresh-token rejection, and the full listener flow with httptest.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tapir list — table of stored summaries (date, title|id, channel, AI,
fallback), recent-first. tapir show <video-id> — full summary with
highlights and takeaways. DSN + user id from TAPIR_DB_DSN/TAPIR_USER_ID,
never hardcoded. main.go gains a minimal os.Args[1] dispatcher kept flat
so Worker F's auth/run cases union cleanly at merge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ListSummaries (recent-first, user-scoped, limit) and GetSummaryByVideo
LEFT JOIN videos for title/url/published_at, null-safe when no videos
row exists. Channel mirrors provider for now — channel_title lives on
the not-yet-migrated subscriptions table (data-model.md). New file so it
does not collide with Worker F's concurrent edits to store.go.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements ports.SecretStore over a 0600 JSON file as a stand-in for op/ESO so
the demo runs without live op. Put persists atomically (temp + rename) and
merges; Get returns ErrNotFound for unknown refs so a missing token fails loud.
Behind the port, so swapping to op/ESO later is wiring, not code (ADR-002).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Parse TAPIR_* env into a typed Config with homelab defaults (gateway URL,
summarizer model, token ref, redirect addr). Secrets (gateway key, OAuth
client secret) come from env only; the refresh token never lives here — it is
addressed by an opaque ref behind the SecretStore port. Per-command validation
(ValidateForAuth/ValidateForRun) so each command demands only what it needs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The copied OpenAI-compatible client sent no max_tokens. Thinking models
(qwen3, deepseek-r1) spend their budget on the reasoning trace and return
EMPTY content when max_tokens is unset, which the summarizer treats as an
error. ADR-004 says change Tapir's copy rather than the hyperguild upstream,
so set a generous default (8192) leaving room for both reasoning and output.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements ports.Sink over Postgres (pgx/v5 + pgxpool, DSN from env per
estate convention). This is the primary sink (ADR-003) and the source of
the engine's durable, cross-restart dedup — the in-engine processed map is
process-lifetime only.
- Migrations (golang-migrate, NNN_name.up/down.sql per estate convention,
applied from an embedded FS on New): users, videos, transcripts,
summaries, sink_deliveries. Every user-owned table carries user_id
(Stage-0 per-user isolation promise, data-model.md). summaries has
UNIQUE(user_id, video_id) — at most one summary per video; highlights /
takeaways are jsonb.
- Deliver upserts the summary idempotently on (user_id, video_id)
(ON CONFLICT DO UPDATE) inside one tx with its sink_delivery row. Re-
delivering the same summary updates in place, never duplicates or errors.
- Dedup reads (store methods, not a new port): HasSummary(ctx,userID,
videoID) and SeenVideoIDs(ctx,userID) — both user_id-scoped, so one
user never sees another's videos.
summaries.video_id is intentionally not FK-constrained to videos at Stage 0:
the sink receives only a Summary, so the dedup key stands alone; video-row
persistence is the engine/source's concern, deferred.
Tested against a real in-process Postgres via embedded-postgres (real SQL:
constraints, ON CONFLICT, jsonb, user_id scoping) — no docker, no live
cluster, no creds, fully offline.
Deps: golang-migrate/migrate/v4 and jackc/pgx/v5 (runtime),
fergusstrange/embedded-postgres + stretchr/testify (test-only). go mod tidy
raised the go directive to 1.25.0 (minimum required by the dep graph;
estate elsewhere already runs 1.26.1).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements ports.VideoSource against the YouTube Data API v3:
ListSubscriptions (paginated), NewVideos (recent per channel), and
captions-first FetchTranscript — an absent caption track yields
domain.SourceNone (not an error) per ADR-007, with no audio download
or speech-to-text.
OAuth is written fresh on golang.org/x/oauth2 (ADR-006, distinct from
ingestion's inbound MCP auth); the Google token endpoint is inlined to
avoid the heavy x/oauth2/google dep. The per-connection refresh token is
resolved through the SecretStore port from an opaque TokenSecretRef and
is never stored on the adapter or logged.
Unit-tested against an httptest server + fake SecretStore (no live
googleapis egress): subscriptions list/pagination, new-video detection,
captions present -> Source set, captions absent -> SourceNone no error,
and secret-ref resolution failure surfacing as an error.
oauth2 pinned to v0.30.0 to keep the go directive at 1.23.x (koala
runner), not the v0.36 line that requires a newer toolchain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implement ports.Summarizer in internal/adapters/summarizer. It routes through a
local Primary endpoint first and an optional BYO Fallback, owning the routing
itself (not delegating to llm.Router) so it can record AIProvider, AIModel, and
FallbackUsed on domain.Summary. Prompt asks for JSON {summary, highlights,
takeaways}; the parser tolerates thinking-model fences/reasoning and rejects an
empty summary.
The summarizer is the single egress point for content toward an AI model, so it
enforces the local-first guarantee from ai_routing.feature: with no BYO
configured (nil fallback) there is no external endpoint, so content reaches the
local stack and nowhere else. Tests assert all four scenarios via a fake client.
Model alias is config (TAPIR_SUMMARIZER_MODEL, host/name) — not hardcoded;
docs/homelab-integration.md notes it stays `confirm` and that thinking models
need an explicit max_tokens or they return empty content.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copy hyperguild/ingestion/internal/llm into internal/adapters/llm and own it.
Tapir owes that repo nothing at the dependency level — no module dep added.
Router gives the local-Primary -> BYO-Fallback path needed for ai_routing.feature.
Copied tests rewritten from testify to stdlib testing to keep go.mod
dependency-free (repo has zero deps; acceptance tests are stdlib too).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Conductor verified against the live koala k3s cluster before launching the
adapter workers: LiteLLM is in ns ai-stack, ClusterIP 10.43.159.89:4000,
off-cluster NodePort 30401. The doc's 31234 was actually llama-swap. Also
flag that sk-local-123 now 401s — use LITELLM_MASTER_KEY from the vault.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ProcessNewVideos walks a user's subscriptions and processes each newly
seen video. Two remaining .feature scenarios are now covered: a channel
the user is not subscribed to is never surfaced (so never processed), and
a video already processed in this engine's lifetime is not summarized
twice. Dedup is in-memory and per-user; durable cross-restart dedup stays
the store's concern (no new port), per docs/data-model.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve transcript -> summarize -> deliver, or skip when no usable
transcript. Sinks fail independently: a failing sink does not abort the
others and successful deliveries are kept; per-sink errors are returned
joined. Makes the two scaffolded acceptance scenarios GREEN.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
go-task 3.51.1 read the unquoted scalar 'echo "gofmt: files..."' as a
key:value mapping, failing every task with 'invalid keys in command' and
blocking the whole quality gate (not just the acceptance suite). Wrap the
command in an explicit double-quoted YAML scalar.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lists the egress the build assumes (Go module proxy, toolchain download,
raw.githubusercontent for golangci-lint, github for non-proxied modules) and the
runtime assumes (LiteLLM gateway, brain-mcp, YouTube/Vimeo APIs, per-user BYO-AI
hosts only on opt-in), so a locked-down koala act_runner or dev env knows what to
allow or which GOPROXY to set. Notes the claude.ai sandbox allowlist is separate
and unrelated.
Tells an agent to wire skills via `task skills` (gitignored symlinks, never
committed) and which skills matter for Tapir; documents the scaffolded-and-RED
state with the first build task spelled out; flags the unverified setup items
(Go version, brain-mcp URL, secret-ref naming, model alias) to resolve against
the live cluster.
Excludes build artifacts and the .claude/skills symlink (wired by the skills
installer, never committed — matches the mathias/skills convention) and local
env files.
check -> build -> mirror, self-hosted runner, buildah to localhost:5000, k3s
smoke test. Follows the gitea-ci skill template and its act_runner gotchas
(secrets inlined in run:, no heredocs). check will be RED until the engine is
implemented (acceptance suite). Deploy job omitted until k3s manifests exist in
infra. GH_DEPLOY_KEY secret must be set before mirror succeeds.
Defines `task check` (fmt-check + vet + lint + test) — the gate ADR-009 and CI
both invoke. Also `task skills` to wire the engineering skills library via the
canonical installer (symlinks, gitignored). lint no-ops locally when
golangci-lint is absent; CI installs it.
Minimal main that identifies the binary (gives the CI smoke test something to
grep). HTTP server, watcher, adapter wiring, and config come with the build.
Executable translation of docs/use-cases/summarize_new_video.feature: captioned
video is summarized and delivered to the store sink; no-transcript video is
skipped with reason "no transcript" and no delivery. Drives the engine through
fake adapters (no live YouTube/brain). Intentionally RED until the engine is
implemented — this is the swarm's target.
Engine wires the ports and exposes ProcessNewVideo, the core use case. Returns
ErrNotImplemented on purpose so the acceptance suite fails RED — implementing it
to make those tests pass is the first build task. Depends only on ports + domain
(dependencies point inward).
The hexagonal interfaces the engine depends on. Keeps the engine provider- and
sink-agnostic: YouTube/Vimeo implement VideoSource, the AI router implements
Summarizer, store/brain implement Sink, ESO implements SecretStore. This is what
makes standalone-vs-homelab a wiring choice (ADR-003).
Pure domain types matching docs/data-model.md: User, Subscription, Video,
Transcript, Summary, plus Provider and TranscriptSource enums. Stdlib-only,
no outward dependencies — the innermost layer. Video carries user_id per the
per-user-isolation decision (no global dedup).
Go module root for Tapir. Go 1.23 — confirm against the koala act_runner
toolchain; bump to match the estate (ingestion uses 1.26.1) if the runner has it.
Adds a consolidated table of approaches considered and deliberately not taken
(Python, Supabase, living in the monolith, shared-lib lift, filesystem brain
package, reusing inbound oauth, global dedup table, Whisper-in-core, SaaS-now,
swarm-delegating the spike), each mapped to the ADR that settles it. Prevents a
later session from re-proposing settled rejections as fresh ideas.
Operational context for independent agent sessions: orientation order, TBD +
conventional-commit workflow, the three "looks reusable but isn't" traps (llm is
copied not imported, brain sink is HTTP not filesystem, OAuth is fresh), the
settled decisions not to reopen, the Clean Architecture/BDD stance, and a
provenance trail back to the S5 spike and the llm source. Distills the operating
knowledge surfaced during the 2026-06-02 planning+grill session.
Points anyone (or any agent) landing cold at the vision, decisions, architecture,
data model, and BDD feature specs; states the Clean Architecture / TDD-BDD / TBD
approach and the homelab conventions reused. Notes the repo is pre-code and the
docs are the version-controlled design intent.
Gherkin spec for connecting YouTube/Vimeo accounts and configuring optional
per-provider BYO AI credentials: connections sync subscriptions, tokens/keys are
stored only as secret references (never in the clear), revocation stops watching
but preserves history. Encodes the secrets-by-reference and data-isolation
guardrails (ADR-002, ADR-006, data-model).
Gherkin spec for the routing guardrail: local produces the summary by default;
on local failure, fall back only to a user-configured BYO provider; with no BYO,
queue for retry and never send content to a third-party model. Encodes the
local-first/user-owned principle (VISION) as executable behavior.
Gherkin spec for the core use case: captioned video is summarized and delivered;
no-transcript video is recorded as skipped; unsubscribed channels are ignored;
already-summarized videos are not reprocessed. These scenarios seed the use-case
test suite (Clean Architecture core tested through fake adapters).
Per-user-isolated entities (no global cross-tenant video table per the S5/grill
correction), secrets stored by reference only (ESO/1Password, never the token),
brain delivery modelled as a sink_delivery row rather than brain-specific tables.
fallback_used recorded per summary as the Stage 0 quality signal. Future C dedup
and sharding explicitly out of scope.
Context and container diagrams, sequence diagrams for the core summarize-new-video
use case and the local-first/BYO AI routing fallback, and the Clean Architecture
layering. Ports & adapters keep the engine provider- and sink-agnostic, making
standalone-vs-homelab a wiring choice (ADR-003), not two codebases.
Records the decisions from the S5 spike and the Full Grill as append-only ADRs:
Go not Python; no Supabase; standalone-first with brain as one sink; copy the
llm package; brain sink via HTTP brain-mcp; fresh outbound OAuth; captions-first
with STT deferred; Future C deferred behind the Stage 0 gate; trunk-based dev.
The top-level guardrail for Tapir: the problem, the product, the principles
(local-first, standalone-first, attention-as-scarce-resource, data isolation),
who it's for (now / Future B / deferred Future C), and a staged, falsifiable
definition of success with Stage 0 ("useful to me") as the gate.
mathiasandClaude Opus 4.8