Implements ports.Sink over Postgres (pgx/v5 + pgxpool, DSN from env per
estate convention). This is the primary sink (ADR-003) and the source of
the engine's durable, cross-restart dedup — the in-engine processed map is
process-lifetime only.
- Migrations (golang-migrate, NNN_name.up/down.sql per estate convention,
applied from an embedded FS on New): users, videos, transcripts,
summaries, sink_deliveries. Every user-owned table carries user_id
(Stage-0 per-user isolation promise, data-model.md). summaries has
UNIQUE(user_id, video_id) — at most one summary per video; highlights /
takeaways are jsonb.
- Deliver upserts the summary idempotently on (user_id, video_id)
(ON CONFLICT DO UPDATE) inside one tx with its sink_delivery row. Re-
delivering the same summary updates in place, never duplicates or errors.
- Dedup reads (store methods, not a new port): HasSummary(ctx,userID,
videoID) and SeenVideoIDs(ctx,userID) — both user_id-scoped, so one
user never sees another's videos.
summaries.video_id is intentionally not FK-constrained to videos at Stage 0:
the sink receives only a Summary, so the dedup key stands alone; video-row
persistence is the engine/source's concern, deferred.
Tested against a real in-process Postgres via embedded-postgres (real SQL:
constraints, ON CONFLICT, jsonb, user_id scoping) — no docker, no live
cluster, no creds, fully offline.
Deps: golang-migrate/migrate/v4 and jackc/pgx/v5 (runtime),
fergusstrange/embedded-postgres + stretchr/testify (test-only). go mod tidy
raised the go directive to 1.25.0 (minimum required by the dep graph;
estate elsewhere already runs 1.26.1).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements ports.VideoSource against the YouTube Data API v3:
ListSubscriptions (paginated), NewVideos (recent per channel), and
captions-first FetchTranscript — an absent caption track yields
domain.SourceNone (not an error) per ADR-007, with no audio download
or speech-to-text.
OAuth is written fresh on golang.org/x/oauth2 (ADR-006, distinct from
ingestion's inbound MCP auth); the Google token endpoint is inlined to
avoid the heavy x/oauth2/google dep. The per-connection refresh token is
resolved through the SecretStore port from an opaque TokenSecretRef and
is never stored on the adapter or logged.
Unit-tested against an httptest server + fake SecretStore (no live
googleapis egress): subscriptions list/pagination, new-video detection,
captions present -> Source set, captions absent -> SourceNone no error,
and secret-ref resolution failure surfacing as an error.
oauth2 pinned to v0.30.0 to keep the go directive at 1.23.x (koala
runner), not the v0.36 line that requires a newer toolchain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implement ports.Summarizer in internal/adapters/summarizer. It routes through a
local Primary endpoint first and an optional BYO Fallback, owning the routing
itself (not delegating to llm.Router) so it can record AIProvider, AIModel, and
FallbackUsed on domain.Summary. Prompt asks for JSON {summary, highlights,
takeaways}; the parser tolerates thinking-model fences/reasoning and rejects an
empty summary.
The summarizer is the single egress point for content toward an AI model, so it
enforces the local-first guarantee from ai_routing.feature: with no BYO
configured (nil fallback) there is no external endpoint, so content reaches the
local stack and nowhere else. Tests assert all four scenarios via a fake client.
Model alias is config (TAPIR_SUMMARIZER_MODEL, host/name) — not hardcoded;
docs/homelab-integration.md notes it stays `confirm` and that thinking models
need an explicit max_tokens or they return empty content.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copy hyperguild/ingestion/internal/llm into internal/adapters/llm and own it.
Tapir owes that repo nothing at the dependency level — no module dep added.
Router gives the local-Primary -> BYO-Fallback path needed for ai_routing.feature.
Copied tests rewritten from testify to stdlib testing to keep go.mod
dependency-free (repo has zero deps; acceptance tests are stdlib too).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Conductor verified against the live koala k3s cluster before launching the
adapter workers: LiteLLM is in ns ai-stack, ClusterIP 10.43.159.89:4000,
off-cluster NodePort 30401. The doc's 31234 was actually llama-swap. Also
flag that sk-local-123 now 401s — use LITELLM_MASTER_KEY from the vault.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ProcessNewVideos walks a user's subscriptions and processes each newly
seen video. Two remaining .feature scenarios are now covered: a channel
the user is not subscribed to is never surfaced (so never processed), and
a video already processed in this engine's lifetime is not summarized
twice. Dedup is in-memory and per-user; durable cross-restart dedup stays
the store's concern (no new port), per docs/data-model.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve transcript -> summarize -> deliver, or skip when no usable
transcript. Sinks fail independently: a failing sink does not abort the
others and successful deliveries are kept; per-sink errors are returned
joined. Makes the two scaffolded acceptance scenarios GREEN.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
go-task 3.51.1 read the unquoted scalar 'echo "gofmt: files..."' as a
key:value mapping, failing every task with 'invalid keys in command' and
blocking the whole quality gate (not just the acceptance suite). Wrap the
command in an explicit double-quoted YAML scalar.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mathiasandClaude Opus 4.8