tapir/docs/homelab-integration.md

Tapir — Homelab Integration Facts

The concrete endpoints, conventions, and identifiers Tapir depends on, so an independent session doesn't have to rediscover them. Verify anything marked "confirm" before relying on it — endpoints and aliases drift, and this file is a snapshot (2026-06-06), not a live source.

Local AI (the Primary in llm.Router)

• LiteLLM gateway: OpenAI-compatible, lives in k3s namespace ai-stack (verified 2026-06-02 from koala via kubectl get svc -n ai-stack):
  • In-cluster: http://litellm.ai-stack.svc.cluster.local:4000/v1/ (ClusterIP 10.43.159.89:4000).
  • Off-cluster (host / NodePort): http://koala:30401/v1/ — service litellm-nodeport (4000:30401).
  • NOTE: the earlier 31234 was wrong — 31234 is the llama-swap NodePort (8080:31234), not LiteLLM. Use 30401 for off-cluster LiteLLM. confirm resolved.
• Speech-to-text (ADR-033): the same gateway serves POST /v1/audio/transcriptions for iguana/kb-whisper (primary, local) and berget/kb-whisper (fallback, EU cloud — private audio leaves the homelab if it is used). Call it via the NodePort / in-cluster address above, NOT https://llm-api.d-ma.be: that public ingress (openresty) returns 413 for bodies over roughly 8–10 MB (~5 minutes of 16 kHz WAV), while the NodePort took 174 MB (90 minutes) in one request (measured 2026-10-01, docs/research/audio-length-ceiling-2026-10-01.md). Send multipart field file, response_format=verbose_json, no Expect: 100-continue. iguana's whisper-server can wedge permanently after a Metal OOM (infra#273); never branch on its {"error":"failed to process audio"} string — it has had at least five causes. Tapir's knobs: TAPIR_UPLOADS_ENABLED (off by default), TAPIR_UPLOAD_MAX_MB (200), TAPIR_UPLOAD_MAX_MINUTES (90), TAPIR_TRANSCRIBE_MODEL, TAPIR_TRANSCRIBE_FALLBACK_MODEL (empty disables the cloud fallback), TAPIR_TRANSCRIBE_TIMEOUT (60m, the whole job), TAPIR_TRANSCRIBE_ENGINE_TIMEOUT (20m, one engine call: a wedged engine then fails while the job lives and the fallback can run), TAPIR_UPLOAD_TMP_DIR, TAPIR_FFMPEG_BIN, TAPIR_FFPROBE_BIN, TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS (90000 — the measured ceiling of iguana/gemma4-26b, which silently drops the head above it), TAPIR_UPLOAD_CHAT_MODEL (iguana/gemma4-26b; must be a local koala/ or iguana/ alias, since the whole private transcript is sent to it; koala/phi4-mini's 8,192-token context 400s on an hour of speech). ffmpeg + ffprobe are in the runtime image (static, pinned by sha256; #34 decided A on 2026-10-02); Tapir's own ingress body limit for a 200 MB upload is unverified (infra). Two-engine mode (ADR-034), off unless TAPIR_RECONCILE_BASE_MODEL is set: it runs iguana/pianissimo (the base text) and TAPIR_TRANSCRIBE_MODEL (kb-whisper) on the same audio, then a LOCAL chat model (TAPIR_RECONCILE_MODEL) chooses between their disagreements. Deployment choice (2026-10-02): iguana/gemma4-31b (97% against a speaker's blind answers; 92% for iguana/gemma4-26b). With iguana's macOS Low Power Mode off the 31B decodes at 22 tok/s and reconciled a 63.6-minute recording in 21 min 49 s end to end; with it on it ran at 8 tok/s. Check pmset -g | grep lowpowermode before blaming a model or an engine for slowness. TAPIR_RECONCILE_CHECK_ORDER=true asks twice per dispute with the order swapped (doubles the time). Every model must start koala/ or iguana/: a cloud alias is a start-up error. The default job timeout is 60 minutes. iguana/pianissimo is KlangAI's Swedish ASR (a Parakeet/FastConformer on MLX, served by a small FastAPI shell on iguana:8102) and speaks the same /v1/audio/transcriptions API.
• Auth key: treat as a secret via the SecretStore port; do not hardcode in committed code. NOTE: the documented sk-local-123 returned 401 on 30401/v1/models (2026-06-02) — it is stale or the gateway now enforces the LITELLM_MASTER_KEY (1Password op://HomeLab/LITELLM_MASTER_KEY). Resolve the live key from the vault when wiring; sk-local-123 is no longer valid. confirm partially resolved.
• Model alias format: host/name, e.g. koala/qwen3-coder-30b, koala/phi4-mini, iguana/devstral, iguana/deepseek-r1-14b. Not the ollama/ prefix form.
• Which alias for summarization: alias-as-config, confirm resolved (2026-06-02, Worker F). The alias is never hardcoded: it is TAPIR_SUMMARIZER_MODEL (format host/name), wired through the summarizer's Endpoint.Model. Default: koala/phi4-mini — a non-thinking instruct model chosen for safety: it cannot fall into the empty-content trap below, so the demo summarizes even if no one tunes it. It is provisional and overridable; final live alias selection happens at demo time when the gateway is reachable, where a more capable model (e.g. iguana/deepseek-r1-14b) is preferred for summary quality if its latency/output is acceptable. The max_tokens fix below means thinking models no longer return empty content, so they are now viable choices, not blocked ones. Do not assume a coder alias is right for prose.
• Summarizer fallback chain (ADR-022). The primary alias is the first of an ordered chain; on failure or unparseable output the summarizer advances to the next model. All reached through the same gateway by alias.
  • TAPIR_FALLBACK_MODEL — local fallback. Default iguana/gemma4-26b — on iguana, NOT koala, so the fallback does not compete with koala's other GPU loads (and runs from a different egress IP). Empty disables it.
  • TAPIR_CLOUD_FALLBACK_MODEL — worst-case EXTERNAL fallback. Default berget/mistral-small. Set this empty ("") for any client/NDA deployment so content never leaves the local stack — the chain then contains only local endpoints.
  • TAPIR_SUMMARY_MAX_TOKENS — per-summary completion budget. Default 1500. Small on purpose: with the old 8192 budget, prompt + completion overflowed phi4-mini's 8k window.
  • TAPIR_MAX_TRANSCRIPT_CHARS — transcript truncation budget sent to the model. Default 18000 (~fits an 8k-context model). 0 disables truncation. Prevents the context-overflow HTTP 400 a long transcript caused on phi4-mini.
• Discovery low-value filter (ADR-023). TAPIR_MIN_VIDEO_SECONDS — default 60. At discovery, NewVideos enriches candidates with one cheap videos.list call (quota API, NOT the timedtext 429 path) and drops videos shorter than this plus any live/upcoming broadcast, so the scarce caption-fetch budget isn't spent on Shorts. 0 disables the filter. The paste-a-URL path is never filtered.
• Per-channel caption memory (ADR-024). TAPIR_CHANNEL_CAPTIONLESS_THRESHOLD — default 5 consecutive no-caption results before a channel is suppressed (its videos listed but not caption-fetched). TAPIR_CHANNEL_CAPTIONLESS_WINDOW — default 336h (14d) suppression before one video is re-probed. THRESHOLD=0 disables. A successful fetch resets the channel; a 429 does not count; an explicit manual request bypasses suppression.
• Thinking models need an explicit max_tokens. qwen3 / deepseek-r1 spend the budget on reasoning and return empty content if max_tokens is too low (or unset). The summarizer's parser treats an empty summary as an error for exactly this reason. Done (2026-06-02, Worker F): the copied llm.Client now sends a generous max_tokens (8192) on every request per ADR-004, so thinking models no longer return empty content. A non-thinking instruct model remains the safe default (koala/phi4-mini), but thinking aliases are now viable.

This maps directly onto the copied llm package: Client is the OpenAI-compatible caller, Router.Primary points at this gateway with a chosen alias, Router.Fallback is the user's BYO.

BYO AI (the Fallback in llm.Router)

• Per-user, opt-in: Anthropic / OpenAI / Gemini. All reachable as OpenAI-compatible or via a thin adapter. The user's key is stored only as a secret reference (data-model AI_CREDENTIAL).
• Used only when Primary fails and the user has configured a provider (see docs/use-cases/ai_routing.feature). With no BYO, work is queued for retry — content is never sent externally.

Brain sink (optional delivery target)

• Reached via brain-mcp over HTTP, calling the brain_ingest tool (ADR-005). Do not use the filesystem brain package from hyperguild/ingestion.
• brain-mcp base URL: confirm — not pinned in this snapshot. The homelab convention is *-mcp.d-ma.be ingress hostnames (e.g. git-mcp.d-ma.be, brain-mcp.d-ma.be is the likely form), but verify before wiring.
• brain distinguishes knowledge/ (session-derived) from wiki/ (stable, manually promoted). Tapir summaries are session-derived signal → knowledge/-style ingestion. brain_ingest runs the LLM extraction pipeline; there is a raw-ingest path (brain_ingest_raw) and a reliable brain_write fallback when the local extraction model is unavailable — relevant if ingestion reliability becomes an issue.

Auth / identity (Stage 1+, dormant at Stage 0)

• Dex (auth.d-ma.be) is the homelab OIDC provider. When Tapir needs real user identity (Stage 1, Future B), it authenticates via Dex — not a new auth system (ADR-002).
• If Tapir ever exposes an MCP surface, use the mcp-chassis shared Go library (gitea.d-ma.be/mathias/mcp-chassis) for the Dex-JWT validator + Bearer middleware + RFC 9728 metadata, rather than hand-rolling it.

Secrets (OAuth tokens + BYO keys)

• Convention: ESO + 1Password (the homelab's External Secrets Operator against a 1Password vault). Tables store only an opaque *_secret_ref; the secret material lives in the vault and is surfaced to the workload via ESO.
• Exact ref format / vault item naming for Tapir: confirm / decide during build. Follow the pattern existing services use (e.g. how gitea-mcp references GITEA_MCP_DEFAULT_TOKEN) rather than inventing a new scheme.
  • Adapter status (YouTube): the YouTube VideoSource adapter does not assume a scheme. The OAuth refresh-token reference is an opaque Config.TokenSecretRef resolved through the SecretStore port (youtube.New(cfg, secrets)). Pinning the actual vault-item name only changes wiring/config, not the adapter — so this confirm does not block the adapter. Decide the name when wiring the live connection and record it here.
  • Per-user token-ref scheme (Stage 1 web connect): the web connect flow (/oauth/youtube/connect → /oauth/youtube/callback) persists each user's refresh token under a per-user ref youtube/<userID>/refresh_token (web.YouTubeTokenRef), not the Stage-0 single youtube/refresh_token. This is what keeps tokens isolated across tenants behind the SecretStore port; the video_connections row stores only this opaque token_ref, never the token. The connect callback URL is TAPIR_YT_CONNECT_REDIRECT_URL (default https://tapir.d-ma.be/oauth/youtube/callback) and must be in the Google OAuth client's authorized redirects for live connect.

Hosts (for reference)

• koala — Arch, RTX 5070, k3s single-node, Gitea, GPU inference. Where Tapir most likely runs and where the local models live.
• iguana — Mac Studio M2 Ultra, ollama + mlx-whisper, also serves models via the gateway.
• piguard — Pi, NPM perimeter (public HTTPS edge). Note: the May-2026 architecture review has LiteLLM relocating from piguard into k3s/ai-stack on koala — confirm current location if latency or endpoint matters.

Network egress the build and runtime assume

If the koala act_runner (or a dev environment) has locked-down egress, these must be reachable or the build/run fails. Split into build-time and runtime because they're different lists and fail at different stages.

Build-time (CI check/build, and local go/task work)

• proxy.golang.org, sum.golang.org — Go module proxy + checksum DB. The default GOPROXY. If a homelab GOPROXY (e.g. Athens) exists, point GOPROXY at it and this pair is not needed directly — confirm whether one is run.
• go.dev, dl.google.com — Go toolchain download (the CI setup-go step / go-version-file). Not needed if the runner already has the matching Go installed.
• raw.githubusercontent.com — the CI installs golangci-lint via its install script from here (.gitea/workflows/ci.yml).
• github.com, codeload.github.com — any module not served by the proxy (incl. golangci-lint and, later, golang.org/x/oauth2 and its deps) is fetched from GitHub.
• objects.githubusercontent.com — GitHub release asset downloads (some installers redirect here).

A configured GOPROXY collapses most of the module-fetch entries into one host. Recording the homelab's actual GOPROXY/GONOSUMCHECK policy here once known would remove the confirm.

Runtime (the running service)

• koala:4000 (or the NodePort / post-relocation address) — LiteLLM gateway, the Primary.
• brain-mcp host (*-mcp.d-ma.be, confirm) — only when the brain sink is enabled.
• www.googleapis.com / YouTube Data API + oauth2.googleapis.com (token exchange/refresh) — the YouTube provider adapter (internal/adapters/youtube). OAuth is hand-rolled on golang.org/x/oauth2 (ADR-006); the Google token endpoint is hardcoded inline to avoid the heavy golang.org/x/oauth2/google dependency.
  • confirm — captions.download ownership limitation. The Data API captions.download endpoint only authorizes the owner of the video; for arbitrary subscribed channels it typically returns 403. The adapter lists tracks via captions.list (no track → SourceNone, ADR-007) and downloads the selected track as WebVTT; a download failure surfaces as an error (retryable), not a silent skip. If the owner-only limit proves blocking in practice, the fallback is the unofficial timedtext endpoint or the deferred STT path (ADR-007) — a new ADR, not a default. Verify against a real non-owned video before relying on live caption download.
• api.vimeo.com — the Vimeo provider adapter.
• BYO-AI endpoints, per configured provider and only when a user opts in: api.anthropic.com, api.openai.com, generativelanguage.googleapis.com.
• 1Password / ESO reach for secret resolution follows the homelab's existing ESO egress, not Tapir-specific — no new rule expected.

Note: a user with no BYO configured means the service never needs the third-party AI hosts (the local-first guarantee, docs/use-cases/ai_routing.feature). Don't allow them by default; allow per-provider when a user connects one.

The claude.ai chat sandbox (where planning sessions run code) has its own separate egress allowlist — unrelated to koala. It blocked go.dev during scaffolding, which is why the scaffold was reviewed but not compile-verified in-chat; verification belongs on koala via task check regardless.

Deployment / GitOps (when Tapir reaches deploy)

• The homelab is Flux GitOps: manifests in mathias/infra under k3s/, Flux watches main.
• Tapir's k3s manifests will live in infra/k3s/apps/tapir/ (by convention) — not in this repo. This repo is the application; infra is the deployment source of truth.
• Multi-tenancy primitives (NetworkPolicy per namespace, Kyverno, postgres role-per-tenant, tenant= label) exist in the architecture review (SC7/P6) and apply at Stage 1 — not Stage 0.

---

Snapshot date 2026-06-06. Items marked confirm were not verified to a pinned source at snapshot time — check brain or the live cluster before depending on them.

Stage 1 — multi-user facts (verified 2026-06-03)

Postgres RLS (ADR-012)

• The deployed DSN MUST connect as a non-superuser, non-BYPASSRLS role. The app uses the tapir role (table owner, non-superuser). FORCE ROW LEVEL SECURITY is applied on all user-owned tables; a superuser DSN silently bypasses FORCE and isolation is dead in prod. Verify: SELECT rolsuper FROM pg_roles WHERE rolname = 'tapir' must return f.
• Scoping is via set_config('tapir.current_user_id', $userID, true) (transaction- local, auto-resets on commit — never leaks across a pooled connection).

Per-user YouTube token persistence

• Stage-1 uses the file-backed SecretStore at TAPIR_SECRETS_FILE=/data/secrets.json mounted from a PVC (tapir-secrets, 64Mi, RWO). Tokens survive pod restarts. Upgrading to an ESO-backed per-user SecretStore is backlog (infra#86).
• Per-user token ref scheme: youtube/<userID>/refresh_token (Worker C, ADR-006). The Stage-0 single ref youtube/refresh_token is no longer used by serve; it remains valid for the CLI tapir run (single-user, host-side).

Web YouTube connect

• Redirect URI (registered in Google OAuth client, type Web): https://tapir.d-ma.be/oauth/youtube/callback.
• Config env: TAPIR_YT_CONNECT_REDIRECT_URL=https://tapir.d-ma.be/oauth/youtube/callback. TAPIR_YT_CLIENT_ID / TAPIR_YT_CLIENT_SECRET from the Web client (not the Desktop client used for the CLI).

Identity resolution

• user_identities(dex_subject → user_id) table is intentionally NOT RLS-enabled (it's auth plumbing, holds no user data; data isolation is on the user-owned tables). All data access after subject resolution goes through withUser.

Scheduled discovery (ADR-018, verified 2026-06-05)

tapir serve runs discovery for all users in-process on a timer (no CronJob). Three env knobs plus one load-bearing deployment constraint:

• TAPIR_DISCOVERY_INTERVAL — Go duration, e.g. 2h. The cadence the serve process runs a discovery pass for every registered user (run-once-on-startup, then every interval). Unset or 0 = disabled (dev/tests never auto-fetch).
• TAPIR_USAGE_GATE_START — YYYY-MM-DD, default 2026-06-11 (the morning the pilot was unblocked and summaries started flowing). tapir report counts return-usage (distinct active weeks, ADR-016) only from this date, so pre-launch testing and the blocked period are excluded.
• TAPIR_METRICS_ADDR — listen address for the Prometheus /metrics endpoint (ADR-030). Default :9090 — a SEPARATE port from TAPIR_HTTP_ADDR so metrics are never on the public app; scraped in-cluster only (PodMonitor). Empty disables the metrics server. Key series: tapir_summarize_duration_seconds{model,outcome,fallback}, tapir_caption_fetch_duration_seconds{outcome}, tapir_chat_duration_seconds{model}, tapir_llm_tokens_total{model,kind}, tapir_http_request_duration_seconds{method,route}, tapir_logins_total.
• TAPIR_FETCH_RATE — Go duration, default 2s. The process-wide per-egress-IP caption-fetch rate gate (ADR-014 item 2). Every caption fetch — scheduler runners and the web "Summarize" click-path — serialises through this one limiter so the pod cannot collectively trip 429s. 0 = unlimited (dev/tests). This is the precondition that makes auto-summarize-on-a-schedule safe; do not raise it aggressively without watching for 429s.
• TAPIR_FETCH_BACKOFF=4h — per-video rate-limit retry window; default 1h. A video that returns HTTP 429 on a caption fetch is skipped for this duration before being retried. The scheduler checks NOW() - rate_limited_at > TAPIR_FETCH_BACKOFF before attempting to fetch a video marked transcript_status = rate_limited. Longer values reduce 429 pressure at the cost of slower recovery after a throttling episode.
• SINGLE-REPLICA WARNING (load-bearing). The scheduler lives in the web process, so replicas: 1 in the deployment manifest is load-bearing: running tapir serve at >1 replica makes every replica run the discovery loop → every user fetched in parallel from the same egress IP (429s + duplicate work). Do not scale serve past 1 replica without first moving discovery to a k8s CronJob or adding leader election. The process logs a Warn at startup when scheduled discovery is enabled, as a reminder.