tapir/docs/homelab-integration.md

# Tapir — Homelab Integration Facts

The concrete endpoints, conventions, and identifiers Tapir depends on, so an independent
session doesn't have to rediscover them. **Verify anything marked "confirm" before relying on
it** — endpoints and aliases drift, and this file is a snapshot (2026-06-06), not a live source.

## Local AI (the Primary in `llm.Router`)

- **LiteLLM gateway:** OpenAI-compatible, lives in k3s namespace `ai-stack` (verified 2026-06-02
  from koala via `kubectl get svc -n ai-stack`):
  - **In-cluster:** `http://litellm.ai-stack.svc.cluster.local:4000/v1/` (ClusterIP `10.43.159.89:4000`).
  - **Off-cluster (host / NodePort):** `http://koala:30401/v1/` — service `litellm-nodeport` (`4000:30401`).
  - NOTE: the earlier `31234` was **wrong** — `31234` is the `llama-swap` NodePort (`8080:31234`),
    not LiteLLM. Use `30401` for off-cluster LiteLLM. `confirm` resolved.
- **Speech-to-text (ADR-033):** the same gateway serves `POST /v1/audio/transcriptions` for
  `iguana/kb-whisper` (primary, local) and `berget/kb-whisper` (fallback, EU cloud — private audio
  leaves the homelab if it is used). **Call it via the NodePort / in-cluster address above, NOT
  `https://llm-api.d-ma.be`:** that public ingress (openresty) returns `413` for bodies over
  roughly 8–10 MB (~5 minutes of 16 kHz WAV), while the NodePort took 174 MB (90 minutes) in one
  request (measured 2026-10-01, `docs/research/audio-length-ceiling-2026-10-01.md`). Send multipart
  field `file`, `response_format=verbose_json`, no `Expect: 100-continue`. iguana's whisper-server
  can wedge permanently after a Metal OOM (infra#273); never branch on its
  `{"error":"failed to process audio"}` string — it has had at least five causes. Tapir's knobs:
  `TAPIR_UPLOADS_ENABLED` (off by default), `TAPIR_UPLOAD_MAX_MB` (200), `TAPIR_UPLOAD_MAX_MINUTES`
  (90), `TAPIR_TRANSCRIBE_MODEL`, `TAPIR_TRANSCRIBE_FALLBACK_MODEL` (empty disables the cloud
  fallback), `TAPIR_TRANSCRIBE_TIMEOUT` (60m, the whole job), `TAPIR_TRANSCRIBE_ENGINE_TIMEOUT` (20m, one engine call: a wedged engine then fails while the job lives and the fallback can run), `TAPIR_UPLOAD_TMP_DIR`, `TAPIR_FFMPEG_BIN`,
  `TAPIR_FFPROBE_BIN`, `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS` (90000 — the measured ceiling of
  `iguana/gemma4-26b`, which silently drops the head above it), `TAPIR_UPLOAD_CHAT_MODEL`
  (`iguana/gemma4-26b`; must be a local `koala/` or `iguana/` alias, since the whole private transcript is sent to it; `koala/phi4-mini`'s 8,192-token context 400s on an hour of speech). `ffmpeg` + `ffprobe` are in the runtime image (static, pinned by sha256; #34 decided A on 2026-10-02); Tapir's own ingress body limit for a 200 MB upload is unverified (infra).
  **Two-engine mode (ADR-034), off unless `TAPIR_RECONCILE_BASE_MODEL` is set:** it runs
  `iguana/pianissimo` (the base text) and `TAPIR_TRANSCRIBE_MODEL` (kb-whisper) on the same audio, then a
  LOCAL chat model (`TAPIR_RECONCILE_MODEL`) chooses between their disagreements. **Deployment choice (2026-10-02): `iguana/gemma4-31b`** (97% against a speaker's blind answers; 92% for `iguana/gemma4-26b`). With iguana's macOS **Low Power Mode off** the 31B decodes at 22 tok/s and reconciled a 63.6-minute recording in 21 min 49 s end to end; with it on it ran at 8 tok/s. Check `pmset -g | grep lowpowermode` before blaming a model or an engine for slowness.
  `TAPIR_RECONCILE_CHECK_ORDER=true` asks twice per dispute with the order swapped (doubles the time).
  Every model must start `koala/` or `iguana/`: a cloud alias is a start-up error. The default job timeout is
  60 minutes. `iguana/pianissimo` is KlangAI's Swedish ASR (a Parakeet/FastConformer on MLX, served by a small
  FastAPI shell on iguana:8102) and speaks the same `/v1/audio/transcriptions` API.
- **Auth key:** treat as a secret via the SecretStore port; do not hardcode in committed code.
  NOTE: the documented `sk-local-123` returned `401` on `30401/v1/models` (2026-06-02) — it is
  stale or the gateway now enforces the `LITELLM_MASTER_KEY` (1Password `op://HomeLab/LITELLM_MASTER_KEY`).
  Resolve the live key from the vault when wiring; `sk-local-123` is no longer valid. `confirm` partially resolved.
- **Model alias format:** `host/name`, e.g. `koala/qwen3-coder-30b`, `koala/phi4-mini`,
  `iguana/devstral`, `iguana/deepseek-r1-14b`. **Not** the `ollama/` prefix form.
- **Which alias for summarization:** alias-as-config, **`confirm` resolved** (2026-06-02, Worker F).
  The alias is never hardcoded: it is `TAPIR_SUMMARIZER_MODEL` (format `host/name`), wired through
  the summarizer's `Endpoint.Model`. **Default: `koala/phi4-mini`** — a non-thinking instruct model
  chosen for safety: it cannot fall into the empty-content trap below, so the demo summarizes even
  if no one tunes it. It is provisional and overridable; **final live alias selection happens at
  demo time when the gateway is reachable**, where a more capable model (e.g.
  `iguana/deepseek-r1-14b`) is preferred for summary quality if its latency/output is acceptable.
  The `max_tokens` fix below means thinking models no longer return empty content, so they are now
  viable choices, not blocked ones. Do not assume a coder alias is right for prose.
- **Summarizer fallback chain (ADR-022).** The primary alias is the *first* of an ordered chain;
  on failure or unparseable output the summarizer advances to the next model. All reached through
  the same gateway by alias.
  - `TAPIR_FALLBACK_MODEL` — local fallback. **Default `iguana/gemma4-26b`** — on iguana, NOT
    koala, so the fallback does not compete with koala's other GPU loads (and runs from a different
    egress IP). Empty disables it.
  - `TAPIR_CLOUD_FALLBACK_MODEL` — worst-case EXTERNAL fallback. **Default `berget/mistral-small`.**
    **Set this empty (`""`) for any client/NDA deployment** so content never leaves the local
    stack — the chain then contains only local endpoints.
  - `TAPIR_SUMMARY_MAX_TOKENS` — per-summary completion budget. **Default `1500`.** Small on
    purpose: with the old 8192 budget, prompt + completion overflowed `phi4-mini`'s 8k window.
  - `TAPIR_MAX_TRANSCRIPT_CHARS` — transcript truncation budget sent to the model. **Default
    `18000`** (~fits an 8k-context model). `0` disables truncation. Prevents the context-overflow
    HTTP 400 a long transcript caused on `phi4-mini`.
- **Discovery low-value filter (ADR-023).** `TAPIR_MIN_VIDEO_SECONDS` — **default `60`**. At
  discovery, `NewVideos` enriches candidates with one cheap `videos.list` call (quota API, NOT
  the timedtext 429 path) and drops videos shorter than this plus any live/upcoming broadcast,
  so the scarce caption-fetch budget isn't spent on Shorts. `0` disables the filter. The
  paste-a-URL path is never filtered.
- **Per-channel caption memory (ADR-024).** `TAPIR_CHANNEL_CAPTIONLESS_THRESHOLD` — **default
  `5`** consecutive no-caption results before a channel is suppressed (its videos listed but not
  caption-fetched). `TAPIR_CHANNEL_CAPTIONLESS_WINDOW` — **default `336h`** (14d) suppression
  before one video is re-probed. `THRESHOLD=0` disables. A successful fetch resets the channel;
  a 429 does not count; an explicit manual request bypasses suppression.
- **Thinking models need an explicit `max_tokens`.** qwen3 / deepseek-r1 spend the budget on
  reasoning and return **empty content** if `max_tokens` is too low (or unset). The summarizer's
  parser treats an empty summary as an error for exactly this reason. **Done (2026-06-02, Worker F):**
  the copied `llm.Client` now sends a generous `max_tokens` (8192) on every request per ADR-004, so
  thinking models no longer return empty content. A non-thinking instruct model remains the safe
  default (`koala/phi4-mini`), but thinking aliases are now viable.

This maps directly onto the copied `llm` package: `Client` is the OpenAI-compatible caller,
`Router.Primary` points at this gateway with a chosen alias, `Router.Fallback` is the user's BYO.

## BYO AI (the Fallback in `llm.Router`)

- Per-user, opt-in: Anthropic / OpenAI / Gemini. All reachable as OpenAI-compatible or via a thin
  adapter. The user's key is stored only as a secret reference (data-model `AI_CREDENTIAL`).
- Used **only** when Primary fails and the user has configured a provider (see
  `docs/use-cases/ai_routing.feature`). With no BYO, work is queued for retry — content is never
  sent externally.

## Brain sink (optional delivery target)

- Reached via **brain-mcp** over HTTP, calling the `brain_ingest` tool (ADR-005). Do **not** use
  the filesystem `brain` package from `hyperguild/ingestion`.
- **brain-mcp base URL:** **confirm** — not pinned in this snapshot. The homelab convention is
  `*-mcp.d-ma.be` ingress hostnames (e.g. `git-mcp.d-ma.be`, `brain-mcp.d-ma.be` is the likely
  form), but verify before wiring.
- brain distinguishes `knowledge/` (session-derived) from `wiki/` (stable, manually promoted).
  Tapir summaries are session-derived signal → `knowledge/`-style ingestion. `brain_ingest` runs
  the LLM extraction pipeline; there is a raw-ingest path (`brain_ingest_raw`) and a reliable
  `brain_write` fallback when the local extraction model is unavailable — relevant if ingestion
  reliability becomes an issue.

## Auth / identity (Stage 1+, dormant at Stage 0)

- **Dex** (`auth.d-ma.be`) is the homelab OIDC provider. When Tapir needs real user identity
  (Stage 1, Future B), it authenticates via Dex — not a new auth system (ADR-002).
- If Tapir ever exposes an MCP surface, use the **`mcp-chassis`** shared Go library
  (`gitea.d-ma.be/mathias/mcp-chassis`) for the Dex-JWT validator + Bearer middleware + RFC 9728
  metadata, rather than hand-rolling it.

## Secrets (OAuth tokens + BYO keys)

- Convention: **ESO + 1Password** (the homelab's External Secrets Operator against a 1Password
  vault). Tables store only an opaque `*_secret_ref`; the secret material lives in the vault and
  is surfaced to the workload via ESO.
- **Exact ref format / vault item naming for Tapir:** **confirm / decide during build.** Follow
  the pattern existing services use (e.g. how `gitea-mcp` references `GITEA_MCP_DEFAULT_TOKEN`)
  rather than inventing a new scheme.
  - **Adapter status (YouTube):** the YouTube `VideoSource` adapter does **not** assume a scheme.
    The OAuth refresh-token reference is an opaque `Config.TokenSecretRef` resolved through the
    `SecretStore` port (`youtube.New(cfg, secrets)`). Pinning the actual vault-item name only
    changes wiring/config, not the adapter — so this `confirm` does not block the adapter. Decide
    the name when wiring the live connection and record it here.
  - **Per-user token-ref scheme (Stage 1 web connect):** the web connect flow
    (`/oauth/youtube/connect` → `/oauth/youtube/callback`) persists each user's refresh token
    under a **per-user ref `youtube/<userID>/refresh_token`** (`web.YouTubeTokenRef`), not the
    Stage-0 single `youtube/refresh_token`. This is what keeps tokens isolated across tenants
    behind the `SecretStore` port; the `video_connections` row stores only this opaque
    `token_ref`, never the token. The connect callback URL is
    `TAPIR_YT_CONNECT_REDIRECT_URL` (default `https://tapir.d-ma.be/oauth/youtube/callback`) and
    must be in the Google OAuth client's authorized redirects for live connect.

## Hosts (for reference)

- **koala** — Arch, RTX 5070, k3s single-node, Gitea, GPU inference. Where Tapir most likely runs
  and where the local models live.
- **iguana** — Mac Studio M2 Ultra, ollama + mlx-whisper, also serves models via the gateway.
- **piguard** — Pi, NPM perimeter (public HTTPS edge). Note: the May-2026 architecture review has
  LiteLLM relocating from piguard into k3s/ai-stack on koala — **confirm current location** if
  latency or endpoint matters.

## Network egress the build and runtime assume

If the koala act_runner (or a dev environment) has locked-down egress, these must be reachable
or the build/run fails. Split into build-time and runtime because they're different lists and
fail at different stages.

### Build-time (CI `check`/`build`, and local `go`/`task` work)

- `proxy.golang.org`, `sum.golang.org` — Go module proxy + checksum DB. The default `GOPROXY`.
  If a homelab GOPROXY (e.g. Athens) exists, point `GOPROXY` at it and this pair is not needed
  directly — **confirm** whether one is run.
- `go.dev`, `dl.google.com` — Go toolchain download (the CI `setup-go` step / `go-version-file`).
  Not needed if the runner already has the matching Go installed.
- `raw.githubusercontent.com` — the CI installs golangci-lint via its install script from here
  (`.gitea/workflows/ci.yml`).
- `github.com`, `codeload.github.com` — any module not served by the proxy (incl. golangci-lint
  and, later, `golang.org/x/oauth2` and its deps) is fetched from GitHub.
- `objects.githubusercontent.com` — GitHub release asset downloads (some installers redirect here).

A configured `GOPROXY` collapses most of the module-fetch entries into one host. Recording the
homelab's actual `GOPROXY`/`GONOSUMCHECK` policy here once known would remove the **confirm**.

### Runtime (the running service)

- `koala:4000` (or the NodePort / post-relocation address) — LiteLLM gateway, the Primary.
- brain-mcp host (`*-mcp.d-ma.be`, **confirm**) — only when the brain sink is enabled.
- `www.googleapis.com` / YouTube Data API + `oauth2.googleapis.com` (token exchange/refresh) —
  the YouTube provider adapter (`internal/adapters/youtube`). OAuth is hand-rolled on
  `golang.org/x/oauth2` (ADR-006); the Google token endpoint is hardcoded inline to avoid the
  heavy `golang.org/x/oauth2/google` dependency.
  - **`confirm` — captions.download ownership limitation.** The Data API `captions.download`
    endpoint only authorizes the *owner* of the video; for arbitrary subscribed channels it
    typically returns `403`. The adapter lists tracks via `captions.list` (no track → `SourceNone`,
    ADR-007) and downloads the selected track as WebVTT; a download failure surfaces as an error
    (retryable), not a silent skip. If the owner-only limit proves blocking in practice, the
    fallback is the unofficial `timedtext` endpoint or the deferred STT path (ADR-007) — a new
    ADR, not a default. Verify against a real non-owned video before relying on live caption
    download.
- `api.vimeo.com` — the Vimeo provider adapter.
- BYO-AI endpoints, per configured provider and **only when a user opts in**:
  `api.anthropic.com`, `api.openai.com`, `generativelanguage.googleapis.com`.
- 1Password / ESO reach for secret resolution follows the homelab's existing ESO egress, not
  Tapir-specific — no new rule expected.

Note: a user with **no** BYO configured means the service never needs the third-party AI hosts
(the local-first guarantee, `docs/use-cases/ai_routing.feature`). Don't allow them by default;
allow per-provider when a user connects one.

> The claude.ai chat sandbox (where planning sessions run code) has its **own** separate egress
> allowlist — unrelated to koala. It blocked `go.dev` during scaffolding, which is why the
> scaffold was reviewed but not compile-verified in-chat; verification belongs on koala via
> `task check` regardless.

## Deployment / GitOps (when Tapir reaches deploy)

- The homelab is **Flux GitOps**: manifests in `mathias/infra` under `k3s/`, Flux watches `main`.
- Tapir's k3s manifests will live in `infra/k3s/apps/tapir/` (by convention) — not in this repo.
  This repo is the application; `infra` is the deployment source of truth.
- Multi-tenancy primitives (NetworkPolicy per namespace, Kyverno, postgres role-per-tenant,
  `tenant=` label) exist in the architecture review (SC7/P6) and apply at Stage 1 — not Stage 0.

---

_Snapshot date 2026-06-06. Items marked **confirm** were not verified to a pinned source at
snapshot time — check brain or the live cluster before depending on them._

## Stage 1 — multi-user facts (verified 2026-06-03)

### Postgres RLS (ADR-012)
- **The deployed DSN MUST connect as a non-superuser, non-BYPASSRLS role.** The
  app uses the `tapir` role (table owner, non-superuser). `FORCE ROW LEVEL
  SECURITY` is applied on all user-owned tables; a superuser DSN silently bypasses
  FORCE and isolation is dead in prod. Verify: `SELECT rolsuper FROM pg_roles
  WHERE rolname = 'tapir'` must return `f`.
- Scoping is via `set_config('tapir.current_user_id', $userID, true)` (transaction-
  local, auto-resets on commit — never leaks across a pooled connection).

### Per-user YouTube token persistence
- Stage-1 uses the **file-backed SecretStore** at `TAPIR_SECRETS_FILE=/data/secrets.json`
  mounted from a **PVC** (`tapir-secrets`, 64Mi, RWO). Tokens survive pod restarts.
  Upgrading to an ESO-backed per-user SecretStore is backlog (infra#86).
- Per-user token ref scheme: `youtube/<userID>/refresh_token` (Worker C, ADR-006).
  The Stage-0 single ref `youtube/refresh_token` is no longer used by `serve`; it
  remains valid for the CLI `tapir run` (single-user, host-side).

### Web YouTube connect
- Redirect URI (registered in Google OAuth client, type Web): `https://tapir.d-ma.be/oauth/youtube/callback`.
- Config env: `TAPIR_YT_CONNECT_REDIRECT_URL=https://tapir.d-ma.be/oauth/youtube/callback`.
  `TAPIR_YT_CLIENT_ID` / `TAPIR_YT_CLIENT_SECRET` from the Web client (not the Desktop client used for the CLI).

### Identity resolution
- `user_identities(dex_subject → user_id)` table is **intentionally NOT RLS-enabled**
  (it's auth plumbing, holds no user data; data isolation is on the user-owned tables).
  All data access after subject resolution goes through `withUser`.

## Scheduled discovery (ADR-018, verified 2026-06-05)

`tapir serve` runs discovery for **all users** in-process on a timer (no CronJob). Three env
knobs plus one load-bearing deployment constraint:

- `TAPIR_DISCOVERY_INTERVAL` — Go duration, e.g. `2h`. The cadence the serve process runs a
  discovery pass for every registered user (run-once-on-startup, then every interval).
  **Unset or `0` = disabled** (dev/tests never auto-fetch).
- `TAPIR_USAGE_GATE_START` — `YYYY-MM-DD`, default **`2026-06-11`** (the morning the pilot was
  unblocked and summaries started flowing). `tapir report` counts return-usage (distinct active
  weeks, ADR-016) only from this date, so pre-launch testing and the blocked period are excluded.
- `TAPIR_METRICS_ADDR` — listen address for the Prometheus `/metrics` endpoint (ADR-030).
  **Default `:9090`** — a SEPARATE port from `TAPIR_HTTP_ADDR` so metrics are never on the public
  app; scraped in-cluster only (PodMonitor). Empty disables the metrics server. Key series:
  `tapir_summarize_duration_seconds{model,outcome,fallback}`, `tapir_caption_fetch_duration_seconds{outcome}`,
  `tapir_chat_duration_seconds{model}`, `tapir_llm_tokens_total{model,kind}`,
  `tapir_http_request_duration_seconds{method,route}`, `tapir_logins_total`.
- `TAPIR_FETCH_RATE` — Go duration, default `2s`. The **process-wide per-egress-IP caption-fetch
  rate gate** (ADR-014 item 2). Every caption fetch — scheduler runners *and* the web "Summarize"
  click-path — serialises through this one limiter so the pod cannot collectively trip 429s. `0`
  = unlimited (dev/tests). This is the precondition that makes auto-summarize-on-a-schedule safe;
  do not raise it aggressively without watching for 429s.
- `TAPIR_FETCH_BACKOFF=4h` — per-video rate-limit retry window; default `1h`. A video that
  returns HTTP 429 on a caption fetch is skipped for this duration before being retried. The
  scheduler checks `NOW() - rate_limited_at > TAPIR_FETCH_BACKOFF` before attempting to fetch
  a video marked `transcript_status = rate_limited`. Longer values reduce 429 pressure at the
  cost of slower recovery after a throttling episode.
- **SINGLE-REPLICA WARNING (load-bearing).** The scheduler lives in the web process, so
  `replicas: 1` in the deployment manifest is load-bearing: running `tapir serve` at >1 replica
  makes **every** replica run the discovery loop → every user fetched in parallel from the same
  egress IP (429s + duplicate work). Do **not** scale `serve` past 1 replica without first moving
  discovery to a k8s CronJob or adding leader election. The process logs a `Warn` at startup when
  scheduled discovery is enabled, as a reminder.