314 lines
12 KiB
Markdown
314 lines
12 KiB
Markdown
# Tapir — Architecture (C4 + sequences)
|
|
|
|
Diagrams describe **current assumptions**, not a frozen design. They are diffable Mermaid
|
|
so they live in version control and render in Gitea. When a decision here changes, record it
|
|
in `DECISIONS.md` and update the relevant diagram in the same commit.
|
|
|
|
Altitude: C4 Level 1 (Context) and Level 2 (Container), plus sequence diagrams for the two
|
|
behaviours that carry the most design risk. Clean Architecture layering is described after
|
|
the diagrams.
|
|
|
|
---
|
|
|
|
## C4 L1 — System Context
|
|
|
|
Who and what Tapir talks to. (Brain and external AI are dashed — both optional.)
|
|
|
|
```mermaid
|
|
graph TB
|
|
user["User<br/>(maintainer; later, trusted users)"]
|
|
tapir["Tapir<br/>watches subscriptions,<br/>summarizes new videos"]
|
|
yt["YouTube<br/>(Data API + captions)"]
|
|
vimeo["Vimeo<br/>(API + text tracks)"]
|
|
local["Local AI stack<br/>(LiteLLM / piguard alias)"]
|
|
byo["User's BYO AI<br/>(Claude / OpenAI / Gemini)"]
|
|
brain["brain<br/>(brain-mcp)"]
|
|
|
|
user -->|connects accounts,<br/>reads summaries| tapir
|
|
tapir -->|subscriptions,<br/>new-video events,<br/>captions| yt
|
|
tapir -->|subscriptions,<br/>text tracks| vimeo
|
|
tapir -->|summarize<br/>PRIMARY| local
|
|
tapir -.->|summarize<br/>FALLBACK, opt-in| byo
|
|
tapir -.->|optional sink:<br/>brain_ingest| brain
|
|
|
|
classDef opt stroke-dasharray: 5 5;
|
|
class byo,brain opt;
|
|
```
|
|
|
|
---
|
|
|
|
## C4 L2 — Containers
|
|
|
|
Inside Tapir. The **engine** is provider- and sink-agnostic; everything external is an
|
|
adapter behind an interface (Clean Architecture ports & adapters).
|
|
|
|
```mermaid
|
|
graph TB
|
|
subgraph tapir["Tapir (Go)"]
|
|
http["tapir serve<br/>(HTMX+Templ web surface:<br/>read summaries, connect,<br/>account, summarize)"]
|
|
watcher["Watcher<br/>detects new videos<br/>(WebSub + poll)"]
|
|
engine["Summarization engine<br/>(use-case core)"]
|
|
resolver["Transcript resolver<br/>(captions-first)"]
|
|
router["AI router<br/>(llm.Router:<br/>Primary -> Fallback)"]
|
|
store[("User store<br/>(Postgres,<br/>per-user isolated)")]
|
|
|
|
subgraph sinks["Sink adapters (Sink interface)"]
|
|
sink_store["Store sink<br/>(primary)"]
|
|
sink_brain["Brain sink<br/>(HTTP brain-mcp)"]
|
|
end
|
|
|
|
subgraph providers["Provider adapters (VideoSource interface)"]
|
|
p_yt["YouTube adapter"]
|
|
p_vimeo["Vimeo adapter"]
|
|
end
|
|
end
|
|
|
|
http --> store
|
|
watcher --> p_yt
|
|
watcher --> p_vimeo
|
|
watcher -->|NewVideo event| engine
|
|
engine --> resolver
|
|
resolver --> p_yt
|
|
resolver --> p_vimeo
|
|
engine --> router
|
|
engine --> sink_store
|
|
engine -.-> sink_brain
|
|
sink_store --> store
|
|
|
|
classDef opt stroke-dasharray: 5 5;
|
|
class sink_brain opt;
|
|
```
|
|
|
|
**Interfaces that keep the engine pure (the ports):**
|
|
|
|
- `VideoSource` — list subscriptions, detect new videos, fetch transcript. Implemented by
|
|
YouTube and Vimeo adapters.
|
|
- `Summarizer` — turn a transcript + context into highlights/takeaways. Backed by the
|
|
`llm.Router` (Primary local, Fallback BYO).
|
|
- `Sink` — deliver a summary. Implemented by the store sink (primary) and brain sink
|
|
(optional). New sinks add an implementation, nothing else.
|
|
- `SecretStore` — fetch/store per-user OAuth tokens and BYO keys. Backed by ESO/1Password.
|
|
|
|
The engine depends only on these interfaces, never on YouTube, brain, or a concrete model.
|
|
This is what makes "standalone vs homelab" a configuration of which adapters are wired, not
|
|
two codebases (ADR-003).
|
|
|
|
---
|
|
|
|
## Web surface — `tapir serve` (Stage 1, ADR-011 → ADR-012)
|
|
|
|
A later transport added over the **unchanged** engine/ports/sinks core (ADR-003): `tapir serve`
|
|
is an HTMX+Templ reader/writer (`internal/web`) over the existing `store`. It added no business
|
|
logic to the engine — it reads the store and, for one action, kicks the existing engine. ADR-011
|
|
shipped it single-user; ADR-012 opened multi-user with DB-enforced (RLS) isolation.
|
|
|
|
```mermaid
|
|
graph TB
|
|
browser["Browser<br/>(Dex-authenticated user)"]
|
|
subgraph web["internal/web (tapir serve)"]
|
|
oidc["oidc<br/>Dex OIDC session<br/>(authenticate-only)"]
|
|
gate["registration gate<br/>new subject -> /register"]
|
|
pages["summary list + detail<br/>(read) + actions"]
|
|
connect["/oauth/youtube/callback<br/>per-user token connect"]
|
|
account["account<br/>(disconnect, delete)"]
|
|
summarize["Summarize button<br/>-> background goroutine"]
|
|
end
|
|
store[("store<br/>(Postgres, RLS per user)")]
|
|
engine["Summarization engine<br/>(unchanged core)"]
|
|
secrets["SecretStore<br/>(per-user token refs)"]
|
|
|
|
browser --> oidc
|
|
oidc --> gate
|
|
gate --> pages
|
|
pages --> store
|
|
connect --> secrets
|
|
connect --> store
|
|
account --> store
|
|
account --> secrets
|
|
summarize -->|background| engine
|
|
summarize -->|HTMX status poll| store
|
|
engine --> store
|
|
```
|
|
|
|
- **Dex OIDC session layer** (`internal/web/oidc`) — **authenticate-only** (ADR-012). It proves
|
|
*who*; authorization/isolation is the DB's job (RLS), not the session's.
|
|
- **Registration gate** — a Dex subject with no `users` row is routed to `/register`, which
|
|
creates the `users` row + the `user_identities` mapping (migration 004). Returning subjects
|
|
pass straight through.
|
|
- **Web-initiated YouTube connect** — `/oauth/youtube/connect` → `/oauth/youtube/callback`
|
|
persists a **per-user** refresh-token ref (`youtube/<userID>/refresh_token`) via `SecretStore`
|
|
and a `video_connections` row (ADR-006, migration 005). Distinct from the CLI `tapir auth`.
|
|
- **Account management** — `/account` offers disconnect and **delete account**. Delete removes
|
|
only Tapir-side state (cascade across the user's tables + secret refs); the shared Dex identity
|
|
is left intact (ADR-013).
|
|
- **Immediate summarization** — the web "Summarize" button (`POST /v/{id}/summarize`) fires the
|
|
engine in a **background goroutine** inside `serve`; the page HTMX-polls `/v/{id}/status`,
|
|
showing a Charmbracelet spinner while in-flight (and an honest "queued/waiting" state under
|
|
rate-limiting — ADR-014).
|
|
- **Summarization mode** — `users.auto_summarize` (migration 006). Default is **true** for new
|
|
users (migration 011, ADR-018); all existing rows were back-filled via migration 012. Auto:
|
|
every new video is summarized. Manual: new videos appear unsummarized; the button sets
|
|
`videos.summarize_requested`, which the next `tapir run` processes and clears. Both the click
|
|
path and the batch `tapir run` drive the same unchanged engine.
|
|
|
|
The engine, ports, and sink adapters are **untouched** by all of the above — the web surface only
|
|
reads the store and triggers the existing engine. Adding it changed wiring, not the core (ADR-003).
|
|
|
|
---
|
|
|
|
## In-process scheduler (ADR-018)
|
|
|
|
`cmdServe` launches a background goroutine when `TAPIR_DISCOVERY_INTERVAL > 0`. On each tick
|
|
it calls `store.ListAllUsers` (un-RLS'd admin query), builds a per-user `runner.Runner`, and
|
|
calls `RunOnce` for each registered user in sequence.
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant S as Scheduler goroutine
|
|
participant DB as Postgres (RLS)
|
|
participant YT as YouTube timedtext
|
|
participant LLM as LiteLLM gateway
|
|
|
|
loop every TAPIR_DISCOVERY_INTERVAL
|
|
S->>DB: ListAllUsers() [un-RLS'd]
|
|
loop per user
|
|
S->>DB: GetAutoSummarize(userID)
|
|
S->>YT: ListSubscriptions + NewVideos
|
|
Note over S,YT: WaitFetchGate(ctx) throttles<br/>all fetches to TAPIR_FETCH_RATE
|
|
alt transcript available
|
|
S->>LLM: Summarize
|
|
S->>DB: Deliver(summary)
|
|
else 429
|
|
S->>DB: SetTranscriptStatus(rate_limited)
|
|
end
|
|
end
|
|
end
|
|
```
|
|
|
|
**Single-replica constraint (load-bearing).** The scheduler lives in the web process;
|
|
`replicas: 1` in the k3s deployment manifest is not cosmetic — running `tapir serve` at
|
|
>1 replica makes every replica run the full discovery loop, causing every registered user
|
|
to be fetched in parallel from the same egress IP (429s + duplicate work). Do not scale
|
|
`serve` past 1 replica without first moving discovery to a k8s CronJob or adding leader
|
|
election. The process logs a `Warn` at startup when scheduled discovery is enabled as a
|
|
reminder.
|
|
|
|
---
|
|
|
|
## Process-wide timedtext rate gate
|
|
|
|
**`internal/adapters/youtube/gate.go`** (ADR-014 item 2): a single `rate.Limiter`
|
|
(`golang.org/x/time/rate`) shared across **all** Adapter instances. Every `httpDo` call for
|
|
a caption fetch passes through `WaitFetchGate(ctx)` before hitting YouTube. This serialises
|
|
the scheduler loop AND the web click-path through the same per-egress-IP budget. Configured
|
|
via `TAPIR_FETCH_RATE` (Go duration, default `2s`). Setting it to `0` disables the gate
|
|
(dev/tests only).
|
|
|
|
This is the precondition that makes scheduled auto-summarize safe: without the gate, a
|
|
multi-user scheduler pass could fire many concurrent timedtext requests from the same IP
|
|
within seconds, triggering 429s for all users.
|
|
|
|
---
|
|
|
|
## Sequence — core use case: new video summarized
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Src as VideoSource<br/>(YouTube/Vimeo)
|
|
participant W as Watcher
|
|
participant E as Engine
|
|
participant R as Transcript resolver
|
|
participant AI as AI router
|
|
participant S as Sink(s)
|
|
|
|
Src->>W: new upload (WebSub push / poll)
|
|
W->>E: NewVideo{user, channel, videoID}
|
|
E->>R: resolve transcript(videoID)
|
|
R->>Src: fetch captions
|
|
alt captions available
|
|
Src-->>R: captions text
|
|
R-->>E: Transcript{source: captions}
|
|
E->>AI: summarize(transcript, userContext)
|
|
AI-->>E: Summary{highlights, takeaways}
|
|
E->>S: deliver(summary)
|
|
S-->>E: ok
|
|
else no transcript
|
|
Src-->>R: none
|
|
R-->>E: NoTranscript
|
|
E->>S: deliver(skipped: no transcript)
|
|
end
|
|
```
|
|
|
|
---
|
|
|
|
## Sequence — AI routing (local-first, BYO fallback)
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant E as Engine
|
|
participant R as llm.Router
|
|
participant P as Primary<br/>(local: LiteLLM/piguard)
|
|
participant F as Fallback<br/>(user BYO key)
|
|
|
|
E->>R: summarize(transcript)
|
|
R->>P: complete(prompt)
|
|
alt local succeeds
|
|
P-->>R: summary
|
|
R-->>E: summary (fallback_used = false)
|
|
else local fails (error/timeout/unavailable)
|
|
P-->>R: error
|
|
alt user has BYO configured
|
|
R->>F: complete(prompt)
|
|
F-->>R: summary
|
|
R-->>E: summary (fallback_used = true)
|
|
else no BYO
|
|
R-->>E: error (queued for retry)
|
|
end
|
|
end
|
|
```
|
|
|
|
> "Reliably" (VISION / the BYO trigger) is operationalized as: **Primary returned without
|
|
> error within timeout.** Richer quality scoring can layer on later without changing the
|
|
> interface. `fallback_used` is recorded per summary so "is the local stack good enough?"
|
|
> becomes a query, not a guess.
|
|
|
|
---
|
|
|
|
## Clean Architecture layering
|
|
|
|
```mermaid
|
|
graph LR
|
|
subgraph domain["Domain (entities)"]
|
|
d["User, Subscription, Video,<br/>Transcript, Summary"]
|
|
end
|
|
subgraph usecase["Use cases (engine)"]
|
|
u["SummarizeNewVideo,<br/>ConnectAccount,<br/>RouteAI"]
|
|
end
|
|
subgraph ports["Ports (interfaces)"]
|
|
po["VideoSource, Summarizer,<br/>Sink, SecretStore"]
|
|
end
|
|
subgraph adapters["Adapters (infra)"]
|
|
a["YouTube, Vimeo, llm.Router,<br/>Store sink, Brain sink,<br/>ESO secret store, HTTP, Postgres"]
|
|
end
|
|
|
|
a --> po
|
|
po --> u
|
|
u --> d
|
|
```
|
|
|
|
Dependencies point inward only. Domain knows nothing of YouTube, brain, Postgres, or any
|
|
model. Adapters are swappable; tests target the use-case core through fake adapters (see the
|
|
Gherkin features in `docs/use-cases/`).
|
|
|
|
---
|
|
|
|
## Out of scope in these diagrams (deferred per ADRs)
|
|
|
|
- Audio-download + speech-to-text resolver (ADR-007) — would be an additional `VideoSource`
|
|
fallback path, drawn when built.
|
|
- Per-user isolation is **live, not deferred**: Postgres RLS `FORCE`d on every user-owned table
|
|
(ADR-012, migration 003), realising ADR-002's per-tenant intent at the DB layer. The coarser
|
|
multi-tenant primitives (per-namespace NetworkPolicy, Kyverno, tenant label) remain a
|
|
Stage-2 hardening item, not exercised yet.
|
|
- Public SaaS surface (sign-up, billing) — Future C, not built (ADR-008).
|