Tapir — Architecture (C4 + sequences)

Diagrams describe current assumptions, not a frozen design. They are diffable Mermaid so they live in version control and render in Gitea. When a decision here changes, record it in DECISIONS.md and update the relevant diagram in the same commit.

Altitude: C4 Level 1 (Context) and Level 2 (Container), plus sequence diagrams for the two behaviours that carry the most design risk. Clean Architecture layering is described after the diagrams.

C4 L1 — System Context

Who and what Tapir talks to. (Brain and external AI are dashed — both optional.)

graph TB
    user["User<br/>(maintainer; later, trusted users)"]
    tapir["Tapir<br/>watches subscriptions,<br/>summarizes new videos"]
    yt["YouTube<br/>(Data API + captions)"]
    vimeo["Vimeo<br/>(API + text tracks)"]
    local["Local AI stack<br/>(LiteLLM / piguard alias)"]
    byo["User's BYO AI<br/>(Claude / OpenAI / Gemini)"]
    brain["brain<br/>(brain-mcp)"]

    user -->|connects accounts,<br/>reads summaries| tapir
    tapir -->|subscriptions,<br/>new-video events,<br/>captions| yt
    tapir -->|subscriptions,<br/>text tracks| vimeo
    tapir -->|summarize<br/>PRIMARY| local
    tapir -.->|summarize<br/>FALLBACK, opt-in| byo
    tapir -.->|optional sink:<br/>brain_ingest| brain

    classDef opt stroke-dasharray: 5 5;
    class byo,brain opt;

C4 L2 — Containers

Inside Tapir. The engine is provider- and sink-agnostic; everything external is an adapter behind an interface (Clean Architecture ports & adapters).

graph TB
    subgraph tapir["Tapir (Go)"]
        http["tapir serve<br/>(HTMX+Templ web surface:<br/>read summaries, connect,<br/>account, summarize)"]
        watcher["Watcher<br/>detects new videos<br/>(WebSub + poll)"]
        engine["Summarization engine<br/>(use-case core)"]
        resolver["Transcript resolver<br/>(captions-first)"]
        router["AI router<br/>(llm.Router:<br/>Primary -> Fallback)"]
        store[("User store<br/>(Postgres,<br/>per-user isolated)")]

        subgraph sinks["Sink adapters (Sink interface)"]
            sink_store["Store sink<br/>(primary)"]
            sink_brain["Brain sink<br/>(HTTP brain-mcp)"]
        end

        subgraph providers["Provider adapters (VideoSource interface)"]
            p_yt["YouTube adapter"]
            p_vimeo["Vimeo adapter"]
        end
    end

    http --> store
    watcher --> p_yt
    watcher --> p_vimeo
    watcher -->|NewVideo event| engine
    engine --> resolver
    resolver --> p_yt
    resolver --> p_vimeo
    engine --> router
    engine --> sink_store
    engine -.-> sink_brain
    sink_store --> store

    classDef opt stroke-dasharray: 5 5;
    class sink_brain opt;

Interfaces that keep the engine pure (the ports):

VideoSource — list subscriptions, detect new videos, fetch transcript. Implemented by YouTube and Vimeo adapters.
Summarizer — turn a transcript + context into highlights/takeaways. Backed by the llm.Router (Primary local, Fallback BYO).
Sink — deliver a summary. Implemented by the store sink (primary) and brain sink (optional). New sinks add an implementation, nothing else.
SecretStore — fetch/store per-user OAuth tokens and BYO keys. Backed by ESO/1Password.

The engine depends only on these interfaces, never on YouTube, brain, or a concrete model. This is what makes "standalone vs homelab" a configuration of which adapters are wired, not two codebases (ADR-003).

Web surface — `tapir serve` (Stage 1, ADR-011 → ADR-012)

A later transport added over the unchanged engine/ports/sinks core (ADR-003): tapir serve is an HTMX+Templ reader/writer (internal/web) over the existing store. It added no business logic to the engine — it reads the store and, for one action, kicks the existing engine. ADR-011 shipped it single-user; ADR-012 opened multi-user with DB-enforced (RLS) isolation.

graph TB
    browser["Browser<br/>(Dex-authenticated user)"]
    subgraph web["internal/web (tapir serve)"]
        oidc["oidc<br/>Dex OIDC session<br/>(authenticate-only)"]
        gate["registration gate<br/>new subject -> /register"]
        pages["summary list + detail<br/>(read) + actions"]
        connect["/oauth/youtube/callback<br/>per-user token connect"]
        account["account<br/>(disconnect, delete)"]
        summarize["Summarize button<br/>-> background goroutine"]
    end
    store[("store<br/>(Postgres, RLS per user)")]
    engine["Summarization engine<br/>(unchanged core)"]
    secrets["SecretStore<br/>(per-user token refs)"]

    browser --> oidc
    oidc --> gate
    gate --> pages
    pages --> store
    connect --> secrets
    connect --> store
    account --> store
    account --> secrets
    summarize -->|background| engine
    summarize -->|HTMX status poll| store
    engine --> store

Dex OIDC session layer (internal/web/oidc) — authenticate-only (ADR-012). It proves who; authorization/isolation is the DB's job (RLS), not the session's.
Registration gate — a Dex subject with no users row is routed to /register, which creates the users row + the user_identities mapping (migration 004). Returning subjects pass straight through.
Web-initiated YouTube connect — /oauth/youtube/connect → /oauth/youtube/callback persists a per-user refresh-token ref (youtube/<userID>/refresh_token) via SecretStore and a video_connections row (ADR-006, migration 005). Distinct from the CLI tapir auth.
Account management — /account offers disconnect and delete account. Delete removes only Tapir-side state (cascade across the user's tables + secret refs); the shared Dex identity is left intact (ADR-013).
Immediate summarization — the web "Summarize" button (POST /v/{id}/summarize) fires the engine in a background goroutine inside serve; the page HTMX-polls /v/{id}/status, showing a Charmbracelet spinner while in-flight (and an honest "queued/waiting" state under rate-limiting — ADR-014).
Summarization mode — users.auto_summarize (migration 006). Default is true for new users (migration 011, ADR-018); all existing rows were back-filled via migration 012. Auto: every new video is summarized. Manual: new videos appear unsummarized; the button sets videos.summarize_requested, which the next tapir run processes and clears. Both the click path and the batch tapir run drive the same unchanged engine.

The engine, ports, and sink adapters are untouched by all of the above — the web surface only reads the store and triggers the existing engine. Adding it changed wiring, not the core (ADR-003).

In-process scheduler (ADR-018)

cmdServe launches a background goroutine when TAPIR_DISCOVERY_INTERVAL > 0. On each tick it calls store.ListAllUsers (un-RLS'd admin query), builds a per-user runner.Runner, and calls RunOnce for each registered user in sequence.

sequenceDiagram
    participant S as Scheduler goroutine
    participant DB as Postgres (RLS)
    participant YT as YouTube timedtext
    participant LLM as LiteLLM gateway

    loop every TAPIR_DISCOVERY_INTERVAL
        S->>DB: ListAllUsers() [un-RLS'd]
        loop per user
            S->>DB: GetAutoSummarize(userID)
            S->>YT: ListSubscriptions + NewVideos
            Note over S,YT: WaitFetchGate(ctx) throttles<br/>all fetches to TAPIR_FETCH_RATE
            alt transcript available
                S->>LLM: Summarize
                S->>DB: Deliver(summary)
            else 429
                S->>DB: SetTranscriptStatus(rate_limited)
            end
        end
    end

Single-replica constraint (load-bearing). The scheduler lives in the web process; replicas: 1 in the k3s deployment manifest is not cosmetic — running tapir serve at

1 replica makes every replica run the full discovery loop, causing every registered user to be fetched in parallel from the same egress IP (429s + duplicate work). Do not scale serve past 1 replica without first moving discovery to a k8s CronJob or adding leader election. The process logs a Warn at startup when scheduled discovery is enabled as a reminder.

Process-wide timedtext rate gate

internal/adapters/youtube/gate.go (ADR-014 item 2): a single rate.Limiter (golang.org/x/time/rate) shared across all Adapter instances. Every httpDo call for a caption fetch passes through WaitFetchGate(ctx) before hitting YouTube. This serialises the scheduler loop AND the web click-path through the same per-egress-IP budget. Configured via TAPIR_FETCH_RATE (Go duration, default 2s). Setting it to 0 disables the gate (dev/tests only).

This is the precondition that makes scheduled auto-summarize safe: without the gate, a multi-user scheduler pass could fire many concurrent timedtext requests from the same IP within seconds, triggering 429s for all users.

Sequence — core use case: new video summarized

sequenceDiagram
    participant Src as VideoSource<br/>(YouTube/Vimeo)
    participant W as Watcher
    participant E as Engine
    participant R as Transcript resolver
    participant AI as AI router
    participant S as Sink(s)

    Src->>W: new upload (WebSub push / poll)
    W->>E: NewVideo{user, channel, videoID}
    E->>R: resolve transcript(videoID)
    R->>Src: fetch captions
    alt captions available
        Src-->>R: captions text
        R-->>E: Transcript{source: captions}
        E->>AI: summarize(transcript, userContext)
        AI-->>E: Summary{highlights, takeaways}
        E->>S: deliver(summary)
        S-->>E: ok
    else no transcript
        Src-->>R: none
        R-->>E: NoTranscript
        E->>S: deliver(skipped: no transcript)
    end

Sequence — AI routing (local-first, BYO fallback)

sequenceDiagram
    participant E as Engine
    participant R as llm.Router
    participant P as Primary<br/>(local: LiteLLM/piguard)
    participant F as Fallback<br/>(user BYO key)

    E->>R: summarize(transcript)
    R->>P: complete(prompt)
    alt local succeeds
        P-->>R: summary
        R-->>E: summary (fallback_used = false)
    else local fails (error/timeout/unavailable)
        P-->>R: error
        alt user has BYO configured
            R->>F: complete(prompt)
            F-->>R: summary
            R-->>E: summary (fallback_used = true)
        else no BYO
            R-->>E: error (queued for retry)
        end
    end

"Reliably" (VISION / the BYO trigger) is operationalized as: Primary returned without error within timeout. Richer quality scoring can layer on later without changing the interface. fallback_used is recorded per summary so "is the local stack good enough?" becomes a query, not a guess.

Clean Architecture layering

graph LR
    subgraph domain["Domain (entities)"]
        d["User, Subscription, Video,<br/>Transcript, Summary"]
    end
    subgraph usecase["Use cases (engine)"]
        u["SummarizeNewVideo,<br/>ConnectAccount,<br/>RouteAI"]
    end
    subgraph ports["Ports (interfaces)"]
        po["VideoSource, Summarizer,<br/>Sink, SecretStore"]
    end
    subgraph adapters["Adapters (infra)"]
        a["YouTube, Vimeo, llm.Router,<br/>Store sink, Brain sink,<br/>ESO secret store, HTTP, Postgres"]
    end

    a --> po
    po --> u
    u --> d

Dependencies point inward only. Domain knows nothing of YouTube, brain, Postgres, or any model. Adapters are swappable; tests target the use-case core through fake adapters (see the Gherkin features in docs/use-cases/).

Out of scope in these diagrams (deferred per ADRs)

Audio-download + speech-to-text resolver (ADR-007) — would be an additional VideoSource fallback path, drawn when built.
Per-user isolation is live, not deferred: Postgres RLS FORCEd on every user-owned table (ADR-012, migration 003), realising ADR-002's per-tenant intent at the DB layer. The coarser multi-tenant primitives (per-namespace NetworkPolicy, Kyverno, tenant label) remain a Stage-2 hardening item, not exercised yet.
Public SaaS surface (sign-up, billing) — Future C, not built (ADR-008).

12 KiB Raw Blame History