diff --git a/docs/data-model.md b/docs/data-model.md new file mode 100644 index 0000000..954fd64 --- /dev/null +++ b/docs/data-model.md @@ -0,0 +1,150 @@ +# Tapir — Data Model + +Current assumptions for the persistence model. Scoped to **Stage 0 (single user)** and +**Stage 1 (Future B, 1–5 trusted users)**. Future C concerns (sharding, cross-tenant +dedup) are explicitly excluded and noted at the end. + +Persistence is Postgres (ADR-002), reusing the homelab's instance with a per-tenant role +when Stage 1 arrives. Secrets (OAuth tokens, BYO keys) are **not** stored in these tables — +only opaque references to them; the secret material lives in ESO/1Password (ADR-002, ADR-006). + +## Design decisions baked into this model + +- **Per-user isolation, not a shared global video table.** The earlier draft proposed a + global `videos`/`transcripts` table deduped across tenants. Rejected for Future B: it + reintroduces exactly the cross-domain coupling the homelab architecture review is + removing, and at 1–5 users the cost of occasionally re-summarizing the same video is + trivial compared to the isolation it would cost. Each user's data is self-contained. + (Revisit only if Future C makes GPU/transcription cost dominate — a new ADR, not a default.) +- **Secrets by reference only.** Tables hold a `secret_ref` (opaque string/UUID resolved via + the `SecretStore` port), never tokens or keys. +- **The brain sink is just a delivery target.** No brain-specific tables. Whether a summary + was also delivered to brain is recorded as sink status on the summary. + +## Entities + +```mermaid +erDiagram + USER ||--o{ VIDEO_CONNECTION : has + USER ||--o{ AI_CREDENTIAL : has + VIDEO_CONNECTION ||--o{ SUBSCRIPTION : exposes + SUBSCRIPTION ||--o{ VIDEO : "produces (per user)" + VIDEO ||--o| TRANSCRIPT : "has at most one" + VIDEO ||--o| SUMMARY : "has at most one" + SUMMARY ||--o{ SINK_DELIVERY : "delivered via" + + USER { + uuid id PK + text display_name + timestamptz created_at + } + VIDEO_CONNECTION { + uuid id PK + uuid user_id FK + text provider "youtube | vimeo" + text provider_account + text token_secret_ref "-> SecretStore, never the token" + text status "active | revoked | error" + timestamptz connected_at + } + AI_CREDENTIAL { + uuid id PK + uuid user_id FK + text provider "anthropic | openai | gemini" + text key_secret_ref "-> SecretStore, never the key" + text status + timestamptz created_at + } + SUBSCRIPTION { + uuid id PK + uuid user_id FK + uuid connection_id FK + text channel_id + text channel_title + timestamptz websub_expires "nullable; youtube push lease" + bool active + } + VIDEO { + uuid id PK + uuid user_id FK + uuid subscription_id FK + text provider + text provider_video_id + text title + int duration_s + timestamptz published_at + text url + timestamptz seen_at + } + TRANSCRIPT { + uuid video_id PK_FK + uuid user_id FK + text source "captions | none" + text language + text content "null when source = none" + timestamptz resolved_at + } + SUMMARY { + uuid id PK + uuid user_id FK + uuid video_id FK + text summary + jsonb highlights + jsonb takeaways + text ai_provider "local | anthropic | openai | gemini" + text ai_model + bool fallback_used + timestamptz created_at + } + SINK_DELIVERY { + uuid id PK + uuid summary_id FK + text sink "store | brain" + text status "pending | delivered | error" + text detail "nullable; error message etc" + timestamptz updated_at + } +``` + +## Notes per entity + +- **USER** — at Stage 0 there is exactly one row. At Stage 1, identity comes via Dex; this + table holds the Tapir-side profile keyed to the Dex subject. +- **VIDEO_CONNECTION** — a connected YouTube/Vimeo account. `token_secret_ref` resolves to + the OAuth refresh token via `SecretStore`. Revocation flips `status`, doesn't delete history. +- **AI_CREDENTIAL** — optional, per provider, per user (ADR-004's Fallback). Absent for users + who only use the local stack. One row per provider max. +- **SUBSCRIPTION** — a watched channel. `websub_expires` tracks the YouTube push lease so the + watcher knows when to re-subscribe; null for poll-based (Vimeo). +- **VIDEO** — one row per (user, video) — note `user_id`, reflecting the per-user-isolation + decision. The same video seen by two users is two rows. `seen_at` is when Tapir detected it. +- **TRANSCRIPT** — at most one per video. `source = none` records "checked, no usable + transcript" so the watcher doesn't reprocess (ADR-007). `content` null in that case. +- **SUMMARY** — at most one per video. `fallback_used` + `ai_provider`/`ai_model` make the + "is local good enough?" question queryable (the Stage 0 quality signal). `highlights`/ + `takeaways` as jsonb to stay schema-flexible while the output format settles. +- **SINK_DELIVERY** — one row per (summary, sink) attempt. This is where "also sent to brain" + lives — no brain tables, just a delivery row with `sink = brain`. Sinks fail independently; + a failed brain delivery doesn't fail the store delivery. + +## Isolation invariant (Stage 1+) + +Every user-owned table carries `user_id`. At Stage 1, this is enforced at the DB layer via a +per-tenant Postgres role + row grants (architecture review SC7), not only in application code. +At Stage 0 (single user) the column exists but the enforcement is dormant. The isolation test +in VISION Stage 2 asserts user A cannot read user B's rows. + +## Job / processing state + +Execution state (queued / running / retrying) for the watch→resolve→summarize→deliver +pipeline is owned by the worker runtime, not modelled as first-class domain tables here. +`SINK_DELIVERY.status` and `TRANSCRIPT.source` capture the durable, queryable outcomes; a +thin `jobs` table may be added when a "what's processing" view is needed (mirrors the worker +queue, doesn't replace it). Deferred until there's a reason. + +## Explicitly out of scope (Future C) + +- Global cross-tenant video/transcript dedup (rejected above). +- Sharding / per-tenant physical databases. +- Soft-delete + full audit trail on connections/credentials (a Stage 2 hardening item; add + via ADR when Stage 2 work starts).