docs: add data model (Stage 0 / Stage 1 scope)
Per-user-isolated entities (no global cross-tenant video table per the S5/grill correction), secrets stored by reference only (ESO/1Password, never the token), brain delivery modelled as a sink_delivery row rather than brain-specific tables. fallback_used recorded per summary as the Stage 0 quality signal. Future C dedup and sharding explicitly out of scope.
This commit is contained in:
mathias
@@ -0,0 +1,150 @@
|
||||
# Tapir — Data Model
|
||||
|
||||
Current assumptions for the persistence model. Scoped to **Stage 0 (single user)** and
|
||||
**Stage 1 (Future B, 1–5 trusted users)**. Future C concerns (sharding, cross-tenant
|
||||
dedup) are explicitly excluded and noted at the end.
|
||||
|
||||
Persistence is Postgres (ADR-002), reusing the homelab's instance with a per-tenant role
|
||||
when Stage 1 arrives. Secrets (OAuth tokens, BYO keys) are **not** stored in these tables —
|
||||
only opaque references to them; the secret material lives in ESO/1Password (ADR-002, ADR-006).
|
||||
|
||||
## Design decisions baked into this model
|
||||
|
||||
- **Per-user isolation, not a shared global video table.** The earlier draft proposed a
|
||||
global `videos`/`transcripts` table deduped across tenants. Rejected for Future B: it
|
||||
reintroduces exactly the cross-domain coupling the homelab architecture review is
|
||||
removing, and at 1–5 users the cost of occasionally re-summarizing the same video is
|
||||
trivial compared to the isolation it would cost. Each user's data is self-contained.
|
||||
(Revisit only if Future C makes GPU/transcription cost dominate — a new ADR, not a default.)
|
||||
- **Secrets by reference only.** Tables hold a `secret_ref` (opaque string/UUID resolved via
|
||||
the `SecretStore` port), never tokens or keys.
|
||||
- **The brain sink is just a delivery target.** No brain-specific tables. Whether a summary
|
||||
was also delivered to brain is recorded as sink status on the summary.
|
||||
|
||||
## Entities
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
USER ||--o{ VIDEO_CONNECTION : has
|
||||
USER ||--o{ AI_CREDENTIAL : has
|
||||
VIDEO_CONNECTION ||--o{ SUBSCRIPTION : exposes
|
||||
SUBSCRIPTION ||--o{ VIDEO : "produces (per user)"
|
||||
VIDEO ||--o| TRANSCRIPT : "has at most one"
|
||||
VIDEO ||--o| SUMMARY : "has at most one"
|
||||
SUMMARY ||--o{ SINK_DELIVERY : "delivered via"
|
||||
|
||||
USER {
|
||||
uuid id PK
|
||||
text display_name
|
||||
timestamptz created_at
|
||||
}
|
||||
VIDEO_CONNECTION {
|
||||
uuid id PK
|
||||
uuid user_id FK
|
||||
text provider "youtube | vimeo"
|
||||
text provider_account
|
||||
text token_secret_ref "-> SecretStore, never the token"
|
||||
text status "active | revoked | error"
|
||||
timestamptz connected_at
|
||||
}
|
||||
AI_CREDENTIAL {
|
||||
uuid id PK
|
||||
uuid user_id FK
|
||||
text provider "anthropic | openai | gemini"
|
||||
text key_secret_ref "-> SecretStore, never the key"
|
||||
text status
|
||||
timestamptz created_at
|
||||
}
|
||||
SUBSCRIPTION {
|
||||
uuid id PK
|
||||
uuid user_id FK
|
||||
uuid connection_id FK
|
||||
text channel_id
|
||||
text channel_title
|
||||
timestamptz websub_expires "nullable; youtube push lease"
|
||||
bool active
|
||||
}
|
||||
VIDEO {
|
||||
uuid id PK
|
||||
uuid user_id FK
|
||||
uuid subscription_id FK
|
||||
text provider
|
||||
text provider_video_id
|
||||
text title
|
||||
int duration_s
|
||||
timestamptz published_at
|
||||
text url
|
||||
timestamptz seen_at
|
||||
}
|
||||
TRANSCRIPT {
|
||||
uuid video_id PK_FK
|
||||
uuid user_id FK
|
||||
text source "captions | none"
|
||||
text language
|
||||
text content "null when source = none"
|
||||
timestamptz resolved_at
|
||||
}
|
||||
SUMMARY {
|
||||
uuid id PK
|
||||
uuid user_id FK
|
||||
uuid video_id FK
|
||||
text summary
|
||||
jsonb highlights
|
||||
jsonb takeaways
|
||||
text ai_provider "local | anthropic | openai | gemini"
|
||||
text ai_model
|
||||
bool fallback_used
|
||||
timestamptz created_at
|
||||
}
|
||||
SINK_DELIVERY {
|
||||
uuid id PK
|
||||
uuid summary_id FK
|
||||
text sink "store | brain"
|
||||
text status "pending | delivered | error"
|
||||
text detail "nullable; error message etc"
|
||||
timestamptz updated_at
|
||||
}
|
||||
```
|
||||
|
||||
## Notes per entity
|
||||
|
||||
- **USER** — at Stage 0 there is exactly one row. At Stage 1, identity comes via Dex; this
|
||||
table holds the Tapir-side profile keyed to the Dex subject.
|
||||
- **VIDEO_CONNECTION** — a connected YouTube/Vimeo account. `token_secret_ref` resolves to
|
||||
the OAuth refresh token via `SecretStore`. Revocation flips `status`, doesn't delete history.
|
||||
- **AI_CREDENTIAL** — optional, per provider, per user (ADR-004's Fallback). Absent for users
|
||||
who only use the local stack. One row per provider max.
|
||||
- **SUBSCRIPTION** — a watched channel. `websub_expires` tracks the YouTube push lease so the
|
||||
watcher knows when to re-subscribe; null for poll-based (Vimeo).
|
||||
- **VIDEO** — one row per (user, video) — note `user_id`, reflecting the per-user-isolation
|
||||
decision. The same video seen by two users is two rows. `seen_at` is when Tapir detected it.
|
||||
- **TRANSCRIPT** — at most one per video. `source = none` records "checked, no usable
|
||||
transcript" so the watcher doesn't reprocess (ADR-007). `content` null in that case.
|
||||
- **SUMMARY** — at most one per video. `fallback_used` + `ai_provider`/`ai_model` make the
|
||||
"is local good enough?" question queryable (the Stage 0 quality signal). `highlights`/
|
||||
`takeaways` as jsonb to stay schema-flexible while the output format settles.
|
||||
- **SINK_DELIVERY** — one row per (summary, sink) attempt. This is where "also sent to brain"
|
||||
lives — no brain tables, just a delivery row with `sink = brain`. Sinks fail independently;
|
||||
a failed brain delivery doesn't fail the store delivery.
|
||||
|
||||
## Isolation invariant (Stage 1+)
|
||||
|
||||
Every user-owned table carries `user_id`. At Stage 1, this is enforced at the DB layer via a
|
||||
per-tenant Postgres role + row grants (architecture review SC7), not only in application code.
|
||||
At Stage 0 (single user) the column exists but the enforcement is dormant. The isolation test
|
||||
in VISION Stage 2 asserts user A cannot read user B's rows.
|
||||
|
||||
## Job / processing state
|
||||
|
||||
Execution state (queued / running / retrying) for the watch→resolve→summarize→deliver
|
||||
pipeline is owned by the worker runtime, not modelled as first-class domain tables here.
|
||||
`SINK_DELIVERY.status` and `TRANSCRIPT.source` capture the durable, queryable outcomes; a
|
||||
thin `jobs` table may be added when a "what's processing" view is needed (mirrors the worker
|
||||
queue, doesn't replace it). Deferred until there's a reason.
|
||||
|
||||
## Explicitly out of scope (Future C)
|
||||
|
||||
- Global cross-tenant video/transcript dedup (rejected above).
|
||||
- Sharding / per-tenant physical databases.
|
||||
- Soft-delete + full audit trail on connections/credentials (a Stage 2 hardening item; add
|
||||
via ADR when Stage 2 work starts).
|
||||
Reference in New Issue
Block a user