docs: add data model (Stage 0 / Stage 1 scope)
Per-user-isolated entities (no global cross-tenant video table per the S5/grill correction), secrets stored by reference only (ESO/1Password, never the token), brain delivery modelled as a sink_delivery row rather than brain-specific tables. fallback_used recorded per summary as the Stage 0 quality signal. Future C dedup and sharding explicitly out of scope.
This commit is contained in:
mathias
@@ -0,0 +1,150 @@
|
|||||||
|
# Tapir — Data Model
|
||||||
|
|
||||||
|
Current assumptions for the persistence model. Scoped to **Stage 0 (single user)** and
|
||||||
|
**Stage 1 (Future B, 1–5 trusted users)**. Future C concerns (sharding, cross-tenant
|
||||||
|
dedup) are explicitly excluded and noted at the end.
|
||||||
|
|
||||||
|
Persistence is Postgres (ADR-002), reusing the homelab's instance with a per-tenant role
|
||||||
|
when Stage 1 arrives. Secrets (OAuth tokens, BYO keys) are **not** stored in these tables —
|
||||||
|
only opaque references to them; the secret material lives in ESO/1Password (ADR-002, ADR-006).
|
||||||
|
|
||||||
|
## Design decisions baked into this model
|
||||||
|
|
||||||
|
- **Per-user isolation, not a shared global video table.** The earlier draft proposed a
|
||||||
|
global `videos`/`transcripts` table deduped across tenants. Rejected for Future B: it
|
||||||
|
reintroduces exactly the cross-domain coupling the homelab architecture review is
|
||||||
|
removing, and at 1–5 users the cost of occasionally re-summarizing the same video is
|
||||||
|
trivial compared to the isolation it would cost. Each user's data is self-contained.
|
||||||
|
(Revisit only if Future C makes GPU/transcription cost dominate — a new ADR, not a default.)
|
||||||
|
- **Secrets by reference only.** Tables hold a `secret_ref` (opaque string/UUID resolved via
|
||||||
|
the `SecretStore` port), never tokens or keys.
|
||||||
|
- **The brain sink is just a delivery target.** No brain-specific tables. Whether a summary
|
||||||
|
was also delivered to brain is recorded as sink status on the summary.
|
||||||
|
|
||||||
|
## Entities
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
erDiagram
|
||||||
|
USER ||--o{ VIDEO_CONNECTION : has
|
||||||
|
USER ||--o{ AI_CREDENTIAL : has
|
||||||
|
VIDEO_CONNECTION ||--o{ SUBSCRIPTION : exposes
|
||||||
|
SUBSCRIPTION ||--o{ VIDEO : "produces (per user)"
|
||||||
|
VIDEO ||--o| TRANSCRIPT : "has at most one"
|
||||||
|
VIDEO ||--o| SUMMARY : "has at most one"
|
||||||
|
SUMMARY ||--o{ SINK_DELIVERY : "delivered via"
|
||||||
|
|
||||||
|
USER {
|
||||||
|
uuid id PK
|
||||||
|
text display_name
|
||||||
|
timestamptz created_at
|
||||||
|
}
|
||||||
|
VIDEO_CONNECTION {
|
||||||
|
uuid id PK
|
||||||
|
uuid user_id FK
|
||||||
|
text provider "youtube | vimeo"
|
||||||
|
text provider_account
|
||||||
|
text token_secret_ref "-> SecretStore, never the token"
|
||||||
|
text status "active | revoked | error"
|
||||||
|
timestamptz connected_at
|
||||||
|
}
|
||||||
|
AI_CREDENTIAL {
|
||||||
|
uuid id PK
|
||||||
|
uuid user_id FK
|
||||||
|
text provider "anthropic | openai | gemini"
|
||||||
|
text key_secret_ref "-> SecretStore, never the key"
|
||||||
|
text status
|
||||||
|
timestamptz created_at
|
||||||
|
}
|
||||||
|
SUBSCRIPTION {
|
||||||
|
uuid id PK
|
||||||
|
uuid user_id FK
|
||||||
|
uuid connection_id FK
|
||||||
|
text channel_id
|
||||||
|
text channel_title
|
||||||
|
timestamptz websub_expires "nullable; youtube push lease"
|
||||||
|
bool active
|
||||||
|
}
|
||||||
|
VIDEO {
|
||||||
|
uuid id PK
|
||||||
|
uuid user_id FK
|
||||||
|
uuid subscription_id FK
|
||||||
|
text provider
|
||||||
|
text provider_video_id
|
||||||
|
text title
|
||||||
|
int duration_s
|
||||||
|
timestamptz published_at
|
||||||
|
text url
|
||||||
|
timestamptz seen_at
|
||||||
|
}
|
||||||
|
TRANSCRIPT {
|
||||||
|
uuid video_id PK_FK
|
||||||
|
uuid user_id FK
|
||||||
|
text source "captions | none"
|
||||||
|
text language
|
||||||
|
text content "null when source = none"
|
||||||
|
timestamptz resolved_at
|
||||||
|
}
|
||||||
|
SUMMARY {
|
||||||
|
uuid id PK
|
||||||
|
uuid user_id FK
|
||||||
|
uuid video_id FK
|
||||||
|
text summary
|
||||||
|
jsonb highlights
|
||||||
|
jsonb takeaways
|
||||||
|
text ai_provider "local | anthropic | openai | gemini"
|
||||||
|
text ai_model
|
||||||
|
bool fallback_used
|
||||||
|
timestamptz created_at
|
||||||
|
}
|
||||||
|
SINK_DELIVERY {
|
||||||
|
uuid id PK
|
||||||
|
uuid summary_id FK
|
||||||
|
text sink "store | brain"
|
||||||
|
text status "pending | delivered | error"
|
||||||
|
text detail "nullable; error message etc"
|
||||||
|
timestamptz updated_at
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Notes per entity
|
||||||
|
|
||||||
|
- **USER** — at Stage 0 there is exactly one row. At Stage 1, identity comes via Dex; this
|
||||||
|
table holds the Tapir-side profile keyed to the Dex subject.
|
||||||
|
- **VIDEO_CONNECTION** — a connected YouTube/Vimeo account. `token_secret_ref` resolves to
|
||||||
|
the OAuth refresh token via `SecretStore`. Revocation flips `status`, doesn't delete history.
|
||||||
|
- **AI_CREDENTIAL** — optional, per provider, per user (ADR-004's Fallback). Absent for users
|
||||||
|
who only use the local stack. One row per provider max.
|
||||||
|
- **SUBSCRIPTION** — a watched channel. `websub_expires` tracks the YouTube push lease so the
|
||||||
|
watcher knows when to re-subscribe; null for poll-based (Vimeo).
|
||||||
|
- **VIDEO** — one row per (user, video) — note `user_id`, reflecting the per-user-isolation
|
||||||
|
decision. The same video seen by two users is two rows. `seen_at` is when Tapir detected it.
|
||||||
|
- **TRANSCRIPT** — at most one per video. `source = none` records "checked, no usable
|
||||||
|
transcript" so the watcher doesn't reprocess (ADR-007). `content` null in that case.
|
||||||
|
- **SUMMARY** — at most one per video. `fallback_used` + `ai_provider`/`ai_model` make the
|
||||||
|
"is local good enough?" question queryable (the Stage 0 quality signal). `highlights`/
|
||||||
|
`takeaways` as jsonb to stay schema-flexible while the output format settles.
|
||||||
|
- **SINK_DELIVERY** — one row per (summary, sink) attempt. This is where "also sent to brain"
|
||||||
|
lives — no brain tables, just a delivery row with `sink = brain`. Sinks fail independently;
|
||||||
|
a failed brain delivery doesn't fail the store delivery.
|
||||||
|
|
||||||
|
## Isolation invariant (Stage 1+)
|
||||||
|
|
||||||
|
Every user-owned table carries `user_id`. At Stage 1, this is enforced at the DB layer via a
|
||||||
|
per-tenant Postgres role + row grants (architecture review SC7), not only in application code.
|
||||||
|
At Stage 0 (single user) the column exists but the enforcement is dormant. The isolation test
|
||||||
|
in VISION Stage 2 asserts user A cannot read user B's rows.
|
||||||
|
|
||||||
|
## Job / processing state
|
||||||
|
|
||||||
|
Execution state (queued / running / retrying) for the watch→resolve→summarize→deliver
|
||||||
|
pipeline is owned by the worker runtime, not modelled as first-class domain tables here.
|
||||||
|
`SINK_DELIVERY.status` and `TRANSCRIPT.source` capture the durable, queryable outcomes; a
|
||||||
|
thin `jobs` table may be added when a "what's processing" view is needed (mirrors the worker
|
||||||
|
queue, doesn't replace it). Deferred until there's a reason.
|
||||||
|
|
||||||
|
## Explicitly out of scope (Future C)
|
||||||
|
|
||||||
|
- Global cross-tenant video/transcript dedup (rejected above).
|
||||||
|
- Sharding / per-tenant physical databases.
|
||||||
|
- Soft-delete + full audit trail on connections/credentials (a Stage 2 hardening item; add
|
||||||
|
via ADR when Stage 2 work starts).
|
||||||
Reference in New Issue
Block a user