Compare commits
7
Commits
v0.32.1
...
836a501576
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
836a501576 | ||
|
|
c29c0a7127 | ||
|
|
4b557209c8 | ||
|
|
e36c4c1e0c | ||
|
|
066ae6966a | ||
|
|
38b3f1cf33 | ||
|
|
feb108def3 |
+191
-1
@@ -123,7 +123,9 @@ secret surface (see VISION Stage 2); it gets the existing, vetted secret path.
|
|||||||
## ADR-007 — Captions-first; audio-download + speech-to-text deferred
|
## ADR-007 — Captions-first; audio-download + speech-to-text deferred
|
||||||
|
|
||||||
**Status:** Accepted (2026-06-02). Acquisition mechanism superseded by ADR-010 (Data API
|
**Status:** Accepted (2026-06-02). Acquisition mechanism superseded by ADR-010 (Data API
|
||||||
`captions.download` → player/timedtext baseUrl); captions-first stance and STT deferral stand.
|
`captions.download` → player/timedtext baseUrl); captions-first stance and STT deferral stand
|
||||||
|
**for YouTube**. STT deferral superseded **for user-supplied audio only** by ADR-033; the
|
||||||
|
YouTube audio-download rejection is unchanged.
|
||||||
|
|
||||||
**Context.** YouTube's Data API does not expose transcripts. Options: official captions
|
**Context.** YouTube's Data API does not expose transcripts. Options: official captions
|
||||||
(clean, limited coverage) vs. audio download + local Whisper (broad coverage, ToS-grey,
|
(clean, limited coverage) vs. audio download + local Whisper (broad coverage, ToS-grey,
|
||||||
@@ -1314,6 +1316,191 @@ roll back. No schema, no structural change.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## ADR-033 — Upload audio, get a private Swedish transcript, discuss it (user-supplied STT)
|
||||||
|
|
||||||
|
**Status:** Proposed (2026-10-01). **Decider: Mathias — pending.** Issue #33 (audio-only subset of
|
||||||
|
#28). **Supersedes ADR-007's speech-to-text deferral for user-supplied audio only.** The
|
||||||
|
YouTube audio-download rejection in ADR-007 **stands untouched**. Builds on ADR-012 (isolation),
|
||||||
|
ADR-018 (single replica), ADR-021 (**the constraint**), ADR-022 (endpoint chain), ADR-025
|
||||||
|
(honest status), ADR-027 (chat). ADR-014 item 4 ("Whisper reconsideration is a future ADR,
|
||||||
|
gated on caption data") is a different question — whether to transcribe YouTube videos with no
|
||||||
|
captions — and this ADR does not answer it. Measurements: `docs/research/audio-length-ceiling-2026-10-01.md`.
|
||||||
|
|
||||||
|
**Context.** The maintainer has a concrete need: a Swedish Teams voice-only export, 30–90
|
||||||
|
minutes, to upload and then *discuss* (what was decided, what is open, who owes what). Every
|
||||||
|
piece exists — a Whisper endpoint on the gateway (infra#274), chat-with-transcript (ADR-027) —
|
||||||
|
and nothing connects them.
|
||||||
|
|
||||||
|
ADR-007 deferred STT for three reasons: yt-dlp fragility, ToS-grey scraping, and GPU contention
|
||||||
|
with the JEPA PoC. Upload-STT shares **none** of them: the user owns the file, nothing is
|
||||||
|
scraped, there is no caption rate gate, and the contention moved from koala to iguana. What it
|
||||||
|
does share is landing Whisper in the codebase, which ADR-007 deferred — so a new ADR is required,
|
||||||
|
and ADR-017's process note records the one time a guardrail-reversing change shipped without one.
|
||||||
|
|
||||||
|
**The constraint that shapes everything.** ADR-021 made `transcripts` a shared, **non-RLS**
|
||||||
|
table on the explicit grounds that it holds only public caption content and nothing
|
||||||
|
user-identifying. A recorded internal meeting is the opposite: private third-party content owned
|
||||||
|
by one user. The existing isolation test asserts `transcripts` *is* non-RLS, so a naive reuse
|
||||||
|
would leak across tenants **and pass CI green**.
|
||||||
|
|
||||||
|
**What was measured, not argued** (research doc above):
|
||||||
|
- **No transcription ceiling up to 90 minutes.** One 174 MB request through the LiteLLM NodePort
|
||||||
|
took 440 s (~12× realtime). The first failure is the public ingress `llm-api.d-ma.be`
|
||||||
|
returning `413` above roughly 8–10 MB (~5 min of WAV) — not whisper, and not a Metal OOM.
|
||||||
|
The 8:20 failure recorded in #33 is consistent with that cap.
|
||||||
|
- **`response_format=srt` survives LiteLLM, wrapped** in a JSON envelope (`{"text": "<srt>"}`).
|
||||||
|
- **The two backends differ.** `iguana/kb-whisper` returns whole-second segments;
|
||||||
|
`berget/kb-whisper` returns sub-second times and a different `language` token.
|
||||||
|
- **The existing chat path truncates at 18,000 chars (~22 min of speech), keeping the head.**
|
||||||
|
#33 assumed chat "retrieves rather than generates" and so has no length ceiling. It does
|
||||||
|
not retrieve; it stuffs the transcript into the prompt and cuts the tail.
|
||||||
|
|
||||||
|
**Unknowns, stated.** Real-speech accuracy; compressed-input behaviour; concurrent long jobs on
|
||||||
|
iguana; whether the models chat offers can hold an hour of Swedish (~50 k chars); and the Metal
|
||||||
|
OOM, deliberately not provoked.
|
||||||
|
|
||||||
|
**Decision.**
|
||||||
|
|
||||||
|
1. **Audio only; transcript is the only thing kept.** An authenticated user uploads an audio
|
||||||
|
file; Tapir transcribes it and stores the transcript. The audio itself is deleted after
|
||||||
|
transcription (temp file, removed in a `defer`, swept at start-up) — no custody problem to
|
||||||
|
solve, and nothing to hand to #31. Video is rejected with a message naming #28. YouTube
|
||||||
|
audio-download stays rejected. No public/anonymous upload (ADR-008).
|
||||||
|
|
||||||
|
2. **Private store: a separate RLS-scoped table, `uploaded_transcripts`** (resolves #28's
|
||||||
|
decision 1). `user_id NOT NULL REFERENCES users ON DELETE CASCADE`, `ENABLE` + `FORCE ROW LEVEL
|
||||||
|
SECURITY`, the same `tapir.current_user_id` policy as every other user-owned table. It carries
|
||||||
|
the transcript text and its timestamped segments (re-transcribing 90 minutes costs ~7
|
||||||
|
minutes; discarding the timestamps is irreversible), `status`, the `endpoint` that served
|
||||||
|
it, `audio_seconds`, and a typed `error_kind`. **Not** a `user_id` column on `transcripts`
|
||||||
|
with a partial policy, even though that would reuse more code: ADR-021's exception is
|
||||||
|
precisely that *that table holds no private row*, the isolation test asserts it has no
|
||||||
|
`user_id`, and a partial policy would turn "single exception" into "single exception, with a
|
||||||
|
private lane inside it" — the shape under which a future query drops the policy clause and
|
||||||
|
leaks. Two tables keep both boundaries independently assertable.
|
||||||
|
|
||||||
|
3. **The isolation test asserts both boundaries and the set.** (a) User A cannot read, update or
|
||||||
|
delete user B's uploaded transcript; no GUC set → deny-all. (b) `transcripts` is still
|
||||||
|
non-RLS with no `user_id`. (c) **The set of public-schema tables with row security off is
|
||||||
|
asserted exactly** — so a new table added without RLS fails CI instead of passing it. (c) is
|
||||||
|
new; the existing test only checks the named tables.
|
||||||
|
|
||||||
|
4. **Validate by probing, reject clearly.** Format and duration cannot be read from a filename
|
||||||
|
or MIME type — a Teams "voice-only" recording is typically an `.mp4` container with no video
|
||||||
|
stream, which is indistinguishable from video by extension. The upload is probed
|
||||||
|
(`ffprobe`): accept any container with an audio stream and **no** video stream; reject video
|
||||||
|
with "video is #28"; reject unreadable files. Limits, all configurable, all rejected
|
||||||
|
synchronously with a specific message rather than by timeout: **200 MB** upload and
|
||||||
|
**90 minutes** duration. 90 minutes is *the longest length measured*, not a ceiling that was
|
||||||
|
found; raising it is a measurement, not an edit.
|
||||||
|
|
||||||
|
5. **Transcode server-side, always, to 16 kHz mono PCM WAV** — even when the input already is,
|
||||||
|
rather than assuming. whisper-server's `--convert` is rejected (its `--tmp-dir` is created at
|
||||||
|
process start and rots under macOS periodic cleanup). The transcoder sits behind a port; the
|
||||||
|
tests use a fake. **This puts `ffmpeg` and `ffprobe` in the runtime image**, which is
|
||||||
|
distroless-static today. **Provisional: copy static ffmpeg/ffprobe binaries into the existing
|
||||||
|
distroless image** (no shell, no package manager retained). This is a heavy new dependency
|
||||||
|
and is **the maintainer's call** — see Open items.
|
||||||
|
|
||||||
|
6. **Transcription adapter via the gateway, never the public hostname.** A `Transcriber` port;
|
||||||
|
the implementation calls `POST {TAPIR_GATEWAY_URL}/audio/transcriptions` where the gateway
|
||||||
|
URL is the in-cluster NodePort (`koala:30401/v1`). Primary `iguana/kb-whisper`, fallback
|
||||||
|
`berget/kb-whisper` (immune to iguana contention, which can wedge permanently — infra#273).
|
||||||
|
- **Fallback triggers on any primary failure** (transport error, timeout, non-2xx). It never
|
||||||
|
branches on the body: `{"error":"failed to process audio"}` has had at least five distinct
|
||||||
|
causes.
|
||||||
|
- **`response_format=verbose_json`**, normalised to one segment shape: times clamped to the
|
||||||
|
audio duration (the final segment can end past it), one language token. Not `srt`: the
|
||||||
|
envelope adds a parse step and iguana's segment timing is the same either way.
|
||||||
|
- **No chunking in v1.** Nothing failed to 90 minutes (measured). Chunk-and-stitch is the
|
||||||
|
answer *if* the cap is raised or smaller requests become necessary — see triggers.
|
||||||
|
- **The serving endpoint is recorded per row**, and surfaced, because the two backends
|
||||||
|
differ in timestamp granularity: which one served a transcript is a quality fact.
|
||||||
|
- **Berget is a cloud endpoint.** On fallback the audio of a private meeting leaves the
|
||||||
|
homelab. #33 asks for this fallback; it is configurable (`TAPIR_TRANSCRIBE_FALLBACK_MODEL`,
|
||||||
|
empty disables), recorded per row, and called out in Open items rather than defaulted
|
||||||
|
silently.
|
||||||
|
- Swedish only (`language=sv`). Language detection and routing (#28 decision 2) are not
|
||||||
|
touched.
|
||||||
|
- `-H 'Expect:'`, multipart field `file` — encoded in the adapter and its tests, not rediscovered.
|
||||||
|
|
||||||
|
7. **Jobs run in-process, one at a time (ADR-018 stands).** Transcription is minutes (~7.5 min
|
||||||
|
for 90) but a single goroutine behind a one-slot semaphore: two simultaneous 90-minute jobs on
|
||||||
|
iguana were not measured, so concurrency is 1 until they are. No worker, no new deployable,
|
||||||
|
no second replica. Honest status per ADR-025: `queued → transcribing → done | failed`, polled
|
||||||
|
by the same `/status` pattern, every terminal state a rendered card, never a stuck spinner.
|
||||||
|
Audio is not retained, so a restart mid-job **cannot resume**: at start-up any
|
||||||
|
`queued`/`transcribing` row is marked `failed` with `error_kind = interrupted`, and the user
|
||||||
|
re-uploads. The single-replica assumption gains a second dependent (the semaphore is
|
||||||
|
process-local) and is recorded here.
|
||||||
|
|
||||||
|
8. **Chat reuses `chat.Service`; it is not forked.** The uploaded-transcript chat handlers
|
||||||
|
look the row up through the RLS-scoped store, so a non-owner gets 404 by the same mechanism
|
||||||
|
as ADR-027's summary gate. ADR-027's decisions (ephemeral, switchable models, trust-the-model)
|
||||||
|
carry over unchanged. **One thing must differ:** the 18,000-char default cuts a one-hour
|
||||||
|
meeting to its first third, so uploaded transcripts take their own cap,
|
||||||
|
`TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS`. **Provisional default 120,000 chars (~2.4 h at
|
||||||
|
measured density)**, with the existing `Truncated` flag surfaced honestly. The default is a
|
||||||
|
guess until the chat slice measures what the offered models actually accept; it is not
|
||||||
|
evidence that they can.
|
||||||
|
|
||||||
|
9. **Not decided here.** Rich `AVSNITT`/`ALTERNATIV` analysis (gated on #30 — `analyze_srt.py`
|
||||||
|
hardcodes `max_tokens: 6000`); video, keyframes, synced playback (#28); language detection;
|
||||||
|
per-transcript deletion UX (account deletion cascades; see Open items); retention of audio.
|
||||||
|
|
||||||
|
**On the sequencing question #28 asks of itself** — *is this the thing that best serves the
|
||||||
|
Stage-0 gate, or the most interesting thing to build?* Recorded answer: it proceeds **ahead of
|
||||||
|
#32** (the demand test) as the maintainer's explicit, knowing call (#28, comment of 2026-10-01),
|
||||||
|
on a concrete need rather than speculation. It is not a return-nudge (ADR-020/027): it adds
|
||||||
|
nothing that prompts a user to come back, so it does not contaminate the unprompted-return signal
|
||||||
|
ADR-016 measures. The honest risk is the one ADR-016 names: it is larger than anything ADR-020
|
||||||
|
deferred. The scope cut — chat first, no rich analysis, no video — is what keeps it bounded.
|
||||||
|
|
||||||
|
**Consequences.**
|
||||||
|
- **Positive.** The user's request is met end to end with existing parts. ADR-021's exception stays
|
||||||
|
singular and now provably so. The measured ceiling removes chunking from v1. No audio is stored.
|
||||||
|
- **Negative / watch.**
|
||||||
|
- Tapir's runtime image gains `ffmpeg`: a larger image and a new CVE surface to patch.
|
||||||
|
- iguana becomes load-bearing for a user-facing feature (#28 decision 3; infra#273 is still the
|
||||||
|
dependency that matters) — a Metal OOM now means *every upload fails over to berget*, i.e.
|
||||||
|
private audio leaving the homelab, silently unless the per-row endpoint is surfaced.
|
||||||
|
- A 90-minute job holds the single slot for ~7.5 minutes; a second user's upload queues behind it.
|
||||||
|
- Chat over a long transcript is bounded by model context, not by anything in Tapir.
|
||||||
|
- Tapir's own ingress body limit for a 200 MB upload is **unverified** (it lives in `infra`).
|
||||||
|
|
||||||
|
**Validation / resolution triggers** (each observable by someone other than the author):
|
||||||
|
- *Chunking*: build it when the duration cap is raised above 90 min, or when a measured
|
||||||
|
request at the current cap fails with whisper-side (not ingress) errors.
|
||||||
|
- *Berget-first*: if more than a quarter of uploads in any week are served by the fallback,
|
||||||
|
swap the primary/fallback order and revisit infra#273.
|
||||||
|
- *Retrieval chat*: if the chat slice finds an offered model cannot answer a question whose
|
||||||
|
answer is in the last third of a 60-minute transcript, chunked retrieval replaces the larger cap.
|
||||||
|
- *Worker*: if a transcription job ever degrades the reading UI, or Tapir runs more than one
|
||||||
|
replica, ADR-018 is revisited.
|
||||||
|
- *The decision itself*: if the maintainer has not used the feature unprompted within the
|
||||||
|
ADR-016 window, it was the interesting thing, not the useful one — remove it.
|
||||||
|
|
||||||
|
**Open items this ADR creates.**
|
||||||
|
- **DECISION NEEDED — ffmpeg in the runtime image.** *Options:* (A) static ffmpeg/ffprobe copied
|
||||||
|
into distroless **(recommended — the only option that satisfies "accept m4a/mp3/opus/mp4,
|
||||||
|
validate duration, reject video"; pure-Go decoders cover wav/mp3 only)**; (B) accept WAV
|
||||||
|
only, convert client-side (fails "I just want to upload it"); (C) a transcode sidecar
|
||||||
|
(second deployable, ADR-018's reasoning against). *Blocks:* the Dockerfile slice, not the
|
||||||
|
tests, which use the port.
|
||||||
|
- **DECISION NEEDED — berget fallback for private audio.** *Options:* (A) enabled by default as
|
||||||
|
#33 specifies, with the serving endpoint shown **(recommended, matches the issue)**; (B)
|
||||||
|
disabled by default, a failed iguana job fails honestly and the user retries. *Blocks:*
|
||||||
|
the production default only; the adapter ships either way.
|
||||||
|
- Verify Tapir's ingress `proxy-body-size` against the 200 MB cap (`infra`).
|
||||||
|
- Per-transcript deletion is not built; account deletion cascades. A private-content store
|
||||||
|
without a delete button is a gap worth a ticket, not a silent omission.
|
||||||
|
|
||||||
|
**Reversibility.** Additive: one table (down-migration drops it), one adapter, one upload
|
||||||
|
surface, one image change. Removing the upload route removes the feature; `transcripts`, `videos`
|
||||||
|
and ADR-021 are untouched.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Rejected alternatives
|
## Rejected alternatives
|
||||||
|
|
||||||
Approaches considered during the 2026-06-02 planning + grill session and **deliberately not
|
Approaches considered during the 2026-06-02 planning + grill session and **deliberately not
|
||||||
@@ -1338,6 +1525,9 @@ maps to the ADR that settles it.
|
|||||||
| Reverse the Dex-write invite flow (Google OIDC only) | Some intended Future-B users won't use Google; OIDC-only leaves them with no onboarding path — invite flow is load-bearing | ADR-017 |
|
| Reverse the Dex-write invite flow (Google OIDC only) | Some intended Future-B users won't use Google; OIDC-only leaves them with no onboarding path — invite flow is load-bearing | ADR-017 |
|
||||||
| k8s CronJob for scheduled discovery (vs in-process) | At Future-B scale the in-process scheduler is simpler to deploy; CronJob's failure-isolation benefit was weighed and traded away knowingly (revisit if >1 replica or load grows) | ADR-018 |
|
| k8s CronJob for scheduled discovery (vs in-process) | At Future-B scale the in-process scheduler is simpler to deploy; CronJob's failure-isolation benefit was weighed and traded away knowingly (revisit if >1 replica or load grows) | ADR-018 |
|
||||||
| Cached-transcript-first onboarding burst (instant, zero-fetch picks) | Live pilot DB: ~3% cross-user video overlap, 0 cached-and-unsummarized, a new user's newest-20 are 20/20 uncached — newest-first and cached-first are structurally incompatible. Empty lever at pilot scale | ADR-028 |
|
| Cached-transcript-first onboarding burst (instant, zero-fetch picks) | Live pilot DB: ~3% cross-user video overlap, 0 cached-and-unsummarized, a new user's newest-20 are 20/20 uncached — newest-first and cached-first are structurally incompatible. Empty lever at pilot scale | ADR-028 |
|
||||||
|
| Uploaded transcripts as a `user_id` column + partial RLS policy on `transcripts` | Turns ADR-021's single exception into an exception with a private lane inside it; the isolation test asserts `transcripts` has no `user_id`. A separate RLS table keeps both boundaries independently assertable | ADR-033 |
|
||||||
|
| whisper-server `--convert` for server-side transcode | Its `--tmp-dir` is created at process start and rots under macOS periodic cleanup | ADR-033 |
|
||||||
|
| Calling the public `llm-api.d-ma.be` from in-cluster Tapir | Its ingress returns 413 above ~8–10 MB (~5 min of WAV); the NodePort has no such cap | ADR-033 |
|
||||||
|
|
||||||
If a future case genuinely reopens one of these, that's a new ADR superseding the relevant one —
|
If a future case genuinely reopens one of these, that's a new ADR superseding the relevant one —
|
||||||
not a silent reversal.
|
not a silent reversal.
|
||||||
|
|||||||
@@ -0,0 +1,110 @@
|
|||||||
|
# Audio length ceiling — measured before designing chunking (tapir#33, 2026-10-01)
|
||||||
|
|
||||||
|
Question (#33, "the main technical risk"): at what recording length does Swedish
|
||||||
|
transcription through the LiteLLM gateway first fail, and with what failure mode?
|
||||||
|
Measured **before** any chunking design, with iguana not holding a large ollama model,
|
||||||
|
so the result is the ceiling and not the Metal OOM of infra#273.
|
||||||
|
|
||||||
|
## Answer
|
||||||
|
|
||||||
|
**No transcription ceiling was found at or below 90 minutes.** A single request carrying
|
||||||
|
90 minutes of 16 kHz mono PCM (174 MB) transcribed in one pass in 440 s. The first thing
|
||||||
|
that actually fails is **not whisper**: it is the public gateway ingress, which rejects the
|
||||||
|
upload with `413` at roughly 5.5 minutes of WAV.
|
||||||
|
|
||||||
|
| Route | First failure | Failure mode |
|
||||||
|
|---|---|---|
|
||||||
|
| `https://llm-api.d-ma.be` (openresty ingress) | between 8.0 MB and 10.5 MB body (~4–5.5 min of WAV) | `413 Request Entity Too Large`, HTML body from `openresty`, returned instantly |
|
||||||
|
| `http://koala:30401` (LiteLLM NodePort) → `iguana/kb-whisper` | **none up to 90 min / 174 MB** | — |
|
||||||
|
| `http://koala:30401` → `berget/kb-whisper` | **none up to 22 min / 42 MB** (largest tried) | — |
|
||||||
|
|
||||||
|
The ingress limit is the reason the 8:20 / 15 MB recording in #33 "failed repeatedly" and a
|
||||||
|
30-second slice succeeded. That failure was attributed to a Metal OOM; at least in the
|
||||||
|
size-dependent form described, a body-size cap explains it without one. (The OOM
|
||||||
|
mechanism in infra#273 is separately real; this run did not provoke it.)
|
||||||
|
|
||||||
|
The exact ingress cap was bracketed (8.0 MB passed, 10.5 MB failed), not read from the
|
||||||
|
ingress configuration, which lives in `infra`. "About 10 MB" is an inference.
|
||||||
|
|
||||||
|
## Method
|
||||||
|
|
||||||
|
- **Audio.** No known-good Swedish recording is available that is safe to use: the only real
|
||||||
|
one (`IMG_1233.wav`) is the private `/bygge` recording (#31 custody still open) and was
|
||||||
|
deliberately not touched. Instead: macOS `say -v Alva` (Swedish) on iguana, a 1,171-char
|
||||||
|
five-paragraph meeting script → 82.5 s of 16 kHz mono PCM WAV (`ffmpeg -ar 16000 -ac 1`).
|
||||||
|
Concatenated ×4, ×8, ×16, ×32, ×44, ×66 with Python `wave` (the issue's own recipe).
|
||||||
|
Nothing here is a real transcript; none of it is committed.
|
||||||
|
- **Request.** `POST /v1/audio/transcriptions`, multipart `file=`, `model=iguana/kb-whisper`,
|
||||||
|
`response_format=verbose_json`, `-H 'Expect:'` (per #33's traps), master key supplied only
|
||||||
|
through `op run` and never on argv.
|
||||||
|
- **Confound control.** `ollama /api/ps` on iguana was read before every step. Resident:
|
||||||
|
nothing, or `nomic-embed-text` (0.37 GB). Never a large model.
|
||||||
|
|
||||||
|
## Results (`iguana/kb-whisper`, NodePort)
|
||||||
|
|
||||||
|
| Audio | WAV size | HTTP | Wall | Realtime factor | Segments | Chars |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| 0:30 | 0.96 MB | 200 | 2.8 s | 11× | — | — |
|
||||||
|
| 2:30 / 3:20 / 4:10 | 4.8 / 6.4 / 8.0 MB | 200 (ingress) | 12.9 / 16.0 / 19.9 s | 12× | — | — |
|
||||||
|
| 5:30 (×4) | 10.5 MB | **413 ingress**, 200 NodePort in 27 s | 27 s | 12× | 121 | 4,557 |
|
||||||
|
| 11:00 (×8) | 21.1 MB | 200 | 53 s | 12× | 241 | 9,094 |
|
||||||
|
| 22:00 (×16) | 42.3 MB | 200 | 105 s | 13× | 482 | 18,201 |
|
||||||
|
| 44:01 (×32) | 84.5 MB | 200 | 209 s | 13× | 966 | 36,438 |
|
||||||
|
| 60:32 (×44) | 116.2 MB | 200 | 291 s | 12× | 1,332 | 50,145 |
|
||||||
|
| 90:48 (×66) | 174.3 MB | 200 | 440 s | 12× | 2,005 | 75,165 |
|
||||||
|
|
||||||
|
- Throughput is flat at ~12× realtime. 90 minutes costs ~7.5 minutes of wall time. This is
|
||||||
|
a minutes-long job, not a request/response one (see ADR-033 on ADR-018).
|
||||||
|
- Segments per source repetition stay at 30.1–30.4 from ×4 to ×66, and segment times are
|
||||||
|
monotonic with `last_end` within 0.2 s of `duration`. No drift and no repetition-loop
|
||||||
|
hallucination appeared across 90 minutes. This is a weak check — the input is a repeated
|
||||||
|
loop and synthetic voice — not evidence of accuracy.
|
||||||
|
- **Only `iguana/kb-whisper`'s `x4` run reported `last_end` 360.0 s against `duration`
|
||||||
|
330.2 s.** Whisper's final segment can end past the audio; clamp to `duration`.
|
||||||
|
|
||||||
|
## Findings that change the design
|
||||||
|
|
||||||
|
1. **Use the NodePort route, not the public hostname.** Tapir runs in-cluster beside the
|
||||||
|
gateway; `llm-api.d-ma.be` is the external door and has a body cap. #33 said "use the
|
||||||
|
gateway, never the host" and named the public URL; the gateway is right, that URL is not.
|
||||||
|
`TAPIR_GATEWAY_URL` already points at `koala:30401/v1`.
|
||||||
|
2. **`response_format=srt` survives the LiteLLM hop, but wrapped.** The response is JSON
|
||||||
|
`{"text": "<the SRT>", "usage": …}`, not a bare SRT body — a client that parses the body
|
||||||
|
as SRT gets a parse error on `{`. #33 question 2 is answered: it passes through, inside
|
||||||
|
an envelope.
|
||||||
|
3. **The two backends return different timestamp granularity.** `iguana/kb-whisper`
|
||||||
|
segments are **whole seconds** (`0→1`, `1→4`, `4→7`); `berget/kb-whisper` returns real
|
||||||
|
sub-second times (`0.051→1.192`) and a different `language` token (`swedish` vs `sv`).
|
||||||
|
`verbose_json` and the default JSON were byte-identical on iguana. An adapter that
|
||||||
|
falls back between the two must normalise both, and "which endpoint served this" is
|
||||||
|
a quality fact, not just a log line.
|
||||||
|
4. **The existing chat path cannot answer about most of a long recording.** At
|
||||||
|
828 chars/min (this fixture; real speech is plausibly a little denser),
|
||||||
|
`TAPIR_MAX_TRANSCRIPT_CHARS`'s default of **18,000** is ~21.7 minutes. A 60-minute
|
||||||
|
transcript is 50 k chars, a 90-minute one 75 k. `chat.Service` truncates by keeping
|
||||||
|
the head (`internal/adapters/chat/chat.go`), so questions about the last two-thirds of
|
||||||
|
a one-hour meeting get an answer from a transcript that does not contain it. #33 says
|
||||||
|
chat "retrieves rather than generates" and therefore has no ceiling; it **stuffs and
|
||||||
|
truncates**, and its ceiling is the model context. See ADR-033.
|
||||||
|
5. **Chunking is not needed for v1 on length.** Nothing failed up to 90 minutes. The
|
||||||
|
chunking-and-stitching recipe in #33 (offset timestamps, renumber cues, strip `\r`) is
|
||||||
|
not built. It stays the answer *if* a longer cap is wanted or the OOM mechanism forces
|
||||||
|
smaller requests — see ADR-033's resolution triggers.
|
||||||
|
|
||||||
|
## What this did not measure
|
||||||
|
|
||||||
|
- **Metal OOM under contention.** Deliberately avoided. A big resident ollama model
|
||||||
|
during a long transcription is still the known way to wedge iguana permanently.
|
||||||
|
- **Real speech.** TTS is clean, single-voice, no crosstalk or room noise. WER and
|
||||||
|
speaker overlap in a Teams export are untested. Speed is probably representative;
|
||||||
|
accuracy is not claimed.
|
||||||
|
- **Compressed input.** Only WAV was sent. Whether `m4a`/`mp3`/`opus` can reach whisper
|
||||||
|
without a server-side transcode is not established here (ADR-033 treats it as needing one).
|
||||||
|
- **Concurrency.** One request at a time. Two simultaneous 90-minute jobs on iguana were
|
||||||
|
not tried.
|
||||||
|
- **`berget/kb-whisper` beyond 22 min / 42 MB.** Enough to know the fallback is not
|
||||||
|
size-capped at meeting length for 22 minutes; not proven at 90.
|
||||||
|
- **Time-of-day / other tenants.** One afternoon, one run per length.
|
||||||
|
|
||||||
|
Scripts were throwaway shell/Python in a session scratchpad and are not committed; the
|
||||||
|
method above reproduces them.
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
-- Drops the private uploaded-transcript store and every transcript in it. There
|
||||||
|
-- is no revert of the data: the original audio was never kept (ADR-033).
|
||||||
|
DROP TABLE uploaded_transcripts;
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
-- Migration 017: uploaded_transcripts — PRIVATE transcripts of audio a user
|
||||||
|
-- uploaded themselves (ADR-033, tapir#33).
|
||||||
|
--
|
||||||
|
-- This is deliberately NOT the shared `transcripts` table (migration 015,
|
||||||
|
-- ADR-021). That table is non-RLS on the explicit grounds that it holds only
|
||||||
|
-- public caption content and nothing user-identifying. A recorded meeting is the
|
||||||
|
-- opposite: private third-party content owned by exactly one user. Reusing
|
||||||
|
-- `transcripts` — or adding a user_id to it with a partial policy — would put a
|
||||||
|
-- private row in the one table the isolation test asserts is shared, and would
|
||||||
|
-- pass CI green. A separate table keeps ADR-021's exception singular.
|
||||||
|
--
|
||||||
|
-- Row-level security follows migration 003 exactly: ENABLE + FORCE (the app
|
||||||
|
-- connects as the table owner, which bypasses RLS unless forced), and a policy
|
||||||
|
-- keyed off the per-request GUC tapir.current_user_id. With no GUC set the
|
||||||
|
-- predicate is NULL and the table is deny-all.
|
||||||
|
--
|
||||||
|
-- The original audio is never stored (ADR-033): this row is all that is kept.
|
||||||
|
-- segments holds the timestamped cues; re-transcribing 90 minutes costs ~7
|
||||||
|
-- minutes, so discarding the timestamps would be irreversible.
|
||||||
|
CREATE TABLE uploaded_transcripts (
|
||||||
|
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||||
|
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||||
|
title TEXT NOT NULL,
|
||||||
|
status TEXT NOT NULL DEFAULT 'queued'
|
||||||
|
CHECK (status IN ('queued', 'transcribing', 'done', 'failed')),
|
||||||
|
content TEXT,
|
||||||
|
segments JSONB,
|
||||||
|
endpoint TEXT, -- which backend served it (iguana | berget); a quality fact
|
||||||
|
audio_seconds DOUBLE PRECISION,
|
||||||
|
error_kind TEXT, -- typed, never a provider's error string (ADR-033)
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||||
|
finished_at TIMESTAMPTZ,
|
||||||
|
CONSTRAINT uploaded_transcripts_done_has_content
|
||||||
|
CHECK (status <> 'done' OR content IS NOT NULL),
|
||||||
|
CONSTRAINT uploaded_transcripts_failed_has_error_kind
|
||||||
|
CHECK (status <> 'failed' OR error_kind IS NOT NULL)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX idx_uploaded_transcripts_user ON uploaded_transcripts (user_id, created_at DESC);
|
||||||
|
|
||||||
|
ALTER TABLE uploaded_transcripts ENABLE ROW LEVEL SECURITY;
|
||||||
|
ALTER TABLE uploaded_transcripts FORCE ROW LEVEL SECURITY;
|
||||||
|
CREATE POLICY uploaded_transcripts_isolation ON uploaded_transcripts
|
||||||
|
FOR ALL
|
||||||
|
USING (user_id = current_setting('tapir.current_user_id', true)::uuid);
|
||||||
|
|
||||||
|
COMMENT ON TABLE uploaded_transcripts IS
|
||||||
|
'PRIVATE per-user transcripts of user-supplied audio. RLS-scoped (ADR-012). Never merge into the shared transcripts table (ADR-021, ADR-033).';
|
||||||
@@ -0,0 +1,199 @@
|
|||||||
|
package store
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ErrUploadNotFound is returned when an uploaded transcript does not exist OR is
|
||||||
|
// not the caller's. The two are deliberately indistinguishable: a non-owner learns
|
||||||
|
// nothing about whether someone else's upload exists (ADR-033).
|
||||||
|
var ErrUploadNotFound = errors.New("store: uploaded transcript not found")
|
||||||
|
|
||||||
|
// Uploaded transcripts are PRIVATE per-user content, stored in their own
|
||||||
|
// RLS-scoped table (migration 017) — never in the shared `transcripts` (ADR-021).
|
||||||
|
// Every method goes through withUser, so row-level security confines it to the
|
||||||
|
// caller's rows; each statement ALSO names user_id in its predicate, so the
|
||||||
|
// isolation does not rest on a single mechanism and stays testable under the
|
||||||
|
// superuser connection the unit tests use.
|
||||||
|
|
||||||
|
// CreateUpload records a new upload in the queued state and returns its id.
|
||||||
|
func (s *Store) CreateUpload(ctx context.Context, userID, title string) (string, error) {
|
||||||
|
var id string
|
||||||
|
if err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
|
||||||
|
return tx.QueryRow(ctx,
|
||||||
|
`INSERT INTO uploaded_transcripts (user_id, title) VALUES ($1, $2) RETURNING id`,
|
||||||
|
userID, title).Scan(&id)
|
||||||
|
}); err != nil {
|
||||||
|
return "", fmt.Errorf("store: create upload: %w", err)
|
||||||
|
}
|
||||||
|
return id, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// StartTranscribing moves an upload to transcribing.
|
||||||
|
func (s *Store) StartTranscribing(ctx context.Context, userID, id string) error {
|
||||||
|
return s.updateUpload(ctx, userID, id, "start transcribing",
|
||||||
|
`UPDATE uploaded_transcripts SET status = 'transcribing'
|
||||||
|
WHERE id = $1 AND user_id = $2`)
|
||||||
|
}
|
||||||
|
|
||||||
|
// CompleteUpload stores the finished transcription and moves the upload to done.
|
||||||
|
func (s *Store) CompleteUpload(ctx context.Context, userID, id string, t domain.Transcription) error {
|
||||||
|
segs, err := json.Marshal(t.Segments)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("store: complete upload: marshal segments: %w", err)
|
||||||
|
}
|
||||||
|
return s.updateUpload(ctx, userID, id, "complete upload",
|
||||||
|
`UPDATE uploaded_transcripts
|
||||||
|
SET status = 'done', content = $3, segments = $4, endpoint = $5,
|
||||||
|
audio_seconds = $6, error_kind = NULL, finished_at = NOW()
|
||||||
|
WHERE id = $1 AND user_id = $2`,
|
||||||
|
t.Text, segs, t.Endpoint, t.AudioSeconds)
|
||||||
|
}
|
||||||
|
|
||||||
|
// FailUpload moves an upload to failed with a typed reason.
|
||||||
|
func (s *Store) FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error {
|
||||||
|
return s.updateUpload(ctx, userID, id, "fail upload",
|
||||||
|
`UPDATE uploaded_transcripts
|
||||||
|
SET status = 'failed', error_kind = $3, finished_at = NOW()
|
||||||
|
WHERE id = $1 AND user_id = $2`,
|
||||||
|
string(kind))
|
||||||
|
}
|
||||||
|
|
||||||
|
// updateUpload runs one UPDATE keyed on ($1 = id, $2 = userID) plus extra args,
|
||||||
|
// and maps "no row touched" to ErrUploadNotFound — which is what a write to another
|
||||||
|
// user's upload looks like under RLS (the row is invisible, not forbidden).
|
||||||
|
func (s *Store) updateUpload(ctx context.Context, userID, id, op, sql string, args ...any) error {
|
||||||
|
all := append([]any{id, userID}, args...)
|
||||||
|
return s.withUser(ctx, userID, func(tx pgx.Tx) error {
|
||||||
|
ct, err := tx.Exec(ctx, sql, all...)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("store: %s: %w", op, err)
|
||||||
|
}
|
||||||
|
if ct.RowsAffected() == 0 {
|
||||||
|
return ErrUploadNotFound
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetUpload returns one of the caller's uploads, including the transcript text.
|
||||||
|
func (s *Store) GetUpload(ctx context.Context, userID, id string) (*domain.UploadedTranscript, error) {
|
||||||
|
var u domain.UploadedTranscript
|
||||||
|
err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
|
||||||
|
row := tx.QueryRow(ctx, uploadSelect+` WHERE id = $1 AND user_id = $2`, id, userID)
|
||||||
|
return scanUpload(row, &u, true)
|
||||||
|
})
|
||||||
|
if errors.Is(err, pgx.ErrNoRows) {
|
||||||
|
return nil, ErrUploadNotFound
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("store: get upload: %w", err)
|
||||||
|
}
|
||||||
|
return &u, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ListUploads returns the caller's uploads, newest first, WITHOUT transcript text
|
||||||
|
// or segments: a list view must not drag every transcript along.
|
||||||
|
func (s *Store) ListUploads(ctx context.Context, userID string) ([]domain.UploadedTranscript, error) {
|
||||||
|
var out []domain.UploadedTranscript
|
||||||
|
err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
|
||||||
|
rows, err := tx.Query(ctx, uploadSelect+` WHERE user_id = $1 ORDER BY created_at DESC, id`, userID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
for rows.Next() {
|
||||||
|
var u domain.UploadedTranscript
|
||||||
|
if err := scanUpload(rows, &u, false); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
out = append(out, u)
|
||||||
|
}
|
||||||
|
return rows.Err()
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("store: list uploads: %w", err)
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// FailInterruptedUploads is the start-up sweep. The audio of an upload is never
|
||||||
|
// retained (ADR-033), so a job that was queued or transcribing when the process
|
||||||
|
// stopped can never resume; leaving it would be a stuck spinner. Every such row,
|
||||||
|
// for every user, becomes failed/interrupted and the user re-uploads. It
|
||||||
|
// enumerates users from the un-RLS'd identity map and scopes each pass with
|
||||||
|
// withUser, exactly as scheduled discovery does (ADR-018). Returns rows failed.
|
||||||
|
func (s *Store) FailInterruptedUploads(ctx context.Context) (int, error) {
|
||||||
|
users, err := s.ListAllUsers(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return 0, fmt.Errorf("store: fail interrupted uploads: %w", err)
|
||||||
|
}
|
||||||
|
total := 0
|
||||||
|
for _, u := range users {
|
||||||
|
err := s.withUser(ctx, u.UserID, func(tx pgx.Tx) error {
|
||||||
|
ct, err := tx.Exec(ctx,
|
||||||
|
`UPDATE uploaded_transcripts
|
||||||
|
SET status = 'failed', error_kind = $2, finished_at = NOW()
|
||||||
|
WHERE user_id = $1 AND status IN ('queued', 'transcribing')`,
|
||||||
|
u.UserID, string(domain.UploadErrInterrupted))
|
||||||
|
total += int(ct.RowsAffected())
|
||||||
|
return err
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return total, fmt.Errorf("store: fail interrupted uploads: user %s: %w", u.UserID, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return total, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
const uploadSelect = `SELECT id, user_id, title, status, content, segments, endpoint,
|
||||||
|
audio_seconds, error_kind, created_at, finished_at FROM uploaded_transcripts`
|
||||||
|
|
||||||
|
// scanUpload scans one uploadSelect row. withText controls whether the transcript
|
||||||
|
// body and segments are kept (GetUpload) or dropped (ListUploads).
|
||||||
|
func scanUpload(row pgx.Row, u *domain.UploadedTranscript, withText bool) error {
|
||||||
|
var (
|
||||||
|
status, errKind, endpoint, content *string
|
||||||
|
segments []byte
|
||||||
|
seconds *float64
|
||||||
|
finished *time.Time
|
||||||
|
)
|
||||||
|
if err := row.Scan(&u.ID, &u.UserID, &u.Title, &status, &content, &segments, &endpoint,
|
||||||
|
&seconds, &errKind, &u.CreatedAt, &finished); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
u.Status = domain.UploadStatus(deref(status))
|
||||||
|
u.Endpoint = deref(endpoint)
|
||||||
|
u.ErrorKind = domain.UploadErrorKind(deref(errKind))
|
||||||
|
if seconds != nil {
|
||||||
|
u.AudioSeconds = *seconds
|
||||||
|
}
|
||||||
|
if finished != nil {
|
||||||
|
u.FinishedAt = *finished
|
||||||
|
}
|
||||||
|
if !withText {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
u.Content = deref(content)
|
||||||
|
if len(segments) > 0 {
|
||||||
|
if err := json.Unmarshal(segments, &u.Segments); err != nil {
|
||||||
|
return fmt.Errorf("unmarshal segments: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func deref(s *string) string {
|
||||||
|
if s == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return *s
|
||||||
|
}
|
||||||
@@ -0,0 +1,196 @@
|
|||||||
|
package store_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"sort"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5/pgxpool"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Isolation proof for ADR-033: an uploaded transcript is PRIVATE content owned by
|
||||||
|
// one user, so it lives in its own RLS-scoped table — never in the shared,
|
||||||
|
// non-RLS `transcripts` (ADR-021). These tests drive the database directly (raw
|
||||||
|
// SQL as the non-superuser `app` role) so they observe row-level security itself,
|
||||||
|
// not any application-layer WHERE clause. The superuser pool is used only to seed
|
||||||
|
// and to read pg_catalog; it bypasses RLS and so must never be the pool whose
|
||||||
|
// answers the assertions rest on.
|
||||||
|
|
||||||
|
// seedUploadedTranscript inserts one finished upload for userID as the superuser
|
||||||
|
// (RLS bypassed so both users' rows land regardless of any GUC) and returns its id.
|
||||||
|
// The text is synthetic — this repo is public, and a real transcript must never
|
||||||
|
// appear in it (#33).
|
||||||
|
func seedUploadedTranscript(t *testing.T, super *pgxpool.Pool, userID, text string) string {
|
||||||
|
t.Helper()
|
||||||
|
ctx := context.Background()
|
||||||
|
_, err := super.Exec(ctx, `INSERT INTO users (id) VALUES ($1) ON CONFLICT DO NOTHING`, userID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
var id string
|
||||||
|
require.NoError(t, super.QueryRow(ctx,
|
||||||
|
`INSERT INTO uploaded_transcripts (user_id, title, status, content)
|
||||||
|
VALUES ($1, 'synthetic fixture', 'done', $2) RETURNING id`,
|
||||||
|
userID, text).Scan(&id))
|
||||||
|
return id
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadedTranscriptsRLS_ScopedReadsSeeOnlyOwnRows(t *testing.T) {
|
||||||
|
newStore(t)
|
||||||
|
super := rawPool(t)
|
||||||
|
resetDB(t, super)
|
||||||
|
seedUploadedTranscript(t, super, userA, "text owned by A")
|
||||||
|
seedUploadedTranscript(t, super, userB, "text owned by B")
|
||||||
|
app := appPool(t, super)
|
||||||
|
|
||||||
|
require.Equal(t, 0, scopedCount(t, app, "", "uploaded_transcripts"),
|
||||||
|
"no tapir.current_user_id must be deny-all: this proves RLS is live, not bypassed")
|
||||||
|
require.Equal(t, 1, scopedCount(t, app, userA, "uploaded_transcripts"),
|
||||||
|
"user A must see exactly its own upload")
|
||||||
|
require.Equal(t, 1, scopedCount(t, app, userB, "uploaded_transcripts"),
|
||||||
|
"user B must see exactly its own upload")
|
||||||
|
|
||||||
|
// Count alone cannot distinguish "A's row" from "B's row" — read the content.
|
||||||
|
var got string
|
||||||
|
ctx := context.Background()
|
||||||
|
tx, err := app.Begin(ctx)
|
||||||
|
require.NoError(t, err)
|
||||||
|
defer tx.Rollback(ctx) //nolint:errcheck
|
||||||
|
_, err = tx.Exec(ctx, `SELECT set_config('tapir.current_user_id', $1, true)`, userA)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, tx.QueryRow(ctx, `SELECT content FROM uploaded_transcripts`).Scan(&got))
|
||||||
|
require.Equal(t, "text owned by A", got, "A's scoped read must return A's text, never B's")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadedTranscriptsRLS_CrossUserWritesAffectZeroRows(t *testing.T) {
|
||||||
|
newStore(t)
|
||||||
|
super := rawPool(t)
|
||||||
|
resetDB(t, super)
|
||||||
|
seedUploadedTranscript(t, super, userA, "text owned by A")
|
||||||
|
bID := seedUploadedTranscript(t, super, userB, "text owned by B")
|
||||||
|
app := appPool(t, super)
|
||||||
|
|
||||||
|
require.Equal(t, int64(0), scopedRowsAffected(t, app, userA,
|
||||||
|
`UPDATE uploaded_transcripts SET content = 'hacked' WHERE id = $1`, bID),
|
||||||
|
"user A scoped UPDATE must touch zero of B's uploads")
|
||||||
|
require.Equal(t, int64(0), scopedRowsAffected(t, app, userA,
|
||||||
|
`DELETE FROM uploaded_transcripts WHERE id = $1`, bID),
|
||||||
|
"user A scoped DELETE must touch zero of B's uploads")
|
||||||
|
|
||||||
|
var content string
|
||||||
|
require.NoError(t, super.QueryRow(context.Background(),
|
||||||
|
`SELECT content FROM uploaded_transcripts WHERE id = $1`, bID).Scan(&content))
|
||||||
|
require.Equal(t, "text owned by B", content, "B's upload must survive A's writes unchanged")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadedTranscriptsRLS_InsertForAnotherUserIsRejected(t *testing.T) {
|
||||||
|
newStore(t)
|
||||||
|
super := rawPool(t)
|
||||||
|
resetDB(t, super)
|
||||||
|
seedUploadedTranscript(t, super, userA, "text owned by A")
|
||||||
|
_, err := super.Exec(context.Background(), `INSERT INTO users (id) VALUES ($1)`, userB)
|
||||||
|
require.NoError(t, err)
|
||||||
|
app := appPool(t, super)
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
tx, err := app.Begin(ctx)
|
||||||
|
require.NoError(t, err)
|
||||||
|
defer tx.Rollback(ctx) //nolint:errcheck
|
||||||
|
_, err = tx.Exec(ctx, `SELECT set_config('tapir.current_user_id', $1, true)`, userA)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
// Scoped as A, plant a row that claims to belong to B. The policy's WITH CHECK
|
||||||
|
// (defaulting to USING under FOR ALL) must refuse it.
|
||||||
|
_, err = tx.Exec(ctx,
|
||||||
|
`INSERT INTO uploaded_transcripts (user_id, title, status, content)
|
||||||
|
VALUES ($1, 'planted', 'done', 'planted')`, userB)
|
||||||
|
require.Error(t, err, "a row scoped as A must not be insertable on behalf of B")
|
||||||
|
require.Contains(t, err.Error(), "row-level security")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadedTranscripts_RowSecurityIsEnabledForcedAndKeyedByUser(t *testing.T) {
|
||||||
|
newStore(t)
|
||||||
|
super := rawPool(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
var enabled, forced bool
|
||||||
|
require.NoError(t, super.QueryRow(ctx,
|
||||||
|
`SELECT relrowsecurity, relforcerowsecurity FROM pg_class
|
||||||
|
WHERE oid = 'public.uploaded_transcripts'::regclass`).Scan(&enabled, &forced))
|
||||||
|
require.True(t, enabled, "uploaded_transcripts must have row-level security enabled")
|
||||||
|
require.True(t, forced,
|
||||||
|
"FORCE is what binds the table-owner role tapir; without it the policy is dead in production")
|
||||||
|
|
||||||
|
var userIDNotNull bool
|
||||||
|
require.NoError(t, super.QueryRow(ctx,
|
||||||
|
`SELECT is_nullable = 'NO' FROM information_schema.columns
|
||||||
|
WHERE table_name = 'uploaded_transcripts' AND column_name = 'user_id'`).Scan(&userIDNotNull))
|
||||||
|
require.True(t, userIDNotNull, "an upload with no owner would match no policy and be unreachable")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadedTranscripts_DeleteUserRemovesTheirUploads(t *testing.T) {
|
||||||
|
s := newStore(t)
|
||||||
|
super := rawPool(t)
|
||||||
|
resetDB(t, super)
|
||||||
|
seedUploadedTranscript(t, super, userA, "text owned by A")
|
||||||
|
bID := seedUploadedTranscript(t, super, userB, "text owned by B")
|
||||||
|
|
||||||
|
require.NoError(t, s.DeleteUser(context.Background(), userA))
|
||||||
|
|
||||||
|
var aRows, bRows int
|
||||||
|
ctx := context.Background()
|
||||||
|
require.NoError(t, super.QueryRow(ctx,
|
||||||
|
`SELECT count(*) FROM uploaded_transcripts WHERE user_id = $1`, userA).Scan(&aRows))
|
||||||
|
require.NoError(t, super.QueryRow(ctx,
|
||||||
|
`SELECT count(*) FROM uploaded_transcripts WHERE id = $1`, bID).Scan(&bRows))
|
||||||
|
require.Equal(t, 0, aRows, "deleting a user must remove their private transcripts (ON DELETE CASCADE)")
|
||||||
|
require.Equal(t, 1, bRows, "deleting A must not touch B's transcripts")
|
||||||
|
}
|
||||||
|
|
||||||
|
// rowSecurityOffTables are the public-schema tables that deliberately have row
|
||||||
|
// security OFF. The list is exact on purpose: a NEW table added without RLS must
|
||||||
|
// fail this test and force a decision, instead of passing CI by omission — the
|
||||||
|
// failure mode ADR-033 names, where a naive reuse "leaks private content across
|
||||||
|
// tenants and passes CI green".
|
||||||
|
//
|
||||||
|
// - transcripts: shared public caption content, the single exception for CONTENT
|
||||||
|
// (ADR-021). It is the one that must never gain a private row.
|
||||||
|
// - user_identities: the Dex-subject -> user_id map, read at login before any
|
||||||
|
// user is known (migration 004). Holds no user content.
|
||||||
|
// - invitations: exists before the user does, so there is no user to scope by;
|
||||||
|
// the single-use token is the capability (migration 009).
|
||||||
|
// - schema_migrations: golang-migrate bookkeeping.
|
||||||
|
//
|
||||||
|
// #33 describes `transcripts` as "the only non-RLS table". That is true of content
|
||||||
|
// and false of tables; this test asserts the real set.
|
||||||
|
var rowSecurityOffTables = []string{
|
||||||
|
"invitations",
|
||||||
|
"schema_migrations",
|
||||||
|
"transcripts",
|
||||||
|
"user_identities",
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRowSecurityOffTablesAreExactlyTheKnownSet(t *testing.T) {
|
||||||
|
newStore(t)
|
||||||
|
super := rawPool(t)
|
||||||
|
|
||||||
|
rows, err := super.Query(context.Background(),
|
||||||
|
`SELECT c.relname FROM pg_class c
|
||||||
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
||||||
|
WHERE n.nspname = 'public' AND c.relkind IN ('r', 'p') AND NOT c.relrowsecurity
|
||||||
|
ORDER BY c.relname`)
|
||||||
|
require.NoError(t, err)
|
||||||
|
defer rows.Close()
|
||||||
|
var got []string
|
||||||
|
for rows.Next() {
|
||||||
|
var name string
|
||||||
|
require.NoError(t, rows.Scan(&name))
|
||||||
|
got = append(got, name)
|
||||||
|
}
|
||||||
|
require.NoError(t, rows.Err())
|
||||||
|
|
||||||
|
want := append([]string{}, rowSecurityOffTables...)
|
||||||
|
sort.Strings(want)
|
||||||
|
require.Equal(t, want, got,
|
||||||
|
"every public table must have row security ON unless it is in rowSecurityOffTables; "+
|
||||||
|
"adding a table here is a decision that needs an ADR")
|
||||||
|
}
|
||||||
@@ -0,0 +1,188 @@
|
|||||||
|
package store_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/adapters/store"
|
||||||
|
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||||
|
)
|
||||||
|
|
||||||
|
// These exercise the Store API over uploaded_transcripts. The test Store connects
|
||||||
|
// as the superuser, which BYPASSES row-level security — so the cross-user cases
|
||||||
|
// here prove the query's own user_id predicate (defence in depth), while the
|
||||||
|
// database-enforced guarantee is proven separately, as the non-superuser app
|
||||||
|
// role, in uploaded_transcripts_isolation_test.go. Text is synthetic; this repo is
|
||||||
|
// public and no real transcript may appear in it (#33).
|
||||||
|
|
||||||
|
func seedUsers(t *testing.T, ids ...string) {
|
||||||
|
t.Helper()
|
||||||
|
super := rawPool(t)
|
||||||
|
resetDB(t, super)
|
||||||
|
for _, id := range ids {
|
||||||
|
_, err := super.Exec(context.Background(), `INSERT INTO users (id) VALUES ($1)`, id)
|
||||||
|
require.NoError(t, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCreateUpload_StartsQueuedAndIsOwnedByTheCreator(t *testing.T) {
|
||||||
|
s := newStore(t)
|
||||||
|
seedUsers(t, userA)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
id, err := s.CreateUpload(ctx, userA, "Teams export")
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
got, err := s.GetUpload(ctx, userA, id)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, id, got.ID)
|
||||||
|
assert.Equal(t, userA, got.UserID)
|
||||||
|
assert.Equal(t, "Teams export", got.Title)
|
||||||
|
assert.Equal(t, domain.UploadQueued, got.Status)
|
||||||
|
assert.Empty(t, got.Content)
|
||||||
|
assert.True(t, got.FinishedAt.IsZero(), "an unfinished upload has no finish time")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadLifecycle_CompletedKeepsTextSegmentsEndpointAndDuration(t *testing.T) {
|
||||||
|
s := newStore(t)
|
||||||
|
seedUsers(t, userA)
|
||||||
|
ctx := context.Background()
|
||||||
|
id, err := s.CreateUpload(ctx, userA, "m")
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
||||||
|
mid, err := s.GetUpload(ctx, userA, id)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, domain.UploadTranscribing, mid.Status)
|
||||||
|
|
||||||
|
want := domain.Transcription{
|
||||||
|
Text: "God morgon allihopa. Vi går igenom läget.",
|
||||||
|
Segments: []domain.Segment{
|
||||||
|
{Start: 0.0, End: 1.5, Text: "God morgon allihopa."},
|
||||||
|
{Start: 1.5, End: 4.25, Text: "Vi går igenom läget."},
|
||||||
|
},
|
||||||
|
Endpoint: "iguana/kb-whisper",
|
||||||
|
AudioSeconds: 4.25,
|
||||||
|
}
|
||||||
|
require.NoError(t, s.CompleteUpload(ctx, userA, id, want))
|
||||||
|
|
||||||
|
got, err := s.GetUpload(ctx, userA, id)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, domain.UploadDone, got.Status)
|
||||||
|
assert.Equal(t, want.Text, got.Content)
|
||||||
|
assert.Equal(t, want.Segments, got.Segments, "segment times must survive the JSONB round trip")
|
||||||
|
assert.Equal(t, "iguana/kb-whisper", got.Endpoint, "the serving endpoint is a quality fact and is recorded per row")
|
||||||
|
assert.InDelta(t, 4.25, got.AudioSeconds, 1e-9)
|
||||||
|
assert.False(t, got.FinishedAt.IsZero())
|
||||||
|
assert.Empty(t, got.ErrorKind)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadLifecycle_FailedRecordsTypedKindAndNoContent(t *testing.T) {
|
||||||
|
s := newStore(t)
|
||||||
|
seedUsers(t, userA)
|
||||||
|
ctx := context.Background()
|
||||||
|
id, err := s.CreateUpload(ctx, userA, "m")
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
||||||
|
|
||||||
|
require.NoError(t, s.FailUpload(ctx, userA, id, domain.UploadErrTranscriptionFailed))
|
||||||
|
|
||||||
|
got, err := s.GetUpload(ctx, userA, id)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, domain.UploadFailed, got.Status)
|
||||||
|
assert.Equal(t, domain.UploadErrTranscriptionFailed, got.ErrorKind)
|
||||||
|
assert.Empty(t, got.Content)
|
||||||
|
assert.False(t, got.FinishedAt.IsZero())
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGetUpload_AnotherUsersIDIsNotFound(t *testing.T) {
|
||||||
|
s := newStore(t)
|
||||||
|
seedUsers(t, userA, userB)
|
||||||
|
ctx := context.Background()
|
||||||
|
id, err := s.CreateUpload(ctx, userA, "A's meeting")
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
_, err = s.GetUpload(ctx, userB, id)
|
||||||
|
|
||||||
|
assert.ErrorIs(t, err, store.ErrUploadNotFound,
|
||||||
|
"a non-owner must get not-found, not forbidden: existence is not disclosed")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadWrites_ByAnotherUserChangeNothing(t *testing.T) {
|
||||||
|
s := newStore(t)
|
||||||
|
seedUsers(t, userA, userB)
|
||||||
|
ctx := context.Background()
|
||||||
|
id, err := s.CreateUpload(ctx, userA, "A's meeting")
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
assert.ErrorIs(t, s.StartTranscribing(ctx, userB, id), store.ErrUploadNotFound)
|
||||||
|
assert.ErrorIs(t, s.CompleteUpload(ctx, userB, id, domain.Transcription{Text: "planted"}), store.ErrUploadNotFound)
|
||||||
|
assert.ErrorIs(t, s.FailUpload(ctx, userB, id, domain.UploadErrTranscriptionFailed), store.ErrUploadNotFound)
|
||||||
|
|
||||||
|
got, err := s.GetUpload(ctx, userA, id)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, domain.UploadQueued, got.Status, "B's writes must leave A's upload exactly as it was")
|
||||||
|
assert.Empty(t, got.Content)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestListUploads_ReturnsOnlyTheCallersNewestFirstWithoutTranscriptText(t *testing.T) {
|
||||||
|
s := newStore(t)
|
||||||
|
seedUsers(t, userA, userB)
|
||||||
|
ctx := context.Background()
|
||||||
|
first, err := s.CreateUpload(ctx, userA, "first")
|
||||||
|
require.NoError(t, err)
|
||||||
|
second, err := s.CreateUpload(ctx, userA, "second")
|
||||||
|
require.NoError(t, err)
|
||||||
|
_, err = s.CreateUpload(ctx, userB, "B's")
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, s.StartTranscribing(ctx, userA, first))
|
||||||
|
require.NoError(t, s.CompleteUpload(ctx, userA, first, domain.Transcription{Text: "long text body", Endpoint: "e"}))
|
||||||
|
|
||||||
|
got, err := s.ListUploads(ctx, userA)
|
||||||
|
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, got, 2, "A must see only A's uploads")
|
||||||
|
assert.Equal(t, second, got[0].ID, "newest first")
|
||||||
|
assert.Equal(t, first, got[1].ID)
|
||||||
|
assert.Empty(t, got[1].Content, "a list row must not drag the whole transcript along")
|
||||||
|
assert.Equal(t, domain.UploadDone, got[1].Status)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFailInterruptedUploads_FailsOnlyUnfinishedRowsForEveryUser(t *testing.T) {
|
||||||
|
s := newStore(t)
|
||||||
|
seedUsers(t, userA, userB)
|
||||||
|
super := rawPool(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
for subject, uid := range map[string]string{"sub-a": userA, "sub-b": userB} {
|
||||||
|
_, err := super.Exec(ctx, `INSERT INTO user_identities (dex_subject, user_id) VALUES ($1, $2)`, subject, uid)
|
||||||
|
require.NoError(t, err)
|
||||||
|
}
|
||||||
|
queued, err := s.CreateUpload(ctx, userA, "queued")
|
||||||
|
require.NoError(t, err)
|
||||||
|
running, err := s.CreateUpload(ctx, userB, "running")
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, s.StartTranscribing(ctx, userB, running))
|
||||||
|
done, err := s.CreateUpload(ctx, userA, "done")
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, s.StartTranscribing(ctx, userA, done))
|
||||||
|
require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "kept", Endpoint: "e"}))
|
||||||
|
|
||||||
|
n, err := s.FailInterruptedUploads(ctx)
|
||||||
|
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, 2, n, "the queued and the transcribing row, across both users")
|
||||||
|
for _, c := range []struct{ user, id string }{{userA, queued}, {userB, running}} {
|
||||||
|
got, err := s.GetUpload(ctx, c.user, c.id)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, domain.UploadFailed, got.Status)
|
||||||
|
assert.Equal(t, domain.UploadErrInterrupted, got.ErrorKind,
|
||||||
|
"audio is not retained, so a restart cannot resume: the honest state is failed/interrupted")
|
||||||
|
}
|
||||||
|
kept, err := s.GetUpload(ctx, userA, done)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, domain.UploadDone, kept.Status, "a finished transcript must survive the sweep")
|
||||||
|
assert.Equal(t, "kept", kept.Content)
|
||||||
|
}
|
||||||
@@ -111,3 +111,62 @@ type Summary struct {
|
|||||||
FallbackUsed bool
|
FallbackUsed bool
|
||||||
CreatedAt time.Time
|
CreatedAt time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Segment is one timed span of a transcription. Times are seconds from the start
|
||||||
|
// of the audio.
|
||||||
|
type Segment struct {
|
||||||
|
Start float64 `json:"start"`
|
||||||
|
End float64 `json:"end"`
|
||||||
|
Text string `json:"text"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Transcription is the result of transcribing user-supplied audio (ADR-033):
|
||||||
|
// the full text, its timed segments, which backend served it, and how long the
|
||||||
|
// audio was. Endpoint is recorded because the backends differ in timestamp
|
||||||
|
// granularity — which one served a transcript is a quality fact.
|
||||||
|
type Transcription struct {
|
||||||
|
Text string
|
||||||
|
Segments []Segment
|
||||||
|
Endpoint string
|
||||||
|
AudioSeconds float64
|
||||||
|
}
|
||||||
|
|
||||||
|
// UploadStatus is the state of an uploaded-audio transcription job.
|
||||||
|
type UploadStatus string
|
||||||
|
|
||||||
|
const (
|
||||||
|
UploadQueued UploadStatus = "queued"
|
||||||
|
UploadTranscribing UploadStatus = "transcribing"
|
||||||
|
UploadDone UploadStatus = "done"
|
||||||
|
UploadFailed UploadStatus = "failed"
|
||||||
|
)
|
||||||
|
|
||||||
|
// UploadErrorKind is the typed reason an upload failed. It is deliberately NOT a
|
||||||
|
// provider's error string: the whisper server's "failed to process audio" has had
|
||||||
|
// at least five distinct causes, so nothing may branch on it (ADR-033).
|
||||||
|
type UploadErrorKind string
|
||||||
|
|
||||||
|
const (
|
||||||
|
// UploadErrInterrupted: the process restarted mid-job. The audio is not
|
||||||
|
// retained, so the job cannot resume; the user re-uploads.
|
||||||
|
UploadErrInterrupted UploadErrorKind = "interrupted"
|
||||||
|
// UploadErrTranscriptionFailed: every configured transcription endpoint failed.
|
||||||
|
UploadErrTranscriptionFailed UploadErrorKind = "transcription_failed"
|
||||||
|
)
|
||||||
|
|
||||||
|
// UploadedTranscript is a user's PRIVATE transcript of audio they uploaded. It is
|
||||||
|
// never stored in, or merged with, the shared public-caption Transcript (ADR-021).
|
||||||
|
// Content and Segments are empty until Status is UploadDone.
|
||||||
|
type UploadedTranscript struct {
|
||||||
|
ID string
|
||||||
|
UserID string
|
||||||
|
Title string
|
||||||
|
Status UploadStatus
|
||||||
|
Content string
|
||||||
|
Segments []Segment
|
||||||
|
Endpoint string
|
||||||
|
AudioSeconds float64
|
||||||
|
ErrorKind UploadErrorKind
|
||||||
|
CreatedAt time.Time
|
||||||
|
FinishedAt time.Time // zero until the job reaches done or failed
|
||||||
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user