Compare commits

...
82 Commits
Author SHA1 Message Date
mathiasandClaude Sonnet 5.5 1c726a810e docs(homelab): upload chat budget is 90000 and the chat model is gemma4-26b
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:08:33 +02:00
mathiasandClaude Sonnet 5.5 77fe6954e5 docs(adr): ADR-033 decision 8 -- chat budget and model from measurement
The proposed 120,000-char budget was a guess; measured, gemma4-26b silently
loses the head at that size and phi4-mini (first in the offered list) 400s on
an hour of speech. Budget is now 90,000, uploads open on gemma4-26b, the
retrieval trigger is concrete, and the iguana contention hazard is recorded.
ADR-033 is still Proposed, so this is an in-place correction, not a supersession.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:08:28 +02:00
mathiasandClaude Sonnet 5.5 4d3e2a471a docs(research): chat over a long transcript -- what the offered models really hold
phi4-mini 400s at ~13k tokens; gemma4-26b is correct to 90,000 chars and
silently loses the HEAD at 120,000 (no error, no truncation flag). The
end-needle test alone passed at 120k -- only a head-needle exposed it.
Records the default-model trap and the iguana contention hazard.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:08:13 +02:00
mathiasandClaude Sonnet 5.5 548bfaee85 test(e2e): use the same wireUploads as serve
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:41 +02:00
mathiasandClaude Sonnet 5.5 ad2e3e0e8e fix(cmd): serve now actually sets the upload chat budget and model
cmdServe assigned Uploads and UploadLimits but not UploadChatMaxChars, so chat
over a long recording would have used the 18,000-char caption default. It now
calls wireUploads, which is tested.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:41 +02:00
mathiasandClaude Sonnet 5.5 e6738bcd3e feat(cmd): wireUploads -- limits, chat budget and chat model into the web App
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:40 +02:00
mathiasandClaude Sonnet 5.5 3bb4e30f4c test(cmd): wireUploads sets the budget and model, and never a typed-nil Intake (red)
I wired Uploads and UploadLimits in cmdServe but not UploadChatMaxChars: chat
over a long recording would have silently used the 18,000-char caption
default. A helper that main and the e2e test both call makes that a tested fact.

Red: wireUploads is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:33 +02:00
mathiasandClaude Sonnet 5.5 6c39ab86cf feat(web): chat over an upload opens with App.UploadChatModel
The Chatter's own default is the summarizer's (koala/phi4-mini, 8,192-token
context), which answers HTTP 400 on an hour of speech. Empty keeps the old
behaviour.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:33 +02:00
mathiasandClaude Sonnet 5.5 4b6ac2a213 test(web): chat over an upload opens with the upload chat model (red)
Red: App.UploadChatModel is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:20 +02:00
mathiasandClaude Sonnet 5.5 af66419a44 feat(config): measured chat budget (90,000) and upload chat model (gemma4-26b)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:20 +02:00
mathiasandClaude Sonnet 5.5 4b09dc8f36 test(config): chat defaults for uploads come from measurement, not a guess (red)
iguana/gemma4-26b answered correctly at 90,000 chars but SILENTLY dropped the
head of a 120,000-char transcript (prompt tokens fell to 16k; 'not in the
text'); koala/phi4-mini has an 8,192-token context and answers HTTP 400 at
~13k tokens. So the budget is 90,000 and the upload chat model defaults to
gemma4-26b. Red: UploadChatModel is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:07:09 +02:00
mathiasandClaude Sonnet 5.5 8f16ab671a docs(claude): audio upload supersedes the STT deferral for user-supplied audio only
The 'no audio-download + STT' bullet is narrowed to YouTube (ADR-007 stands);
adds the upload packages, the exact-set isolation test and the pending ffmpeg
decision to the build state.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:02:11 +02:00
mathiasandClaude Sonnet 5.5 39aeda7c69 docs(homelab): speech-to-text endpoints, the ingress 413, and the upload knobs
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:02:05 +02:00
mathiasandClaude Sonnet 5.5 0fc25a24ed docs(data-model): uploaded_transcripts, and fix the stale claim that transcripts is RLS'd
Adds the private uploaded_transcripts entity and the exact-set isolation rule
(ADR-033). Also corrects two lists that still named `transcripts` as an RLS'd
table: it has been shared non-RLS since ADR-021 / migration 015, and these
lists sit next to the very boundary that matters.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:02:05 +02:00
mathiasandClaude Sonnet 5.5 07375f75de test(acceptance): map audio_upload.feature scenarios to their covering tests
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:01:38 +02:00
mathiasandClaude Sonnet 5.5 f9d6df9f84 docs(use-cases): audio_upload.feature -- scenarios for #33 (ADR-033)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:01:20 +02:00
mathiasandClaude Sonnet 5.5 3464cdf876 test(e2e): opt-in live upload -> transcript -> grounded chat check
Real ffprobe/ffmpeg, the live gateway and a real chat model, a private
embedded Postgres. Skipped unless TAPIR_E2E_AUDIO is set. Run 2026-10-01 with
82 s of synthetic Swedish as AAC .m4a:
- primary: iguana/kb-whisper, 82.6 s audio, 1,186 chars; asked who owns the
  timber order, the answer named Anders and the 15 November deadline from the
  transcript; the spool directory was empty afterwards.
- primary forced to fail (TAPIR_TRANSCRIBE_MODEL=iguana/does-not-exist): the
  fallback served it and the row recorded endpoint=berget/kb-whisper.
Synthetic audio only; this repo is public.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 17:01:06 +02:00
mathiasandClaude Sonnet 5.5 4fdb212cef feat(web): chat with an uploaded transcript, reusing the ADR-027 path (ADR-033)
GET/POST /uploads/{id}/chat, mounted only when both an Uploader and a Chatter
are wired. The row comes from the user-scoped private store, so another user's
upload is a 404 before any model is involved. The owner's transcript goes to
the SAME Chatter with App.UploadChatMaxChars instead of the 18,000-char caption
default. The chat panel is shared, not forked: chatView gains PostURL/Subject
(empty = a YouTube video, so existing callers are unchanged). The done card
links to chat only when chat is wired.

Mutation-checked: dropping the upload budget, ignoring the done-status guard,
or scoping the lookup to the wrong user each turn tests red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:58:25 +02:00
mathiasandClaude Sonnet 5.5 a39f58b730 test(web): match the HTML-escaped tail of the inline chat error
templ escapes the apostrophe in "couldn't", so the assertion could never match
the rendered page. A test defect, not a loosening: the failure must still be
shown inline with the conversation kept.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:58:25 +02:00
mathiasandClaude Sonnet 5.5 10ca45fc0a test(web): chat over an uploaded transcript reuses the ADR-027 path (red)
Owner sees the panel posting to /uploads/{id}/chat; an unfinished upload
redirects to its status; ANOTHER USER'S upload is 404 on GET and POST and no
model is ever called with it; the owner's transcript is sent with the upload
budget (not the 18,000-char caption default); an unoffered model falls back; a
model failure is inline and keeps the conversation; routes are absent without a
chat backend; the done card links to chat only when chat is wired.

Red: App.UploadChatMaxChars is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:57:47 +02:00
mathiasandClaude Sonnet 5.5 c72d366935 feat(chat): per-request transcript budget so a long recording is not cut to its head (ADR-033)
chat.Request.MaxChars overrides the Service cap when positive. The 18,000-char
default (~22 min of speech) suits captions; an uploaded hour is ~50,000 chars
and truncation keeps only the head. This is reuse of the existing chat path,
not a fork.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:57:01 +02:00
mathiasandClaude Sonnet 5.5 ec29c5e761 style(chat): gofumpt the new test
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:57:01 +02:00
mathiasandClaude Sonnet 5.5 867afefb08 test(chat): a request may carry its own transcript budget (red)
The 18,000-char default is ~22 min of speech and keeps only the head; an
uploaded hour is ~50,000 chars. Red: Request.MaxChars is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:56:51 +02:00
mathiasandClaude Sonnet 5.5 1c5d85f5c6 feat(cmd): serve mounts audio upload when enabled (ADR-033)
Wires buildUploads into cmdServe: Intake and limits into the web App, the
worker on the server's ctx. Logs the two facts an operator must not miss: the
queue is process-local (single replica), and a configured fallback is a cloud
endpoint that private audio can leave the homelab through.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:56:22 +02:00
mathiasandClaude Sonnet 5.5 9b3175db09 feat(cmd): buildUploads -- spool, sweep, fail interrupted rows, wire the chain (ADR-033)
Returns nils when disabled (never a typed-nil Intake). When enabled: private
0700 spool dir, delete audio left by a crash, fail rows a restart left queued
or transcribing (the audio is not retained, so they cannot resume), then build
the iguana->berget chain, worker and intake. An empty
TAPIR_TRANSCRIBE_FALLBACK_MODEL keeps private audio on the homelab. Enabling
with no gateway fails at start-up, not at the first upload.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:56:21 +02:00
mathiasandClaude Sonnet 5.5 eb9357a5de test(cmd): start-up removes spooled audio left by a crash (red)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:56:11 +02:00
mathiasandClaude Sonnet 5.5 3e802adade test(cmd): buildUploads wiring -- off by default, sweep first, private spool (red)
Red: buildUploads is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:55:55 +02:00
mathiasandClaude Sonnet 5.5 b8798dbd5d feat(config): upload and transcription settings (ADR-033)
Embedded UploadConfig. Off by default until the image carries ffmpeg; 200 MB /
90 min; iguana primary with a berget fallback that an explicit empty
TAPIR_TRANSCRIBE_FALLBACK_MODEL disables (private audio then never leaves the
homelab); zero or negative limits fail at load instead of rejecting every
upload at runtime.

uploads.go and the two-line hook in config.go are one commit: the tests need both.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:55:32 +02:00
mathiasandClaude Sonnet 5.5 fb521f2b89 test(config): upload and transcription settings (red)
Off by default until the image has ffmpeg; 200 MB / 90 min (the longest length
measured); iguana primary, berget fallback that an explicit empty value
disables; zero or negative limits are refused rather than silently rejecting
every upload.

Red: the Config fields do not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:55:11 +02:00
mathiasandClaude Sonnet 5.5 a8dadb8ca4 feat(uploads): in-process transcription worker, one job at a time (ADR-033)
Queue -> StartTranscribing -> ffmpeg to 16 kHz WAV -> gateway chain ->
CompleteUpload, with a per-job timeout so a hung endpoint ends in failed, not
a spinner. The upload and the WAV are deleted on every path (audio is never
retained). A shutdown mid-job is not a failure: the row is left for the
start-up sweep. Cancellation wins over a non-empty queue, so queued jobs are
never started on shutdown; stop() deletes their audio and refuses new jobs.

Concurrency is 1 on purpose: two simultaneous long jobs on iguana were never
measured, and a Metal OOM there latches whisper (infra#273). The queue is
process-local, so ADR-018's single-replica assumption now has a second
dependent.

Mutation-checked, -race clean: cleanup, shutdown-vs-failure, timeout,
concurrency and stop-cleanup each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:54:52 +02:00
mathiasandClaude Sonnet 5.5 3cfc95640e test(uploads): a queued job at shutdown must never be started; timeout test fails fast
Mutation-testing the worker showed deleting stop()'s cleanup went unnoticed:
Process was incidentally deleting the audio of jobs it should never have
started. The stop test now asserts only the running job was converted (red on
the old code). The timeout test no longer hangs for the 10-minute go test
default when the timeout is removed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:54:52 +02:00
mathiasandClaude Sonnet 5.5 52417e6b47 test(uploads): in-process transcription worker (red)
Success path deletes both the upload and the WAV; failures are typed and still
delete the audio; a shutdown mid-job leaves the row for the start-up sweep
(it is not a transcription failure); a hung endpoint is failed by the job
timeout, never an eternal spinner; strictly one job at a time (two long jobs on
iguana were never measured); stopping deletes queued audio and refuses new jobs.

Red: uploads.Worker is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:47:22 +02:00
mathiasandClaude Sonnet 5.5 4352446f36 feat(audio): ffmpeg adapter -- always transcode to 16 kHz mono PCM (ADR-033)
Replaces whisper-server --convert. A zero exit that wrote nothing is an error.
Also fixes the red test's stub, which wrote an empty file and so (correctly)
tripped the no-output check.

Checked against real ffmpeg on iguana with the same argv: AAC .m4a in,
pcm_s16le 16000 Hz mono WAV out, duration preserved.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:46:34 +02:00
mathiasandClaude Sonnet 5.5 3bc33fc34c test(audio): ffmpeg transcode to 16 kHz mono PCM, always (red)
Red: audio.FFmpeg is undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:46:14 +02:00
mathiasandClaude Sonnet 5.5 2227cfff7e test(transcribe): opt-in live check against the real gateway
The httptest fake proves only what we THINK the gateway accepts. Skipped
unless TAPIR_LIVE_WAV and TAPIR_LIVE_GATEWAY_URL are set. Run once on
2026-10-01 with 11 min of synthetic Swedish (21 MB, above the 8-10 MB ingress
cap, via the NodePort): both iguana/kb-whisper (52.8 s) and berget/kb-whisper
(35.4 s) accepted the streamed, computed-Content-Length body and returned
usable verbose_json. Synthetic audio only; this repo is public.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:46:14 +02:00
mathiasandClaude Sonnet 5.5 cae423750c feat(transcribe): gateway client with an iguana->berget fallback chain (ADR-033)
Streams the WAV from disk with a computed Content-Length (no buffering, no
chunking), posts verbose_json with field `file` and language=sv, clamps the
final segment to the audio duration, and treats an empty transcript as a
failure. The chain falls back on ANY primary error, records the serving
endpoint, and never falls back once the context is cancelled.

gateway.go and chain.go are one commit: the chain's tests use the gateway's
StatusError and ErrEmptyTranscript, so neither compiles alone.

Mutation-checked: field name, clamp, cancel-fallback, content length and the
recorded endpoint each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:44:34 +02:00
mathiasandClaude Sonnet 5.5 bb285d8757 test(transcribe): gateway client and iguana->berget chain (red)
Gateway: field is `file`, no Expect header, a computed Content-Length (not
chunked, not buffered -- a 90 min WAV is ~170 MB), verbose_json normalised
with the last segment clamped to the audio duration, errors independent of
the body, an empty transcript is a failure. Chain: ANY primary failure falls
back and the row records the fallback endpoint; a cancelled job never falls
back to the cloud.

Red: package transcribe does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:43:51 +02:00
mathiasandClaude Sonnet 5.5 e12a5e7e89 feat(web): upload routes -- form, multipart intake, honest status (ADR-033)
GET/POST /uploads, GET /uploads/{id}, GET /uploads/{id}/status, mounted only
when an Uploader is wired. The multipart body is streamed to Intake, never
buffered. Rejections are synchronous: 422 (video, too long, unreadable) or 413
with the user-facing message and no row. Another user's id is a 404 that does
not leak the title. Unfinished states poll; done/failed render without
hx-trigger so nothing polls forever (ADR-025). A done transcript names the
endpoint that served it.

handlers.go, uploads.go and the templ pair are one commit: the App fields and
Store methods are what make the handlers compile.

Mutation-checked: dropping the 413 mapping, the not-found mapping, or letting
a done card poll each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:42:58 +02:00
mathiasandClaude Sonnet 5.5 1bc98813fc test(web): read the response body once; match the message's real case
Two defects in my red tests, found when going green: body() drains the
recorder so a second call read empty, and two assertions expected a different
case than the user-facing message. The assertions were not weakened.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:42:40 +02:00
mathiasandClaude Sonnet 5.5 e1ed6ea6f4 test(web): upload routes -- form, multipart POST, rejections, owner-only, honest status (red)
Real Router + real Intake + real Postgres; only ffprobe and the worker are
faked. Covers: the form states the limits up front; accepted audio queues a
row and redirects; video / too-long / too-large are rejected synchronously
(422/422/413) with a message and NO row; a non-owner gets 404 without the
title leaking; unfinished states keep polling and terminal states stop (no
stuck spinner); a done transcript names the endpoint that served it.

Red: App.Uploads / App.UploadLimits are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:41:42 +02:00
mathiasandClaude Sonnet 5.5 f84389028d feat(audio): ffprobe adapter implementing uploads.Prober (ADR-033)
Cover art is not video; non-zero exit is the file's fault (ErrUnreadable),
a missing binary / cancelled ctx / bad output is ours. WaitDelay bounds a
cancelled probe whose child holds the pipe open.

Checked against real ffprobe on iguana: JSON shape matches the fixtures and
/etc/hosts exits 1 with 'Invalid data found when processing input'. The unit
tests drive a stub process; ffprobe is not installed on koala.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:40:51 +02:00
mathiasandClaude Sonnet 5.5 0ab067c510 test(audio): ffprobe adapter -- cover art is not video, exit code vs missing binary (red)
Red: package audio does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:40:29 +02:00
mathiasandClaude Sonnet 5.5 1910d3dfb5 feat(uploads): Intake -- spool, probe, validate, queue; clean up on every failure (ADR-033)
Bounded read (cap+1 byte tells 'at the limit' from 'over'); a file that fails
validation creates no row; a prober fault is our error, not the user's bad
file; a row whose job cannot be submitted is failed, never left queued.

Mutation-checked: cap off-by-one, missing cleanup and deleting the handed-over
file each turn a test red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:40:13 +02:00
mathiasandClaude Sonnet 5.5 b98e9df462 test(uploads): Accept -- size cap, probe verdicts, cleanup on every failure path (red)
Red: uploads.Job, Intake and ErrUnreadable are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:40:03 +02:00
mathiasandClaude Sonnet 5.5 e3e1f17c40 feat(uploads): validation policy for probed audio uploads (ADR-033)
Video is checked first so it is always refused as video; limits and reasons
are typed so callers and tests branch on Reason, not message text.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:39:51 +02:00
mathiasandClaude Sonnet 5.5 f42b8e23fc test(uploads): validation policy -- video, no-audio, too-long, unreadable (red)
Probe-based, not extension-based: a Teams voice-only export is typically an
.mp4 with no video stream, indistinguishable from video by name (ADR-033).
Red: package uploads does not exist.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:39:20 +02:00
mathiasandClaude Sonnet 5.5 836a501576 feat(store): Store API for private uploaded transcripts (ADR-033)
Create/Start/Complete/Fail/Get/List plus the start-up sweep that fails
unfinished rows as interrupted (audio is not retained, so no resume). Every
call goes through withUser AND names user_id in its predicate, so isolation
does not rest on RLS alone and stays testable under the superuser test
connection; a non-owner gets ErrUploadNotFound, never a distinct forbidden.
List drops transcript text.

Mutation-checked: removing the user predicate from Get or from the UPDATEs
turns the cross-user tests red.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:38:25 +02:00
mathiasandClaude Sonnet 5.5 c29c0a7127 feat(domain): types for private uploaded transcripts (ADR-033)
Segment, Transcription, UploadStatus, UploadErrorKind and UploadedTranscript.
Error kinds are typed, never a provider's string. UploadedTranscript is a
distinct type from the shared public-caption Transcript on purpose.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:37:39 +02:00
mathiasandClaude Sonnet 5.5 4b557209c8 test(store): Store API for uploaded transcripts (red)
Lifecycle queued -> transcribing -> done|failed, per-row serving endpoint,
segments JSONB round trip, owner-only get/write (not-found, not forbidden),
list without transcript text, and the start-up sweep that fails unfinished
rows as `interrupted` for every user (audio is not retained, so no resume).

Red: CreateUpload and friends are undefined.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:37:30 +02:00
mathiasandClaude Sonnet 5.5 e36c4c1e0c feat(store): migration 017 -- private RLS-scoped uploaded_transcripts (ADR-033)
Separate table, not a user_id on the shared transcripts (ADR-021). ENABLE +
FORCE ROW LEVEL SECURITY with the migration-003 policy, user_id NOT NULL
ON DELETE CASCADE, typed error_kind, per-row serving endpoint, JSONB segments.
CHECK constraints: done => content, failed => error_kind.

Up and down are one commit: golang-migrate pairs them.

Mutation-checked: dropping FORCE, the policy, or the cascade each turn a test
red. FORCE is only observable structurally -- the app role is not the table
owner, so no behavioural test can see it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:36:50 +02:00
mathiasandClaude Sonnet 5.5 066ae6966a test(store): isolation tests for the private uploaded_transcripts table (red)
Drives RLS directly as the non-superuser app role: deny-all with no scope,
own-rows-only scoped reads, zero-row cross-user UPDATE/DELETE, rejected
cross-user INSERT, FORCE + NOT NULL user_id, cascade on DeleteUser. Also pins
the EXACT set of public tables with row security off, so a new table added
without RLS fails CI rather than passing it (ADR-033). The existing test only
named transcripts; #33's "the only non-RLS table" is false -- invitations,
user_identities and schema_migrations are off too.

Red: relation "uploaded_transcripts" does not exist. Set test passes today
and guards the migration that follows.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:36:22 +02:00
mathiasandClaude Sonnet 5.5 38b3f1cf33 docs(adr): ADR-033 upload audio -> private Swedish transcript -> chat (#33)
Proposed, decider pending. Supersedes ADR-007's STT deferral for
user-supplied audio only; the YouTube audio-download rejection stands.
Settles #28 decision 1 as a separate RLS-scoped uploaded_transcripts table
and records the measured length ceiling, the ingress 413, and the 18k-char
chat truncation that #33 assumed did not exist. Two DECISION NEEDED items
(ffmpeg in the runtime image; berget fallback for private audio).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:35:04 +02:00
mathiasandClaude Sonnet 5.5 feb108def3 docs(research): measure the audio length ceiling before designing chunking (#33)
No transcription ceiling found up to 90 min: one 174 MB request through the
LiteLLM NodePort transcribed in 440 s. The first real failure is the public
gateway ingress rejecting bodies over ~8-10 MB with 413, not whisper or a
Metal OOM. Also records that srt arrives wrapped in a JSON envelope, that the
two backends differ in timestamp granularity, and that the existing chat path
truncates at 18k chars (~22 min of speech).

Synthetic Swedish TTS fixture only; no real transcript.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-01 16:33:34 +02:00
mathias 4bafd5f082 fix(web): views.templ imports the pre-rename gitea.d-ma.be host
CI / Build & Import (push) Successful in 11s
CI / Deploy via GitOps (push) Successful in 4s
CI / Lint / Test / Vet (push) Successful in 19s
internal/web/views.templ:7 still imports
  gitea.d-ma.be/mathias/tapir/internal/adapters/store
while go.mod declares
  git.d-ma.be/mathias/tapir

Left behind by the gitea rename (infra ADR-0004). The old hostname still
resolves in DNS but no longer serves — HTTPS to it fails outright — so the
import is genuinely broken, not merely outdated.

Why nothing caught it: the COMMITTED internal/web/views_templ.go is correct,
so builds and CI use a good generated file and stay green. The rot is only in
the source. It surfaces the moment anyone runs `templ generate` — which
regenerates the broken import over the good output, and then the build fails
with

  no required module provides package gitea.d-ma.be/mathias/tapir/internal/adapters/store

Found exactly that way while evaluating this repo for the estate-wide lint
config: regenerating templ as part of the check turned a green repo red, and
the cause was the source, not the check.

Source fixed; regenerating now reproduces the committed generated file
byte-for-byte, so views_templ.go is unchanged here. Verified: go build and
go vet pass.
2026-09-05 14:44:39 +02:00
mathias e3e78e042b style: apply gofumpt
CI / Build & Import (push) Successful in 10s
CI / Lint / Test / Vet (push) Successful in 21s
CI / Deploy via GitOps (push) Successful in 4s
Estate-wide consolidation pass. The canonical .golangci.yml
(infra ci-templates/golangci.yml, infra#309 pattern) enables the gofumpt
formatter; golangci-lint v2 default set enables NO formatter, which is why
this drift accumulated silently -- the same hole that let two files rot in
dma-cli until they were found by hand (dma-cli#12).

Mechanical and deterministic: `gofumpt -w .`, nothing hand-edited. Verified
per repo before commit -- gofumpt -l now reports 0, gofmt -l reports 0, and
go build + go vet both pass.

Clears this repo to adopt the canonical lint config in a later wave.
2026-09-05 12:39:31 +02:00
mathiasandClaude Opus 5 6f1e0be09b ci(deploy): refuse to patch a manifest with an empty IMAGE_TAG (infra#311)
CI / Lint / Test / Vet (push) Successful in 44s
CI / Build & Import (push) Successful in 9s
CI / Deploy via GitOps (push) Successful in 5s
On 2026-08-25 a deploy job ran with an empty IMAGE_TAG, wrote
`image: localhost:5000/gitea-mcp:` into the infra repo and pushed it.
That is invalid YAML, so Flux failed the ENTIRE apps Kustomization --
every app stopped reconciling for about an hour, including three
unrelated deploys queued behind it.

This job's existing check could not catch it. It greps for
`image: <repo>:${IMAGE_TAG}` after the sed, and with IMAGE_TAG empty that
pattern matches the corrupted line itself, so the verification passes
because of the bug rather than despite it.

Asserting the input up front is the fix: an empty tag now fails the job
before anything is written, let alone committed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011PL5XcG68SvQqz3vGVbWbf
2026-08-25 17:25:24 +02:00
mathiasandClaude Opus 5 d45348ad2e ci: call the shared build-image script instead of an inline copy (infra#309)
CI / Lint / Test / Vet (push) Successful in 27s
CI / Build & Import (push) Successful in 11s
CI / Deploy via GitOps (push) Successful in 5s
The buildctl/skopeo step here was one of nineteen near-identical copies
across the estate. That duplication is why moving them off rootless
buildah (infra#241) took eleven commits and then had to be rediscovered
in both repo templates and the gitea-ci skill.

The implementation now lives in .gitea/build-image.sh, byte-identical to
mathias/infra's ci-templates/build-image.sh and held there by a drift
gate. What used to be textual differences between copies -- labels, tags,
build-args, build target -- are environment inputs.

Behaviour is unchanged: same registry, same tags, same labels, same
buildctl-against-host-buildkitd path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011PL5XcG68SvQqz3vGVbWbf
2026-08-25 11:29:55 +02:00
mathiasandClaude Opus 5 1c2ad2990f ci: build via buildctl instead of rootless buildah (infra#241)
CI / Build & Import (push) Successful in 8s
CI / Deploy via GitOps (push) Successful in 1s
CI / Lint / Test / Vet (push) Successful in 34s
Main has been CI-red since 2026-08-13. The build job dies extracting a base
image layer inside the gitea-runner pod:

  ApplyLayer ... potentially insufficient UIDs or GIDs available in user
  namespace (requested 0:42 for /etc/gshadow)

Root cause is the runner pod's securityContext, not this repo: it drops ALL
capabilities and sets allowPrivilegeEscalation:false, so newuidmap/newgidmap
can't gain CAP_SETUID/CAP_SETGID and rootless buildah falls back to a single
non-remapped uid/gid mapping. Adding those capabilities was tried live in the
infra repo and reverted as a confirmed no-op (no_new_privs blocks the
setuid-exec gain regardless of the container's capability set).

buildctl talks to koala's host buildkitd over /run/buildkit/buildkitd.sock,
which the runner pod already mounts, so the privileged layer work happens in
buildkitd on the host and the pod's hardened posture is untouched. Same
migration agentsquad (infra#233) and segment-embedder (infra#243) already made.

Drops the buildah smoke test with it: `buildah from ${REF}` would pull the
image back into the pod's buildah store and hit the very wall this migration
avoids, failing the job for a reason unrelated to the image. The deploy job's
`Verify rollout` is the stronger check — it requires the real pod to reach
Ready in-cluster, rather than the entrypoint printing its own name for 5s.
Tag builds lose their only image-level check (deploy runs on main pushes only);
accepted, since a tag is always cut from a main commit that already deployed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011PL5XcG68SvQqz3vGVbWbf
2026-08-17 12:07:51 +02:00
mathiasandClaude Opus 5 489b4bb2a1 chore(spike): recover the /bygge toolchain from tmpfs (#28)
CI / Lint / Test / Vet (push) Successful in 37s
CI / Build & Import (push) Failing after 5s
CI / Deploy via GitOps (push) Skipped
analyze_srt.py, build_page.py and transcode.yaml produced the /bygge
prototype. They were sitting in a session scratchpad on tmpfs, one reboot
from gone, while spike issues #29-#31 were written as if they had to be
built from scratch.

Committed as found, defects documented rather than fixed: max_tokens=6000
truncates anything past ~6 minutes of audio, the page title is hardcoded,
and the schema still carries fields no consumer renders.

The four-check validator is the part worth keeping — the coverage-gap and
Swedish-number-grounding checks catch errors that timestamp and citation
checks structurally cannot.

Real transcripts and analyses stay out: this repo is public and the
recordings are a named person discussing a client's project. Only the
synthetic fixture is committed, which exercises all four checks with no
model call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
2026-08-13 11:37:03 +02:00
mathias 21e7d6c74b fix(ci): smoke test hung 14min — buildah run doesn't die under plain timeout
CI / Lint / Test / Vet (push) Successful in 15s
CI / Build & Import (push) Successful in 20s
CI / Deploy via GitOps (push) Successful in 5s
Bare `/tapir` runs the long-running server, same as the old ctr-based smoke
test. ctr's --rm reliably force-killed it; a plain `timeout N buildah run`
does not — it only signals the wrapper, and the container process can
survive that and keep the log pipe open, hanging the whole job (observed
live: run 155, 14min before failure). Backgrounds the run and tears it down
with `buildah rm -f`, which forcibly kills regardless of wrapper state, and
captures output via a file instead of a blocking pipe.

Refs infra#132.
2026-07-26 06:17:18 +00:00
mathias 8814ba6673 ci(smoke): use buildah run instead of sudo k3s ctr for smoke test
CI / Lint / Test / Vet (push) Successful in 12s
CI / Build & Import (push) Failing after 13m42s
CI / Deploy via GitOps (push) Has been skipped
Same fix as cobalt-dingo — removes the sudo/host-containerd dependency so
this still works once the act_runner is containerized (infra#132).

Refs infra#132.
2026-07-26 05:57:37 +00:00
mathias 4b557a4325 ci(deploy): add Flux GitOps deploy job, mirroring cobalt-dingo's pattern
CI / Lint / Test / Vet (push) Successful in 27s
CI / Build & Import (push) Successful in 18s
CI / Deploy via GitOps (push) Successful in 5s
Fixes silent stale-deploy gap: CI built+pushed images but nothing bumped
k3s/apps/tapir/deployment.yaml, so merged features sat CI-green with zero
production effect (infra#111, infra#168). Flux native image-automation
can't scan localhost:5000 from inside k3s pods, so this patches the infra
repo directly via the existing INFRA_DEPLOY_KEY org secret (same key
cobalt-dingo and brain-gardener already use) on every push to main.

Refs infra#111.
2026-07-25 21:21:02 +00:00
mathiasandClaude Opus 4.8 72cb25111f test(store): address migrations by version, not step count (#8)
CI / Lint / Test / Vet (push) Successful in 12s
CI / Build & Import (push) Successful in 12s
The up/down migration tests stepped a hard-coded number of Steps(-N)/Steps(+N)
down from HEAD and back. The counts assumed a specific latest migration, so
adding one shifted every count by one and unrelated tests (010/011/014) went
red with confusing off-by-one symptoms — a papercut on every new migration.

Drive the schema to an exact version with m.Migrate(version) via two helpers
(headVersion, migrateTo). Each test now steps to just below its target by
version, asserts the down effect, steps up to the target, asserts the up
effect, then restores to the captured HEAD. A migration added on top changes
HEAD but shifts no count, so no test needs editing.

Verified by adding a throwaway migration 017 on top: all four tests stayed
green with zero edits.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015QbdxXWxLefS5AwLN5eyze
2026-07-02 14:49:02 +02:00
mathiasandClaude Sonnet 4.6 38f222c931 chore: rename Go module path gitea.d-ma.be → git.d-ma.be
CI / Lint / Test / Vet (push) Successful in 11s
CI / Build & Import (push) Successful in 12s
Infra ADR-0004 renamed the Gitea host. Bulk replace across go.mod and
all .go import paths. Build and tests pass unchanged.

Closes #20

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt6aHEDWRjkK14Voi6HnGh
2026-07-02 14:37:33 +02:00
mathias e042b6d26d chore: add .mcp.json (brain + gitea) — wire tapir for the single-harness workflow
CI / Lint / Test / Vet (push) Successful in 16s
CI / Build & Import (push) Successful in 12s
tapir had no MCP config at all — a hyperguild session here couldn't reach brain
or Gitea, breaking the report-back path before it could start. Matches the
shape now standard post-consolidation (hyperguild#75/#76, infra#178 template
refresh). Prerequisite for tapir#20 being run as the first real workflow test.
2026-07-02 07:00:53 +00:00
mathias c85a32e770 Merge pull request 'LANGUAGE.md: fix LiteLLM gateway never-say (stale endpoints, not piguard)' (#21) from language-md-litellm-fix into main
CI / Lint / Test / Vet (push) Successful in 12s
CI / Build & Import (push) Successful in 10s
2026-06-16 15:45:45 +00:00
mathias b246c0e688 fix(language): LiteLLM gateway never-say — stale endpoints, not piguard
CI / Lint / Test / Vet (pull_request) Successful in 12s
CI / Build & Import (pull_request) Has been skipped
piguard is legitimately in the request path as the reverse proxy; only the
stale endpoint forms (piguard:4000, koala:4000) are wrong. Per brain#6 review
amendment #3 / tapir#18 follow-on.
2026-06-16 15:35:31 +00:00
mathias c7896cb3cc Merge pull request 'LANGUAGE.md pilot: hand-written vocabulary + caveman rubric (Phase 1)' (#19) from language-md-pilot into main
CI / Lint / Test / Vet (push) Successful in 12s
CI / Build & Import (push) Successful in 10s
2026-06-16 15:22:14 +00:00
mathiasandClaude Opus 4.8 b79fb892c8 docs(claude): point to LANGUAGE.md + caveman rubric
CI / Lint / Test / Vet (pull_request) Successful in 13s
CI / Build & Import (pull_request) Has been skipped
Wires the Phase 1 vocabulary pilot into the canonical agent instructions
(tapir#18). CLAUDE.md is canonical here (no .context/ source, no generation
header).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 22:54:57 +02:00
mathias c315e3e003 feat(language): add hand-written vocabulary + caveman rubric pilot
Phase 1 of the ubiquitous-language system (tapir#18). Tapir-specific terms
verified against domain/ports code + README/VISION; homelab-core terms (brain
sink, LiteLLM gateway) derived from brain#5 glossary. Tooling-free pilot.
2026-06-12 20:52:23 +00:00
mathiasandClaude Opus 4.8 64e3368f5f feat(web): charm-reader visual refresh with light/dark theme toggle (ADR-032, #17)
CI / Lint / Test / Vet (push) Successful in 11s
CI / Build & Import (push) Successful in 10s
The UI read flat and boring. Reskin to one charm/TUI-inspired layout in two
palettes (CSS custom properties): a warm "reader" light theme (sketch B) and a
"cozy terminal" dark theme (sketch C).

- Palette chosen in cascade order: :root light default; an OS-preference dark
  block scoped to :root:not([data-theme]) so it applies only absent an explicit
  choice; and :root[data-theme="dark"|"light"] set by a header toggle that
  outranks the media query by specificity and persists in localStorage (guarded,
  degrades to OS default). A <head> init script applies the stored choice before
  paint, so no flash of the wrong palette.
- Charm touches via existing classes (no templ structure churn): monospace meta
  lines, accent uppercase section dividers with a trailing rule, pill buttons, a
  lifted/accent-edged expanded card.
- Error/danger shades become --err-* tokens so they follow the theme, replacing
  three per-block prefers-color-scheme dark overrides.
- Theme toggle wired into Layout and PublicLayout headers.

BDD: docs/use-cases/visual_theme.feature un-pended, mapped in scenarioCoverage.
TDD: internal/web/visual_theme_test.go (palettes, OS default, persisted toggle,
expanded-card embed). Verified light+dark on list/reader/welcome via web-shot.
Sketches kept as the design record. ui-spec.md as-built row added.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 12:54:06 +02:00
mathias b01f8d5783 docs(bdd): visual_theme.feature scenarios (@pending until TDD, #17) 2026-06-12 12:29:31 +02:00
mathias 8ca767c144 docs(adr): ADR-032 visual refresh — light(B)+dark(C) charm-reader themes + sketches (issue #17) 2026-06-12 11:54:54 +02:00
mathiasandClaude Opus 4.8 f98b640531 feat(web): inline-expand summary + Q&A in the list (ADR-031, #16)
CI / Lint / Test / Vet (push) Successful in 11s
CI / Build & Import (push) Successful in 11s
Click a summarized card → its full summary (summaryBody) + chat dock (chatReveal)
expand in place via HTMX (GET /v/{id}/expand → expandedCard), collapse back via
GET /v/{id}/card → compact VideoCard. Same <li id>, outerHTML swap — the existing
list-fragment pattern. The card title carries href=/v/{id} as the no-JS fallback
(detail page stays for no-JS + deep links); only summarized cards expand. Reuses
summaryBody + chatReveal so the expanded card never drifts from the detail page.

BDD: inline_expand.feature un-pended + mapped. TDD: 6 handler/fragment tests
(expand/collapse fragments, chat dock, summarized-only, no-JS href, shared body).
Minimal CSS only — the TUI/charm restyle is #17.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 10:47:47 +02:00
mathias 607a8cbe8d docs(bdd): inline_expand.feature scenarios (@pending until TDD, #16) 2026-06-12 10:38:40 +02:00
mathias 54d60e53b9 docs(adr): ADR-031 inline-expand summary + Q&A in the list (issue #16) 2026-06-12 10:37:55 +02:00
mathias 9298e0c686 docs(homelab): TAPIR_METRICS_ADDR + key metric series (ADR-030, #15)
CI / Lint / Test / Vet (push) Successful in 10s
CI / Build & Import (push) Successful in 10s
2026-06-12 08:50:36 +02:00
mathiasandClaude Opus 4.8 cd461b95f8 feat(observability): instrument AI + HTTP paths, serve /metrics on a side port (ADR-030, #15)
CI / Build & Import (push) Successful in 11s
CI / Lint / Test / Vet (push) Successful in 10s
Wire the metrics package into the live paths and serve it:
- summarizer: per-endpoint latency by model/outcome(success|error|parse_error)/fallback + slog.
- youtube.FetchTranscript: latency by outcome (captions|none|rate_limited) + slog.
- chat: answer latency by model + slog.
- llm usage hook → token counts (prompt|completion) per model, wired in buildSummarizer/buildChat.
- oidc callback: login counter.
- cmdServe: wrap Router in metrics.HTTPMiddleware (request count + latency by bounded
  route pattern) and serve /metrics on TAPIR_METRICS_ADDR (default :9090), a SEPARATE
  port — never on the public app mux.

BDD: observability.feature scenarios un-pended + mapped. TDD: summarizer wiring tested
black-box via the /metrics scrape; metrics-not-on-public-mux asserted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 08:49:51 +02:00
mathiasandClaude Opus 4.8 9c2a04406b feat(llm): usage hook to surface token counts (ADR-030, #15)
WithUsageHook callback fires with model + prompt/completion tokens parsed from the
response usage block. Keeps the copied stdlib-only llm package decoupled from
metrics (ADR-004) — the caller wires it to internal/metrics. TDD covered.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 08:36:02 +02:00
mathiasandClaude Opus 4.8 a5a8cf6f6d feat(metrics): Prometheus collectors + typed API + HTTP middleware (ADR-030, #15)
New internal/metrics package: AI metrics (summarize/caption/chat latency, LLM
tokens), session metrics (requests+latency by bounded route pattern, logins), and
a /metrics Handler. Adapters call a typed API; never touch prometheus types.
Dep: github.com/prometheus/client_golang (standard Go client; cluster runs
prometheus-operator). TDD: collectors + middleware route-pattern cardinality
covered.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 08:32:41 +02:00
mathias 64d11af9ef docs(bdd): observability.feature scenarios (@pending until TDD, #15) 2026-06-12 08:28:46 +02:00
mathias b590d2708d docs(adr): ADR-030 observability — slog + Prometheus metrics (issue #15) 2026-06-12 08:27:56 +02:00
129 changed files with 9003 additions and 857 deletions
+121
View File
@@ -0,0 +1,121 @@
#!/usr/bin/env bash
# Build an OCI image with buildctl and push it to the local registry.
#
# CANONICAL COPY: mathias/infra ci-templates/build-image.sh (infra#309).
# Every repo carries a byte-identical copy at .gitea/build-image.sh, written by
# `task ci:sync` and held in place by the drift gate. DO NOT EDIT THE COPIES --
# edit here, re-sync, and the gate will tell you which repos still disagree.
#
# Why this exists: this step was hand-copied into a dozen repos, so migrating it
# off rootless buildah (infra#241) meant eleven near-identical commits, eleven
# verifications, and three separate rediscoveries of the same fix in the two
# repo templates and the gitea-ci skill. The differences between the copies were
# incidental -- a label here, a build-arg there -- never intentional. Those are
# inputs now, not forks of the implementation.
#
# Why buildctl and not buildah: the gitea-runner pod runs with
# `capabilities: drop [ALL]` and `allowPrivilegeEscalation: false`, which
# unconditionally blocks the setuid-root exec that rootless buildah's uid/gid
# remapping needs, and the pod has no OCI runtime at all. buildctl is a thin
# client -- it hands the build to the host's buildkitd over a socket, so the
# privileged layer work happens on the host and this pod's securityContext is
# not in the picture. buildah is absent from the runner image as of baked-v14
# and its absence is asserted at pod start (infra#308).
#
# Inputs, all via environment:
#
# IMAGE (required) image name, e.g. cobalt-dingo
# REGISTRY default localhost:5000
# SHA_TAG default `git rev-parse --short HEAD`
# VERSION_TAG optional extra tag, e.g. v1.2.3 on a tag build
# PUSH_LATEST default 1; set 0 to skip the :latest tag
# CONTEXT default . -- build context directory
# DOCKERFILE_DIR default $CONTEXT -- directory holding the Dockerfile
# DOCKERFILE_NAME optional, when the file is not named `Dockerfile`
# TARGET optional, multi-stage build target (--opt target=)
# BUILD_ARGS optional, space-separated K=V pairs
# BUILD_SECRETS optional, space-separated buildkit secret specs, each
# `id=NAME,src=PATH`. Pass the SPEC only -- the caller
# writes the secret to PATH itself. A secret VALUE must
# never be passed here: it would land in this script's
# argv and therefore in the CI log.
# EXTRA_LABELS optional, space-separated K=V pairs
# GIT_SHA optional, stamped as org.opencontainers.image.revision
# IMAGE_SOURCE optional, stamped as org.opencontainers.image.source
#
# Note on shell style below: every conditional is a full `if` block, never
# `[ -n "$X" ] && cmd`. Under `set -e` the && form exits the script whenever the
# test is false, which is a real bug this estate has already shipped once.
set -euo pipefail
if [ -z "${IMAGE:-}" ]; then
echo "build-image: IMAGE is required" >&2
exit 2
fi
REGISTRY="${REGISTRY:-localhost:5000}"
BUILDKIT_HOST="${BUILDKIT_HOST:-unix:///run/buildkit/buildkitd.sock}"
CONTEXT="${CONTEXT:-.}"
DOCKERFILE_DIR="${DOCKERFILE_DIR:-$CONTEXT}"
PUSH_LATEST="${PUSH_LATEST:-1}"
VERSION_TAG="${VERSION_TAG:-}"
if [ -z "${SHA_TAG:-}" ]; then
SHA_TAG="$(git rev-parse --short HEAD)"
fi
REF="${REGISTRY}/${IMAGE}:${SHA_TAG}"
TAR_FILE="$(mktemp)"
trap 'rm -f "$TAR_FILE"' EXIT
args=(
--frontend dockerfile.v0
--local "context=${CONTEXT}"
--local "dockerfile=${DOCKERFILE_DIR}"
)
if [ -n "${DOCKERFILE_NAME:-}" ]; then
args+=(--opt "filename=${DOCKERFILE_NAME}")
fi
if [ -n "${TARGET:-}" ]; then
args+=(--opt "target=${TARGET}")
fi
for spec in ${BUILD_SECRETS:-}; do
# Opaque id=NAME,src=PATH -- buildkit reads the file, nothing is echoed.
args+=(--secret "${spec}")
done
if [ -n "${GIT_SHA:-}" ]; then
args+=(--opt "label:org.opencontainers.image.revision=${GIT_SHA}")
fi
if [ -n "${IMAGE_SOURCE:-}" ]; then
args+=(--opt "label:org.opencontainers.image.source=${IMAGE_SOURCE}")
fi
for kv in ${BUILD_ARGS:-}; do
args+=(--opt "build-arg:${kv}")
done
for kv in ${EXTRA_LABELS:-}; do
args+=(--opt "label:${kv}")
done
echo "build-image: building ${REF}"
buildctl --addr "$BUILDKIT_HOST" build "${args[@]}" --output "type=oci,dest=${TAR_FILE}"
# buildctl has no registry-push equivalent to `buildah push`, so skopeo copies
# the OCI archive up -- once per tag, from the same archive, so every tag points
# at bit-identical content.
push_tag() {
echo "build-image: pushing ${1}"
skopeo copy --dest-tls-verify=false "oci-archive:${TAR_FILE}" "docker://${1}"
}
push_tag "$REF"
if [ "$PUSH_LATEST" = "1" ]; then
push_tag "${REGISTRY}/${IMAGE}:latest"
fi
if [ -n "$VERSION_TAG" ]; then
push_tag "${REGISTRY}/${IMAGE}:${VERSION_TAG}"
fi
echo "✓ build-image: ${REF} pushed"
+157 -27
View File
@@ -61,33 +61,163 @@ jobs:
echo "version-tag=${REF#refs/tags/}" >> "$GITHUB_OUTPUT" echo "version-tag=${REF#refs/tags/}" >> "$GITHUB_OUTPUT"
fi fi
# infra#241: rootless buildah inside the gitea-runner pod can't remap the
# UID/GID range newuidmap/newgidmap need (the pod drops ALL capabilities
# and sets allowPrivilegeEscalation:false — both deliberate, it also holds
# CI secrets), so extracting a base-image layer that chowns outside the
# single available mapping fails with `ApplyLayer ... requested 0:42 for
# /etc/gshadow`. Main has been red on exactly this since 2026-08-13.
# buildctl instead talks to koala's host buildkitd over the socket the
# runner pod already mounts: the privileged extraction happens inside
# buildkitd on the host, never in this pod. Same migration agentsquad
# (infra#233) and segment-embedder (infra#243) already made — zero
# security tradeoff, unlike loosening the pod's capabilities.
#
# buildctl has no registry-push flag of its own: it writes an OCI archive
# and skopeo pushes that, which is also why every tag below is one copy of
# a single build rather than a rebuild.
- name: Build and push to local registry - name: Build and push to local registry
run: | # infra#309: implementation lives in .gitea/build-image.sh, a
REGISTRY="localhost:5000" # byte-identical copy of mathias/infra's ci-templates/build-image.sh.
REF="${REGISTRY}/${{ env.IMAGE }}:${{ steps.meta.outputs.sha-tag }}" # Do not edit it here -- edit the canonical file, run `task ci:sync`,
buildah build \ # and infra's drift gate names any repo still carrying an old copy.
--label "org.opencontainers.image.revision=${{ github.sha }}" \ env:
--label "org.opencontainers.image.source=${{ github.repositoryUrl }}" \ SHA_TAG: ${{ steps.meta.outputs.sha-tag }}
-t ${REF} \ VERSION_TAG: ${{ steps.meta.outputs.version-tag }}
-t ${REGISTRY}/${{ env.IMAGE }}:latest \ GIT_SHA: ${{ github.sha }}
. IMAGE_SOURCE: ${{ github.repositoryUrl }}
buildah push --tls-verify=false ${REF} run: bash .gitea/build-image.sh
buildah push --tls-verify=false ${REGISTRY}/${{ env.IMAGE }}:latest
[[ -n "${{ steps.meta.outputs.version-tag }}" ]] && \
buildah push --tls-verify=false \
${REGISTRY}/${{ env.IMAGE }}:${{ steps.meta.outputs.version-tag }} || true
echo "Image pushed to ${REF}"
- name: Smoke test # The buildah-based smoke test that used to sit here is gone with the
run: | # buildctl migration above: `buildah from ${REF}` would pull the image
REGISTRY="localhost:5000" # back into the pod's buildah store and hit the very uid/gid wall the
REF="${REGISTRY}/${{ env.IMAGE }}:${{ steps.meta.outputs.sha-tag }}" # migration exists to avoid — it would fail the job for a reason that has
CNAME="smoke-${{ steps.meta.outputs.sha-tag }}" # nothing to do with the image being good. The `Verify rollout` step in
sudo k3s ctr images pull --plain-http ${REF} # the deploy job below is the stronger check anyway: it requires the real
OUTPUT=$(timeout 5 sudo k3s ctr run --rm ${REF} ${CNAME} /tapir 2>&1 || true) # pod to reach Ready in-cluster, not just the entrypoint to print its own
sudo k3s ctr containers delete ${CNAME} 2>/dev/null || true # name for 5s. Tag builds lose their only image-level check (deploy is
echo "$OUTPUT" | grep -q "tapir" \ # main-push only) — accepted; a tag is always built from a main commit
&& echo "Smoke test passed" \ # that already deployed.
|| echo "Smoke test inconclusive: $OUTPUT"
# ── 3. Mirror to GitHub — skipped for now (SSH key rotation pending) ─ # ── 3. Deploy via infra repo + Flux ─────────────────────────────────────────
# Flux native image-automation can't scan localhost:5000 from inside k3s pods
# (mathias/infra k3s/flux/flux-system/image-automation.yaml) — this job
# mirrors cobalt-dingo's proven pattern instead: patch the infra repo's
# manifest directly on every push to main, then annotate Flux for a fast
# reconcile. Fixes infra#111 (image built+pushed but manifest bump was
# manual, so merged features silently didn't deploy).
deploy:
name: Deploy via GitOps
needs: build
runs-on: self-hosted
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
environment: staging
steps:
- name: Update image tag in infra repo
env:
IMAGE_TAG: ${{ needs.build.outputs.image-tag }}
DEPLOY_KEY: ${{ secrets.INFRA_DEPLOY_KEY }}
run: |
set -euo pipefail
# infra#311: fail on an empty tag BEFORE patching anything. The
# grep verification further down CANNOT catch it -- with IMAGE_TAG
# empty its pattern matches the very corruption it exists to reject.
# The result is `image: <repo>:` in the manifest, which is invalid
# YAML, so Flux fails the ENTIRE apps Kustomization and every app
# stops reconciling, not just this one. That happened on 2026-08-25.
[ -n "${IMAGE_TAG:-}" ] || { echo "IMAGE_TAG is empty - refusing to patch"; exit 1; }
# INFRA_DEPLOY_KEY is a Gitea org secret (mathias org), already
# configured per docs/cd-pipeline.md in the infra repo — same key
# cobalt-dingo and brain-gardener use, no new secret needed.
mkdir -p ~/.ssh
echo "$DEPLOY_KEY" > ~/.ssh/id_infra
chmod 600 ~/.ssh/id_infra
ssh-keyscan -p 30022 10.0.1.20 >> ~/.ssh/known_hosts 2>/dev/null
export GIT_SSH_COMMAND="ssh -i ~/.ssh/id_infra -o IdentitiesOnly=yes"
rm -rf /tmp/infra
git clone -b main ssh://git@10.0.1.20:30022/mathias/infra.git /tmp/infra
cd /tmp/infra
DEPLOYMENT="k3s/apps/tapir/deployment.yaml"
# In-place update of the image tag. sed (not yq) so we don't
# depend on additional tooling on the runner — same as cobalt-dingo.
sed -i "s|image: localhost:5000/tapir:.*|image: localhost:5000/tapir:${IMAGE_TAG}|" "$DEPLOYMENT"
# Verify the patch took effect.
grep -q "localhost:5000/tapir:${IMAGE_TAG}" "$DEPLOYMENT" \
|| { echo "✗ image tag patch failed"; exit 1; }
if git diff --quiet "$DEPLOYMENT"; then
echo "ℹ image tag unchanged — skipping push"
else
git -c user.name="tapir CI" \
-c user.email="ci@tapir.local" \
commit -m "chore(deploy): tapir → ${IMAGE_TAG}" "$DEPLOYMENT"
git push origin main
echo "✓ pushed to infra repo"
fi
shred -u ~/.ssh/id_infra
- name: Trigger Flux reconcile (immediate)
run: |
# Without these annotations, Flux would still pick up the change
# within 30s (the apps Kustomization interval). The annotations
# cut latency to ~1s.
kubectl -n flux-system annotate gitrepository flux-system \
reconcile.fluxcd.io/requestedAt="$(date +%s)" --overwrite
kubectl -n flux-system annotate kustomization apps \
reconcile.fluxcd.io/requestedAt="$(date +%s)" --overwrite
- name: Wait for Flux to apply new image
env:
IMAGE_TAG: ${{ needs.build.outputs.image-tag }}
run: |
# Poll the Deployment spec until it reflects the new tag.
# Bound to 60s so a stuck Flux doesn't hang CI.
EXPECTED="localhost:5000/tapir:${IMAGE_TAG}"
for i in $(seq 1 60); do
CURRENT=$(kubectl get deploy tapir -n tapir \
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || echo "")
if [ "$CURRENT" = "$EXPECTED" ]; then
echo "✓ Flux applied new image after ${i}s"
break
fi
sleep 1
done
# Final assertion (in case the loop exited without matching).
kubectl get deploy tapir -n tapir \
-o jsonpath='{.spec.template.spec.containers[0].image}' \
| grep -qx "$EXPECTED" \
|| { echo "✗ Flux did not apply new image within 60s"; exit 1; }
- name: Verify rollout
run: |
kubectl rollout status deployment/tapir \
--namespace tapir \
--timeout=120s \
|| {
echo "── pod status ──"
kubectl get pods -n tapir -o wide
echo "── events ──"
kubectl get events -n tapir --sort-by='.lastTimestamp' | tail -20
echo "── describe ──"
kubectl describe pods -n tapir -l app=tapir | tail -40
exit 1
}
- name: Confirm pod running new image
env:
IMAGE_TAG: ${{ needs.build.outputs.image-tag }}
run: |
kubectl get pods -n tapir \
-l app=tapir \
--field-selector=status.phase=Running \
-o jsonpath='{.items[*].spec.containers[0].image}' \
| grep -q "localhost:5000/tapir:${IMAGE_TAG}" \
&& echo "✓ pod running new image" \
|| { echo "✗ pod image mismatch"; exit 1; }
# ── 4. Mirror to GitHub — skipped for now (SSH key rotation pending) ─
+11
View File
@@ -11,3 +11,14 @@
.env.* .env.*
!.env.example !.env.example
*.local *.local
# Spike media (#28): real recordings, their transcripts and derived analyses are
# private third-party content and this repo is public. Only synthetic fixtures
# are committed — see scripts/spike-media/README.md.
/scripts/spike-media/*.mov
/scripts/spike-media/*.mp4
/scripts/spike-media/*.wav
/scripts/spike-media/*.srt
/scripts/spike-media/*.json
/scripts/spike-media/*.html
!/scripts/spike-media/fixtures/
+18
View File
@@ -0,0 +1,18 @@
{
"mcpServers": {
"brain": {
"type": "http",
"url": "https://brain-mcp.d-ma.be/mcp",
"headers": {
"Authorization": "Bearer ${BRAIN_MCP_TOKEN}"
}
},
"gitea": {
"type": "http",
"url": "https://git-mcp.d-ma.be/mcp",
"headers": {
"Authorization": "Bearer ${GITEA_MCP_TOKEN}"
}
}
}
}
+13 -3
View File
@@ -12,6 +12,7 @@ docs it indexes.
3. `DECISIONS.md` — the ADRs. Decisions are settled here; do not re-litigate without a new ADR. 3. `DECISIONS.md` — the ADRs. Decisions are settled here; do not re-litigate without a new ADR.
4. `docs/architecture/architecture.md`, `docs/data-model.md`, `docs/use-cases/*.feature`. 4. `docs/architecture/architecture.md`, `docs/data-model.md`, `docs/use-cases/*.feature`.
5. `docs/homelab-integration.md` — the concrete endpoints/conventions you'll need. 5. `docs/homelab-integration.md` — the concrete endpoints/conventions you'll need.
6. `LANGUAGE.md` — the project vocabulary. Apply the caveman rubric before destructive operations.
## How to work in this repo ## How to work in this repo
@@ -49,8 +50,11 @@ These caused real mistakes that were caught and corrected; the corrections are l
- **No global cross-tenant *video* table** — videos stay per-user (data-model). Transcripts ARE - **No global cross-tenant *video* table** — videos stay per-user (data-model). Transcripts ARE
shared since ADR-021 (public caption content, keyed by `(provider, provider_video_id)`, non-RLS) shared since ADR-021 (public caption content, keyed by `(provider, provider_video_id)`, non-RLS)
so re-analysis never re-fetches; the *videos* half of cross-tenant dedup stays a Future C concern. so re-analysis never re-fetches; the *videos* half of cross-tenant dedup stays a Future C concern.
- **No audio-download + speech-to-text in the core path** — captions-first (ADR-007). STT is a - **No YouTube audio-download + speech-to-text in the core path** — captions-first (ADR-007).
deferred, bounded optional component. That rejection stands. ADR-033 superseded the STT deferral **for user-supplied audio only**:
an authenticated user may upload a recording and get a *private* transcript (own RLS table,
never the shared `transcripts`). It is off by default (`TAPIR_UPLOADS_ENABLED`) and does not
open the door to scraping audio.
- **No public SaaS / sign-up / billing / Google OAuth verification at scale** — Future C, - **No public SaaS / sign-up / billing / Google OAuth verification at scale** — Future C,
deferred behind the Stage 0 gate (ADR-008). deferred behind the Stage 0 gate (ADR-008).
@@ -96,7 +100,7 @@ The repo is **green and shipping** — last tag `v0.15.0`. `task check` passes (
- Adapters present under `internal/adapters/`: `youtube` (captions-first `VideoSource`, - Adapters present under `internal/adapters/`: `youtube` (captions-first `VideoSource`,
timedtext/InnerTube acquisition per ADR-010), `summarizer` + `llm` (the copied AI router, timedtext/InnerTube acquisition per ADR-010), `summarizer` + `llm` (the copied AI router,
now a resilient endpoint chain — local primary → local fallback → external worst-case, now a resilient endpoint chain — local primary → local fallback → external worst-case,
parse-failure-aware, ADR-004 + ADR-022), `store` (Postgres, golang-migrate migrations 001–015), parse-failure-aware, ADR-004 + ADR-022), `store` (Postgres, golang-migrate migrations 001–017),
`secrets` (file-backed `SecretStore`). The brain HTTP sink (ADR-005) is the remaining `secrets` (file-backed `SecretStore`). The brain HTTP sink (ADR-005) is the remaining
optional sink. optional sink.
- Stage 1 is open (ADR-012): multi-user with **DB-enforced** isolation — Postgres RLS `FORCE`d - Stage 1 is open (ADR-012): multi-user with **DB-enforced** isolation — Postgres RLS `FORCE`d
@@ -108,6 +112,12 @@ The repo is **green and shipping** — last tag `v0.15.0`. `task check` passes (
(`TestTranscriptsTableIsSharedNotRLS`). The engine reads stored transcripts before any caption (`TestTranscriptsTableIsSharedNotRLS`). The engine reads stored transcripts before any caption
fetch (`usecase.resolveTranscript`), so re-analysis — re-summarize, paste-a-URL, onboarding fetch (`usecase.resolveTranscript`), so re-analysis — re-summarize, paste-a-URL, onboarding
burst — never re-touches YouTube. Per-user summaries/videos stay RLS-scoped. burst — never re-touches YouTube. Per-user summaries/videos stay RLS-scoped.
- Audio upload (ADR-033, migration 017): `internal/uploads` (validation by **probing**, intake,
one-at-a-time in-process worker), `internal/adapters/audio` (ffprobe/ffmpeg), and
`internal/adapters/transcribe` (gateway client + iguana→berget chain). Transcripts are private
in `uploaded_transcripts`; the audio is never retained; chat reuses `chat.Service`. The
isolation test pins the **exact set** of non-RLS tables (`TestRowSecurityOffTablesAreExactlyTheKnownSet`).
The runtime image does not yet contain ffmpeg — that is a pending decision recorded in ADR-033.
- `cmd/tapir` subcommands: `list`, `show`, `auth` (interactive host-side OAuth), `run` (batch - `cmd/tapir` subcommands: `list`, `show`, `auth` (interactive host-side OAuth), `run` (batch
watch→summarize), `serve` (the HTMX+Templ web reader/writer under `internal/web`, a new watch→summarize), `serve` (the HTMX+Templ web reader/writer under `internal/web`, a new
transport over the unchanged engine/ports — ADR-003). `tapir env` prints config. transport over the unchanged engine/ports — ADR-003). `tapir env` prints config.
+361 -1
View File
@@ -123,7 +123,9 @@ secret surface (see VISION Stage 2); it gets the existing, vetted secret path.
## ADR-007 — Captions-first; audio-download + speech-to-text deferred ## ADR-007 — Captions-first; audio-download + speech-to-text deferred
**Status:** Accepted (2026-06-02). Acquisition mechanism superseded by ADR-010 (Data API **Status:** Accepted (2026-06-02). Acquisition mechanism superseded by ADR-010 (Data API
`captions.download` → player/timedtext baseUrl); captions-first stance and STT deferral stand. `captions.download` → player/timedtext baseUrl); captions-first stance and STT deferral stand
**for YouTube**. STT deferral superseded **for user-supplied audio only** by ADR-033; the
YouTube audio-download rejection is unchanged.
**Context.** YouTube's Data API does not expose transcripts. Options: official captions **Context.** YouTube's Data API does not expose transcripts. Options: official captions
(clean, limited coverage) vs. audio download + local Whisper (broad coverage, ToS-grey, (clean, limited coverage) vs. audio download + local Whisper (broad coverage, ToS-grey,
@@ -1162,6 +1164,361 @@ re-login now rare (30-day idle or explicit logout), it matters far less.
--- ---
## ADR-030 — Observability: slog timing + Prometheus metrics (AI-focused)
**Status:** Proposed (2026-06-11). Issue #15. **Draft for review — no code yet.**
**Context / requirements.** Nothing measures the activities that drive Tapir's performance and
UX, and the Stage-0 eval gate (ADR-016) needs a *performance* dimension to sit beside the
return-usage one. We need timing for: caption fetches (the scarce op), summarization (which model
won, how long, fallbacks), Q&A latency, LLM token spend, and basic session/usage (request rate,
latency by route, logins). Requirements:
- R1: structured `slog` timing at each AI call site (human-readable, already the logging stack).
- R2: Prometheus metrics for the same, scrapeable by the cluster's prometheus-operator.
- R3: **AI metrics are the priority** — summarize latency by `model`/`outcome`/`fallback`,
caption-fetch latency by `outcome`, chat latency by `model`, and LLM `tokens` by model+kind.
- R4: HTTP/session metrics via middleware — request count + latency by route, logins.
- R5: bounded label cardinality (no per-user, no raw-path labels).
- R6: `/metrics` must NOT be publicly exposed.
**Decision / architecture.**
1. **New package `internal/metrics`** owns all Prometheus collectors + a typed API
(`ObserveSummarize`, `ObserveCaptionFetch`, `ObserveChat`, `RecordTokens`, `IncLogin`,
`HTTPMiddleware`, `Handler`). Adapters call this API; they never import prometheus types.
2. **New dependency `github.com/prometheus/client_golang`.** Justification: it is *the* standard
Go Prometheus client and the cluster already runs prometheus-operator; hand-rolling exposition
is not worth it. (Needs the dep-justification note in the commit per repo rules.)
3. **The copied `llm` package stays stdlib-only (ADR-004).** It must not import `internal/metrics`.
Token usage is surfaced via an **optional callback** `llm.WithUsageHook(func(model string, prompt, completion int))`
set at wiring time (`buildSummarizer`/`buildChat`) to `metrics.RecordTokens`; `llm.Client` only
gains parsing of the response `usage` block. Our own adapters (`summarizer`, `youtube`, `chat`)
may import `internal/metrics` directly.
4. **HTTP middleware** reads `r.Pattern` AFTER routing (Go 1.22 sets it during ServeMux match), so
the `route` label is the bounded registered pattern (`GET /v/{videoId}`), satisfying R5;
unmatched → `other`.
5. **Dedicated metrics port** (`TAPIR_METRICS_ADDR`, default `:9090`) served by a second
`http.Server` in `cmdServe`; `/metrics` is never on the public app mux (R6). A **PodMonitor**
in `mathias/infra` scrapes it; the deployment exposes the port.
6. **slog** elapsed fields are emitted alongside each metric at the call sites (R1).
**Hook points (where the instrumentation lands).**
- `summarizer.Summarize` — per-endpoint timing + outcome (`success`/`parse_error`/`error`) + fallback flag.
- `youtube.FetchTranscript` — fetch timing + outcome from `domain.Transcript.Source`.
- `chat.Service` answer — timing by model.
- `llm.Client.Complete` — parse `usage`, fire the usage hook.
- `oidc.handleCallback` — `IncLogin`.
- `cmdServe` — wrap `Router()` in `metrics.HTTPMiddleware`; start the metrics server.
**Out of scope / later.** Persisting per-summary latency into Postgres for `tapir report`
(derive UX latency — publish/discovery → summary — from existing timestamps first; only persist
op-latency if the scrape proves insufficient). SPA view (#16) and visual refresh (#17).
**Reversibility.** Additive: a new package + middleware + a metrics port. Removing the PodMonitor
stops scraping; the app is unaffected. No schema change.
**Next steps (gated):** on approval of this ADR → BDD scenarios (`docs/use-cases/observability.feature`
+ scenario-coverage map) → TDD → implement → SemVer + docs + PodMonitor.
---
## ADR-031 — SPA-like reader: inline-expand summary + Q&A in the list (HTMX, no framework)
**Status:** Proposed (2026-06-12). Issue #16. **Draft for review — no code yet.**
**Context / requirements.** The reader is multi-page: a list of compact cards (`/`), then a
navigation to a separate detail page (`/v/{id}`) for the full summary + the docked chat (ADR-027).
It feels less fluid than a single integrated view. We want the full summary AND the per-video
Q&A to open **in place in the list**, no page hop. Requirements:
- R1: clicking a summarized card expands it in place to the full summary (summary/highlights/
takeaways) + the chat dock; a collapse returns it to the compact card.
- R2: **no SPA framework** — stay HTMX + Templ (ADR-003); reuse existing fragments, not a rewrite.
- R3: **progressive enhancement** — with JS off, the card link still navigates to `/v/{id}`
(the detail page stays as the no-JS + deep-link surface). Nothing becomes JS-only.
- R4: only **summarized** cards expand; pending/rate-limited/no-caption cards keep their current
footer behaviour (Summarize button, waiting/none states).
- R5: chat inside an expanded card works exactly as on the detail page (reuse `chatReveal`/
`chatSection` + the existing `/v/{id}/chat` endpoints, unchanged).
**Decision / architecture.**
1. **Reuse the existing fragments.** `summaryBody(r)` and `chatReveal(videoID)` already exist and
render the detail page; a new `expandedCard(r, chatEnabled)` composes the compact header + a
collapse control + `summaryBody` + `chatReveal`. `DetailPage` is refactored to also compose
`summaryBody` so the two never drift (DRY).
2. **Two fragment endpoints** (mirroring the existing list/status HTMX fragment pattern):
`GET /v/{videoId}/expand` → `expandedCard`; collapse reuses the existing compact `VideoCard`
via `GET /v/{videoId}/card`. Both are list-card `<li>` fragments with the SAME `id`
(`video-{id}`), swapped `outerHTML` — same mechanism as `processingCard`/`VideoCard` today.
3. **The compact card's title/"Read" affordance** becomes `hx-get=/v/{id}/expand`,
`hx-target=#video-{id}`, `hx-swap=outerHTML`, with `href=/v/{id}` as the no-JS fallback (R3).
The expanded card's collapse control is the inverse (`hx-get=/v/{id}/card`).
4. **Only when `r.Summarized`** does the expand affordance render (R4); the other states are
unchanged.
5. **v1 does NOT push the URL** (`hx-push-url`) — expand/collapse is ephemeral list UI state; the
detail page remains the deep-link/shareable URL. Deep-linking the open state via `hx-push-url`
is noted as a later option (needs list-state restore on back).
**Out of scope / later.** URL push / deep-linkable open state; the visual refresh (#17) — though
the expanded-card markup is where #17's TUI/charm styling will land, so they pair.
**Reversibility.** Additive: two fragment endpoints + one templ + an affordance swap on the
compact card. Removing the affordance reverts to plain list→detail navigation; the detail page is
untouched. No schema change.
**Next steps (gated):** on approval → BDD (`docs/use-cases/inline_expand.feature` + coverage
map) → TDD → implement → SemVer + docs.
---
## ADR-032 — Visual refresh: one charm-reader layout, light + dark themes
**Status:** Proposed (2026-06-12). Issue #17. **Draft for review — no code yet.** Follows the
sketch-first explore step (3 throwaway mockups in `docs/sketches/`, screenshotted for review).
**Context / decision.** The UI is flat. From the mockups, directions **B (light reader + charm)**
and **C (dark cozy terminal)** are the SAME layout — readable sans body, monospace meta, charm
palette accents, lipgloss-style bordered cards — in two palettes. Direction A (full-monospace TUI)
is dropped as too heavy to read long summaries. Decision: ship that one layout with **both a light
theme (B) and a dark theme (C)**, user-toggleable, defaulting to the OS preference.
**Requirements.**
- R1: one set of markup/structure; the two themes are pure palette (CSS variables), no duplicate templates.
- R2: a **theme toggle** persisted across visits; default to `prefers-color-scheme` when no choice stored.
- R3: charm language in both — mint/purple/pink accents, mono meta + section labels, lipgloss
bordered/gradient cards, the (fixed) ASCII tapir; readable sans body.
- R4: style the existing pieces — list, compact card, **expanded card incl. the already-present
video embed (ADR-031/summaryBody)**, detail page, chat dock, the queue note, the charm spinner.
- R5: WCAG-AA contrast for body text in BOTH themes; keep `prefers-reduced-motion` (already honoured).
- R6: stay HTMX+Templ; no CSS framework.
**Architecture.**
1. **Palette as CSS variables.** `:root` holds the light (B) tokens; `:root[data-theme="dark"]`
holds the dark (C) tokens; a `prefers-color-scheme: dark` media block sets the dark tokens when
no explicit `data-theme` is set. All component CSS references variables only (R1). The existing
`CharmMint/Purple/Pink/Cream/Dim` Go consts remain the source for the spinner's inline colours.
2. **Theme toggle** = a small inline script (a dozen lines, no framework) in `Layout`: on load,
apply stored theme (localStorage) or fall through to the media query; a header toggle button
flips `data-theme` on `<html>` and stores it. This is the one new bit of JS; everything else
stays server-rendered + HTMX. (Considered: cookie + server-render — rejected, a full round-trip
per toggle is clunky for a pure presentation flip.)
3. **Scope = the `styleTag` CSS in `view.go`** (the single style source) plus tiny class hooks in
the templ where needed; content/structure are unchanged, so existing view tests keep passing.
**Verification.** The deployed UI is behind auth (web-shot can't log in), so visual review is via
the mockups now + a styled full-set mockup screenshot before merge, then a live eyeball on device.
Automated tests stay structural/behavioural (theme tokens present, toggle persists, expanded card
embeds the video, no-JS still renders a readable default) — colours are not unit-tested.
**Reversibility.** A CSS theme swap + one small script + a few class hooks; revert `styleTag` to
roll back. No schema, no structural change.
**Next steps (gated):** on approval → BDD (`docs/use-cases/visual_theme.feature` + coverage map)
→ TDD → implement → SemVer + docs.
---
## ADR-033 — Upload audio, get a private Swedish transcript, discuss it (user-supplied STT)
**Status:** Proposed (2026-10-01). **Decider: Mathias — pending.** Issue #33 (audio-only subset of
#28). **Supersedes ADR-007's speech-to-text deferral for user-supplied audio only.** The
YouTube audio-download rejection in ADR-007 **stands untouched**. Builds on ADR-012 (isolation),
ADR-018 (single replica), ADR-021 (**the constraint**), ADR-022 (endpoint chain), ADR-025
(honest status), ADR-027 (chat). ADR-014 item 4 ("Whisper reconsideration is a future ADR,
gated on caption data") is a different question — whether to transcribe YouTube videos with no
captions — and this ADR does not answer it. Measurements: `docs/research/audio-length-ceiling-2026-10-01.md`.
**Context.** The maintainer has a concrete need: a Swedish Teams voice-only export, 30–90
minutes, to upload and then *discuss* (what was decided, what is open, who owes what). Every
piece exists — a Whisper endpoint on the gateway (infra#274), chat-with-transcript (ADR-027) —
and nothing connects them.
ADR-007 deferred STT for three reasons: yt-dlp fragility, ToS-grey scraping, and GPU contention
with the JEPA PoC. Upload-STT shares **none** of them: the user owns the file, nothing is
scraped, there is no caption rate gate, and the contention moved from koala to iguana. What it
does share is landing Whisper in the codebase, which ADR-007 deferred — so a new ADR is required,
and ADR-017's process note records the one time a guardrail-reversing change shipped without one.
**The constraint that shapes everything.** ADR-021 made `transcripts` a shared, **non-RLS**
table on the explicit grounds that it holds only public caption content and nothing
user-identifying. A recorded internal meeting is the opposite: private third-party content owned
by one user. The existing isolation test asserts `transcripts` *is* non-RLS, so a naive reuse
would leak across tenants **and pass CI green**.
**What was measured, not argued** (research doc above):
- **No transcription ceiling up to 90 minutes.** One 174 MB request through the LiteLLM NodePort
took 440 s (~12× realtime). The first failure is the public ingress `llm-api.d-ma.be`
returning `413` above roughly 8–10 MB (~5 min of WAV) — not whisper, and not a Metal OOM.
The 8:20 failure recorded in #33 is consistent with that cap.
- **`response_format=srt` survives LiteLLM, wrapped** in a JSON envelope (`{"text": "<srt>"}`).
- **The two backends differ.** `iguana/kb-whisper` returns whole-second segments;
`berget/kb-whisper` returns sub-second times and a different `language` token.
- **The existing chat path truncates at 18,000 chars (~22 min of speech), keeping the head.**
#33 assumed chat "retrieves rather than generates" and so has no length ceiling. It does
not retrieve; it stuffs the transcript into the prompt and cuts the tail. Measured against
the offered models: `phi4-mini` 400s at ~13k tokens; `gemma4-26b` is right to 90,000 chars and
silently drops the head at 120,000.
**Unknowns, stated.** Real-speech accuracy; compressed-input behaviour; concurrent long jobs on
iguana; and the Metal OOM, deliberately not provoked. (Whether the offered chat models can hold
an hour of Swedish *was* an unknown here; it is now measured, see decision 8.)
**Decision.**
1. **Audio only; transcript is the only thing kept.** An authenticated user uploads an audio
file; Tapir transcribes it and stores the transcript. The audio itself is deleted after
transcription (temp file, removed in a `defer`, swept at start-up) — no custody problem to
solve, and nothing to hand to #31. Video is rejected with a message naming #28. YouTube
audio-download stays rejected. No public/anonymous upload (ADR-008).
2. **Private store: a separate RLS-scoped table, `uploaded_transcripts`** (resolves #28's
decision 1). `user_id NOT NULL REFERENCES users ON DELETE CASCADE`, `ENABLE` + `FORCE ROW LEVEL
SECURITY`, the same `tapir.current_user_id` policy as every other user-owned table. It carries
the transcript text and its timestamped segments (re-transcribing 90 minutes costs ~7
minutes; discarding the timestamps is irreversible), `status`, the `endpoint` that served
it, `audio_seconds`, and a typed `error_kind`. **Not** a `user_id` column on `transcripts`
with a partial policy, even though that would reuse more code: ADR-021's exception is
precisely that *that table holds no private row*, the isolation test asserts it has no
`user_id`, and a partial policy would turn "single exception" into "single exception, with a
private lane inside it" — the shape under which a future query drops the policy clause and
leaks. Two tables keep both boundaries independently assertable.
3. **The isolation test asserts both boundaries and the set.** (a) User A cannot read, update or
delete user B's uploaded transcript; no GUC set → deny-all. (b) `transcripts` is still
non-RLS with no `user_id`. (c) **The set of public-schema tables with row security off is
asserted exactly** — so a new table added without RLS fails CI instead of passing it. (c) is
new; the existing test only checks the named tables.
4. **Validate by probing, reject clearly.** Format and duration cannot be read from a filename
or MIME type — a Teams "voice-only" recording is typically an `.mp4` container with no video
stream, which is indistinguishable from video by extension. The upload is probed
(`ffprobe`): accept any container with an audio stream and **no** video stream; reject video
with "video is #28"; reject unreadable files. Limits, all configurable, all rejected
synchronously with a specific message rather than by timeout: **200 MB** upload and
**90 minutes** duration. 90 minutes is *the longest length measured*, not a ceiling that was
found; raising it is a measurement, not an edit.
5. **Transcode server-side, always, to 16 kHz mono PCM WAV** — even when the input already is,
rather than assuming. whisper-server's `--convert` is rejected (its `--tmp-dir` is created at
process start and rots under macOS periodic cleanup). The transcoder sits behind a port; the
tests use a fake. **This puts `ffmpeg` and `ffprobe` in the runtime image**, which is
distroless-static today. **Provisional: copy static ffmpeg/ffprobe binaries into the existing
distroless image** (no shell, no package manager retained). This is a heavy new dependency
and is **the maintainer's call** — see Open items.
6. **Transcription adapter via the gateway, never the public hostname.** A `Transcriber` port;
the implementation calls `POST {TAPIR_GATEWAY_URL}/audio/transcriptions` where the gateway
URL is the in-cluster NodePort (`koala:30401/v1`). Primary `iguana/kb-whisper`, fallback
`berget/kb-whisper` (immune to iguana contention, which can wedge permanently — infra#273).
- **Fallback triggers on any primary failure** (transport error, timeout, non-2xx). It never
branches on the body: `{"error":"failed to process audio"}` has had at least five distinct
causes.
- **`response_format=verbose_json`**, normalised to one segment shape: times clamped to the
audio duration (the final segment can end past it), one language token. Not `srt`: the
envelope adds a parse step and iguana's segment timing is the same either way.
- **No chunking in v1.** Nothing failed to 90 minutes (measured). Chunk-and-stitch is the
answer *if* the cap is raised or smaller requests become necessary — see triggers.
- **The serving endpoint is recorded per row**, and surfaced, because the two backends
differ in timestamp granularity: which one served a transcript is a quality fact.
- **Berget is a cloud endpoint.** On fallback the audio of a private meeting leaves the
homelab. #33 asks for this fallback; it is configurable (`TAPIR_TRANSCRIBE_FALLBACK_MODEL`,
empty disables), recorded per row, and called out in Open items rather than defaulted
silently.
- Swedish only (`language=sv`). Language detection and routing (#28 decision 2) are not
touched.
- `-H 'Expect:'`, multipart field `file` — encoded in the adapter and its tests, not rediscovered.
7. **Jobs run in-process, one at a time (ADR-018 stands).** Transcription is minutes (~7.5 min
for 90) but a single goroutine behind a one-slot semaphore: two simultaneous 90-minute jobs on
iguana were not measured, so concurrency is 1 until they are. No worker, no new deployable,
no second replica. Honest status per ADR-025: `queued → transcribing → done | failed`, polled
by the same `/status` pattern, every terminal state a rendered card, never a stuck spinner.
Audio is not retained, so a restart mid-job **cannot resume**: at start-up any
`queued`/`transcribing` row is marked `failed` with `error_kind = interrupted`, and the user
re-uploads. The single-replica assumption gains a second dependent (the semaphore is
process-local) and is recorded here.
8. **Chat reuses `chat.Service`; it is not forked.** The uploaded-transcript chat handlers
look the row up through the RLS-scoped store, so a non-owner gets 404 by the same mechanism
as ADR-027's summary gate, **before any model is called**. ADR-027's decisions (ephemeral,
switchable models, trust-the-model) carry over unchanged. **Two things must differ, and both
were measured** (`docs/research/chat-context-over-long-transcripts-2026-10-01.md`):
- **The transcript budget.** The 18,000-char caption default cuts a one-hour meeting to its
first third. Uploaded transcripts take their own `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS`,
default **90,000** — the measured ceiling of the best offered model (`iguana/gemma4-26b`),
which answers correctly there but **silently loses the head of a 120,000-char transcript**
with no error and no truncation flag. (An earlier draft of this ADR proposed 120,000 as a
guess; the measurement refuted it.) The mechanism is a per-request `chat.Request.MaxChars`.
- **The opening model.** The offered list leads with the summarizer default `koala/phi4-mini`,
whose 8,192-token context returns HTTP 400 on an hour of speech (~13,000 tokens). Chat over
an upload opens on `TAPIR_UPLOAD_CHAT_MODEL` (default `iguana/gemma4-26b`); the switcher
still lists every model, and picking `phi4-mini` on a long recording fails loudly and
inline. A per-model budget is not built.
Past ~90,000 chars (≈100 minutes) no offered model can answer by stuffing; see triggers.
9. **Not decided here.** Rich `AVSNITT`/`ALTERNATIV` analysis (gated on #30 — `analyze_srt.py`
hardcodes `max_tokens: 6000`); video, keyframes, synced playback (#28); language detection;
per-transcript deletion UX (account deletion cascades; see Open items); retention of audio.
**On the sequencing question #28 asks of itself** — *is this the thing that best serves the
Stage-0 gate, or the most interesting thing to build?* Recorded answer: it proceeds **ahead of
#32** (the demand test) as the maintainer's explicit, knowing call (#28, comment of 2026-10-01),
on a concrete need rather than speculation. It is not a return-nudge (ADR-020/027): it adds
nothing that prompts a user to come back, so it does not contaminate the unprompted-return signal
ADR-016 measures. The honest risk is the one ADR-016 names: it is larger than anything ADR-020
deferred. The scope cut — chat first, no rich analysis, no video — is what keeps it bounded.
**Consequences.**
- **Positive.** The user's request is met end to end with existing parts. ADR-021's exception stays
singular and now provably so. The measured ceiling removes chunking from v1. No audio is stored.
- **Negative / watch.**
- Tapir's runtime image gains `ffmpeg`: a larger image and a new CVE surface to patch.
- iguana becomes load-bearing for a user-facing feature (#28 decision 3; infra#273 is still the
dependency that matters) — a Metal OOM now means *every upload fails over to berget*, i.e.
private audio leaving the homelab, silently unless the per-row endpoint is surfaced.
- A 90-minute job holds the single slot for ~7.5 minutes; a second user's upload queues behind it.
- Chat over a long transcript is bounded by model context, not by anything in Tapir, and the
failure at the edge is *silent* for gemma (the head is dropped, no error). The 90,000 default
sits just inside the measured limit and is only as good as that single measurement.
- **The default chat model for uploads is a large ollama model on iguana — the same host as
`iguana/kb-whisper`.** infra#273 is exactly that a large resident ollama model can Metal-OOM
whisper-server permanently. Chatting with a transcript can therefore load the model that
wedges the transcriber. Not solved here; the berget fallback limits the blast radius.
- Tapir's own ingress body limit for a 200 MB upload is **unverified** (it lives in `infra`).
**Validation / resolution triggers** (each observable by someone other than the author):
- *Chunking*: build it when the duration cap is raised above 90 min, or when a measured
request at the current cap fails with whisper-side (not ingress) errors.
- *Berget-first*: if more than a quarter of uploads in any week are served by the fallback,
swap the primary/fallback order and revisit infra#273.
- *Retrieval chat*: build chunked retrieval when a recording's transcript exceeds the measured
90,000-char ceiling (≈100 minutes) — i.e. the moment the duration cap is raised above that, or a
real recording turns out denser than the fixture — or when the offered model list changes and
the ceiling is re-measured lower.
- *Worker*: if a transcription job ever degrades the reading UI, or Tapir runs more than one
replica, ADR-018 is revisited.
- *The decision itself*: if the maintainer has not used the feature unprompted within the
ADR-016 window, it was the interesting thing, not the useful one — remove it.
**Open items this ADR creates.**
- **DECISION NEEDED — ffmpeg in the runtime image.** *Options:* (A) static ffmpeg/ffprobe copied
into distroless **(recommended — the only option that satisfies "accept m4a/mp3/opus/mp4,
validate duration, reject video"; pure-Go decoders cover wav/mp3 only)**; (B) accept WAV
only, convert client-side (fails "I just want to upload it"); (C) a transcode sidecar
(second deployable, ADR-018's reasoning against). *Blocks:* the Dockerfile slice, not the
tests, which use the port.
- **DECISION NEEDED — berget fallback for private audio.** *Options:* (A) enabled by default as
#33 specifies, with the serving endpoint shown **(recommended, matches the issue)**; (B)
disabled by default, a failed iguana job fails honestly and the user retries. *Blocks:*
the production default only; the adapter ships either way.
- Verify Tapir's ingress `proxy-body-size` against the 200 MB cap (`infra`).
- Per-transcript deletion is not built; account deletion cascades. A private-content store
without a delete button is a gap worth a ticket, not a silent omission.
**Reversibility.** Additive: one table (down-migration drops it), one adapter, one upload
surface, one image change. Removing the upload route removes the feature; `transcripts`, `videos`
and ADR-021 are untouched.
---
## Rejected alternatives ## Rejected alternatives
Approaches considered during the 2026-06-02 planning + grill session and **deliberately not Approaches considered during the 2026-06-02 planning + grill session and **deliberately not
@@ -1186,6 +1543,9 @@ maps to the ADR that settles it.
| Reverse the Dex-write invite flow (Google OIDC only) | Some intended Future-B users won't use Google; OIDC-only leaves them with no onboarding path — invite flow is load-bearing | ADR-017 | | Reverse the Dex-write invite flow (Google OIDC only) | Some intended Future-B users won't use Google; OIDC-only leaves them with no onboarding path — invite flow is load-bearing | ADR-017 |
| k8s CronJob for scheduled discovery (vs in-process) | At Future-B scale the in-process scheduler is simpler to deploy; CronJob's failure-isolation benefit was weighed and traded away knowingly (revisit if >1 replica or load grows) | ADR-018 | | k8s CronJob for scheduled discovery (vs in-process) | At Future-B scale the in-process scheduler is simpler to deploy; CronJob's failure-isolation benefit was weighed and traded away knowingly (revisit if >1 replica or load grows) | ADR-018 |
| Cached-transcript-first onboarding burst (instant, zero-fetch picks) | Live pilot DB: ~3% cross-user video overlap, 0 cached-and-unsummarized, a new user's newest-20 are 20/20 uncached — newest-first and cached-first are structurally incompatible. Empty lever at pilot scale | ADR-028 | | Cached-transcript-first onboarding burst (instant, zero-fetch picks) | Live pilot DB: ~3% cross-user video overlap, 0 cached-and-unsummarized, a new user's newest-20 are 20/20 uncached — newest-first and cached-first are structurally incompatible. Empty lever at pilot scale | ADR-028 |
| Uploaded transcripts as a `user_id` column + partial RLS policy on `transcripts` | Turns ADR-021's single exception into an exception with a private lane inside it; the isolation test asserts `transcripts` has no `user_id`. A separate RLS table keeps both boundaries independently assertable | ADR-033 |
| whisper-server `--convert` for server-side transcode | Its `--tmp-dir` is created at process start and rots under macOS periodic cleanup | ADR-033 |
| Calling the public `llm-api.d-ma.be` from in-cluster Tapir | Its ingress returns 413 above ~8–10 MB (~5 min of WAV); the NodePort has no such cap | ADR-033 |
If a future case genuinely reopens one of these, that's a new ADR superseding the relevant one — If a future case genuinely reopens one of these, that's a new ADR superseding the relevant one —
not a silent reversal. not a silent reversal.
+26
View File
@@ -0,0 +1,26 @@
# Vocabulary (hand-written pilot 2026-06 — will be generated by langgen in Phase 2)
| Term | Means | Never say |
|---|---|---|
| transcript | Captions-first text of a video; the source for summarizing. Shared store keyed by `(provider, video_id)`. | "subtitles file", "the audio" |
| video | Per-user record of a seen video; the unit of work. Not globally deduped. | "the shared video" |
| subscription | A watched channel on a user's connection that Tapir polls for new videos. | "feed", "follow" |
| summary | A video's produced output: summary text + highlights + takeaways + AI provenance. | "transcript" |
| highlight | A notable point pulled from a video (`Summary.Highlights`). | "takeaway" |
| takeaway | An actionable conclusion from a video (`Summary.Takeaways`). | "highlight" |
| sink | A delivery destination for a summary (`store`, `brain`). New one = new adapter. | "the database" |
| AI router | Local-first chain: local Primary → local fallback → external worst-case. | "the API" |
| BYO-AI fallback | User's own external AI key, used only when local fails; opt-in. Without it, content is never sent externally. | "the default AI" |
| brain | Persistent homelab knowledge store; in Tapir, one optional HTTP sink (`brain_ingest`), not the filesystem package. | "the database", "the store" |
| LiteLLM gateway | Local AI gateway (`koala:30401/v1`); the Primary in the AI router. | "piguard:4000", "koala:4000", "the cloud" |
| connection | A connected video account a user authorizes via OAuth; subscriptions hang off it. | "login", "session" |
## Caveman rubric
Before any HIGH/CRITICAL operation, output one line:
`caveman: me <verb> <object>, not <excluded thing>`
Valid iff: (1) only vocabulary terms + plain verbs, (2) a stranger could identify
the exact operation, (3) names one thing explicitly NOT being done.
Examples:
- `caveman: me delete user connection, not the shared transcript`
- `caveman: me send transcript to BYO-AI fallback, not the LiteLLM gateway`
+1 -1
View File
@@ -4,7 +4,7 @@ import (
"strings" "strings"
"testing" "testing"
"gitea.d-ma.be/mathias/tapir/internal/config" "git.d-ma.be/mathias/tapir/internal/config"
) )
// TestChatModelsReuseTheChainLocalFirst: the switcher offers the ADR-022 chain in // TestChatModelsReuseTheChainLocalFirst: the switcher offers the ADR-022 chain in
+1 -1
View File
@@ -5,7 +5,7 @@ import (
"log/slog" "log/slog"
"sync" "sync"
"gitea.d-ma.be/mathias/tapir/internal/runner" "git.d-ma.be/mathias/tapir/internal/runner"
) )
// discoveryRunner runs one user's discovery pass. // discoveryRunner runs one user's discovery pass.
+1 -1
View File
@@ -10,7 +10,7 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/runner" "git.d-ma.be/mathias/tapir/internal/runner"
) )
// serialize must guarantee at most one discovery pass runs at a time, so a // serialize must guarantee at most one discovery pass runs at a time, so a
+170
View File
@@ -0,0 +1,170 @@
package main
import (
"context"
"fmt"
"io"
"mime/multipart"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"strings"
"testing"
"time"
embeddedpostgres "github.com/fergusstrange/embedded-postgres"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/store"
"git.d-ma.be/mathias/tapir/internal/config"
"git.d-ma.be/mathias/tapir/internal/web"
)
// TestE2E_UploadTranscribeAndChat is the end-to-end check for tapir#33: upload a
// real audio file, watch it become a transcript through the REAL ffprobe, ffmpeg
// and gateway, then ask a question and get a grounded answer from a REAL model.
// It is opt-in — it needs real binaries and the live gateway — and skipped unless
// TAPIR_E2E_AUDIO is set:
//
// TAPIR_E2E_AUDIO=/path/meeting.m4a TAPIR_E2E_FFMPEG=/path/ffmpeg \
// TAPIR_E2E_FFPROBE=/path/ffprobe TAPIR_E2E_QUESTION='Vem ansvarar för virket?' \
// TAPIR_E2E_EXPECT=anders op run --env-file ~/.op-env -- \
// go test ./cmd/tapir -run E2E_Upload -v -count=1
//
// Use SYNTHETIC audio only: this repo is public, and a fallback to berget sends
// the audio to a cloud endpoint (#33).
func TestE2E_UploadTranscribeAndChat(t *testing.T) {
audioPath := os.Getenv("TAPIR_E2E_AUDIO")
if audioPath == "" {
t.Skip("set TAPIR_E2E_AUDIO (and friends) to run the live end-to-end check")
}
key := os.Getenv("LITELLM_MASTER_KEY")
require.NotEmpty(t, key, "LITELLM_MASTER_KEY must be set (via op run)")
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
// A private embedded Postgres: nothing here may touch a real database.
port := uint32(55000 + os.Getpid()%1000)
rt := filepath.Join(os.TempDir(), fmt.Sprintf("tapir-epg-e2e-%d", os.Getpid()))
pg := embeddedpostgres.NewDatabase(embeddedpostgres.DefaultConfig().Port(port).
RuntimePath(rt).DataPath(filepath.Join(rt, "data")).BinariesPath(filepath.Join(rt, "bin")).
CachePath(filepath.Join(os.TempDir(), "tapir-epg-cache")))
require.NoError(t, pg.Start())
t.Cleanup(func() { _ = pg.Stop(); _ = os.RemoveAll(rt) })
dsn := fmt.Sprintf("postgres://postgres:postgres@localhost:%d/postgres?sslmode=disable", port)
st, err := store.New(ctx, dsn)
require.NoError(t, err)
t.Cleanup(st.Close)
raw, err := pgxpool.New(ctx, dsn)
require.NoError(t, err)
t.Cleanup(raw.Close)
const uid = "11111111-1111-1111-1111-111111111111"
_, err = raw.Exec(ctx, `INSERT INTO users (id) VALUES ($1)`, uid)
require.NoError(t, err)
_, err = raw.Exec(ctx, `INSERT INTO user_identities (dex_subject, user_id) VALUES ('e2e', $1)`, uid)
require.NoError(t, err)
cfg, err := config.Load()
require.NoError(t, err)
cfg.GatewayKey = key
cfg.UploadsEnabled = true
cfg.UploadTmpDir = filepath.Join(t.TempDir(), "spool")
cfg.FFmpegBin = envOrDefault("TAPIR_E2E_FFMPEG", "ffmpeg")
cfg.FFprobeBin = envOrDefault("TAPIR_E2E_FFPROBE", "ffprobe")
intake, worker, err := buildUploads(ctx, cfg, st, quietLog())
require.NoError(t, err)
go worker.Run(ctx)
app := &web.App{
Store: st, Identity: st, Auth: web.StubAuth{U: web.User{Subject: "e2e"}},
Chat: buildChat(cfg),
}
wireUploads(app, cfg, intake) // the same wiring cmdServe uses
srv := httptest.NewServer(app.Router())
t.Cleanup(srv.Close)
noRedirect := &http.Client{CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
// 1. Upload.
f, err := os.Open(audioPath)
require.NoError(t, err)
defer f.Close()
pr, pw := io.Pipe()
mw := multipart.NewWriter(pw)
go func() {
_ = mw.WriteField("title", "E2E recording")
fw, _ := mw.CreateFormFile("file", filepath.Base(audioPath))
_, _ = io.Copy(fw, f)
_ = mw.Close()
_ = pw.Close()
}()
req, err := http.NewRequest(http.MethodPost, srv.URL+"/uploads", pr)
require.NoError(t, err)
req.Header.Set("Content-Type", mw.FormDataContentType())
resp, err := noRedirect.Do(req)
require.NoError(t, err)
_ = resp.Body.Close()
require.Equal(t, http.StatusSeeOther, resp.StatusCode, "an accepted upload redirects to its page")
loc := resp.Header.Get("Location")
require.True(t, strings.HasPrefix(loc, "/uploads/"), loc)
t.Logf("accepted -> %s", loc)
// 2. Wait for the transcript: unfinished states poll, terminal states do not.
var page string
require.Eventually(t, func() bool {
r, err := http.Get(srv.URL + loc + "/status")
if err != nil {
return false
}
b, _ := io.ReadAll(r.Body)
_ = r.Body.Close()
page = string(b)
return !strings.Contains(page, "hx-trigger")
}, 12*time.Minute, 3*time.Second, "the upload never reached a terminal state")
require.Contains(t, page, "Transcribed", "terminal state was not 'done': %.300s", page)
var status, endpoint, content string
var secs float64
require.NoError(t, raw.QueryRow(ctx,
`SELECT status, endpoint, content, audio_seconds FROM uploaded_transcripts WHERE user_id = $1`, uid).
Scan(&status, &endpoint, &content, &secs))
t.Logf("done: endpoint=%s audio_s=%.1f transcript_chars=%d", endpoint, secs, len(content))
assert.Equal(t, "done", status)
assert.NotEmpty(t, endpoint)
assert.NotEmpty(t, content)
entries, err := os.ReadDir(cfg.UploadTmpDir)
require.NoError(t, err)
assert.Empty(t, entries, "the audio must be deleted once transcribed")
// 3. Ask a question; the answer must come from the transcript.
q := envOrDefault("TAPIR_E2E_QUESTION", "Who is responsible for the timber order?")
chatResp, err := http.PostForm(srv.URL+loc+"/chat", url.Values{"question": {q}})
require.NoError(t, err)
answerPage, _ := io.ReadAll(chatResp.Body)
_ = chatResp.Body.Close()
require.Equal(t, http.StatusOK, chatResp.StatusCode)
t.Logf("question: %s", q)
if want := os.Getenv("TAPIR_E2E_EXPECT"); want != "" {
assert.Contains(t, strings.ToLower(string(answerPage)), strings.ToLower(want),
"the answer should be grounded in the transcript")
}
t.Logf("answer page excerpt: %.600s", afterMarker(string(answerPage), `class="chat-turn chat-a"`))
}
func envOrDefault(k, d string) string {
if v := os.Getenv(k); v != "" {
return v
}
return d
}
func afterMarker(s, m string) string {
if i := strings.Index(s, m); i >= 0 {
return s[i:]
}
return "(no answer turn found)"
}
+1 -1
View File
@@ -5,7 +5,7 @@ import (
"fmt" "fmt"
"os" "os"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
// Env var names for the read-only CLI. DSN and user id are never hardcoded — the // Env var names for the read-only CLI. DSN and user id are never hardcoded — the
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
func TestFormatListColumnsAndOrdering(t *testing.T) { func TestFormatListColumnsAndOrdering(t *testing.T) {
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"strings" "strings"
"text/tabwriter" "text/tabwriter"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
// runList prints the user's stored summaries as a table, most recent first. // runList prints the user's stored summaries as a table, most recent first.
+48 -9
View File
@@ -23,14 +23,15 @@ import (
"sync" "sync"
"time" "time"
"gitea.d-ma.be/mathias/tapir/internal/adapters/secrets" "git.d-ma.be/mathias/tapir/internal/adapters/secrets"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
"gitea.d-ma.be/mathias/tapir/internal/adapters/youtube" "git.d-ma.be/mathias/tapir/internal/adapters/youtube"
"gitea.d-ma.be/mathias/tapir/internal/auth" "git.d-ma.be/mathias/tapir/internal/auth"
"gitea.d-ma.be/mathias/tapir/internal/config" "git.d-ma.be/mathias/tapir/internal/config"
"gitea.d-ma.be/mathias/tapir/internal/runner" "git.d-ma.be/mathias/tapir/internal/metrics"
"gitea.d-ma.be/mathias/tapir/internal/web" "git.d-ma.be/mathias/tapir/internal/runner"
"gitea.d-ma.be/mathias/tapir/internal/web/oidc" "git.d-ma.be/mathias/tapir/internal/web"
"git.d-ma.be/mathias/tapir/internal/web/oidc"
) )
func main() { func main() {
@@ -206,6 +207,26 @@ func cmdServe(ctx context.Context, log *slog.Logger) error {
log.Info("web chat enabled (stored-transcript only)", "models", chatModels(cfg)) log.Info("web chat enabled (stored-transcript only)", "models", chatModels(cfg))
} }
// Audio upload -> private transcript (ADR-033). Off unless TAPIR_UPLOADS_ENABLED:
// the image must carry ffmpeg first. The worker is IN-PROCESS and one job at a
// time, so it inherits the single-replica assumption below (its queue is
// process-local); it stops with the same ctx as the server.
intake, worker, err := buildUploads(ctx, cfg, st, log)
if err != nil {
return err
}
wireUploads(app, cfg, intake)
if intake != nil {
go worker.Run(ctx)
log.Info("audio upload enabled", "model", cfg.TranscribeModel, "fallback", cfg.TranscribeFallbackModel,
"max_mb", cfg.UploadMaxBytes>>20, "max_minutes", int(cfg.UploadMaxDuration.Minutes()))
log.Warn("audio upload assumes a SINGLE replica: the transcription queue is process-local (ADR-033, ADR-018)")
if cfg.TranscribeFallbackModel != "" {
log.Warn("audio upload fallback is a CLOUD endpoint: if the primary fails, private audio leaves the homelab",
"fallback", cfg.TranscribeFallbackModel)
}
}
// User onboarding is handled by the IdP (Authentik invite flow), not Tapir — // User onboarding is handled by the IdP (Authentik invite flow), not Tapir —
// the Dex local-password provisioning path was removed (ADR-019). An // the Dex local-password provisioning path was removed (ADR-019). An
// authenticated subject with no Tapir user is routed to /register. // authenticated subject with no Tapir user is routed to /register.
@@ -310,16 +331,34 @@ func cmdServe(ctx context.Context, log *slog.Logger) error {
srv := &http.Server{ srv := &http.Server{
Addr: cfg.HTTPAddr, Addr: cfg.HTTPAddr,
Handler: app.Router(), Handler: metrics.HTTPMiddleware(app.Router()),
ReadHeaderTimeout: 10 * time.Second, ReadHeaderTimeout: 10 * time.Second,
} }
// Prometheus /metrics on a SEPARATE port (ADR-030) — never on the public app
// mux, so a scrape is in-cluster only. Empty TAPIR_METRICS_ADDR disables it.
var metricsSrv *http.Server
if cfg.MetricsAddr != "" {
mmux := http.NewServeMux()
mmux.Handle("GET /metrics", metrics.Handler())
metricsSrv = &http.Server{Addr: cfg.MetricsAddr, Handler: mmux, ReadHeaderTimeout: 10 * time.Second}
go func() {
log.Info("serving metrics", "addr", cfg.MetricsAddr)
if err := metricsSrv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Error("metrics server", "err", err)
}
}()
}
// Graceful shutdown on signal: stop accepting, drain in-flight requests. // Graceful shutdown on signal: stop accepting, drain in-flight requests.
go func() { go func() {
<-ctx.Done() <-ctx.Done()
shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel() defer cancel()
_ = srv.Shutdown(shutdownCtx) _ = srv.Shutdown(shutdownCtx)
if metricsSrv != nil {
_ = metricsSrv.Shutdown(shutdownCtx)
}
}() }()
log.Info("serving web ui", "addr", cfg.HTTPAddr, "user", cfg.UserID) log.Info("serving web ui", "addr", cfg.HTTPAddr, "user", cfg.UserID)
+14 -13
View File
@@ -5,17 +5,18 @@ import (
"fmt" "fmt"
"strings" "strings"
"gitea.d-ma.be/mathias/tapir/internal/adapters/chat" "git.d-ma.be/mathias/tapir/internal/adapters/chat"
"gitea.d-ma.be/mathias/tapir/internal/adapters/llm" "git.d-ma.be/mathias/tapir/internal/adapters/llm"
"gitea.d-ma.be/mathias/tapir/internal/adapters/secrets" "git.d-ma.be/mathias/tapir/internal/adapters/secrets"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
"gitea.d-ma.be/mathias/tapir/internal/adapters/summarizer" "git.d-ma.be/mathias/tapir/internal/adapters/summarizer"
"gitea.d-ma.be/mathias/tapir/internal/adapters/youtube" "git.d-ma.be/mathias/tapir/internal/adapters/youtube"
"gitea.d-ma.be/mathias/tapir/internal/config" "git.d-ma.be/mathias/tapir/internal/config"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
"gitea.d-ma.be/mathias/tapir/internal/ports" "git.d-ma.be/mathias/tapir/internal/metrics"
"gitea.d-ma.be/mathias/tapir/internal/usecase" "git.d-ma.be/mathias/tapir/internal/ports"
"gitea.d-ma.be/mathias/tapir/internal/web" "git.d-ma.be/mathias/tapir/internal/usecase"
"git.d-ma.be/mathias/tapir/internal/web"
) )
// videoFetcher adapts the YouTube adapter to web.VideoFetcher for the paste flow // videoFetcher adapts the YouTube adapter to web.VideoFetcher for the paste flow
@@ -69,7 +70,7 @@ func buildSummarizer(cfg config.Config) *summarizer.Summarizer {
func summarizerEndpoint(cfg config.Config) func(model string) summarizer.Endpoint { func summarizerEndpoint(cfg config.Config) func(model string) summarizer.Endpoint {
return func(model string) summarizer.Endpoint { return func(model string) summarizer.Endpoint {
return summarizer.Endpoint{ return summarizer.Endpoint{
Client: llm.New(cfg.GatewayURL, cfg.GatewayKey, model, cfg.SummarizerTimeout, llm.WithMaxTokens(cfg.SummaryMaxTokens)), Client: llm.New(cfg.GatewayURL, cfg.GatewayKey, model, cfg.SummarizerTimeout, llm.WithMaxTokens(cfg.SummaryMaxTokens), llm.WithUsageHook(metrics.RecordTokens)),
Provider: providerOf(model), Provider: providerOf(model),
Model: model, Model: model,
} }
@@ -150,7 +151,7 @@ func buildChat(cfg config.Config) *chat.Service {
return nil return nil
} }
newClient := func(model string) chat.Completer { newClient := func(model string) chat.Completer {
return llm.New(cfg.GatewayURL, cfg.GatewayKey, model, cfg.SummarizerTimeout, llm.WithMaxTokens(cfg.SummaryMaxTokens)) return llm.New(cfg.GatewayURL, cfg.GatewayKey, model, cfg.SummarizerTimeout, llm.WithMaxTokens(cfg.SummaryMaxTokens), llm.WithUsageHook(metrics.RecordTokens))
} }
return chat.New(newClient, models, cfg.MaxTranscriptChars) return chat.New(newClient, models, cfg.MaxTranscriptChars)
} }
+1 -1
View File
@@ -3,7 +3,7 @@ package main
import ( import (
"testing" "testing"
"gitea.d-ma.be/mathias/tapir/internal/config" "git.d-ma.be/mathias/tapir/internal/config"
) )
// TestBuildProcessorNilOnIncompleteConfig asserts the queue-only fallback: when a // TestBuildProcessorNilOnIncompleteConfig asserts the queue-only fallback: when a
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"text/tabwriter" "text/tabwriter"
"time" "time"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
// gateThreshold is the Stage-0 gate (VISION/ADR-016): usage in >= 2 distinct // gateThreshold is the Stage-0 gate (VISION/ADR-016): usage in >= 2 distinct
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
var testSince = time.Date(2026, 6, 11, 0, 0, 0, 0, time.UTC) var testSince = time.Date(2026, 6, 11, 0, 0, 0, 0, time.UTC)
+7 -7
View File
@@ -6,13 +6,13 @@ import (
"log/slog" "log/slog"
"time" "time"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
"gitea.d-ma.be/mathias/tapir/internal/adapters/youtube" "git.d-ma.be/mathias/tapir/internal/adapters/youtube"
"gitea.d-ma.be/mathias/tapir/internal/config" "git.d-ma.be/mathias/tapir/internal/config"
"gitea.d-ma.be/mathias/tapir/internal/ports" "git.d-ma.be/mathias/tapir/internal/ports"
"gitea.d-ma.be/mathias/tapir/internal/runner" "git.d-ma.be/mathias/tapir/internal/runner"
"gitea.d-ma.be/mathias/tapir/internal/usecase" "git.d-ma.be/mathias/tapir/internal/usecase"
"gitea.d-ma.be/mathias/tapir/internal/web" "git.d-ma.be/mathias/tapir/internal/web"
) )
// buildUserRunner constructs a runner.Runner for one user, reusing the same // buildUserRunner constructs a runner.Runner for one user, reusing the same
+2 -2
View File
@@ -11,8 +11,8 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
"gitea.d-ma.be/mathias/tapir/internal/runner" "git.d-ma.be/mathias/tapir/internal/runner"
) )
func quietLog() *slog.Logger { func quietLog() *slog.Logger {
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"os" "os"
"strings" "strings"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
// runShow prints the full summary for one video: text, highlights, takeaways, // runShow prints the full summary for one video: text, highlights, takeaways,
+100
View File
@@ -0,0 +1,100 @@
package main
import (
"context"
"fmt"
"log/slog"
"net/http"
"os"
"path/filepath"
"git.d-ma.be/mathias/tapir/internal/adapters/audio"
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
"git.d-ma.be/mathias/tapir/internal/config"
"git.d-ma.be/mathias/tapir/internal/uploads"
"git.d-ma.be/mathias/tapir/internal/web"
)
// uploadsStore is the store surface the upload wiring needs. *store.Store
// satisfies it.
type uploadsStore interface {
uploads.Store
uploads.WorkerStore
FailInterruptedUploads(ctx context.Context) (int, error)
}
// buildUploads wires audio upload and private transcription (ADR-033). It returns
// nils when uploads are disabled — never a typed-nil Intake, which would mount the
// /uploads routes over nothing. When enabled it, in order: prepares a private spool
// directory, deletes any audio a crashed run left in it, fails the rows such a run
// left queued or transcribing (the audio is not retained, so they can never
// resume), and only then builds the intake and the worker. The caller runs the
// worker. The fallback endpoint is a cloud service; an empty
// TAPIR_TRANSCRIBE_FALLBACK_MODEL keeps private audio on the homelab.
func buildUploads(ctx context.Context, cfg config.Config, st uploadsStore, log *slog.Logger) (*uploads.Intake, *uploads.Worker, error) {
if !cfg.UploadsEnabled {
return nil, nil, nil
}
if cfg.GatewayURL == "" {
return nil, nil, fmt.Errorf("uploads: TAPIR_UPLOADS_ENABLED is set but there is no TAPIR_GATEWAY_URL to transcribe through")
}
if err := os.MkdirAll(cfg.UploadTmpDir, 0o700); err != nil {
return nil, nil, fmt.Errorf("uploads: spool dir: %w", err)
}
removeSpooledAudio(cfg.UploadTmpDir, log)
n, err := st.FailInterruptedUploads(ctx)
if err != nil {
return nil, nil, fmt.Errorf("uploads: fail interrupted rows: %w", err)
}
if n > 0 {
log.Warn("uploads: marked interrupted by a restart (audio is not retained; they must be re-uploaded)", "count", n)
}
// No overall client timeout: a 90 minute recording takes minutes. The deadline
// is the worker's per-job context.
client := &http.Client{}
primary := transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, cfg.TranscribeModel, client)
var fallback transcribe.Backend
if cfg.TranscribeFallbackModel != "" && cfg.TranscribeFallbackModel != cfg.TranscribeModel {
fallback = transcribe.NewGateway(cfg.GatewayURL, cfg.GatewayKey, cfg.TranscribeFallbackModel, client)
}
worker := uploads.NewWorker(st, &audio.FFmpeg{Bin: cfg.FFmpegBin}, transcribe.NewChain(primary, fallback),
cfg.UploadTmpDir, cfg.TranscribeTimeout)
intake := uploads.NewIntake(st, &audio.FFprobe{Bin: cfg.FFprobeBin}, worker,
uploadLimits(cfg), cfg.UploadTmpDir)
return intake, worker, nil
}
func uploadLimits(cfg config.Config) uploads.Limits {
return uploads.Limits{MaxBytes: cfg.UploadMaxBytes, MaxDuration: cfg.UploadMaxDuration}
}
// removeSpooledAudio deletes the spool files this process family creates
// (upload-*, wav-*) and nothing else.
func removeSpooledAudio(dir string, log *slog.Logger) {
for _, pat := range []string{"upload-*", "wav-*"} {
matches, _ := filepath.Glob(filepath.Join(dir, pat))
for _, m := range matches {
if err := os.Remove(m); err != nil {
log.Warn("uploads: remove leftover audio", "path", m, "err", err)
}
}
}
}
// wireUploads connects a built Intake to the web App: the routes, the limits the
// form states, and the chat settings for an uploaded recording. A nil intake
// leaves the App untouched — assigning a nil *Intake to the Uploader interface
// would produce a non-nil interface and mount the routes over nothing.
func wireUploads(app *web.App, cfg config.Config, in *uploads.Intake) {
if in == nil {
return
}
app.Uploads = in
app.UploadLimits = uploadLimits(cfg)
app.UploadChatMaxChars = cfg.MaxUploadedTranscriptChars
app.UploadChatModel = cfg.UploadChatModel
}
+156
View File
@@ -0,0 +1,156 @@
package main
import (
"context"
"errors"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/config"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/web"
)
// fakeUploadsStore satisfies the store surface buildUploads needs, and counts the
// start-up sweep.
type fakeUploadsStore struct {
sweeps int
sweepErr error
}
func (f *fakeUploadsStore) CreateUpload(context.Context, string, string) (string, error) {
return "id", nil
}
func (f *fakeUploadsStore) FailUpload(context.Context, string, string, domain.UploadErrorKind) error {
return nil
}
func (f *fakeUploadsStore) StartTranscribing(context.Context, string, string) error { return nil }
func (f *fakeUploadsStore) CompleteUpload(context.Context, string, string, domain.Transcription) error {
return nil
}
func (f *fakeUploadsStore) FailInterruptedUploads(context.Context) (int, error) {
f.sweeps++
return 2, f.sweepErr
}
func uploadCfg(t *testing.T, enabled bool) config.Config {
t.Helper()
var cfg config.Config
cfg.GatewayURL = "http://gateway.invalid/v1"
cfg.UploadsEnabled = enabled
cfg.UploadMaxBytes = 200 << 20
cfg.UploadTmpDir = filepath.Join(t.TempDir(), "spool")
cfg.TranscribeModel = "iguana/kb-whisper"
cfg.TranscribeFallbackModel = "berget/kb-whisper"
return cfg
}
func TestBuildUploads_DisabledMountsNothingAndSweepsNothing(t *testing.T) {
st := &fakeUploadsStore{}
in, w, err := buildUploads(context.Background(), uploadCfg(t, false), st, quietLog())
require.NoError(t, err)
assert.Nil(t, in, "a typed-nil Intake must never reach web.App.Uploads: it would mount the routes over nothing")
assert.Nil(t, w)
assert.Zero(t, st.sweeps)
}
func TestBuildUploads_EnabledFailsInterruptedRowsBeforeAnythingElseStarts(t *testing.T) {
st := &fakeUploadsStore{}
in, w, err := buildUploads(context.Background(), uploadCfg(t, true), st, quietLog())
require.NoError(t, err)
require.NotNil(t, in)
require.NotNil(t, w)
assert.Equal(t, 1, st.sweeps,
"audio is not retained, so rows left queued/transcribing by a restart can never resume: fail them at start-up, not never")
}
func TestBuildUploads_SweepFailureStopsStartup(t *testing.T) {
st := &fakeUploadsStore{sweepErr: errors.New("db down")}
_, _, err := buildUploads(context.Background(), uploadCfg(t, true), st, quietLog())
require.Error(t, err)
assert.Contains(t, err.Error(), "db down")
}
func TestBuildUploads_CreatesTheSpoolDirectoryPrivately(t *testing.T) {
cfg := uploadCfg(t, true)
_, _, err := buildUploads(context.Background(), cfg, &fakeUploadsStore{}, quietLog())
require.NoError(t, err)
info, err := os.Stat(cfg.UploadTmpDir)
require.NoError(t, err)
assert.True(t, info.IsDir())
assert.Equal(t, os.FileMode(0o700), info.Mode().Perm(), "spooled audio is private third-party content")
}
func TestBuildUploads_EnabledWithoutAGatewayIsAnError(t *testing.T) {
cfg := uploadCfg(t, true)
cfg.GatewayURL = ""
_, _, err := buildUploads(context.Background(), cfg, &fakeUploadsStore{}, quietLog())
assert.Error(t, err, "uploads on with nowhere to transcribe must fail loudly at start-up, not at the first upload")
}
func TestBuildUploads_RemovesAudioLeftBehindByACrash(t *testing.T) {
cfg := uploadCfg(t, true)
require.NoError(t, os.MkdirAll(cfg.UploadTmpDir, 0o700))
leftovers := []string{"upload-123", "wav-456"}
for _, n := range leftovers {
require.NoError(t, os.WriteFile(filepath.Join(cfg.UploadTmpDir, n), []byte("audio"), 0o600))
}
unrelated := filepath.Join(cfg.UploadTmpDir, "someone-elses-file")
require.NoError(t, os.WriteFile(unrelated, []byte("keep"), 0o600))
_, _, err := buildUploads(context.Background(), cfg, &fakeUploadsStore{}, quietLog())
require.NoError(t, err)
for _, n := range leftovers {
_, err := os.Stat(filepath.Join(cfg.UploadTmpDir, n))
assert.True(t, os.IsNotExist(err), "%s: audio is never retained, so a spooled file from a crashed run must not survive start-up", n)
}
_, err = os.Stat(unrelated)
assert.NoError(t, err, "only our own spool files are removed")
}
func TestWireUploads_SetsEverythingTheWebAppNeedsForUploadsAndChat(t *testing.T) {
cfg := uploadCfg(t, true)
cfg.UploadMaxDuration = 90 * 60 * 1e9
cfg.MaxUploadedTranscriptChars = 90000
cfg.UploadChatModel = "iguana/gemma4-26b"
in, _, err := buildUploads(context.Background(), cfg, &fakeUploadsStore{}, quietLog())
require.NoError(t, err)
app := &web.App{}
wireUploads(app, cfg, in)
assert.NotNil(t, app.Uploads)
assert.Equal(t, cfg.UploadMaxBytes, app.UploadLimits.MaxBytes, "the form states the limits Intake enforces")
assert.Equal(t, cfg.UploadMaxDuration, app.UploadLimits.MaxDuration)
assert.Equal(t, 90000, app.UploadChatMaxChars,
"without this the chat would use the 18,000-char caption default and cut an hour to its first third")
assert.Equal(t, "iguana/gemma4-26b", app.UploadChatModel)
}
func TestWireUploads_NilIntakeLeavesTheAppUntouched(t *testing.T) {
app := &web.App{}
wireUploads(app, uploadCfg(t, false), nil)
assert.Nil(t, app.Uploads, "a nil *Intake must not become a non-nil interface that mounts the routes")
assert.Zero(t, app.UploadChatMaxChars)
}
+35 -3
View File
@@ -18,7 +18,12 @@ only opaque references to them; the secret material lives in ESO/1Password (ADR-
not LLM re-summarization but a rate-gated, reputation-risky caption fetch (ADR-010/014), which not LLM re-summarization but a rate-gated, reputation-risky caption fetch (ADR-010/014), which
is paid per re-fetch regardless of user count — so persisting public caption content once and is paid per re-fetch regardless of user count — so persisting public caption content once and
sharing it strictly beats the coupling it removes. Everything else each user owns is sharing it strictly beats the coupling it removes. Everything else each user owns is
self-contained; `rls_test.go` proves transcripts is the single exception. self-contained; `rls_test.go` proves transcripts is the single exception for *content*.
**Uploaded-audio transcripts are the opposite of public caption content** (private, owned by
one user) and therefore live in their own RLS-scoped `uploaded_transcripts` table, never in
`transcripts` (ADR-033). The isolation test now also pins the **exact set** of tables with row
security off (`transcripts`, `user_identities`, `invitations`, `schema_migrations`), so a new
table added without RLS fails CI.
- **Secrets by reference only.** Tables hold a `secret_ref` (opaque string/UUID resolved via - **Secrets by reference only.** Tables hold a `secret_ref` (opaque string/UUID resolved via
the `SecretStore` port), never tokens or keys. the `SecretStore` port), never tokens or keys.
- **The brain sink is just a delivery target.** No brain-specific tables. Whether a summary - **The brain sink is just a delivery target.** No brain-specific tables. Whether a summary
@@ -41,6 +46,7 @@ erDiagram
VIDEO ||--o| SUMMARY : "has at most one" VIDEO ||--o| SUMMARY : "has at most one"
SUMMARY ||--o{ SINK_DELIVERY : "delivered via" SUMMARY ||--o{ SINK_DELIVERY : "delivered via"
USER ||--o{ CHANNEL_ERROR : "reports unavailable channels" USER ||--o{ CHANNEL_ERROR : "reports unavailable channels"
USER ||--o{ UPLOADED_TRANSCRIPT : "uploads (private, ADR-033)"
USER { USER {
uuid id PK uuid id PK
@@ -136,12 +142,27 @@ erDiagram
timestamptz first_seen timestamptz first_seen
timestamptz last_seen timestamptz last_seen
} }
UPLOADED_TRANSCRIPT {
uuid id PK
uuid user_id FK "-> USER, ON DELETE CASCADE, NOT NULL; RLS ENABLE+FORCE (migration 017)"
text title
text status "queued | transcribing | done | failed"
text content "nullable until done; the transcript text"
jsonb segments "nullable until done; [{start,end,text}] in seconds"
text endpoint "which backend served it: iguana/... | berget/... (berget = audio left the homelab)"
float audio_seconds
text error_kind "typed: interrupted | transcription_failed — never a provider string"
timestamptz created_at
timestamptz finished_at
}
``` ```
`SUMMARY_ACTION` has `UNIQUE (user_id, video_id, action)`; `VIDEO_CONNECTION` has `SUMMARY_ACTION` has `UNIQUE (user_id, video_id, action)`; `VIDEO_CONNECTION` has
`UNIQUE (user_id, provider)` (one connection per provider — reconnect upserts in place). `UNIQUE (user_id, provider)` (one connection per provider — reconnect upserts in place).
RLS (`ENABLE` + `FORCE`) is on **every solid user-owned table above** — `users`, `videos`, RLS (`ENABLE` + `FORCE`) is on **every solid user-owned table above** — `users`, `videos`,
`transcripts`, `summaries`, `summary_actions`, `video_connections`, `channel_errors`. `summaries`, `summary_actions`, `video_connections`, `channel_errors`, `uploaded_transcripts`.
(`transcripts` is deliberately **not** in this list: since ADR-021 it is shared public caption
content with no `user_id`.)
`sink_deliveries` is RLS'd via an `EXISTS` on its parent summary; `user_identities` is `sink_deliveries` is RLS'd via an `EXISTS` on its parent summary; `user_identities` is
intentionally **not** RLS'd (auth plumbing). See the *Isolation invariant* section for the intentionally **not** RLS'd (auth plumbing). See the *Isolation invariant* section for the
mechanism. mechanism.
@@ -197,6 +218,14 @@ mechanism.
the most recent discovery pass. Upserted per scheduler pass (`last_seen` refreshed each run); the most recent discovery pass. Upserted per scheduler pass (`last_seen` refreshed each run);
surfaced on the account page as a warning. Cascades on user deletion. Primary key is surfaced on the account page as a warning. Cascades on user deletion. Primary key is
`(user_id, channel_id)`. FORCE RLS'd. `(user_id, channel_id)`. FORCE RLS'd.
- **UPLOADED_TRANSCRIPTS** (migration 017, ADR-033) — the **private** transcript of audio a user
uploaded. Deliberately a separate table from the shared `transcripts`: that one is non-RLS on the
grounds that it holds only public caption content, and a recorded meeting is the opposite.
`user_id NOT NULL ... ON DELETE CASCADE`, FORCE RLS'd. `status` runs `queued → transcribing →
done | failed`; `done` requires `content`, `failed` requires a typed `error_kind` (CHECK
constraints). `endpoint` records which backend served it — `berget/...` means the audio left the
homelab. The audio itself is never stored. `segments` keeps the timestamps (re-transcribing 90
minutes costs ~7). Unlike `summary_actions`/`login_events` it **does** cascade on user deletion.
- **LOGIN_EVENTS** (migration 010) — throttled one-row-per-(user, date) login stamp. Used by the - **LOGIN_EVENTS** (migration 010) — throttled one-row-per-(user, date) login stamp. Used by the
Stage-0 gate query (VISION §Stage 0: "returned and used in ≥2 distinct weeks"). Stage-0 gate query (VISION §Stage 0: "returned and used in ≥2 distinct weeks").
@@ -209,7 +238,10 @@ enforcement dormant); **ADR-012 opened Stage 1 and turned enforcement on in the
Enforcement is **Postgres Row-Level Security** (migration `003_rls.up.sql`): Enforcement is **Postgres Row-Level Security** (migration `003_rls.up.sql`):
- RLS is `ENABLE`d **and** `FORCE`d on every user-owned table — `users`, `videos`, - RLS is `ENABLE`d **and** `FORCE`d on every user-owned table — `users`, `videos`,
`transcripts`, `summaries`, `summary_actions`, `video_connections`, `channel_errors`. `summaries`, `summary_actions`, `video_connections`, `channel_errors`, and (migration 017)
`uploaded_transcripts`. `transcripts` is **not** one of them: ADR-021 made it shared public
content, and migration 003's original per-user `transcripts` policy was dropped with that
table in migration 015.
`FORCE` is load-bearing: the app connects as the table **owner** (`tapir` role), and owners `FORCE` is load-bearing: the app connects as the table **owner** (`tapir` role), and owners
bypass RLS unless forced. bypass RLS unless forced.
- Each policy keys off the per-request GUC `tapir.current_user_id`, set transaction-locally by - Each policy keys off the per-request GUC `tapir.current_user_id`, set transaction-locally by
+22
View File
@@ -12,6 +12,22 @@ it** — endpoints and aliases drift, and this file is a snapshot (2026-06-06),
- **Off-cluster (host / NodePort):** `http://koala:30401/v1/` — service `litellm-nodeport` (`4000:30401`). - **Off-cluster (host / NodePort):** `http://koala:30401/v1/` — service `litellm-nodeport` (`4000:30401`).
- NOTE: the earlier `31234` was **wrong** — `31234` is the `llama-swap` NodePort (`8080:31234`), - NOTE: the earlier `31234` was **wrong** — `31234` is the `llama-swap` NodePort (`8080:31234`),
not LiteLLM. Use `30401` for off-cluster LiteLLM. `confirm` resolved. not LiteLLM. Use `30401` for off-cluster LiteLLM. `confirm` resolved.
- **Speech-to-text (ADR-033):** the same gateway serves `POST /v1/audio/transcriptions` for
`iguana/kb-whisper` (primary, local) and `berget/kb-whisper` (fallback, EU cloud — private audio
leaves the homelab if it is used). **Call it via the NodePort / in-cluster address above, NOT
`https://llm-api.d-ma.be`:** that public ingress (openresty) returns `413` for bodies over
roughly 8–10 MB (~5 minutes of 16 kHz WAV), while the NodePort took 174 MB (90 minutes) in one
request (measured 2026-10-01, `docs/research/audio-length-ceiling-2026-10-01.md`). Send multipart
field `file`, `response_format=verbose_json`, no `Expect: 100-continue`. iguana's whisper-server
can wedge permanently after a Metal OOM (infra#273); never branch on its
`{"error":"failed to process audio"}` string — it has had at least five causes. Tapir's knobs:
`TAPIR_UPLOADS_ENABLED` (off by default), `TAPIR_UPLOAD_MAX_MB` (200), `TAPIR_UPLOAD_MAX_MINUTES`
(90), `TAPIR_TRANSCRIBE_MODEL`, `TAPIR_TRANSCRIBE_FALLBACK_MODEL` (empty disables the cloud
fallback), `TAPIR_TRANSCRIBE_TIMEOUT` (30m), `TAPIR_UPLOAD_TMP_DIR`, `TAPIR_FFMPEG_BIN`,
`TAPIR_FFPROBE_BIN`, `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS` (90000 — the measured ceiling of
`iguana/gemma4-26b`, which silently drops the head above it), `TAPIR_UPLOAD_CHAT_MODEL`
(`iguana/gemma4-26b`; `koala/phi4-mini`'s 8,192-token context 400s on an hour of speech). Needs `ffmpeg` + `ffprobe`
in the runtime image; Tapir's own ingress body limit for a 200 MB upload is unverified (infra).
- **Auth key:** treat as a secret via the SecretStore port; do not hardcode in committed code. - **Auth key:** treat as a secret via the SecretStore port; do not hardcode in committed code.
NOTE: the documented `sk-local-123` returned `401` on `30401/v1/models` (2026-06-02) — it is NOTE: the documented `sk-local-123` returned `401` on `30401/v1/models` (2026-06-02) — it is
stale or the gateway now enforces the `LITELLM_MASTER_KEY` (1Password `op://HomeLab/LITELLM_MASTER_KEY`). stale or the gateway now enforces the `LITELLM_MASTER_KEY` (1Password `op://HomeLab/LITELLM_MASTER_KEY`).
@@ -227,6 +243,12 @@ knobs plus one load-bearing deployment constraint:
- `TAPIR_USAGE_GATE_START` — `YYYY-MM-DD`, default **`2026-06-11`** (the morning the pilot was - `TAPIR_USAGE_GATE_START` — `YYYY-MM-DD`, default **`2026-06-11`** (the morning the pilot was
unblocked and summaries started flowing). `tapir report` counts return-usage (distinct active unblocked and summaries started flowing). `tapir report` counts return-usage (distinct active
weeks, ADR-016) only from this date, so pre-launch testing and the blocked period are excluded. weeks, ADR-016) only from this date, so pre-launch testing and the blocked period are excluded.
- `TAPIR_METRICS_ADDR` — listen address for the Prometheus `/metrics` endpoint (ADR-030).
**Default `:9090`** — a SEPARATE port from `TAPIR_HTTP_ADDR` so metrics are never on the public
app; scraped in-cluster only (PodMonitor). Empty disables the metrics server. Key series:
`tapir_summarize_duration_seconds{model,outcome,fallback}`, `tapir_caption_fetch_duration_seconds{outcome}`,
`tapir_chat_duration_seconds{model}`, `tapir_llm_tokens_total{model,kind}`,
`tapir_http_request_duration_seconds{method,route}`, `tapir_logins_total`.
- `TAPIR_FETCH_RATE` — Go duration, default `2s`. The **process-wide per-egress-IP caption-fetch - `TAPIR_FETCH_RATE` — Go duration, default `2s`. The **process-wide per-egress-IP caption-fetch
rate gate** (ADR-014 item 2). Every caption fetch — scheduler runners *and* the web "Summarize" rate gate** (ADR-014 item 2). Every caption fetch — scheduler runners *and* the web "Summarize"
click-path — serialises through this one limiter so the pod cannot collectively trip 429s. `0` click-path — serialises through this one limiter so the pod cannot collectively trip 429s. `0`
@@ -0,0 +1,110 @@
# Audio length ceiling — measured before designing chunking (tapir#33, 2026-10-01)
Question (#33, "the main technical risk"): at what recording length does Swedish
transcription through the LiteLLM gateway first fail, and with what failure mode?
Measured **before** any chunking design, with iguana not holding a large ollama model,
so the result is the ceiling and not the Metal OOM of infra#273.
## Answer
**No transcription ceiling was found at or below 90 minutes.** A single request carrying
90 minutes of 16 kHz mono PCM (174 MB) transcribed in one pass in 440 s. The first thing
that actually fails is **not whisper**: it is the public gateway ingress, which rejects the
upload with `413` at roughly 5.5 minutes of WAV.
| Route | First failure | Failure mode |
|---|---|---|
| `https://llm-api.d-ma.be` (openresty ingress) | between 8.0 MB and 10.5 MB body (~4–5.5 min of WAV) | `413 Request Entity Too Large`, HTML body from `openresty`, returned instantly |
| `http://koala:30401` (LiteLLM NodePort) → `iguana/kb-whisper` | **none up to 90 min / 174 MB** | — |
| `http://koala:30401` → `berget/kb-whisper` | **none up to 22 min / 42 MB** (largest tried) | — |
The ingress limit is the reason the 8:20 / 15 MB recording in #33 "failed repeatedly" and a
30-second slice succeeded. That failure was attributed to a Metal OOM; at least in the
size-dependent form described, a body-size cap explains it without one. (The OOM
mechanism in infra#273 is separately real; this run did not provoke it.)
The exact ingress cap was bracketed (8.0 MB passed, 10.5 MB failed), not read from the
ingress configuration, which lives in `infra`. "About 10 MB" is an inference.
## Method
- **Audio.** No known-good Swedish recording is available that is safe to use: the only real
one (`IMG_1233.wav`) is the private `/bygge` recording (#31 custody still open) and was
deliberately not touched. Instead: macOS `say -v Alva` (Swedish) on iguana, a 1,171-char
five-paragraph meeting script → 82.5 s of 16 kHz mono PCM WAV (`ffmpeg -ar 16000 -ac 1`).
Concatenated ×4, ×8, ×16, ×32, ×44, ×66 with Python `wave` (the issue's own recipe).
Nothing here is a real transcript; none of it is committed.
- **Request.** `POST /v1/audio/transcriptions`, multipart `file=`, `model=iguana/kb-whisper`,
`response_format=verbose_json`, `-H 'Expect:'` (per #33's traps), master key supplied only
through `op run` and never on argv.
- **Confound control.** `ollama /api/ps` on iguana was read before every step. Resident:
nothing, or `nomic-embed-text` (0.37 GB). Never a large model.
## Results (`iguana/kb-whisper`, NodePort)
| Audio | WAV size | HTTP | Wall | Realtime factor | Segments | Chars |
|---|---|---|---|---|---|---|
| 0:30 | 0.96 MB | 200 | 2.8 s | 11× | — | — |
| 2:30 / 3:20 / 4:10 | 4.8 / 6.4 / 8.0 MB | 200 (ingress) | 12.9 / 16.0 / 19.9 s | 12× | — | — |
| 5:30 (×4) | 10.5 MB | **413 ingress**, 200 NodePort in 27 s | 27 s | 12× | 121 | 4,557 |
| 11:00 (×8) | 21.1 MB | 200 | 53 s | 12× | 241 | 9,094 |
| 22:00 (×16) | 42.3 MB | 200 | 105 s | 13× | 482 | 18,201 |
| 44:01 (×32) | 84.5 MB | 200 | 209 s | 13× | 966 | 36,438 |
| 60:32 (×44) | 116.2 MB | 200 | 291 s | 12× | 1,332 | 50,145 |
| 90:48 (×66) | 174.3 MB | 200 | 440 s | 12× | 2,005 | 75,165 |
- Throughput is flat at ~12× realtime. 90 minutes costs ~7.5 minutes of wall time. This is
a minutes-long job, not a request/response one (see ADR-033 on ADR-018).
- Segments per source repetition stay at 30.1–30.4 from ×4 to ×66, and segment times are
monotonic with `last_end` within 0.2 s of `duration`. No drift and no repetition-loop
hallucination appeared across 90 minutes. This is a weak check — the input is a repeated
loop and synthetic voice — not evidence of accuracy.
- **Only `iguana/kb-whisper`'s `x4` run reported `last_end` 360.0 s against `duration`
330.2 s.** Whisper's final segment can end past the audio; clamp to `duration`.
## Findings that change the design
1. **Use the NodePort route, not the public hostname.** Tapir runs in-cluster beside the
gateway; `llm-api.d-ma.be` is the external door and has a body cap. #33 said "use the
gateway, never the host" and named the public URL; the gateway is right, that URL is not.
`TAPIR_GATEWAY_URL` already points at `koala:30401/v1`.
2. **`response_format=srt` survives the LiteLLM hop, but wrapped.** The response is JSON
`{"text": "<the SRT>", "usage": …}`, not a bare SRT body — a client that parses the body
as SRT gets a parse error on `{`. #33 question 2 is answered: it passes through, inside
an envelope.
3. **The two backends return different timestamp granularity.** `iguana/kb-whisper`
segments are **whole seconds** (`0→1`, `1→4`, `4→7`); `berget/kb-whisper` returns real
sub-second times (`0.051→1.192`) and a different `language` token (`swedish` vs `sv`).
`verbose_json` and the default JSON were byte-identical on iguana. An adapter that
falls back between the two must normalise both, and "which endpoint served this" is
a quality fact, not just a log line.
4. **The existing chat path cannot answer about most of a long recording.** At
828 chars/min (this fixture; real speech is plausibly a little denser),
`TAPIR_MAX_TRANSCRIPT_CHARS`'s default of **18,000** is ~21.7 minutes. A 60-minute
transcript is 50 k chars, a 90-minute one 75 k. `chat.Service` truncates by keeping
the head (`internal/adapters/chat/chat.go`), so questions about the last two-thirds of
a one-hour meeting get an answer from a transcript that does not contain it. #33 says
chat "retrieves rather than generates" and therefore has no ceiling; it **stuffs and
truncates**, and its ceiling is the model context. See ADR-033.
5. **Chunking is not needed for v1 on length.** Nothing failed up to 90 minutes. The
chunking-and-stitching recipe in #33 (offset timestamps, renumber cues, strip `\r`) is
not built. It stays the answer *if* a longer cap is wanted or the OOM mechanism forces
smaller requests — see ADR-033's resolution triggers.
## What this did not measure
- **Metal OOM under contention.** Deliberately avoided. A big resident ollama model
during a long transcription is still the known way to wedge iguana permanently.
- **Real speech.** TTS is clean, single-voice, no crosstalk or room noise. WER and
speaker overlap in a Teams export are untested. Speed is probably representative;
accuracy is not claimed.
- **Compressed input.** Only WAV was sent. Whether `m4a`/`mp3`/`opus` can reach whisper
without a server-side transcode is not established here (ADR-033 treats it as needing one).
- **Concurrency.** One request at a time. Two simultaneous 90-minute jobs on iguana were
not tried.
- **`berget/kb-whisper` beyond 22 min / 42 MB.** Enough to know the fallback is not
size-capped at meeting length for 22 minutes; not proven at 90.
- **Time-of-day / other tenants.** One afternoon, one run per length.
Scripts were throwaway shell/Python in a session scratchpad and are not committed; the
method above reproduces them.
@@ -0,0 +1,66 @@
# Chat over a long transcript — what the offered models really hold (tapir#33, 2026-10-01)
Question: ADR-033 reuses `chat.Service`, which stuffs the transcript into the prompt and
cuts the tail. #33 assumed chat "retrieves rather than generates" and so has no length
ceiling. It does not retrieve, so its ceiling is the model's context. What is it, for the
models chat actually offers?
## Answer
| Model (via the NodePort gateway) | Result | Limit found |
|---|---|---|
| `koala/phi4-mini` | correct to 30,000 chars; **HTTP 400 at 50,000** (13,332 tokens vs an 8,192-token context) | ~22,000 chars in production, where the client adds `max_tokens: 1500` |
| `iguana/gemma4-26b` | correct at 18k, 50k and **90,000 chars** (26.5 k tokens); **at 120,000 chars it silently loses the head** | **90,000 chars** |
| `berget/mistral-small` | answered "the transcript does not contain it" for a fact **at the end** of an 18,000-char and a 50,000-char transcript | unreliable here, one run each — see caveats |
The dangerous one is gemma at 120,000 chars: **no error, no truncation flag** — the reported
prompt tokens fell from 26,548 (90 k chars) to 16,387 (120 k chars), and the model answered
"it does not say who the chair is" about a sentence at the very start of the transcript. The
serving layer truncated the prompt from the front. `chat.Service` reports `Truncated: false`
because *it* did not cut anything.
## What this changes
1. **The budget for an uploaded transcript is 90,000 chars, not a round 120,000.** That is the
best offered model's measured limit, ~100 minutes of speech at the fixture's density.
`TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS` defaults to it.
2. **Chat over an upload must not open on the summarizer default.** The offered list is
`[SummarizerModel, FallbackModel, CloudFallbackModel]`, so `koala/phi4-mini` is first — and
400s on an hour of speech. `TAPIR_UPLOAD_CHAT_MODEL` (default `iguana/gemma4-26b`) is what
the upload chat opens with; the switcher still lists every model.
3. **A per-model budget does not exist yet.** Someone who switches the dropdown to
`phi4-mini` on a long recording gets a loud HTTP 400, shown inline as "couldn't answer".
That fails honestly but unhelpfully. Filed as a follow-up.
4. **The "retrieval chat" trigger in ADR-033 is now concrete:** any recording past 90,000 chars
(≈100 min) cannot be answered by stuffing, for any offered model.
## Method
`POST /v1/chat/completions` through `http://koala:30401/v1`, `temperature: 0`,
`max_tokens: 200` (production sends 1500 for chat via the summary token cap), the same system
framing as `chat.Service`. The transcript is the 60-minute synthetic fixture from
`audio-length-ceiling-2026-10-01.md` truncated/repeated to N chars, with one unique sentence
planted either at the **end** ("lagret på Kvarngatan ska tömmas … av Pelle", question: who/when)
or at the **head** ("kvällens ordförande är Ingrid", question: who chairs). Needle at the
end alone would have missed the gemma failure — a serving layer that truncates from the front
keeps the tail, so the end-needle passed at 120,000 chars; only the head-needle exposed it.
Grounded = the answer contains the planted fact.
## What this did not measure
- **Real speech**: the fixture is a repeated 82-second script, so retrieval "difficulty" is
artificially easy except for the planted needle. One question per run, one run per cell.
- **`berget/mistral-small`'s failure**: it missed an end-needle at 18,000 chars where the
others succeeded. One run; could be the model, the prompt framing, or position bias. Not
investigated; do not read it as a ranking.
- **Latency under load**, **concurrent** chat while whisper is transcribing, and in particular
the hazard below.
- **`koala/qwen36-35b-a3b` or any frontier model** — not in the offered list.
## A hazard this surfaces and does not solve
`iguana/gemma4-26b` is a large ollama model on the same host as `iguana/kb-whisper`. infra#273
is precisely that a large resident ollama model can Metal-OOM whisper-server permanently. Making
gemma the default *chat* model for uploads means chatting with a transcript can load the model
that wedges the transcriber. The primary-fails-over-to-berget design limits the blast radius but
does not remove it. Recorded in ADR-033 and filed.
+63
View File
@@ -0,0 +1,63 @@
<!DOCTYPE html><html lang="en"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>tapir — A · TUI panels</title>
<style>
:root{--bg:#0d0d12;--panel:#14141c;--line:#2a2a3a;--mint:#0EF9B6;--pink:#F740A0;--purple:#7653fc;--cream:#F5E9D6;--dim:#6b6b86;--text:#d9d9e6}
*{box-sizing:border-box}
body{margin:0;background:var(--bg);color:var(--text);font:14px/1.5 ui-monospace,SFMono-Regular,Menlo,"Cascadia Code",monospace}
.wrap{max-width:760px;margin:0 auto;padding:18px 14px 60px}
header{display:flex;align-items:center;gap:10px;border-bottom:1px solid var(--line);padding-bottom:12px;margin-bottom:18px}
.brand{color:var(--mint);font-weight:700;letter-spacing:.5px}
.brand b{color:var(--cream)}
.tip{margin-left:auto;color:var(--dim);font-size:12px}
.note{color:var(--dim);font-size:12.5px;border-left:2px solid var(--purple);padding:6px 10px;margin:0 0 16px;background:#11111a}
/* lipgloss-style bordered panel */
.card{border:1px solid var(--line);border-radius:8px;background:var(--panel);padding:12px 14px;margin:0 0 12px;position:relative}
.card::before{content:"";position:absolute;left:0;top:10px;bottom:10px;width:3px;border-radius:3px;background:var(--purple)}
.card.ready::before{background:var(--mint)}
.title{color:var(--cream);font-size:15px;font-weight:600;margin:0 0 4px}
.meta{color:var(--dim);font-size:12px}
.chip{display:inline-block;border:1px solid var(--mint);color:var(--mint);border-radius:4px;padding:0 6px;font-size:11px;margin-left:6px}
.chip.q{border-color:var(--pink);color:var(--pink)}
.preview{color:var(--dim);margin:6px 0 0;font-size:13px}
/* expanded */
.exp{border-color:var(--purple)}
.exp .head{display:flex;justify-content:space-between;align-items:baseline}
.collapse{color:var(--mint);font-size:12px;text-decoration:none;border:1px solid var(--line);border-radius:4px;padding:1px 7px}
.sec h2{color:var(--mint);font-size:12px;text-transform:uppercase;letter-spacing:1px;margin:16px 0 6px;border-bottom:1px dashed var(--line);padding-bottom:3px}
.sec ul{margin:0;padding-left:18px}.sec li{margin:3px 0}
.body{color:var(--text)}
.dock{margin-top:16px;border-top:1px solid var(--line);padding-top:12px}
.ask{display:inline-block;background:linear-gradient(90deg,var(--purple),var(--pink));color:#fff;border:0;border-radius:6px;padding:7px 12px;font:inherit;font-size:13px;cursor:pointer}
pre.tapir{margin:0;color:var(--mint);font-size:10px;line-height:1.05}
</style></head><body><div class="wrap">
<header>
<pre class="tapir"> ▄█▓▓█▄ ∩
█▓( ◕ ◕ )▓█──┘</pre>
<span class="brand">◆ <b>tapir</b> · watch less, know more</span>
<span class="tip">261 in queue</span>
</header>
<p class="note">Tapir fetches captions slowly on purpose, to respect YouTube's limits — new summaries land gradually.</p>
<div class="card ready"><div class="title">How the Attention Economy Rewires Your Brain</div>
<div class="meta">youtube · 2026-06-11 · 18 min <span class="chip">ready</span></div>
<div class="preview">A tour of the incentive loops behind infinite feeds and three concrete ways to claw back focus…</div></div>
<article class="card exp ready">
<div class="head"><div class="title">Postgres 18 — What's Actually New</div><a class="collapse" href="#">collapse ↑</a></div>
<div class="meta">youtube · 2026-06-10 · 42 min · phi4-mini</div>
<div class="sec"><h2>Takeaways</h2><ul>
<li>Async I/O cuts cold-cache read latency materially on NVMe.</li>
<li>Skip-scan makes more multicolumn indexes usable without rewrites.</li>
<li>Upgrade path is smooth; test the new planner stats first.</li></ul></div>
<div class="sec"><h2>Highlights</h2><ul>
<li>Async I/O subsystem (effective_io_concurrency now matters more).</li>
<li>B-tree skip scan for leading-column gaps.</li>
<li>Better partition-wise joins.</li></ul></div>
<div class="sec"><h2>Summary</h2><p class="body">Postgres 18 is an incremental but meaningful release: the headline is the new asynchronous I/O path, with skip-scan and planner improvements close behind. For most homelabs the upgrade is low-risk and worth it for the read-latency wins.</p></div>
<div class="dock"><button class="ask">Dig deeper — ask about this video →</button></div>
</article>
<div class="card"><div class="title">RAG is dead, right?? — Conference Talk</div>
<div class="meta">youtube · 2026-06-09 · 31 min <span class="chip q">queued</span></div></div>
</div></body></html>
+62
View File
@@ -0,0 +1,62 @@
<!DOCTYPE html><html lang="en"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>tapir — B · reader + charm accents</title>
<style>
:root{--bg:#faf7f2;--card:#fff;--ink:#1c1b22;--soft:#6a6878;--line:#e7e2d8;--mint:#0bbf8c;--purple:#6a4cf0;--pink:#e0379a}
*{box-sizing:border-box}
body{margin:0;background:var(--bg);color:var(--ink);font:16px/1.6 -apple-system,BlinkMacSystemFont,"Segoe UI",Inter,sans-serif}
.mono{font-family:ui-monospace,SFMono-Regular,Menlo,monospace}
.wrap{max-width:680px;margin:0 auto;padding:20px 16px 60px}
header{display:flex;align-items:center;gap:10px;margin-bottom:18px}
.brand{font-weight:800;font-size:18px;letter-spacing:-.3px}
.brand .dot{color:var(--mint)}
.tag{color:var(--soft);font-size:13px}
.tip{margin-left:auto;color:var(--soft);font-size:12px}
.note{color:var(--soft);font-size:13px;background:#fff;border:1px solid var(--line);border-left:3px solid var(--mint);border-radius:8px;padding:8px 12px;margin:0 0 18px}
.card{background:var(--card);border:1px solid var(--line);border-radius:12px;padding:16px 18px;margin:0 0 14px;box-shadow:0 1px 2px rgba(20,18,40,.04)}
.title{font-size:18px;font-weight:700;letter-spacing:-.2px;margin:0 0 4px;line-height:1.3}
.meta{color:var(--soft);font-size:13px}
.meta .mono{font-size:12.5px}
.chip{display:inline-block;background:rgba(11,191,140,.12);color:var(--mint);border-radius:999px;padding:1px 9px;font-size:12px;font-weight:600;margin-left:6px}
.chip.q{background:rgba(224,55,154,.12);color:var(--pink)}
.preview{color:var(--soft);margin:8px 0 0}
.exp{border-color:#d9d0ee;box-shadow:0 6px 24px rgba(106,76,240,.10)}
.head{display:flex;justify-content:space-between;align-items:baseline;gap:10px}
.collapse{color:var(--purple);font-size:13px;font-weight:600;text-decoration:none;white-space:nowrap}
.sec h2{font-size:12px;text-transform:uppercase;letter-spacing:1.2px;color:var(--purple);margin:18px 0 6px}
.sec ul{margin:0;padding-left:20px}.sec li{margin:5px 0}
.body{line-height:1.7}
.dock{margin-top:18px;border-top:1px solid var(--line);padding-top:14px}
.ask{display:inline-flex;align-items:center;gap:6px;background:var(--ink);color:#fff;border:0;border-radius:999px;padding:9px 16px;font:inherit;font-size:14px;font-weight:600;cursor:pointer}
pre.tapir{margin:0;color:var(--mint);font-size:10px;line-height:1.05}
</style></head><body><div class="wrap">
<header>
<pre class="tapir"> ▄█▓▓█▄ ∩
█▓( ◕ ◕ )▓█──┘</pre>
<span class="brand">tapir<span class="dot">.</span></span><span class="tag">watch less, know more</span>
<span class="tip mono">261 in queue</span>
</header>
<p class="note">Tapir fetches captions slowly on purpose, to respect YouTube's limits — new summaries land gradually.</p>
<div class="card"><div class="title">How the Attention Economy Rewires Your Brain</div>
<div class="meta mono">youtube · 2026-06-11 · 18 min <span class="chip">ready</span></div>
<div class="preview">A tour of the incentive loops behind infinite feeds and three concrete ways to claw back focus…</div></div>
<article class="card exp">
<div class="head"><div class="title">Postgres 18 — What's Actually New</div><a class="collapse" href="#">collapse ↑</a></div>
<div class="meta mono">youtube · 2026-06-10 · 42 min · phi4-mini</div>
<div class="sec"><h2>Takeaways</h2><ul>
<li>Async I/O cuts cold-cache read latency materially on NVMe.</li>
<li>Skip-scan makes more multicolumn indexes usable without rewrites.</li>
<li>Upgrade path is smooth; test the new planner stats first.</li></ul></div>
<div class="sec"><h2>Highlights</h2><ul>
<li>Async I/O subsystem (effective_io_concurrency now matters more).</li>
<li>B-tree skip scan for leading-column gaps.</li>
<li>Better partition-wise joins.</li></ul></div>
<div class="sec"><h2>Summary</h2><p class="body">Postgres 18 is an incremental but meaningful release: the headline is the new asynchronous I/O path, with skip-scan and planner improvements close behind. For most homelabs the upgrade is low-risk and worth it for the read-latency wins.</p></div>
<div class="dock"><button class="ask">✦ Ask about this video</button></div>
</article>
<div class="card"><div class="title">RAG is dead, right?? — Conference Talk</div>
<div class="meta mono">youtube · 2026-06-09 · 31 min <span class="chip q">queued</span></div></div>
</div></body></html>
+65
View File
@@ -0,0 +1,65 @@
<!DOCTYPE html><html lang="en"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>tapir — C · cozy terminal</title>
<style>
:root{--bg:#16131d;--card:#1e1a28;--ink:#ece7f5;--soft:#9a92b4;--line:#322b44;--mint:#2ee6b6;--purple:#9d7bff;--pink:#ff6bdb;--cream:#f3ead8}
*{box-sizing:border-box}
body{margin:0;background:radial-gradient(1200px 600px at 70% -10%,#231b33,transparent),var(--bg);color:var(--ink);font:15px/1.6 -apple-system,BlinkMacSystemFont,"Segoe UI",Inter,sans-serif}
.mono{font-family:ui-monospace,SFMono-Regular,Menlo,monospace}
.wrap{max-width:700px;margin:0 auto;padding:20px 16px 60px}
header{display:flex;align-items:center;gap:12px;margin-bottom:18px}
.brand{font-weight:800;font-size:18px}.brand .c{color:var(--mint)}
.tag{color:var(--soft);font-size:13px}
.tip{margin-left:auto;color:var(--soft);font-size:12px}
.note{color:var(--soft);font-size:13px;background:#1b1726;border:1px solid var(--line);border-radius:10px;padding:9px 12px;margin:0 0 18px}
.note b{color:var(--mint);font-weight:600}
.card{position:relative;background:var(--card);border:1px solid var(--line);border-radius:12px;padding:14px 16px 14px 18px;margin:0 0 13px;overflow:hidden}
.card::before{content:"";position:absolute;left:0;top:0;bottom:0;width:4px;background:var(--purple)}
.card.ready::before{background:linear-gradient(var(--mint),var(--purple))}
.title{font-size:16px;font-weight:700;margin:0 0 4px;color:var(--cream)}
.meta{color:var(--soft);font-size:12.5px}
.chip{display:inline-block;border-radius:999px;padding:1px 9px;font-size:11px;font-weight:600;margin-left:6px;background:rgba(46,230,182,.14);color:var(--mint)}
.chip.q{background:rgba(157,123,255,.18);color:var(--purple)}
.preview{color:var(--soft);margin:7px 0 0;font-size:14px}
.exp{border-color:#473a63;box-shadow:0 10px 40px rgba(0,0,0,.35)}
.head{display:flex;justify-content:space-between;align-items:baseline;gap:10px}
.collapse{color:var(--mint);font-size:12px;text-decoration:none;border:1px solid var(--line);border-radius:6px;padding:2px 8px;white-space:nowrap}
.sec h2{font-size:11px;text-transform:uppercase;letter-spacing:1.3px;color:var(--mint);margin:18px 0 7px;display:flex;align-items:center;gap:8px}
.sec h2::after{content:"";flex:1;height:1px;background:var(--line)}
.sec ul{margin:0;padding-left:18px}.sec li{margin:5px 0}
.body{line-height:1.7;color:var(--ink)}
.dock{margin-top:18px;border-top:1px dashed var(--line);padding-top:14px;display:flex;align-items:center;gap:10px}
.ask{display:inline-flex;align-items:center;gap:7px;background:linear-gradient(90deg,var(--purple),var(--mint));color:#10101a;border:0;border-radius:999px;padding:9px 16px;font:inherit;font-weight:700;font-size:14px;cursor:pointer}
.dockhint{color:var(--soft);font-size:12px}
pre.tapir{margin:0;color:var(--mint);font-size:11px;line-height:1.05}
</style></head><body><div class="wrap">
<header>
<pre class="tapir"> ▄█▓▓█▄ ∩
█▓( ◕ ◕ )▓█──┘</pre>
<span class="brand">tapir<span class="c">_</span></span><span class="tag">watch less, know more</span>
<span class="tip mono">261 in queue</span>
</header>
<p class="note">Tapir fetches captions slowly on purpose, to respect YouTube's limits — <b>new summaries land gradually</b>.</p>
<div class="card ready"><div class="title">How the Attention Economy Rewires Your Brain</div>
<div class="meta mono">youtube · 2026-06-11 · 18 min <span class="chip">ready</span></div>
<div class="preview">A tour of the incentive loops behind infinite feeds and three concrete ways to claw back focus…</div></div>
<article class="card exp ready">
<div class="head"><div class="title">Postgres 18 — What's Actually New</div><a class="collapse" href="#">collapse ↑</a></div>
<div class="meta mono">youtube · 2026-06-10 · 42 min · phi4-mini</div>
<div class="sec"><h2>Takeaways</h2><ul>
<li>Async I/O cuts cold-cache read latency materially on NVMe.</li>
<li>Skip-scan makes more multicolumn indexes usable without rewrites.</li>
<li>Upgrade path is smooth; test the new planner stats first.</li></ul></div>
<div class="sec"><h2>Highlights</h2><ul>
<li>Async I/O subsystem (effective_io_concurrency now matters more).</li>
<li>B-tree skip scan for leading-column gaps.</li>
<li>Better partition-wise joins.</li></ul></div>
<div class="sec"><h2>Summary</h2><p class="body">Postgres 18 is an incremental but meaningful release: the headline is the new asynchronous I/O path, with skip-scan and planner improvements close behind. For most homelabs the upgrade is low-risk and worth it for the read-latency wins.</p></div>
<div class="dock"><button class="ask">◆ Ask about this video</button><span class="dockhint mono">answers only from this video's transcript</span></div>
</article>
<div class="card"><div class="title">RAG is dead, right?? — Conference Talk</div>
<div class="meta mono">youtube · 2026-06-09 · 31 min <span class="chip q">queued</span></div></div>
</div></body></html>
+1
View File
@@ -179,4 +179,5 @@ distinguishable.
| **"Summarize now" foreground path** | Unified quiet nudge button on actionable non-summarized cards. Five explicit card states — (1) summarized: chip + no button; (2) no captions (`transcript_status = 'none'`): "No transcript available", no button; (3) queued: "Queued" chip, no button; (4) rate-limited: "Fetching soon…" + "Summarize now" → `POST /v/{id}/retry-now` (clears `rate_limited_at`, triggers engine); (5) pending: "Not summarized" + "Summarize now" → `POST /v/{id}/summarize` (queues + triggers engine). One verb, one style (`.btn-quiet`); backend difference invisible to user. Both handlers call `ProcessVideo` through `globalFetchGate`. Rate gate respected, not bypassed — this is onboarding prioritisation. | Fast onboarding value; honest dead-end for no-captions videos (no button that fails). | `internal/web/handlers.go` (`handleRetryNow`, `handleRequestSummarize`); `internal/web/views.templ` (`VideoCard`) | | **"Summarize now" foreground path** | Unified quiet nudge button on actionable non-summarized cards. Five explicit card states — (1) summarized: chip + no button; (2) no captions (`transcript_status = 'none'`): "No transcript available", no button; (3) queued: "Queued" chip, no button; (4) rate-limited: "Fetching soon…" + "Summarize now" → `POST /v/{id}/retry-now` (clears `rate_limited_at`, triggers engine); (5) pending: "Not summarized" + "Summarize now" → `POST /v/{id}/summarize` (queues + triggers engine). One verb, one style (`.btn-quiet`); backend difference invisible to user. Both handlers call `ProcessVideo` through `globalFetchGate`. Rate gate respected, not bypassed — this is onboarding prioritisation. | Fast onboarding value; honest dead-end for no-captions videos (no button that fails). | `internal/web/handlers.go` (`handleRetryNow`, `handleRequestSummarize`); `internal/web/views.templ` (`VideoCard`) |
| **Pipeline stats bar** | A one-line status bar above the video list: `N summarized · M fetching soon · K no captions`. Computed from the unfiltered row set; hidden when all videos are summarized. Gives the user a clear read on pipeline state without any interaction. | Replaces the "why is nothing happening?" confusion when most videos are pending or rate-limited. | `internal/web/view.go` (`PipelineStats`, `pipelineStats`) | | **Pipeline stats bar** | A one-line status bar above the video list: `N summarized · M fetching soon · K no captions`. Computed from the unfiltered row set; hidden when all videos are summarized. Gives the user a clear read on pipeline state without any interaction. | Replaces the "why is nothing happening?" confusion when most videos are pending or rate-limited. | `internal/web/view.go` (`PipelineStats`, `pipelineStats`) |
| **Unavailable channels (account page)** | The `/account` page shows a "Unavailable channels" section when any channels returned HTTP 404 on the last discovery pass. Lists channel name, an "unavailable" badge, and the first-seen date. Data sourced from the `channel_errors` table (migration 013). | Surfaces silent failures so users know why some subscribed channels produce no new videos. | migration 013; `internal/web/account.go`; `internal/adapters/youtube/youtube.go` (`domain.ErrChannelUnavailable`) | | **Unavailable channels (account page)** | The `/account` page shows a "Unavailable channels" section when any channels returned HTTP 404 on the last discovery pass. Lists channel name, an "unavailable" badge, and the first-seen date. Data sourced from the `channel_errors` table (migration 013). | Surfaces silent failures so users know why some subscribed channels produce no new videos. | migration 013; `internal/web/account.go`; `internal/adapters/youtube/youtube.go` (`domain.ErrChannelUnavailable`) |
| **Visual refresh — charm-reader theme + light/dark toggle (ADR-032)** | One layout in two palettes expressed as CSS custom properties: a warm "reader" light theme (sketch B) and a "cozy terminal" dark theme (sketch C). Palette is chosen in cascade order — `:root` light default, an OS-preference dark block scoped to `:root:not([data-theme])` so it only applies absent an explicit choice, and `:root[data-theme="dark"\|"light"]` set by a header toggle that outranks the media query and persists in `localStorage` (guarded; degrades to OS default). An init script in `<head>` applies the stored choice before paint (no flash). Charm touches: monospace meta lines, accent uppercase section dividers with a trailing rule, pill buttons, a lifted/accent-edged expanded card. Error/danger shades became `--err-*` tokens so they follow the theme without per-block dark overrides. Sketches kept as the design record under `docs/sketches/`. | The UI read "flat and boring"; the charm/TUI aesthetic makes it distinctive and gives a real light/dark choice rather than OS-only. | ADR-032; `docs/use-cases/visual_theme.feature`; `internal/web/visual_theme_test.go`; `internal/web/view.go` (`stylesheet`, `themeScript`), `internal/web/views.templ` (Layout/PublicLayout) |
| **Recency window + sparse-state honesty (ADR-020)** | Supersedes the copy/sort in the rows above. Auto-summarize is bounded to videos published within `TAPIR_AUTO_SUMMARIZE_WINDOW` (~7d); older un-summarized videos collapse behind a single "Show N older videos — summarize on demand" disclosure, and caption-less videos collapse to a one-line count (not N cards). List order is now `summarized-first, published_at DESC NULLS LAST`. Copy reframed for honest scarcity: pipeline bar reads "N ready · M in queue · K no captions" (no "fetching soon"); a gradual-fill note explains the rate limit; the nudge verb is "Summarize" (not "Summarize now"); the queued card says "summarizing shortly"; the empty-connected state drops the impossible `tapir run` instruction. Detail leads with Takeaways. Filters slimmed (no date pickers; hidden when empty); watched/skipped segmented; back link on detail; empty terms checkbox removed. | Make the sparse reality legible and honest instead of implying abundance/imminence; bound auto load so the back-catalogue doesn't re-drive the caption gate. Never fetch harder — scarcity is surfaced, not engineered around. | ADR-020; `2384c47`, `3df0459`, `40b703e`, `a1a5217`, `4a0a56e`, `9bf1c31`, `980638d`, `12fb031`, `f775441`, `51aa5d9` | | **Recency window + sparse-state honesty (ADR-020)** | Supersedes the copy/sort in the rows above. Auto-summarize is bounded to videos published within `TAPIR_AUTO_SUMMARIZE_WINDOW` (~7d); older un-summarized videos collapse behind a single "Show N older videos — summarize on demand" disclosure, and caption-less videos collapse to a one-line count (not N cards). List order is now `summarized-first, published_at DESC NULLS LAST`. Copy reframed for honest scarcity: pipeline bar reads "N ready · M in queue · K no captions" (no "fetching soon"); a gradual-fill note explains the rate limit; the nudge verb is "Summarize" (not "Summarize now"); the queued card says "summarizing shortly"; the empty-connected state drops the impossible `tapir run` instruction. Detail leads with Takeaways. Filters slimmed (no date pickers; hidden when empty); watched/skipped segmented; back link on detail; empty terms checkbox removed. | Make the sparse reality legible and honest instead of implying abundance/imminence; bound auto load so the back-catalogue doesn't re-drive the caption gate. Never fetch harder — scarcity is surfaced, not engineered around. | ADR-020; `2384c47`, `3df0459`, `40b703e`, `a1a5217`, `4a0a56e`, `9bf1c31`, `980638d`, `12fb031`, `f775441`, `51aa5d9` |
+101
View File
@@ -0,0 +1,101 @@
Feature: Upload an audio recording, get a private Swedish transcript, discuss it
As someone with a recorded meeting on my computer
I want to upload it and then ask questions about what was said
So that I can find out what was decided, what is open and who owes what
# ADR-033, tapir#33. The load-bearing constraint: an uploaded transcript is
# PRIVATE content, so it lives in its own RLS-scoped store and never in the
# shared public-caption `transcripts` table (ADR-021). Rejection is synchronous,
# the audio is never retained, and a failed job ends in a terminal state with a
# message — never a spinner (ADR-025). Audio only: video is #28.
Background:
Given I am a registered user with audio upload enabled
Scenario: The upload form states the limits up front
When I open the upload page
Then I see the size limit and the length limit before I choose a file
Scenario: An accepted recording is queued and I am taken to its page
When I upload an audio file within the limits
Then it is recorded as queued for me
And I am redirected to its page
Scenario: A recording with no title is named after its file
When I upload an audio file and give no title
Then its title is the filename without its extension
Scenario: A file containing video is rejected, naming issue 28
When I upload a file that has a video track
Then it is rejected with a message that video is tracked in issue 28
And nothing is recorded
Scenario: A recording over the length limit is rejected with both lengths
When I upload a recording longer than the limit
Then it is rejected stating the recording's length and the limit
And nothing is recorded
Scenario: A file over the size limit is rejected as too large
When I upload a file larger than the size limit
Then it is rejected with 413 and a message about the limit
And nothing is recorded
Scenario: An audio-only MP4 is not mistaken for video
When I upload an mp4 container that has an audio stream and no video stream
Then it is accepted
And an mp3 whose only video stream is cover art is accepted too
Scenario: A file that is not media is rejected as unreadable
When I upload a file ffprobe cannot read
Then it is rejected as the file's fault
Scenario: A broken ffprobe is reported as our fault, not as a bad file
When the prober itself fails
Then the upload is an error rather than a rejection of my file
Scenario: A recording becomes a transcript and the audio is deleted
Given I have uploaded a recording
When it has been transcribed
Then the transcript is stored privately for me
And the audio is deleted
Scenario: The endpoint that served a transcript is recorded and shown
Given a recording was transcribed by the fallback endpoint
When I view its page
Then it names the fallback endpoint, because that means the audio left the homelab
Scenario: The local endpoint is tried first and the fallback only on failure
When the primary transcription endpoint fails for any reason
Then the fallback endpoint transcribes it
But when the primary succeeds the fallback is never contacted
Scenario: A failed transcription ends in a message, never a spinner
When every transcription endpoint fails
Then the upload is failed with a typed reason
And its page shows that reason and stops polling
Scenario: A restart fails the jobs it interrupted
Given the service restarted while a recording was queued or transcribing
When it starts again
Then those uploads are failed as interrupted
And any audio left in the spool is deleted
Scenario: Only one recording is transcribed at a time
When several recordings are queued
Then they are transcribed one after another
Scenario: My uploads are invisible to other users
Given another user has uploaded a recording
When I ask for it by its id
Then I get 404 and no part of its title or text
And the chat for it calls no model
Scenario: I can ask about my recording and get a grounded answer
Given my recording has been transcribed
When I ask a question about it
Then the answer comes from my transcript through the same chat as a video's
Scenario: A long recording is not cut to its first third
Given my recording's transcript is longer than the caption default
When I ask a question about it
Then the transcript is sent with the upload budget, not the 18,000-character caption default
+37
View File
@@ -0,0 +1,37 @@
Feature: Inline-expand summary + Q&A in the list (ADR-031, #16)
As a reader skimming my summaries
I want to open a summary and its Q&A in place in the list
So that I get the full read and follow-up without leaving the list (SPA-like, no page hop)
# HTMX inline-expand, no SPA framework (ADR-031). Each scenario maps to a Go test
# in scenario_coverage_test.go (the BDD name-coverage gate).
Scenario: A summarized card expands to the full summary in place
Given a summarized video in my list
When I expand its card
Then the full summary, highlights, and takeaways are returned as an in-place card fragment, not a full page
Scenario: An expanded card collapses back to the compact card
Given an expanded card
When I collapse it
Then the compact card fragment is returned in its place
Scenario: The expanded card offers the Q&A dock
Given chat is enabled
When a summarized card is expanded
Then the expanded card includes the deeper-dive chat affordance for that video
Scenario: Only a summarized card offers expand
Given a discovered but not-yet-summarized card
When the card is rendered
Then it shows its summarize/queue footer and no expand affordance
Scenario: With JS off the card still reaches the full summary
Given a summarized card
When it is rendered
Then its expand affordance carries an href to the detail page as a no-JS fallback
Scenario: The detail page and the expanded card show the same summary
Given a summarized video
When I view it on the detail page and as an expanded card
Then both render the same summary body (one shared fragment, no drift)
+46
View File
@@ -0,0 +1,46 @@
Feature: Observability — timing and metrics for performance and UX (ADR-030, #15)
As the maintainer running Tapir for pilot users
I want timing and Prometheus metrics for the activities that drive performance and UX
So that I can see latency, model behaviour, and usage — and feed the Stage-0 eval gate
# AI metrics are the priority (ADR-030 R3). Each scenario maps to a Go test in
# test/acceptance/scenario_coverage_test.go (the BDD name-coverage gate).
Scenario: Summarization latency is recorded per endpoint
Given the summarizer runs a transcript through its endpoint chain
When an endpoint returns a parseable summary
Then the summarize latency is recorded with the model, outcome "success", and whether it was a fallback
Scenario: A failing summarizer endpoint records its failure outcome
Given the summarizer runs a transcript through its endpoint chain
When an endpoint errors or returns unparseable output
Then the summarize latency is recorded with outcome "error" or "parse_error" before the chain advances
Scenario: Caption fetch latency is recorded by outcome
Given a caption fetch is attempted for a video
When it resolves to captions, no captions, or a rate limit
Then the caption-fetch latency is recorded labelled by that outcome
Scenario: LLM token usage is recorded from the completion
Given an LLM completion returns a usage block with prompt and completion tokens
When the client finishes the call
Then the prompt and completion tokens are recorded for that model
Scenario: Q&A answer latency is recorded
Given a user asks a question about a video
When the answer is produced from the stored transcript
Then the chat answer latency is recorded for the answering model
Scenario: HTTP requests are counted by route, method, and status
Given the metrics HTTP middleware wraps the app
When a request is served against a registered route
Then it is counted and timed under the bounded route pattern, not the raw path
Scenario: A successful login is counted
Given a user completes the OIDC callback and a session is established
Then the login counter is incremented
Scenario: The metrics endpoint is not on the public app port
Given the service is running
When the public app mux is inspected
Then it exposes no /metrics route — metrics are served on the dedicated metrics port only
+29
View File
@@ -0,0 +1,29 @@
Feature: Visual refresh — one charm-reader layout, light + dark themes (ADR-032, #17)
As a reader who likes a TUI/charm aesthetic
I want a fresh look with a light and a dark theme
So that the app feels distinctive and reads well in either mode
# Direction B (light) + C (dark) are one layout in two palettes (CSS variables),
# plus a persisted toggle and an OS-preference default. Colours are reviewed via
# the mockups in docs/sketches/, not unit-tested; these scenarios cover the
# testable structure. Each maps to a Go test in scenario_coverage_test.go.
Scenario: Light and dark themes share one layout via CSS variables
Given the app stylesheet
When it is rendered
Then it defines a light palette on the root and a dark palette under data-theme="dark", with no duplicate markup
Scenario: Without a stored choice the theme follows the OS preference
Given a visitor with no saved theme
When the page loads
Then a prefers-color-scheme dark block applies the dark palette automatically
Scenario: A persisted toggle switches light and dark
Given any page
When it is rendered
Then it includes a theme-toggle control and a small script that flips data-theme and persists the choice
Scenario: The expanded card embeds the video player
Given a summarized video with a valid provider id
When its card is expanded
Then the expanded card includes the embedded video player
+12 -2
View File
@@ -1,4 +1,4 @@
module gitea.d-ma.be/mathias/tapir module git.d-ma.be/mathias/tapir
go 1.26.1 go 1.26.1
@@ -9,23 +9,33 @@ require (
github.com/go-jose/go-jose/v4 v4.1.4 github.com/go-jose/go-jose/v4 v4.1.4
github.com/golang-migrate/migrate/v4 v4.19.1 github.com/golang-migrate/migrate/v4 v4.19.1
github.com/jackc/pgx/v5 v5.9.2 github.com/jackc/pgx/v5 v5.9.2
github.com/prometheus/client_golang v1.23.2
github.com/prometheus/client_model v0.6.2
github.com/stretchr/testify v1.11.1 github.com/stretchr/testify v1.11.1
golang.org/x/crypto v0.45.0
golang.org/x/oauth2 v0.36.0 golang.org/x/oauth2 v0.36.0
golang.org/x/time v0.15.0 golang.org/x/time v0.15.0
) )
require ( require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa // indirect github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/lib/pq v1.10.9 // indirect github.com/lib/pq v1.10.9 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/common v0.66.1 // indirect
github.com/prometheus/procfs v0.16.1 // indirect
github.com/rogpeppe/go-internal v1.15.0 // indirect github.com/rogpeppe/go-internal v1.15.0 // indirect
github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 // indirect github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect
golang.org/x/sync v0.18.0 // indirect golang.org/x/sync v0.18.0 // indirect
golang.org/x/sys v0.41.0 // indirect
golang.org/x/text v0.31.0 // indirect golang.org/x/text v0.31.0 // indirect
google.golang.org/protobuf v1.36.8 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect
) )
+26 -6
View File
@@ -4,6 +4,10 @@ github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERo
github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU= github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
github.com/a-h/templ v0.3.1020 h1:ypAT/L5ySWEnZ6Zft/5yfoWXYYkhFNvEFOeeqecg4tw= github.com/a-h/templ v0.3.1020 h1:ypAT/L5ySWEnZ6Zft/5yfoWXYYkhFNvEFOeeqecg4tw=
github.com/a-h/templ v0.3.1020/go.mod h1:A2DlK61v+K+NRoGnhmYbNYVmtYHcFO5/AisMvBdDxTM= github.com/a-h/templ v0.3.1020/go.mod h1:A2DlK61v+K+NRoGnhmYbNYVmtYHcFO5/AisMvBdDxTM=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI= github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI=
github.com/containerd/errdefs v1.0.0/go.mod h1:+YBYIdtsnF4Iw6nWZhJcqGSg/dwvV7tyJ/kCkyJ2k+M= github.com/containerd/errdefs v1.0.0/go.mod h1:+YBYIdtsnF4Iw6nWZhJcqGSg/dwvV7tyJ/kCkyJ2k+M=
github.com/containerd/errdefs/pkg v0.3.0 h1:9IKJ06FvyNlexW690DXuQNx2KA2cUJXx151Xdx3ZPPE= github.com/containerd/errdefs/pkg v0.3.0 h1:9IKJ06FvyNlexW690DXuQNx2KA2cUJXx151Xdx3ZPPE=
@@ -37,8 +41,8 @@ github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang-migrate/migrate/v4 v4.19.1 h1:OCyb44lFuQfYXYLx1SCxPZQGU7mcaZ7gH9yH4jSFbBA= github.com/golang-migrate/migrate/v4 v4.19.1 h1:OCyb44lFuQfYXYLx1SCxPZQGU7mcaZ7gH9yH4jSFbBA=
github.com/golang-migrate/migrate/v4 v4.19.1/go.mod h1:CTcgfjxhaUtsLipnLoQRWCrjYXycRz/g5+RWDuYgPrE= github.com/golang-migrate/migrate/v4 v4.19.1/go.mod h1:CTcgfjxhaUtsLipnLoQRWCrjYXycRz/g5+RWDuYgPrE=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa h1:s+4MhCQ6YrzisK6hFJUX53drDT4UsSW3DEhKn0ifuHw= github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa h1:s+4MhCQ6YrzisK6hFJUX53drDT4UsSW3DEhKn0ifuHw=
github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa/go.mod h1:a/s9Lp5W7n/DD0VrVoyJ00FbP2ytTPDVOivvn2bMlds= github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa/go.mod h1:a/s9Lp5W7n/DD0VrVoyJ00FbP2ytTPDVOivvn2bMlds=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM= github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
@@ -49,10 +53,14 @@ github.com/jackc/pgx/v5 v5.9.2 h1:3ZhOzMWnR4yJ+RW1XImIPsD1aNSz4T4fyP7zlQb56hw=
github.com/jackc/pgx/v5 v5.9.2/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4= github.com/jackc/pgx/v5 v5.9.2/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo= github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0= github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk= github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw= github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw=
github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o= github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0= github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0=
@@ -61,6 +69,8 @@ github.com/moby/term v0.5.0 h1:xt8Q1nalod/v7BqbG21f8mQPqH+xAaC9C3N3wfWbVP0=
github.com/moby/term v0.5.0/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y= github.com/moby/term v0.5.0/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y=
github.com/morikuni/aec v1.0.0 h1:nP9CBfwrvYnBRgY6qfDQkygYDmYwOilePFkwzv4dU8A= github.com/morikuni/aec v1.0.0 h1:nP9CBfwrvYnBRgY6qfDQkygYDmYwOilePFkwzv4dU8A=
github.com/morikuni/aec v1.0.0/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc= github.com/morikuni/aec v1.0.0/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM= github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
github.com/opencontainers/image-spec v1.1.0 h1:8SG7/vwALn54lVB/0yZ/MMwhFrPYtpEHQb2IpWsCzug= github.com/opencontainers/image-spec v1.1.0 h1:8SG7/vwALn54lVB/0yZ/MMwhFrPYtpEHQb2IpWsCzug=
@@ -70,6 +80,14 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o=
github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9ZoGs=
github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA=
github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg=
github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is=
github.com/rogpeppe/go-internal v1.15.0 h1:D0RCU5rMAp+SpgkiNdrjfJ+LX4J1M32V2NeCY7EJ6hc= github.com/rogpeppe/go-internal v1.15.0 h1:D0RCU5rMAp+SpgkiNdrjfJ+LX4J1M32V2NeCY7EJ6hc=
github.com/rogpeppe/go-internal v1.15.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs= github.com/rogpeppe/go-internal v1.15.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
@@ -91,8 +109,8 @@ go.opentelemetry.io/otel/trace v1.37.0 h1:HLdcFNbRQBE2imdSEgm/kwqmQj1Or1l/7bW6mx
go.opentelemetry.io/otel/trace v1.37.0/go.mod h1:TlgrlQ+PtQO5XFerSPUYG0JSgGyryXewPGyayAWSBS0= go.opentelemetry.io/otel/trace v1.37.0/go.mod h1:TlgrlQ+PtQO5XFerSPUYG0JSgGyryXewPGyayAWSBS0=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
golang.org/x/crypto v0.45.0 h1:jMBrvKuj23MTlT0bQEOBcAE0mjg8mK9RXFhRH6nyF3Q= go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=
golang.org/x/crypto v0.45.0/go.mod h1:XTGrrkGJve7CYK7J8PEww4aY7gM3qMCElcJQ8n8JdX4= go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs= golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q= golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
golang.org/x/sync v0.18.0 h1:kr88TuHDroi+UVf+0hZnirlk8o8T+4MrK6mr60WkH/I= golang.org/x/sync v0.18.0 h1:kr88TuHDroi+UVf+0hZnirlk8o8T+4MrK6mr60WkH/I=
@@ -103,6 +121,8 @@ golang.org/x/text v0.31.0 h1:aC8ghyu4JhP8VojJ2lEHBnochRno1sgL6nEi9WGFGMM=
golang.org/x/text v0.31.0/go.mod h1:tKRAlv61yKIjGGHX/4tP1LTbc13YSec1pxVEWXzfoeM= golang.org/x/text v0.31.0/go.mod h1:tKRAlv61yKIjGGHX/4tP1LTbc13YSec1pxVEWXzfoeM=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U= golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno= golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc=
google.golang.org/protobuf v1.36.8/go.mod h1:fuxRtAxBytpl4zzqUh6/eyUujkJdNiuEkXntxiD/uRU=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
+45
View File
@@ -0,0 +1,45 @@
package audio
import (
"context"
"fmt"
"os"
"os/exec"
"strings"
)
// FFmpeg converts any audio container to the strict format whisper wants: 16 kHz
// mono signed 16-bit PCM WAV (ADR-033). It ALWAYS transcodes, even when the input
// already is that format — a Teams export often is, and the transcode is then
// nearly free; assuming it would be the bug. This replaces whisper-server's
// --convert, whose --tmp-dir is created at process start and rots under macOS
// periodic cleanup.
type FFmpeg struct {
// Bin is the ffmpeg executable ("ffmpeg" resolves via PATH).
Bin string
}
// ToWAV writes src, converted, to dst. Video streams are dropped (-vn): by now the
// file has already been probed as audio-only, but a cover-art stream must not end
// up in the WAV.
func (f *FFmpeg) ToWAV(ctx context.Context, src, dst string) error {
bin := f.Bin
if bin == "" {
bin = "ffmpeg"
}
cmd := exec.CommandContext(ctx, bin,
"-y", "-v", "error", "-i", src, "-vn", "-ac", "1", "-ar", "16000", "-c:a", "pcm_s16le", "-f", "wav", dst)
cmd.WaitDelay = killGrace
if out, err := cmd.CombinedOutput(); err != nil {
if ctxErr := ctx.Err(); ctxErr != nil {
return fmt.Errorf("audio: ffmpeg: %w", ctxErr)
}
return fmt.Errorf("audio: ffmpeg: %w: %s", err, strings.TrimSpace(string(out)))
}
// A zero exit that wrote nothing is still a failure: never hand whisper a
// missing file and let the error surface three layers later.
if info, err := os.Stat(dst); err != nil || info.Size() == 0 {
return fmt.Errorf("audio: ffmpeg wrote no output at %s", dst)
}
return nil
}
+60
View File
@@ -0,0 +1,60 @@
package audio_test
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/audio"
)
func TestFFmpegToWAV_AsksForSixteenKilohertzMonoPCMAlways(t *testing.T) {
// Always transcode, even when the input already is 16 kHz mono: a Teams export
// often is, and the point is to not ASSUME (ADR-033). Records the argv.
rec := filepath.Join(t.TempDir(), "argv")
bin := stub(t, `printf '%s\n' "$@" > `+rec+`
for a; do last="$a"; done
printf data > "$last"`)
err := (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/spool/in", filepath.Join(t.TempDir(), "out.wav"))
require.NoError(t, err)
raw, err := os.ReadFile(rec)
require.NoError(t, err)
argv := strings.Split(strings.TrimSpace(string(raw)), "\n")
joined := " " + strings.Join(argv, " ") + " "
for _, want := range []string{" -ac 1 ", " -ar 16000 ", " -c:a pcm_s16le ", " -vn ", " -i /spool/in "} {
assert.Contains(t, joined, want)
}
assert.True(t, strings.HasSuffix(strings.TrimSpace(joined), "out.wav"), "the output path is the last argument")
}
func TestFFmpegToWAV_NonzeroExitIsAnErrorCarryingStderr(t *testing.T) {
bin := stub(t, `echo "moov atom not found" >&2; exit 1`)
err := (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/in", filepath.Join(t.TempDir(), "o.wav"))
require.Error(t, err)
assert.Contains(t, err.Error(), "moov atom not found")
}
func TestFFmpegToWAV_ZeroExitWithNoOutputIsAnError(t *testing.T) {
bin := stub(t, `exit 0`)
err := (&audio.FFmpeg{Bin: bin}).ToWAV(context.Background(), "/in", filepath.Join(t.TempDir(), "never-written.wav"))
require.Error(t, err)
}
func TestFFmpegToWAV_MissingBinaryIsAnError(t *testing.T) {
err := (&audio.FFmpeg{Bin: filepath.Join(t.TempDir(), "no-ffmpeg")}).ToWAV(context.Background(), "/in", "/out.wav")
require.Error(t, err)
assert.False(t, errors.Is(err, context.Canceled))
}
+95
View File
@@ -0,0 +1,95 @@
// Package audio holds the adapters that inspect and convert user-supplied audio
// with ffmpeg's tools (ADR-033). They sit behind small ports so the use case is
// tested with fakes and only this package knows a subprocess is involved.
package audio
import (
"context"
"encoding/json"
"errors"
"fmt"
"os/exec"
"strconv"
"time"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
// FFprobe implements uploads.Prober by running ffprobe on the spooled file.
type FFprobe struct {
// Bin is the ffprobe executable ("ffprobe" resolves via PATH).
Bin string
}
// killGrace bounds how long Probe waits for a cancelled ffprobe's pipes to close.
const killGrace = 2 * time.Second
// Probe runs ffprobe and classifies the outcome. A non-zero exit means ffprobe
// looked at the file and could not read it: that is the FILE's fault and returns
// uploads.ErrUnreadable. A missing binary, a cancelled context or unparsable
// output is OURS and is returned as an ordinary error, so it is never reported to
// the user as their file being bad.
func (f *FFprobe) Probe(ctx context.Context, path string) (uploads.Probe, error) {
bin := f.Bin
if bin == "" {
bin = "ffprobe"
}
cmd := exec.CommandContext(ctx, bin,
"-v", "error", "-print_format", "json", "-show_format", "-show_streams", path)
cmd.WaitDelay = killGrace
out, err := cmd.Output()
if err != nil {
if ctxErr := ctx.Err(); ctxErr != nil {
return uploads.Probe{}, fmt.Errorf("audio: ffprobe: %w", ctxErr)
}
var exit *exec.ExitError
if errors.As(err, &exit) {
return uploads.Probe{}, fmt.Errorf("%w: ffprobe: %s", uploads.ErrUnreadable, exit.Stderr)
}
return uploads.Probe{}, fmt.Errorf("audio: run ffprobe: %w", err)
}
p, err := ParseProbe(out)
if err != nil {
return uploads.Probe{}, fmt.Errorf("audio: ffprobe output: %w", err)
}
return p, nil
}
// probeJSON is the subset of `ffprobe -print_format json` output that matters.
type probeJSON struct {
Streams []struct {
CodecType string `json:"codec_type"`
Disposition struct {
AttachedPic int `json:"attached_pic"`
} `json:"disposition"`
} `json:"streams"`
Format struct {
Duration string `json:"duration"`
} `json:"format"`
}
// ParseProbe turns ffprobe's JSON into a uploads.Probe. A video stream that is
// only attached cover art (an mp3/m4a with an album picture) is not video. A
// missing or "N/A" duration yields zero, which uploads.Validate rejects as
// unreadable rather than accepting an empty recording.
func ParseProbe(data []byte) (uploads.Probe, error) {
var j probeJSON
if err := json.Unmarshal(data, &j); err != nil {
return uploads.Probe{}, fmt.Errorf("decode: %w", err)
}
var p uploads.Probe
for _, s := range j.Streams {
switch s.CodecType {
case "audio":
p.HasAudio = true
case "video":
if s.Disposition.AttachedPic == 0 {
p.HasVideo = true
}
}
}
if secs, err := strconv.ParseFloat(j.Format.Duration, 64); err == nil && secs > 0 {
p.Duration = time.Duration(secs * float64(time.Second))
}
return p, nil
}
+129
View File
@@ -0,0 +1,129 @@
package audio_test
import (
"context"
"errors"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/audio"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
func TestParseProbe(t *testing.T) {
tests := []struct {
name string
json string
want uploads.Probe
}{
{
name: "audio-only m4a",
json: `{"streams":[{"codec_type":"audio","disposition":{"attached_pic":0}}],"format":{"duration":"2701.5"}}`,
want: uploads.Probe{Duration: 2701*time.Second + 500*time.Millisecond, HasAudio: true},
},
{
name: "mp4 with a real video stream",
json: `{"streams":[{"codec_type":"video","disposition":{"attached_pic":0}},{"codec_type":"audio"}],"format":{"duration":"240.0"}}`,
want: uploads.Probe{Duration: 240 * time.Second, HasAudio: true, HasVideo: true},
},
{
name: "mp3 whose only video stream is attached cover art is NOT video",
json: `{"streams":[{"codec_type":"audio"},{"codec_type":"video","disposition":{"attached_pic":1}}],"format":{"duration":"60"}}`,
want: uploads.Probe{Duration: 60 * time.Second, HasAudio: true},
},
{
name: "no streams at all",
json: `{"streams":[],"format":{"duration":"10"}}`,
want: uploads.Probe{Duration: 10 * time.Second},
},
{
name: "duration absent means zero, which Validate rejects as unreadable",
json: `{"streams":[{"codec_type":"audio"}],"format":{}}`,
want: uploads.Probe{HasAudio: true},
},
{
name: "duration reported as N/A means zero",
json: `{"streams":[{"codec_type":"audio"}],"format":{"duration":"N/A"}}`,
want: uploads.Probe{HasAudio: true},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := audio.ParseProbe([]byte(tt.json))
require.NoError(t, err)
assert.Equal(t, tt.want, got)
})
}
}
func TestParseProbe_GarbageIsAnError(t *testing.T) {
_, err := audio.ParseProbe([]byte("not json"))
assert.Error(t, err)
}
// stub writes an executable shell script standing in for ffprobe. This is a fake
// at the process boundary: the real binary is not on every host (it is not on
// koala), and what is under test is how Probe drives and interprets a process.
func stub(t *testing.T, body string) string {
t.Helper()
p := filepath.Join(t.TempDir(), "ffprobe")
require.NoError(t, os.WriteFile(p, []byte("#!/bin/sh\n"+body+"\n"), 0o755))
return p
}
func TestFFprobeProbe_ParsesWhatTheProcessPrints(t *testing.T) {
bin := stub(t, `echo '{"streams":[{"codec_type":"audio"}],"format":{"duration":"90"}}'`)
got, err := (&audio.FFprobe{Bin: bin}).Probe(context.Background(), "/ignored")
require.NoError(t, err)
assert.Equal(t, uploads.Probe{Duration: 90 * time.Second, HasAudio: true}, got)
}
func TestFFprobeProbe_PassesTheFilePathAsTheLastArgument(t *testing.T) {
out := filepath.Join(t.TempDir(), "args")
bin := stub(t, `for a; do last="$a"; done; printf '%s' "$last" > `+out+`
echo '{"streams":[{"codec_type":"audio"}],"format":{"duration":"1"}}'`)
_, err := (&audio.FFprobe{Bin: bin}).Probe(context.Background(), "/spool/upload-123")
require.NoError(t, err)
got, err := os.ReadFile(out)
require.NoError(t, err)
assert.Equal(t, "/spool/upload-123", string(got))
}
func TestFFprobeProbe_NonzeroExitMeansTheFileIsUnreadable(t *testing.T) {
bin := stub(t, `echo "Invalid data found when processing input" >&2; exit 1`)
_, err := (&audio.FFprobe{Bin: bin}).Probe(context.Background(), "/x")
assert.ErrorIs(t, err, uploads.ErrUnreadable)
}
func TestFFprobeProbe_MissingBinaryIsOurFaultNotAnUnreadableFile(t *testing.T) {
_, err := (&audio.FFprobe{Bin: filepath.Join(t.TempDir(), "no-such-ffprobe")}).Probe(context.Background(), "/x")
require.Error(t, err)
assert.False(t, errors.Is(err, uploads.ErrUnreadable),
"a missing ffprobe must not be reported to the user as their file being bad")
}
func TestFFprobeProbe_StopsWhenTheContextIsCancelled(t *testing.T) {
bin := stub(t, `sleep 30`)
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
start := time.Now()
_, err := (&audio.FFprobe{Bin: bin}).Probe(ctx, "/x")
require.Error(t, err)
assert.Less(t, time.Since(start), 5*time.Second)
assert.False(t, errors.Is(err, uploads.ErrUnreadable), "a timeout is not a verdict on the file")
}
+18 -1
View File
@@ -13,8 +13,12 @@ package chat
import ( import (
"context" "context"
"fmt" "fmt"
"log/slog"
"strings" "strings"
"time"
"unicode/utf8" "unicode/utf8"
"git.d-ma.be/mathias/tapir/internal/metrics"
) )
// Completer is the minimal LLM chat surface the Service needs. *llm.Client // Completer is the minimal LLM chat surface the Service needs. *llm.Client
@@ -37,6 +41,11 @@ type Request struct {
Transcript string Transcript string
History []Turn History []Turn
Question string Question string
// MaxChars, when positive, overrides the Service's transcript budget for this
// request. The Service default is sized for YouTube captions (~22 minutes of
// speech); an uploaded recording is much longer, and truncation keeps only the
// head (ADR-033). Zero or negative means "use the Service default".
MaxChars int
} }
// Reply is the model's answer plus whether the transcript was bounded to fit the // Reply is the model's answer plus whether the transcript was bounded to fit the
@@ -109,14 +118,22 @@ func (s *Service) Answer(ctx context.Context, req Request) (Reply, error) {
model = s.DefaultModel("") model = s.DefaultModel("")
} }
transcript, truncated := truncate(req.Transcript, s.maxChars) budget := s.maxChars
if req.MaxChars > 0 {
budget = req.MaxChars
}
transcript, truncated := truncate(req.Transcript, budget)
system := buildSystem(transcript, truncated) system := buildSystem(transcript, truncated)
user := buildUser(req.History, req.Question) user := buildUser(req.History, req.Question)
start := time.Now()
out, err := s.newClient(model).Complete(ctx, system, user) out, err := s.newClient(model).Complete(ctx, system, user)
if err != nil { if err != nil {
return Reply{}, fmt.Errorf("chat: %s: %w", model, err) return Reply{}, fmt.Errorf("chat: %s: %w", model, err)
} }
dur := time.Since(start)
metrics.ObserveChat(model, dur)
slog.Default().Info("chat answer", "model", model, "elapsed_ms", dur.Milliseconds())
answer := strings.TrimSpace(out) answer := strings.TrimSpace(out)
if answer == "" { if answer == "" {
return Reply{}, fmt.Errorf("chat: %s returned an empty answer", model) return Reply{}, fmt.Errorf("chat: %s returned an empty answer", model)
+52
View File
@@ -0,0 +1,52 @@
package chat
import (
"context"
"strings"
"testing"
)
// A Request may carry its own transcript budget. The Service's cap (18,000 chars by
// default) is sized for YouTube captions; an uploaded recording is much longer
// (~50,000 chars an hour), and keeping only the head of it would answer questions
// about the last two-thirds of a meeting from text that does not contain them
// (ADR-033). The per-request value overrides; zero means "the service default".
func TestAnswerMaxCharsOverridesTheServiceCap(t *testing.T) {
tests := []struct {
name string
serviceCap int
requestCap int
transcriptLen int
wantKept int
wantTruncated bool
}{
{"request cap above the service cap keeps the whole transcript", 10, 100, 50, 50, false},
{"request cap below the service cap bounds tighter", 100, 10, 50, 10, true},
{"zero request cap falls back to the service cap", 20, 0, 50, 20, true},
{"negative request cap is treated as unset", 20, -1, 50, 20, true},
{"request cap exceeded is reported as truncated", 10, 30, 50, 30, true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
f := &factory{answer: "answer"}
s := New(f.make, []string{"koala/phi4-mini"}, tt.serviceCap)
reply, err := s.Answer(context.Background(), Request{
Model: "koala/phi4-mini",
Transcript: strings.Repeat("x", tt.transcriptLen),
Question: "what was decided?",
MaxChars: tt.requestCap,
})
if err != nil {
t.Fatalf("Answer: %v", err)
}
if got := strings.Count(f.used.lastSystem, "x"); got != tt.wantKept {
t.Errorf("transcript chars sent = %d, want %d", got, tt.wantKept)
}
if reply.Truncated != tt.wantTruncated {
t.Errorf("Truncated = %v, want %v", reply.Truncated, tt.wantTruncated)
}
})
}
}
+16
View File
@@ -32,6 +32,7 @@ type Client struct {
model string model string
maxTokens int maxTokens int
httpClient *http.Client httpClient *http.Client
usageHook func(model string, prompt, completion int)
} }
// Option configures a Client at construction. Variadic so the existing 4-arg // Option configures a Client at construction. Variadic so the existing 4-arg
@@ -50,6 +51,14 @@ func WithMaxTokens(n int) Option {
} }
} }
// WithUsageHook registers a callback fired after a successful completion with the
// model and the prompt/completion token counts from the response usage block. It
// keeps this copied, stdlib-only package (ADR-004) decoupled from metrics: the
// caller wires it to internal/metrics, the client imports nothing. nil is ignored.
func WithUsageHook(fn func(model string, prompt, completion int)) Option {
return func(c *Client) { c.usageHook = fn }
}
// New constructs a Client. // New constructs a Client.
func New(baseURL, apiKey, model string, timeout time.Duration, opts ...Option) *Client { func New(baseURL, apiKey, model string, timeout time.Duration, opts ...Option) *Client {
c := &Client{ c := &Client{
@@ -81,6 +90,10 @@ type chatResponse struct {
Choices []struct { Choices []struct {
Message message `json:"message"` Message message `json:"message"`
} `json:"choices"` } `json:"choices"`
Usage struct {
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
} `json:"usage"`
} }
// Complete sends a system + user message and returns the assistant's reply. // Complete sends a system + user message and returns the assistant's reply.
@@ -152,5 +165,8 @@ func (c *Client) Complete(ctx context.Context, system, user string) (string, err
if len(cr.Choices) == 0 { if len(cr.Choices) == 0 {
return "", fmt.Errorf("LLM returned no choices") return "", fmt.Errorf("LLM returned no choices")
} }
if c.usageHook != nil {
c.usageHook(c.model, cr.Usage.PromptTokens, cr.Usage.CompletionTokens)
}
return cr.Choices[0].Message.Content, nil return cr.Choices[0].Message.Content, nil
} }
+24
View File
@@ -85,6 +85,30 @@ func TestClient_WithMaxTokens(t *testing.T) {
} }
} }
// TestClient_UsageHookRecordsTokens: the usage hook fires with the model and the
// prompt/completion token counts parsed from the response usage block.
func TestClient_UsageHookRecordsTokens(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{
"choices": []map[string]any{{"message": map[string]any{"content": "ok"}}},
"usage": map[string]any{"prompt_tokens": 123, "completion_tokens": 45},
})
}))
defer srv.Close()
var gotModel string
var gotPrompt, gotCompletion int
c := New(srv.URL, "", "test-model", 10*time.Second, WithUsageHook(func(model string, p, comp int) {
gotModel, gotPrompt, gotCompletion = model, p, comp
}))
if _, err := c.Complete(context.Background(), "sys", "user"); err != nil {
t.Fatalf("Complete: %v", err)
}
if gotModel != "test-model" || gotPrompt != 123 || gotCompletion != 45 {
t.Errorf("usage hook got (%q, %d, %d), want (test-model, 123, 45)", gotModel, gotPrompt, gotCompletion)
}
}
func TestClient_ReturnsErrorOnNon200(t *testing.T) { func TestClient_ReturnsErrorOnNon200(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Error(w, "overloaded", http.StatusServiceUnavailable) http.Error(w, "overloaded", http.StatusServiceUnavailable)
+1 -1
View File
@@ -18,7 +18,7 @@ import (
"path/filepath" "path/filepath"
"sync" "sync"
"gitea.d-ma.be/mathias/tapir/internal/ports" "git.d-ma.be/mathias/tapir/internal/ports"
) )
// FileStore is a SecretStore backed by a single 0600 JSON file mapping opaque // FileStore is a SecretStore backed by a single 0600 JSON file mapping opaque
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"path/filepath" "path/filepath"
"testing" "testing"
"gitea.d-ma.be/mathias/tapir/internal/adapters/secrets" "git.d-ma.be/mathias/tapir/internal/adapters/secrets"
) )
func TestPutThenGet(t *testing.T) { func TestPutThenGet(t *testing.T) {
+1 -1
View File
@@ -6,7 +6,7 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
// seedUserRow inserts a bare users row (FK target for a connection) as the // seedUserRow inserts a bare users row (FK target for a connection) as the
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
const ( const (
+48 -37
View File
@@ -3,6 +3,7 @@ package store_test
import ( import (
"context" "context"
"database/sql" "database/sql"
"errors"
"os" "os"
"testing" "testing"
@@ -34,6 +35,30 @@ func fileMigrator(t *testing.T) *migrate.Migrate {
return m return m
} }
// headVersion reports the current (HEAD) schema version so a test can restore
// to it after stepping down, without hard-coding what HEAD is. Adding a
// migration on top changes HEAD but no test that uses this needs editing.
func headVersion(t *testing.T, m *migrate.Migrate) uint {
t.Helper()
v, dirty, err := m.Version()
require.NoError(t, err)
require.False(t, dirty, "schema must not be dirty")
return v
}
// migrateTo drives the schema to an exact version *by version number*, not by
// step count. This is the whole point of the migrate-test design: a migration
// added above the target does not shift any count here, so unrelated tests stay
// green (see issue #8). ErrNoChange (already at that version) is not a failure.
func migrateTo(t *testing.T, m *migrate.Migrate, version uint) {
t.Helper()
err := m.Migrate(version)
if errors.Is(err, migrate.ErrNoChange) {
return
}
require.NoError(t, err)
}
// loginEventsExists reports whether the login_events relation is present. // loginEventsExists reports whether the login_events relation is present.
func loginEventsExists(t *testing.T) bool { func loginEventsExists(t *testing.T) bool {
t.Helper() t.Helper()
@@ -53,25 +78,15 @@ func TestMigration010LoginEventsUpDown(t *testing.T) {
require.True(t, loginEventsExists(t), "login_events must exist at latest migration") require.True(t, loginEventsExists(t), "login_events must exist at latest migration")
m := fileMigrator(t) m := fileMigrator(t)
// 011..016 sit above 010; step them down first so 010 is exercised in isolation. head := headVersion(t, m)
require.NoError(t, m.Steps(-1), "down 016 drops channel_caption_state, login_events intact")
require.True(t, loginEventsExists(t), "016 down leaves login_events intact")
require.NoError(t, m.Steps(-1), "down 015 reshapes transcripts, login_events intact")
require.True(t, loginEventsExists(t), "015 down leaves login_events intact")
require.NoError(t, m.Steps(-1), "down 014 drops channel_title, login_events intact")
require.True(t, loginEventsExists(t), "014 down leaves login_events intact")
require.NoError(t, m.Steps(-1), "down 013 drops channel_errors, login_events intact")
require.True(t, loginEventsExists(t), "013 down leaves login_events intact")
require.NoError(t, m.Steps(-1), "down 012 is a no-op, login_events intact")
require.True(t, loginEventsExists(t), "012 down leaves login_events intact")
require.NoError(t, m.Steps(-1), "down 011 must not touch login_events")
require.True(t, loginEventsExists(t), "011 down leaves login_events intact")
require.NoError(t, m.Steps(-1), "down 010 must drop login_events") migrateTo(t, m, 9) // just below 010 — everything above steps down
require.False(t, loginEventsExists(t), "login_events must be gone after the down migration") require.False(t, loginEventsExists(t), "login_events must be gone after the down migration")
require.NoError(t, m.Steps(7), "up must recreate 010 then re-apply 011..016") migrateTo(t, m, 10) // up 010
require.True(t, loginEventsExists(t), "login_events must be restored after the up migration") require.True(t, loginEventsExists(t), "login_events must be restored after the up migration")
migrateTo(t, m, head) // restore to HEAD for sibling tests
} }
// autoSummarizeDefault reads the users.auto_summarize column default as text // autoSummarizeDefault reads the users.auto_summarize column default as text
@@ -93,21 +108,15 @@ func TestMigration011AutoSummarizeDefaultUpDown(t *testing.T) {
require.Equal(t, "true", autoSummarizeDefault(t), "011 sets the default to TRUE") require.Equal(t, "true", autoSummarizeDefault(t), "011 sets the default to TRUE")
m := fileMigrator(t) m := fileMigrator(t)
require.NoError(t, m.Steps(-1), "down 016 drops channel_caption_state") head := headVersion(t, m)
require.NoError(t, m.Steps(-1), "down 015 reshapes transcripts")
require.NoError(t, m.Steps(-1), "down 014 drops channel_title") migrateTo(t, m, 10) // just below 011 — reverts the column default
require.NoError(t, m.Steps(-1), "down 013 drops channel_errors")
require.NoError(t, m.Steps(-1), "down 012 is a no-op")
require.NoError(t, m.Steps(-1), "down 011 reverts the column default")
require.Equal(t, "false", autoSummarizeDefault(t), "default is FALSE after the down migration") require.Equal(t, "false", autoSummarizeDefault(t), "default is FALSE after the down migration")
require.NoError(t, m.Steps(1), "up 011 re-applies the TRUE default") migrateTo(t, m, 11) // up 011 re-applies the TRUE default
require.Equal(t, "true", autoSummarizeDefault(t)) require.Equal(t, "true", autoSummarizeDefault(t))
require.NoError(t, m.Steps(1), "up 012 runs clean (no FORCE RLS on fresh schema)")
require.NoError(t, m.Steps(1), "up 013 creates channel_errors") migrateTo(t, m, head) // restore to HEAD for sibling tests
require.NoError(t, m.Steps(1), "up 014 recreates channel_title")
require.NoError(t, m.Steps(1), "up 015 reshapes transcripts to shared")
require.NoError(t, m.Steps(1), "up 016 recreates channel_caption_state (HEAD)")
} }
// channelTitleExists reports whether videos.channel_title is present. // channelTitleExists reports whether videos.channel_title is present.
@@ -127,17 +136,15 @@ func TestMigration014VideoChannelTitleUpDown(t *testing.T) {
require.True(t, channelTitleExists(t), "channel_title exists at latest migration") require.True(t, channelTitleExists(t), "channel_title exists at latest migration")
m := fileMigrator(t) m := fileMigrator(t)
require.NoError(t, m.Steps(-1), "down 016 drops channel_caption_state, channel_title intact") head := headVersion(t, m)
require.True(t, channelTitleExists(t), "016 down leaves channel_title intact")
require.NoError(t, m.Steps(-1), "down 015 reshapes transcripts, channel_title intact") migrateTo(t, m, 13) // just below 014 — drops channel_title
require.True(t, channelTitleExists(t), "015 down leaves channel_title intact")
require.NoError(t, m.Steps(-1), "down 014 must drop channel_title")
require.False(t, channelTitleExists(t), "channel_title must be gone after the down migration") require.False(t, channelTitleExists(t), "channel_title must be gone after the down migration")
require.NoError(t, m.Steps(1), "up 014 must recreate channel_title") migrateTo(t, m, 14) // up 014 recreates channel_title
require.True(t, channelTitleExists(t), "channel_title must be restored after the up migration") require.True(t, channelTitleExists(t), "channel_title must be restored after the up migration")
require.NoError(t, m.Steps(1), "up 015 restores the shared transcripts shape")
require.NoError(t, m.Steps(1), "up 016 recreates channel_caption_state (HEAD)") migrateTo(t, m, head) // restore to HEAD for sibling tests
} }
// TestMigration012FixAutoSummarizeRLS proves 012 runs cleanly and flips any // TestMigration012FixAutoSummarizeRLS proves 012 runs cleanly and flips any
@@ -148,7 +155,11 @@ func TestMigration012FixAutoSummarizeRLS(t *testing.T) {
// Round-trip: down 012, then up 012 — must be idempotent. // Round-trip: down 012, then up 012 — must be idempotent.
m := fileMigrator(t) m := fileMigrator(t)
require.NoError(t, m.Steps(-1), "down 012 must not error") head := headVersion(t, m)
require.NoError(t, m.Steps(1), "up 012 must re-apply cleanly")
migrateTo(t, m, 11) // down 012 must not error
migrateTo(t, m, 12) // up 012 must re-apply cleanly
require.Equal(t, "true", autoSummarizeDefault(t), "default still TRUE after 012 re-applied") require.Equal(t, "true", autoSummarizeDefault(t), "default still TRUE after 012 re-applied")
migrateTo(t, m, head) // restore to HEAD for sibling tests
} }
@@ -0,0 +1,3 @@
-- Drops the private uploaded-transcript store and every transcript in it. There
-- is no revert of the data: the original audio was never kept (ADR-033).
DROP TABLE uploaded_transcripts;
@@ -0,0 +1,48 @@
-- Migration 017: uploaded_transcripts — PRIVATE transcripts of audio a user
-- uploaded themselves (ADR-033, tapir#33).
--
-- This is deliberately NOT the shared `transcripts` table (migration 015,
-- ADR-021). That table is non-RLS on the explicit grounds that it holds only
-- public caption content and nothing user-identifying. A recorded meeting is the
-- opposite: private third-party content owned by exactly one user. Reusing
-- `transcripts` — or adding a user_id to it with a partial policy — would put a
-- private row in the one table the isolation test asserts is shared, and would
-- pass CI green. A separate table keeps ADR-021's exception singular.
--
-- Row-level security follows migration 003 exactly: ENABLE + FORCE (the app
-- connects as the table owner, which bypasses RLS unless forced), and a policy
-- keyed off the per-request GUC tapir.current_user_id. With no GUC set the
-- predicate is NULL and the table is deny-all.
--
-- The original audio is never stored (ADR-033): this row is all that is kept.
-- segments holds the timestamped cues; re-transcribing 90 minutes costs ~7
-- minutes, so discarding the timestamps would be irreversible.
CREATE TABLE uploaded_transcripts (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
title TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'queued'
CHECK (status IN ('queued', 'transcribing', 'done', 'failed')),
content TEXT,
segments JSONB,
endpoint TEXT, -- which backend served it (iguana | berget); a quality fact
audio_seconds DOUBLE PRECISION,
error_kind TEXT, -- typed, never a provider's error string (ADR-033)
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
finished_at TIMESTAMPTZ,
CONSTRAINT uploaded_transcripts_done_has_content
CHECK (status <> 'done' OR content IS NOT NULL),
CONSTRAINT uploaded_transcripts_failed_has_error_kind
CHECK (status <> 'failed' OR error_kind IS NOT NULL)
);
CREATE INDEX idx_uploaded_transcripts_user ON uploaded_transcripts (user_id, created_at DESC);
ALTER TABLE uploaded_transcripts ENABLE ROW LEVEL SECURITY;
ALTER TABLE uploaded_transcripts FORCE ROW LEVEL SECURITY;
CREATE POLICY uploaded_transcripts_isolation ON uploaded_transcripts
FOR ALL
USING (user_id = current_setting('tapir.current_user_id', true)::uuid);
COMMENT ON TABLE uploaded_transcripts IS
'PRIVATE per-user transcripts of user-supplied audio. RLS-scoped (ADR-012). Never merge into the shared transcripts table (ADR-021, ADR-033).';
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"github.com/jackc/pgx/v5/pgxpool" "github.com/jackc/pgx/v5/pgxpool"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
// seedVideo inserts a videos row whose id matches a summary's video_id, so the // seedVideo inserts a videos row whose id matches a summary's video_id, so the
+1 -1
View File
@@ -24,7 +24,7 @@ import (
_ "github.com/jackc/pgx/v5/stdlib" // register the "pgx" database/sql driver for migrate _ "github.com/jackc/pgx/v5/stdlib" // register the "pgx" database/sql driver for migrate
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
) )
//go:embed migrations/*.sql //go:embed migrations/*.sql
+3 -3
View File
@@ -11,9 +11,9 @@ import (
"github.com/jackc/pgx/v5/pgxpool" "github.com/jackc/pgx/v5/pgxpool"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
"gitea.d-ma.be/mathias/tapir/internal/ports" "git.d-ma.be/mathias/tapir/internal/ports"
) )
// Static check: Store satisfies the Sink port. // Static check: Store satisfies the Sink port.
@@ -8,7 +8,7 @@ import (
"github.com/jackc/pgx/v5/pgxpool" "github.com/jackc/pgx/v5/pgxpool"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
// seedBareVideo inserts a videos row with no summary, so the all-videos read and // seedBareVideo inserts a videos row with no summary, so the all-videos read and
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
) )
// GetTranscript returns the shared, stored transcript for a video keyed by the // GetTranscript returns the shared, stored transcript for a video keyed by the
@@ -6,7 +6,7 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
func TestSetTranscriptStatus_RoundTrip(t *testing.T) { func TestSetTranscriptStatus_RoundTrip(t *testing.T) {
+3 -3
View File
@@ -6,9 +6,9 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
"gitea.d-ma.be/mathias/tapir/internal/ports" "git.d-ma.be/mathias/tapir/internal/ports"
) )
// Static check: Store satisfies the shared TranscriptStore port (ADR-021). // Static check: Store satisfies the shared TranscriptStore port (ADR-021).
@@ -0,0 +1,199 @@
package store
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
"github.com/jackc/pgx/v5"
"git.d-ma.be/mathias/tapir/internal/domain"
)
// ErrUploadNotFound is returned when an uploaded transcript does not exist OR is
// not the caller's. The two are deliberately indistinguishable: a non-owner learns
// nothing about whether someone else's upload exists (ADR-033).
var ErrUploadNotFound = errors.New("store: uploaded transcript not found")
// Uploaded transcripts are PRIVATE per-user content, stored in their own
// RLS-scoped table (migration 017) — never in the shared `transcripts` (ADR-021).
// Every method goes through withUser, so row-level security confines it to the
// caller's rows; each statement ALSO names user_id in its predicate, so the
// isolation does not rest on a single mechanism and stays testable under the
// superuser connection the unit tests use.
// CreateUpload records a new upload in the queued state and returns its id.
func (s *Store) CreateUpload(ctx context.Context, userID, title string) (string, error) {
var id string
if err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
return tx.QueryRow(ctx,
`INSERT INTO uploaded_transcripts (user_id, title) VALUES ($1, $2) RETURNING id`,
userID, title).Scan(&id)
}); err != nil {
return "", fmt.Errorf("store: create upload: %w", err)
}
return id, nil
}
// StartTranscribing moves an upload to transcribing.
func (s *Store) StartTranscribing(ctx context.Context, userID, id string) error {
return s.updateUpload(ctx, userID, id, "start transcribing",
`UPDATE uploaded_transcripts SET status = 'transcribing'
WHERE id = $1 AND user_id = $2`)
}
// CompleteUpload stores the finished transcription and moves the upload to done.
func (s *Store) CompleteUpload(ctx context.Context, userID, id string, t domain.Transcription) error {
segs, err := json.Marshal(t.Segments)
if err != nil {
return fmt.Errorf("store: complete upload: marshal segments: %w", err)
}
return s.updateUpload(ctx, userID, id, "complete upload",
`UPDATE uploaded_transcripts
SET status = 'done', content = $3, segments = $4, endpoint = $5,
audio_seconds = $6, error_kind = NULL, finished_at = NOW()
WHERE id = $1 AND user_id = $2`,
t.Text, segs, t.Endpoint, t.AudioSeconds)
}
// FailUpload moves an upload to failed with a typed reason.
func (s *Store) FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error {
return s.updateUpload(ctx, userID, id, "fail upload",
`UPDATE uploaded_transcripts
SET status = 'failed', error_kind = $3, finished_at = NOW()
WHERE id = $1 AND user_id = $2`,
string(kind))
}
// updateUpload runs one UPDATE keyed on ($1 = id, $2 = userID) plus extra args,
// and maps "no row touched" to ErrUploadNotFound — which is what a write to another
// user's upload looks like under RLS (the row is invisible, not forbidden).
func (s *Store) updateUpload(ctx context.Context, userID, id, op, sql string, args ...any) error {
all := append([]any{id, userID}, args...)
return s.withUser(ctx, userID, func(tx pgx.Tx) error {
ct, err := tx.Exec(ctx, sql, all...)
if err != nil {
return fmt.Errorf("store: %s: %w", op, err)
}
if ct.RowsAffected() == 0 {
return ErrUploadNotFound
}
return nil
})
}
// GetUpload returns one of the caller's uploads, including the transcript text.
func (s *Store) GetUpload(ctx context.Context, userID, id string) (*domain.UploadedTranscript, error) {
var u domain.UploadedTranscript
err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
row := tx.QueryRow(ctx, uploadSelect+` WHERE id = $1 AND user_id = $2`, id, userID)
return scanUpload(row, &u, true)
})
if errors.Is(err, pgx.ErrNoRows) {
return nil, ErrUploadNotFound
}
if err != nil {
return nil, fmt.Errorf("store: get upload: %w", err)
}
return &u, nil
}
// ListUploads returns the caller's uploads, newest first, WITHOUT transcript text
// or segments: a list view must not drag every transcript along.
func (s *Store) ListUploads(ctx context.Context, userID string) ([]domain.UploadedTranscript, error) {
var out []domain.UploadedTranscript
err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
rows, err := tx.Query(ctx, uploadSelect+` WHERE user_id = $1 ORDER BY created_at DESC, id`, userID)
if err != nil {
return err
}
defer rows.Close()
for rows.Next() {
var u domain.UploadedTranscript
if err := scanUpload(rows, &u, false); err != nil {
return err
}
out = append(out, u)
}
return rows.Err()
})
if err != nil {
return nil, fmt.Errorf("store: list uploads: %w", err)
}
return out, nil
}
// FailInterruptedUploads is the start-up sweep. The audio of an upload is never
// retained (ADR-033), so a job that was queued or transcribing when the process
// stopped can never resume; leaving it would be a stuck spinner. Every such row,
// for every user, becomes failed/interrupted and the user re-uploads. It
// enumerates users from the un-RLS'd identity map and scopes each pass with
// withUser, exactly as scheduled discovery does (ADR-018). Returns rows failed.
func (s *Store) FailInterruptedUploads(ctx context.Context) (int, error) {
users, err := s.ListAllUsers(ctx)
if err != nil {
return 0, fmt.Errorf("store: fail interrupted uploads: %w", err)
}
total := 0
for _, u := range users {
err := s.withUser(ctx, u.UserID, func(tx pgx.Tx) error {
ct, err := tx.Exec(ctx,
`UPDATE uploaded_transcripts
SET status = 'failed', error_kind = $2, finished_at = NOW()
WHERE user_id = $1 AND status IN ('queued', 'transcribing')`,
u.UserID, string(domain.UploadErrInterrupted))
total += int(ct.RowsAffected())
return err
})
if err != nil {
return total, fmt.Errorf("store: fail interrupted uploads: user %s: %w", u.UserID, err)
}
}
return total, nil
}
const uploadSelect = `SELECT id, user_id, title, status, content, segments, endpoint,
audio_seconds, error_kind, created_at, finished_at FROM uploaded_transcripts`
// scanUpload scans one uploadSelect row. withText controls whether the transcript
// body and segments are kept (GetUpload) or dropped (ListUploads).
func scanUpload(row pgx.Row, u *domain.UploadedTranscript, withText bool) error {
var (
status, errKind, endpoint, content *string
segments []byte
seconds *float64
finished *time.Time
)
if err := row.Scan(&u.ID, &u.UserID, &u.Title, &status, &content, &segments, &endpoint,
&seconds, &errKind, &u.CreatedAt, &finished); err != nil {
return err
}
u.Status = domain.UploadStatus(deref(status))
u.Endpoint = deref(endpoint)
u.ErrorKind = domain.UploadErrorKind(deref(errKind))
if seconds != nil {
u.AudioSeconds = *seconds
}
if finished != nil {
u.FinishedAt = *finished
}
if !withText {
return nil
}
u.Content = deref(content)
if len(segments) > 0 {
if err := json.Unmarshal(segments, &u.Segments); err != nil {
return fmt.Errorf("unmarshal segments: %w", err)
}
}
return nil
}
func deref(s *string) string {
if s == nil {
return ""
}
return *s
}
@@ -0,0 +1,196 @@
package store_test
import (
"context"
"sort"
"testing"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/stretchr/testify/require"
)
// Isolation proof for ADR-033: an uploaded transcript is PRIVATE content owned by
// one user, so it lives in its own RLS-scoped table — never in the shared,
// non-RLS `transcripts` (ADR-021). These tests drive the database directly (raw
// SQL as the non-superuser `app` role) so they observe row-level security itself,
// not any application-layer WHERE clause. The superuser pool is used only to seed
// and to read pg_catalog; it bypasses RLS and so must never be the pool whose
// answers the assertions rest on.
// seedUploadedTranscript inserts one finished upload for userID as the superuser
// (RLS bypassed so both users' rows land regardless of any GUC) and returns its id.
// The text is synthetic — this repo is public, and a real transcript must never
// appear in it (#33).
func seedUploadedTranscript(t *testing.T, super *pgxpool.Pool, userID, text string) string {
t.Helper()
ctx := context.Background()
_, err := super.Exec(ctx, `INSERT INTO users (id) VALUES ($1) ON CONFLICT DO NOTHING`, userID)
require.NoError(t, err)
var id string
require.NoError(t, super.QueryRow(ctx,
`INSERT INTO uploaded_transcripts (user_id, title, status, content)
VALUES ($1, 'synthetic fixture', 'done', $2) RETURNING id`,
userID, text).Scan(&id))
return id
}
func TestUploadedTranscriptsRLS_ScopedReadsSeeOnlyOwnRows(t *testing.T) {
newStore(t)
super := rawPool(t)
resetDB(t, super)
seedUploadedTranscript(t, super, userA, "text owned by A")
seedUploadedTranscript(t, super, userB, "text owned by B")
app := appPool(t, super)
require.Equal(t, 0, scopedCount(t, app, "", "uploaded_transcripts"),
"no tapir.current_user_id must be deny-all: this proves RLS is live, not bypassed")
require.Equal(t, 1, scopedCount(t, app, userA, "uploaded_transcripts"),
"user A must see exactly its own upload")
require.Equal(t, 1, scopedCount(t, app, userB, "uploaded_transcripts"),
"user B must see exactly its own upload")
// Count alone cannot distinguish "A's row" from "B's row" — read the content.
var got string
ctx := context.Background()
tx, err := app.Begin(ctx)
require.NoError(t, err)
defer tx.Rollback(ctx) //nolint:errcheck
_, err = tx.Exec(ctx, `SELECT set_config('tapir.current_user_id', $1, true)`, userA)
require.NoError(t, err)
require.NoError(t, tx.QueryRow(ctx, `SELECT content FROM uploaded_transcripts`).Scan(&got))
require.Equal(t, "text owned by A", got, "A's scoped read must return A's text, never B's")
}
func TestUploadedTranscriptsRLS_CrossUserWritesAffectZeroRows(t *testing.T) {
newStore(t)
super := rawPool(t)
resetDB(t, super)
seedUploadedTranscript(t, super, userA, "text owned by A")
bID := seedUploadedTranscript(t, super, userB, "text owned by B")
app := appPool(t, super)
require.Equal(t, int64(0), scopedRowsAffected(t, app, userA,
`UPDATE uploaded_transcripts SET content = 'hacked' WHERE id = $1`, bID),
"user A scoped UPDATE must touch zero of B's uploads")
require.Equal(t, int64(0), scopedRowsAffected(t, app, userA,
`DELETE FROM uploaded_transcripts WHERE id = $1`, bID),
"user A scoped DELETE must touch zero of B's uploads")
var content string
require.NoError(t, super.QueryRow(context.Background(),
`SELECT content FROM uploaded_transcripts WHERE id = $1`, bID).Scan(&content))
require.Equal(t, "text owned by B", content, "B's upload must survive A's writes unchanged")
}
func TestUploadedTranscriptsRLS_InsertForAnotherUserIsRejected(t *testing.T) {
newStore(t)
super := rawPool(t)
resetDB(t, super)
seedUploadedTranscript(t, super, userA, "text owned by A")
_, err := super.Exec(context.Background(), `INSERT INTO users (id) VALUES ($1)`, userB)
require.NoError(t, err)
app := appPool(t, super)
ctx := context.Background()
tx, err := app.Begin(ctx)
require.NoError(t, err)
defer tx.Rollback(ctx) //nolint:errcheck
_, err = tx.Exec(ctx, `SELECT set_config('tapir.current_user_id', $1, true)`, userA)
require.NoError(t, err)
// Scoped as A, plant a row that claims to belong to B. The policy's WITH CHECK
// (defaulting to USING under FOR ALL) must refuse it.
_, err = tx.Exec(ctx,
`INSERT INTO uploaded_transcripts (user_id, title, status, content)
VALUES ($1, 'planted', 'done', 'planted')`, userB)
require.Error(t, err, "a row scoped as A must not be insertable on behalf of B")
require.Contains(t, err.Error(), "row-level security")
}
func TestUploadedTranscripts_RowSecurityIsEnabledForcedAndKeyedByUser(t *testing.T) {
newStore(t)
super := rawPool(t)
ctx := context.Background()
var enabled, forced bool
require.NoError(t, super.QueryRow(ctx,
`SELECT relrowsecurity, relforcerowsecurity FROM pg_class
WHERE oid = 'public.uploaded_transcripts'::regclass`).Scan(&enabled, &forced))
require.True(t, enabled, "uploaded_transcripts must have row-level security enabled")
require.True(t, forced,
"FORCE is what binds the table-owner role tapir; without it the policy is dead in production")
var userIDNotNull bool
require.NoError(t, super.QueryRow(ctx,
`SELECT is_nullable = 'NO' FROM information_schema.columns
WHERE table_name = 'uploaded_transcripts' AND column_name = 'user_id'`).Scan(&userIDNotNull))
require.True(t, userIDNotNull, "an upload with no owner would match no policy and be unreachable")
}
func TestUploadedTranscripts_DeleteUserRemovesTheirUploads(t *testing.T) {
s := newStore(t)
super := rawPool(t)
resetDB(t, super)
seedUploadedTranscript(t, super, userA, "text owned by A")
bID := seedUploadedTranscript(t, super, userB, "text owned by B")
require.NoError(t, s.DeleteUser(context.Background(), userA))
var aRows, bRows int
ctx := context.Background()
require.NoError(t, super.QueryRow(ctx,
`SELECT count(*) FROM uploaded_transcripts WHERE user_id = $1`, userA).Scan(&aRows))
require.NoError(t, super.QueryRow(ctx,
`SELECT count(*) FROM uploaded_transcripts WHERE id = $1`, bID).Scan(&bRows))
require.Equal(t, 0, aRows, "deleting a user must remove their private transcripts (ON DELETE CASCADE)")
require.Equal(t, 1, bRows, "deleting A must not touch B's transcripts")
}
// rowSecurityOffTables are the public-schema tables that deliberately have row
// security OFF. The list is exact on purpose: a NEW table added without RLS must
// fail this test and force a decision, instead of passing CI by omission — the
// failure mode ADR-033 names, where a naive reuse "leaks private content across
// tenants and passes CI green".
//
// - transcripts: shared public caption content, the single exception for CONTENT
// (ADR-021). It is the one that must never gain a private row.
// - user_identities: the Dex-subject -> user_id map, read at login before any
// user is known (migration 004). Holds no user content.
// - invitations: exists before the user does, so there is no user to scope by;
// the single-use token is the capability (migration 009).
// - schema_migrations: golang-migrate bookkeeping.
//
// #33 describes `transcripts` as "the only non-RLS table". That is true of content
// and false of tables; this test asserts the real set.
var rowSecurityOffTables = []string{
"invitations",
"schema_migrations",
"transcripts",
"user_identities",
}
func TestRowSecurityOffTablesAreExactlyTheKnownSet(t *testing.T) {
newStore(t)
super := rawPool(t)
rows, err := super.Query(context.Background(),
`SELECT c.relname FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public' AND c.relkind IN ('r', 'p') AND NOT c.relrowsecurity
ORDER BY c.relname`)
require.NoError(t, err)
defer rows.Close()
var got []string
for rows.Next() {
var name string
require.NoError(t, rows.Scan(&name))
got = append(got, name)
}
require.NoError(t, rows.Err())
want := append([]string{}, rowSecurityOffTables...)
sort.Strings(want)
require.Equal(t, want, got,
"every public table must have row security ON unless it is in rowSecurityOffTables; "+
"adding a table here is a decision that needs an ADR")
}
@@ -0,0 +1,188 @@
package store_test
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/store"
"git.d-ma.be/mathias/tapir/internal/domain"
)
// These exercise the Store API over uploaded_transcripts. The test Store connects
// as the superuser, which BYPASSES row-level security — so the cross-user cases
// here prove the query's own user_id predicate (defence in depth), while the
// database-enforced guarantee is proven separately, as the non-superuser app
// role, in uploaded_transcripts_isolation_test.go. Text is synthetic; this repo is
// public and no real transcript may appear in it (#33).
func seedUsers(t *testing.T, ids ...string) {
t.Helper()
super := rawPool(t)
resetDB(t, super)
for _, id := range ids {
_, err := super.Exec(context.Background(), `INSERT INTO users (id) VALUES ($1)`, id)
require.NoError(t, err)
}
}
func TestCreateUpload_StartsQueuedAndIsOwnedByTheCreator(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "Teams export")
require.NoError(t, err)
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, id, got.ID)
assert.Equal(t, userA, got.UserID)
assert.Equal(t, "Teams export", got.Title)
assert.Equal(t, domain.UploadQueued, got.Status)
assert.Empty(t, got.Content)
assert.True(t, got.FinishedAt.IsZero(), "an unfinished upload has no finish time")
}
func TestUploadLifecycle_CompletedKeepsTextSegmentsEndpointAndDuration(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m")
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id))
mid, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, domain.UploadTranscribing, mid.Status)
want := domain.Transcription{
Text: "God morgon allihopa. Vi går igenom läget.",
Segments: []domain.Segment{
{Start: 0.0, End: 1.5, Text: "God morgon allihopa."},
{Start: 1.5, End: 4.25, Text: "Vi går igenom läget."},
},
Endpoint: "iguana/kb-whisper",
AudioSeconds: 4.25,
}
require.NoError(t, s.CompleteUpload(ctx, userA, id, want))
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, domain.UploadDone, got.Status)
assert.Equal(t, want.Text, got.Content)
assert.Equal(t, want.Segments, got.Segments, "segment times must survive the JSONB round trip")
assert.Equal(t, "iguana/kb-whisper", got.Endpoint, "the serving endpoint is a quality fact and is recorded per row")
assert.InDelta(t, 4.25, got.AudioSeconds, 1e-9)
assert.False(t, got.FinishedAt.IsZero())
assert.Empty(t, got.ErrorKind)
}
func TestUploadLifecycle_FailedRecordsTypedKindAndNoContent(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m")
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id))
require.NoError(t, s.FailUpload(ctx, userA, id, domain.UploadErrTranscriptionFailed))
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, domain.UploadFailed, got.Status)
assert.Equal(t, domain.UploadErrTranscriptionFailed, got.ErrorKind)
assert.Empty(t, got.Content)
assert.False(t, got.FinishedAt.IsZero())
}
func TestGetUpload_AnotherUsersIDIsNotFound(t *testing.T) {
s := newStore(t)
seedUsers(t, userA, userB)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "A's meeting")
require.NoError(t, err)
_, err = s.GetUpload(ctx, userB, id)
assert.ErrorIs(t, err, store.ErrUploadNotFound,
"a non-owner must get not-found, not forbidden: existence is not disclosed")
}
func TestUploadWrites_ByAnotherUserChangeNothing(t *testing.T) {
s := newStore(t)
seedUsers(t, userA, userB)
ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "A's meeting")
require.NoError(t, err)
assert.ErrorIs(t, s.StartTranscribing(ctx, userB, id), store.ErrUploadNotFound)
assert.ErrorIs(t, s.CompleteUpload(ctx, userB, id, domain.Transcription{Text: "planted"}), store.ErrUploadNotFound)
assert.ErrorIs(t, s.FailUpload(ctx, userB, id, domain.UploadErrTranscriptionFailed), store.ErrUploadNotFound)
got, err := s.GetUpload(ctx, userA, id)
require.NoError(t, err)
assert.Equal(t, domain.UploadQueued, got.Status, "B's writes must leave A's upload exactly as it was")
assert.Empty(t, got.Content)
}
func TestListUploads_ReturnsOnlyTheCallersNewestFirstWithoutTranscriptText(t *testing.T) {
s := newStore(t)
seedUsers(t, userA, userB)
ctx := context.Background()
first, err := s.CreateUpload(ctx, userA, "first")
require.NoError(t, err)
second, err := s.CreateUpload(ctx, userA, "second")
require.NoError(t, err)
_, err = s.CreateUpload(ctx, userB, "B's")
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, first))
require.NoError(t, s.CompleteUpload(ctx, userA, first, domain.Transcription{Text: "long text body", Endpoint: "e"}))
got, err := s.ListUploads(ctx, userA)
require.NoError(t, err)
require.Len(t, got, 2, "A must see only A's uploads")
assert.Equal(t, second, got[0].ID, "newest first")
assert.Equal(t, first, got[1].ID)
assert.Empty(t, got[1].Content, "a list row must not drag the whole transcript along")
assert.Equal(t, domain.UploadDone, got[1].Status)
}
func TestFailInterruptedUploads_FailsOnlyUnfinishedRowsForEveryUser(t *testing.T) {
s := newStore(t)
seedUsers(t, userA, userB)
super := rawPool(t)
ctx := context.Background()
for subject, uid := range map[string]string{"sub-a": userA, "sub-b": userB} {
_, err := super.Exec(ctx, `INSERT INTO user_identities (dex_subject, user_id) VALUES ($1, $2)`, subject, uid)
require.NoError(t, err)
}
queued, err := s.CreateUpload(ctx, userA, "queued")
require.NoError(t, err)
running, err := s.CreateUpload(ctx, userB, "running")
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userB, running))
done, err := s.CreateUpload(ctx, userA, "done")
require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, done))
require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "kept", Endpoint: "e"}))
n, err := s.FailInterruptedUploads(ctx)
require.NoError(t, err)
assert.Equal(t, 2, n, "the queued and the transcribing row, across both users")
for _, c := range []struct{ user, id string }{{userA, queued}, {userB, running}} {
got, err := s.GetUpload(ctx, c.user, c.id)
require.NoError(t, err)
assert.Equal(t, domain.UploadFailed, got.Status)
assert.Equal(t, domain.UploadErrInterrupted, got.ErrorKind,
"audio is not retained, so a restart cannot resume: the honest state is failed/interrupted")
}
kept, err := s.GetUpload(ctx, userA, done)
require.NoError(t, err)
assert.Equal(t, domain.UploadDone, kept.Status, "a finished transcript must survive the sweep")
assert.Equal(t, "kept", kept.Content)
}
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
) )
// UpsertVideo persists a video's metadata and returns its durable store id (the // UpsertVideo persists a video's metadata and returns its durable store id (the
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
) )
func ytVideo(userID, provVideoID, title string) domain.Video { func ytVideo(userID, provVideoID, title string) domain.Video {
+11 -2
View File
@@ -13,11 +13,13 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"log/slog"
"strings" "strings"
"time" "time"
"unicode/utf8" "unicode/utf8"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/metrics"
) )
// Completer is the minimal LLM chat surface the Summarizer needs. // Completer is the minimal LLM chat surface the Summarizer needs.
@@ -95,16 +97,23 @@ func (s *Summarizer) Summarize(ctx context.Context, v domain.Video, t domain.Tra
var errs []error var errs []error
for i, ep := range s.endpoints { for i, ep := range s.endpoints {
fallback := i > 0
start := time.Now()
out, err := ep.Client.Complete(ctx, systemPrompt, user) out, err := ep.Client.Complete(ctx, systemPrompt, user)
dur := time.Since(start)
if err != nil { if err != nil {
metrics.ObserveSummarize(ep.Model, "error", fallback, dur)
errs = append(errs, fmt.Errorf("%s/%s call: %w", ep.Provider, ep.Model, err)) errs = append(errs, fmt.Errorf("%s/%s call: %w", ep.Provider, ep.Model, err))
continue continue
} }
sum, perr := s.build(v, ep, i > 0, out) sum, perr := s.build(v, ep, fallback, out)
if perr != nil { if perr != nil {
metrics.ObserveSummarize(ep.Model, "parse_error", fallback, dur)
errs = append(errs, fmt.Errorf("%s/%s output: %w", ep.Provider, ep.Model, perr)) errs = append(errs, fmt.Errorf("%s/%s output: %w", ep.Provider, ep.Model, perr))
continue continue
} }
metrics.ObserveSummarize(ep.Model, "success", fallback, dur)
slog.Default().Info("summarized", "model", ep.Model, "fallback", fallback, "elapsed_ms", dur.Milliseconds())
return sum, nil return sum, nil
} }
return domain.Summary{}, fmt.Errorf("summarize: all %d endpoint(s) failed: %w", len(s.endpoints), errors.Join(errs...)) return domain.Summary{}, fmt.Errorf("summarize: all %d endpoint(s) failed: %w", len(s.endpoints), errors.Join(errs...))
@@ -7,13 +7,36 @@ package summarizer
import ( import (
"context" "context"
"errors" "errors"
"net/http"
"net/http/httptest"
"strings" "strings"
"testing" "testing"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
"gitea.d-ma.be/mathias/tapir/internal/ports" "git.d-ma.be/mathias/tapir/internal/metrics"
"git.d-ma.be/mathias/tapir/internal/ports"
) )
// TestSummarizerRecordsMetric verifies the summarizer→metrics wiring (ADR-030)
// black-box: after a successful summarize, the public /metrics scrape shows a
// success observation for that endpoint's model.
func TestSummarizerRecordsMetric(t *testing.T) {
const model = "metrics-test-model"
s := New(Endpoint{Client: &fakeClient{reply: goodReply}, Provider: "local", Model: model}, nil)
if _, err := s.Summarize(context.Background(), testVideo(), testTranscript()); err != nil {
t.Fatalf("Summarize: %v", err)
}
rec := httptest.NewRecorder()
metrics.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/metrics", nil))
body := rec.Body.String()
if !strings.Contains(body, `tapir_summarize_duration_seconds`) ||
!strings.Contains(body, `model="`+model+`"`) ||
!strings.Contains(body, `outcome="success"`) {
t.Errorf("metrics scrape missing summarize success for %s", model)
}
}
// compile-time check: Summarizer satisfies the port. // compile-time check: Summarizer satisfies the port.
var _ ports.Summarizer = (*Summarizer)(nil) var _ ports.Summarizer = (*Summarizer)(nil)
+53
View File
@@ -0,0 +1,53 @@
package transcribe
import (
"context"
"errors"
"fmt"
"git.d-ma.be/mathias/tapir/internal/domain"
)
// Backend is one transcription endpoint. *Gateway satisfies it; Name is the value
// recorded as the serving endpoint.
type Backend interface {
Name() string
Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error)
}
// Chain tries a primary (local) backend and, if it fails for ANY reason, a
// fallback. The fallback exists because iguana can wedge permanently on a Metal
// OOM (infra#273) — which is also why a fallback is a data-egress decision: the
// berget endpoint is a cloud service, so the result records which one served it.
type Chain struct {
primary Backend
fallback Backend // nil disables the fallback
}
// NewChain builds a Chain. A nil fallback means a primary failure is the result.
func NewChain(primary, fallback Backend) *Chain {
return &Chain{primary: primary, fallback: fallback}
}
// Transcribe returns the first backend's success. It falls back on any primary
// error — transport, timeout, non-2xx, empty transcript — and never inspects a
// response body to decide. A cancelled context stops the chain: a job the user or
// shutdown cancelled must not ship its audio to a cloud endpoint as a last act.
func (c *Chain) Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error) {
t, err := c.primary.Transcribe(ctx, wavPath)
if err == nil {
t.Endpoint = c.primary.Name()
return t, nil
}
if c.fallback == nil || ctx.Err() != nil {
return domain.Transcription{}, fmt.Errorf("%s: %w", c.primary.Name(), err)
}
t, ferr := c.fallback.Transcribe(ctx, wavPath)
if ferr != nil {
return domain.Transcription{}, errors.Join(
fmt.Errorf("%s: %w", c.primary.Name(), err),
fmt.Errorf("%s: %w", c.fallback.Name(), ferr))
}
t.Endpoint = c.fallback.Name()
return t, nil
}
@@ -0,0 +1,99 @@
package transcribe_test
import (
"context"
"errors"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
"git.d-ma.be/mathias/tapir/internal/domain"
)
type fakeBackend struct {
name string
text string
err error
calls int
}
func (f *fakeBackend) Name() string { return f.name }
func (f *fakeBackend) Transcribe(_ context.Context, _ string) (domain.Transcription, error) {
f.calls++
if f.err != nil {
return domain.Transcription{}, f.err
}
return domain.Transcription{Text: f.text, Endpoint: f.name}, nil
}
func TestChain_PrimarySuccessNeverTouchesTheFallback(t *testing.T) {
primary := &fakeBackend{name: "iguana/kb-whisper", text: "primary text"}
fallback := &fakeBackend{name: "berget/kb-whisper", text: "fallback text"}
got, err := transcribe.NewChain(primary, fallback).Transcribe(context.Background(), "/a.wav")
require.NoError(t, err)
assert.Equal(t, "primary text", got.Text)
assert.Equal(t, "iguana/kb-whisper", got.Endpoint)
assert.Zero(t, fallback.calls, "audio must not leave the homelab when the local endpoint worked")
}
func TestChain_AnyPrimaryFailureFallsBackAndRecordsTheFallbackEndpoint(t *testing.T) {
tests := []struct {
name string
err error
}{
{"server error", &transcribe.StatusError{Code: 500}},
{"gateway timeout", &transcribe.StatusError{Code: 504}},
{"client error", &transcribe.StatusError{Code: 400}},
{"transport failure", errors.New("dial tcp: connection refused")},
{"empty transcript", transcribe.ErrEmptyTranscript},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
primary := &fakeBackend{name: "iguana/kb-whisper", err: tt.err}
fallback := &fakeBackend{name: "berget/kb-whisper", text: "fallback text"}
got, err := transcribe.NewChain(primary, fallback).Transcribe(context.Background(), "/a.wav")
require.NoError(t, err)
assert.Equal(t, "fallback text", got.Text)
assert.Equal(t, "berget/kb-whisper", got.Endpoint,
"the row must say berget served it: that means the audio left the homelab")
})
}
}
func TestChain_BothFailingReportsBothErrors(t *testing.T) {
primary := &fakeBackend{name: "iguana/kb-whisper", err: errors.New("iguana down")}
fallback := &fakeBackend{name: "berget/kb-whisper", err: errors.New("berget down")}
_, err := transcribe.NewChain(primary, fallback).Transcribe(context.Background(), "/a.wav")
require.Error(t, err)
assert.Contains(t, err.Error(), "iguana down")
assert.Contains(t, err.Error(), "berget down")
}
func TestChain_WithoutAFallbackAPrimaryFailureIsTheResult(t *testing.T) {
primary := &fakeBackend{name: "iguana/kb-whisper", err: errors.New("iguana down")}
_, err := transcribe.NewChain(primary, nil).Transcribe(context.Background(), "/a.wav")
require.Error(t, err)
assert.Contains(t, err.Error(), "iguana down")
}
func TestChain_ACancelledJobDoesNotFallBack(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
primary := &fakeBackend{name: "iguana/kb-whisper", err: context.Canceled}
fallback := &fakeBackend{name: "berget/kb-whisper", text: "x"}
_, err := transcribe.NewChain(primary, fallback).Transcribe(ctx, "/a.wav")
assert.ErrorIs(t, err, context.Canceled)
assert.Zero(t, fallback.calls, "a cancelled job must not ship the audio to a cloud endpoint")
}
+147
View File
@@ -0,0 +1,147 @@
// Package transcribe turns a WAV file into a timed Swedish transcript by calling
// the LiteLLM gateway's OpenAI-compatible /audio/transcriptions endpoint
// (ADR-033). A Gateway is one model alias; a Chain tries a local one and falls
// back to a second.
package transcribe
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"mime/multipart"
"net/http"
"os"
"strings"
"git.d-ma.be/mathias/tapir/internal/domain"
)
// ErrEmptyTranscript is returned when the endpoint answered 200 but with no text.
// An empty transcript is not a success: chat over it could only answer "nothing".
var ErrEmptyTranscript = errors.New("transcribe: endpoint returned an empty transcript")
// StatusError is a non-2xx answer from the gateway. Nothing may branch on the
// body — {"error":"failed to process audio"} has had at least five distinct
// causes — so only the status code is exposed for logic; Snippet is for logs.
type StatusError struct {
Code int
Snippet string
}
func (e *StatusError) Error() string {
return fmt.Sprintf("transcribe: gateway returned HTTP %d: %s", e.Code, e.Snippet)
}
// Gateway transcribes through one model alias on the gateway. baseURL is the
// gateway's /v1 root — the in-cluster NodePort, not the public hostname, whose
// ingress rejects bodies over ~8-10 MB with 413 (docs/research/audio-length-ceiling).
type Gateway struct {
baseURL string
key string
model string
client *http.Client
}
// NewGateway builds a Gateway. The client should carry no overall Timeout: a 90
// minute recording takes minutes, so the deadline belongs to the caller's context.
func NewGateway(baseURL, key, model string, client *http.Client) *Gateway {
return &Gateway{baseURL: strings.TrimRight(baseURL, "/"), key: key, model: model, client: client}
}
// Name is the model alias; it is what gets recorded as the serving endpoint.
func (g *Gateway) Name() string { return g.model }
// Transcribe uploads the WAV at path and returns the normalised transcription.
// The body is streamed from disk with a COMPUTED Content-Length: a 90 minute WAV
// is ~170 MB, so it is neither buffered in memory nor sent chunked.
func (g *Gateway) Transcribe(ctx context.Context, path string) (domain.Transcription, error) {
f, err := os.Open(path)
if err != nil {
return domain.Transcription{}, fmt.Errorf("transcribe: open audio: %w", err)
}
defer f.Close()
info, err := f.Stat()
if err != nil {
return domain.Transcription{}, fmt.Errorf("transcribe: stat audio: %w", err)
}
var prefix bytes.Buffer
mw := multipart.NewWriter(&prefix)
for _, kv := range [][2]string{{"model", g.model}, {"response_format", "verbose_json"}, {"language", "sv"}} {
if err := mw.WriteField(kv[0], kv[1]); err != nil {
return domain.Transcription{}, fmt.Errorf("transcribe: build request: %w", err)
}
}
// The multipart field is `file`; `audio_file` is rejected as "Invalid request".
if _, err := mw.CreateFormFile("file", "audio.wav"); err != nil {
return domain.Transcription{}, fmt.Errorf("transcribe: build request: %w", err)
}
suffix := []byte("\r\n--" + mw.Boundary() + "--\r\n")
body := io.MultiReader(bytes.NewReader(prefix.Bytes()), f, bytes.NewReader(suffix))
req, err := http.NewRequestWithContext(ctx, http.MethodPost, g.baseURL+"/audio/transcriptions", body)
if err != nil {
return domain.Transcription{}, fmt.Errorf("transcribe: build request: %w", err)
}
req.ContentLength = int64(prefix.Len()) + info.Size() + int64(len(suffix))
req.Header.Set("Content-Type", mw.FormDataContentType())
req.Header.Set("Authorization", "Bearer "+g.key)
// No "Expect: 100-continue": it trips the proxy for any body over 1 KB.
resp, err := g.client.Do(req)
if err != nil {
return domain.Transcription{}, fmt.Errorf("transcribe: %s: %w", g.model, err)
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode > 299 {
snip, _ := io.ReadAll(io.LimitReader(resp.Body, 200))
return domain.Transcription{}, &StatusError{Code: resp.StatusCode, Snippet: strings.TrimSpace(string(snip))}
}
t, err := parseVerbose(resp.Body)
if err != nil {
return domain.Transcription{}, fmt.Errorf("transcribe: %s: %w", g.model, err)
}
t.Endpoint = g.model
return t, nil
}
// verboseJSON is the subset of the verbose_json response used. Both backends
// return it; they differ in segment granularity (iguana: whole seconds, berget:
// sub-second) and in the language token, neither of which matters here.
type verboseJSON struct {
Text string `json:"text"`
Duration float64 `json:"duration"`
Segments []struct {
Start float64 `json:"start"`
End float64 `json:"end"`
Text string `json:"text"`
} `json:"segments"`
}
// parseVerbose normalises a verbose_json body. Whisper's final window can end past
// the audio (9.0 s for 4.25 s of audio was observed), so segment ends are clamped
// to the reported duration.
func parseVerbose(r io.Reader) (domain.Transcription, error) {
var v verboseJSON
if err := json.NewDecoder(r).Decode(&v); err != nil {
return domain.Transcription{}, fmt.Errorf("decode response: %w", err)
}
text := strings.TrimSpace(v.Text)
if text == "" {
return domain.Transcription{}, ErrEmptyTranscript
}
t := domain.Transcription{Text: text, AudioSeconds: v.Duration}
for _, s := range v.Segments {
seg := domain.Segment{Start: s.Start, End: s.End, Text: strings.TrimSpace(s.Text)}
if v.Duration > 0 {
seg.End = min(seg.End, v.Duration)
seg.Start = min(seg.Start, seg.End)
}
t.Segments = append(t.Segments, seg)
}
return t, nil
}
@@ -0,0 +1,174 @@
package transcribe_test
import (
"bytes"
"context"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
"git.d-ma.be/mathias/tapir/internal/domain"
)
// The httptest server stands in for the LiteLLM gateway: it is the process
// boundary, so a fake here is legitimate. The JSON it returns mirrors what the
// real gateway returned for iguana/kb-whisper and berget/kb-whisper on 2026-10-01
// (docs/research/audio-length-ceiling-2026-10-01.md). All text is synthetic.
const verboseJSON = `{
"text": " God morgon allihopa.\n Vi går igenom läget.\n",
"language": "swedish",
"duration": 4.25,
"segments": [
{"id": 0, "start": 0.0, "end": 1.0, "text": " God morgon allihopa."},
{"id": 1, "start": 1.0, "end": 9.0, "text": " Vi går igenom läget."}
]
}`
func wavFile(t *testing.T, content string) string {
t.Helper()
p := filepath.Join(t.TempDir(), "a.wav")
require.NoError(t, os.WriteFile(p, []byte(content), 0o600))
return p
}
type seen struct {
method, path, auth, expect, contentType string
contentLength int64
transferEncoding []string
fields map[string]string
file string
fileField string
bodyLen int
}
// serve starts a gateway fake that records the request it received and replies
// with status and body.
func serve(t *testing.T, status int, reply string, got *seen) *httptest.Server {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if got != nil {
got.method, got.path = r.Method, r.URL.Path
got.auth = r.Header.Get("Authorization")
got.expect = r.Header.Get("Expect")
got.contentType = r.Header.Get("Content-Type")
got.contentLength = r.ContentLength
got.transferEncoding = r.TransferEncoding
raw, _ := io.ReadAll(r.Body)
got.bodyLen = len(raw)
r.Body = io.NopCloser(bytes.NewReader(raw))
require.NoError(t, r.ParseMultipartForm(32<<20))
got.fields = map[string]string{}
for k, v := range r.MultipartForm.Value {
got.fields[k] = v[0]
}
for field, fhs := range r.MultipartForm.File {
got.fileField = field
f, _ := fhs[0].Open()
b, _ := io.ReadAll(f)
got.file = string(b)
}
}
w.WriteHeader(status)
_, _ = io.WriteString(w, reply)
}))
t.Cleanup(srv.Close)
return srv
}
func TestGateway_SendsTheRequestTheGatewayExpects(t *testing.T) {
var got seen
srv := serve(t, 200, verboseJSON, &got)
g := transcribe.NewGateway(srv.URL+"/v1", "secret-key", "iguana/kb-whisper", srv.Client())
_, err := g.Transcribe(context.Background(), wavFile(t, "RIFF-synthetic-pcm"))
require.NoError(t, err)
assert.Equal(t, http.MethodPost, got.method)
assert.Equal(t, "/v1/audio/transcriptions", got.path)
assert.Equal(t, "Bearer secret-key", got.auth)
assert.Equal(t, "file", got.fileField, "the multipart field is `file`; `audio_file` is rejected as 'Invalid request'")
assert.Equal(t, "RIFF-synthetic-pcm", got.file)
assert.Equal(t, "iguana/kb-whisper", got.fields["model"])
assert.Equal(t, "verbose_json", got.fields["response_format"])
assert.Equal(t, "sv", got.fields["language"])
assert.Empty(t, got.expect, "no Expect: 100-continue: it trips the proxy for any body over 1 KB")
}
func TestGateway_SendsAKnownContentLengthNotChunked(t *testing.T) {
var got seen
srv := serve(t, 200, verboseJSON, &got)
g := transcribe.NewGateway(srv.URL+"/v1", "k", "m", srv.Client())
_, err := g.Transcribe(context.Background(), wavFile(t, "0123456789"))
require.NoError(t, err)
assert.Empty(t, got.transferEncoding, "a 90-minute WAV is ~170 MB: stream it with a computed length, not chunked and not buffered")
assert.Equal(t, int64(got.bodyLen), got.contentLength)
}
func TestGateway_NormalisesTheVerboseJSONResponse(t *testing.T) {
srv := serve(t, 200, verboseJSON, nil)
g := transcribe.NewGateway(srv.URL+"/v1", "k", "iguana/kb-whisper", srv.Client())
got, err := g.Transcribe(context.Background(), wavFile(t, "x"))
require.NoError(t, err)
assert.Equal(t, "God morgon allihopa.\n Vi går igenom läget.", got.Text)
assert.Equal(t, "iguana/kb-whisper", got.Endpoint, "the serving endpoint is part of the result")
assert.InDelta(t, 4.25, got.AudioSeconds, 1e-9)
assert.Equal(t, []domain.Segment{
{Start: 0, End: 1, Text: "God morgon allihopa."},
// The final segment ended at 9.0 s for 4.25 s of audio: whisper's last
// window can run past the end. Clamped to the audio duration.
{Start: 1, End: 4.25, Text: "Vi går igenom läget."},
}, got.Segments)
}
func TestGateway_ErrorsDoNotDependOnTheBody(t *testing.T) {
// {"error":"failed to process audio"} has had at least five distinct causes, so
// no code may branch on it. Two different bodies must be treated identically.
for _, reply := range []string{`{"error":"failed to process audio"}`, `{"error":"something else entirely"}`} {
srv := serve(t, 500, reply, nil)
g := transcribe.NewGateway(srv.URL+"/v1", "k", "m", srv.Client())
_, err := g.Transcribe(context.Background(), wavFile(t, "x"))
var se *transcribe.StatusError
require.ErrorAs(t, err, &se)
assert.Equal(t, 500, se.Code)
}
}
func TestGateway_AnEmptyTranscriptIsAFailureNotASuccess(t *testing.T) {
srv := serve(t, 200, `{"text":" ","language":"swedish","duration":3,"segments":[]}`, nil)
g := transcribe.NewGateway(srv.URL+"/v1", "k", "m", srv.Client())
_, err := g.Transcribe(context.Background(), wavFile(t, "x"))
assert.ErrorIs(t, err, transcribe.ErrEmptyTranscript)
}
func TestGateway_MalformedJSONIsAnError(t *testing.T) {
srv := serve(t, 200, `not json`, nil)
g := transcribe.NewGateway(srv.URL+"/v1", "k", "m", srv.Client())
_, err := g.Transcribe(context.Background(), wavFile(t, "x"))
assert.Error(t, err)
}
func TestGateway_MissingAudioFileIsAnError(t *testing.T) {
g := transcribe.NewGateway("http://127.0.0.1:1/v1", "k", "m", http.DefaultClient)
_, err := g.Transcribe(context.Background(), filepath.Join(t.TempDir(), "nope.wav"))
assert.Error(t, err)
}
+49
View File
@@ -0,0 +1,49 @@
package transcribe_test
import (
"context"
"net/http"
"os"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/adapters/transcribe"
)
// TestGateway_LiveGateway is an opt-in check against the REAL gateway, because
// the httptest fake above can only prove what we think the gateway accepts. It
// is skipped unless TAPIR_LIVE_WAV names a 16 kHz mono WAV and the gateway
// coordinates are set:
//
// TAPIR_LIVE_WAV=/path/a.wav TAPIR_LIVE_GATEWAY_URL=http://koala:30401/v1 \
// TAPIR_LIVE_MODEL=iguana/kb-whisper op run --env-file ~/.op-env -- \
// go test ./internal/adapters/transcribe -run LiveGateway -v
//
// Use SYNTHETIC audio only: this repo is public and the gateway may be a cloud
// endpoint (#33).
func TestGateway_LiveGateway(t *testing.T) {
wav := os.Getenv("TAPIR_LIVE_WAV")
url := os.Getenv("TAPIR_LIVE_GATEWAY_URL")
if wav == "" || url == "" {
t.Skip("set TAPIR_LIVE_WAV and TAPIR_LIVE_GATEWAY_URL to run against the real gateway")
}
model := os.Getenv("TAPIR_LIVE_MODEL")
if model == "" {
model = "iguana/kb-whisper"
}
key := os.Getenv("LITELLM_MASTER_KEY")
require.NotEmpty(t, key, "LITELLM_MASTER_KEY must be set (via op run)")
ctx, cancel := context.WithTimeout(context.Background(), 25*time.Minute)
defer cancel()
got, err := transcribe.NewGateway(url, key, model, &http.Client{}).Transcribe(ctx, wav)
require.NoError(t, err)
assert.NotEmpty(t, got.Text)
assert.NotEmpty(t, got.Segments)
assert.Equal(t, model, got.Endpoint)
t.Logf("endpoint=%s audio_s=%.1f segments=%d chars=%d", got.Endpoint, got.AudioSeconds, len(got.Segments), len(got.Text))
}
+32 -1
View File
@@ -7,10 +7,13 @@ import (
"encoding/xml" "encoding/xml"
"fmt" "fmt"
"io" "io"
"log/slog"
"net/http" "net/http"
"strings" "strings"
"time"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/metrics"
) )
// defaultPlayerBaseURL is the InnerTube / watch-page host. Overridable via // defaultPlayerBaseURL is the InnerTube / watch-page host. Overridable via
@@ -43,7 +46,35 @@ const maxCaptionBytes = 16 << 20 // 16 MiB
// fetch, or an unparseable body all yield SourceNone rather than an error. Only // fetch, or an unparseable body all yield SourceNone rather than an error. Only
// genuine transport (network) faults return an error. Audio download and // genuine transport (network) faults return an error. Audio download and
// speech-to-text remain absent (ADR-007). // speech-to-text remain absent (ADR-007).
// FetchTranscript times the caption fetch and records its latency by outcome
// (ADR-030) before returning. Transport errors are surfaced to the caller and not
// recorded as an outcome (logged upstream); the three resolved outcomes
// captions|none|rate_limited are the ones that consume the scarce fetch budget.
func (a *Adapter) FetchTranscript(ctx context.Context, v domain.Video) (domain.Transcript, error) { func (a *Adapter) FetchTranscript(ctx context.Context, v domain.Video) (domain.Transcript, error) {
start := time.Now()
tr, err := a.fetchTranscript(ctx, v)
if err == nil {
dur := time.Since(start)
outcome := captionOutcome(tr.Source)
metrics.ObserveCaptionFetch(outcome, dur)
slog.Default().Info("caption fetch", "video", v.ProviderVideoID, "outcome", outcome, "elapsed_ms", dur.Milliseconds())
}
return tr, err
}
// captionOutcome maps a transcript source to the metric outcome label.
func captionOutcome(s domain.TranscriptSource) string {
switch s {
case domain.SourceCaptions:
return "captions"
case domain.SourceRateLimited:
return "rate_limited"
default:
return "none"
}
}
func (a *Adapter) fetchTranscript(ctx context.Context, v domain.Video) (domain.Transcript, error) {
client := a.plainClient() client := a.plainClient()
tracks, err := a.captionTracks(ctx, client, v.ProviderVideoID) tracks, err := a.captionTracks(ctx, client, v.ProviderVideoID)
+1 -1
View File
@@ -6,7 +6,7 @@ import (
"net/http" "net/http"
"testing" "testing"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
) )
func TestVideoByID(t *testing.T) { func TestVideoByID(t *testing.T) {
+2 -2
View File
@@ -27,8 +27,8 @@ import (
"golang.org/x/oauth2" "golang.org/x/oauth2"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
"gitea.d-ma.be/mathias/tapir/internal/ports" "git.d-ma.be/mathias/tapir/internal/ports"
) )
// defaultBaseURL is the YouTube Data API v3 root. Overridable via Config.BaseURL // defaultBaseURL is the YouTube Data API v3 root. Overridable via Config.BaseURL
+1 -1
View File
@@ -7,7 +7,7 @@ import (
"net/http/httptest" "net/http/httptest"
"testing" "testing"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
) )
// --- fakes ------------------------------------------------------------------ // --- fakes ------------------------------------------------------------------
+1 -1
View File
@@ -12,7 +12,7 @@ import (
"golang.org/x/oauth2" "golang.org/x/oauth2"
"gitea.d-ma.be/mathias/tapir/internal/auth" "git.d-ma.be/mathias/tapir/internal/auth"
) )
// fakeWriter is a TokenWriter capturing the persisted (ref, value). // fakeWriter is a TokenWriter capturing the persisted (ref, value).
+14
View File
@@ -20,6 +20,9 @@ import (
// Config is Tapir's fully-resolved runtime configuration. // Config is Tapir's fully-resolved runtime configuration.
type Config struct { type Config struct {
// UploadConfig is the audio-upload and private-transcription settings (ADR-033).
UploadConfig
// UserID is the Tapir user the run operates as. Stage 0 has exactly one. // UserID is the Tapir user the run operates as. Stage 0 has exactly one.
// It must be a UUID: it keys the UUID user_id/video_id columns (data-model). // It must be a UUID: it keys the UUID user_id/video_id columns (data-model).
UserID string UserID string
@@ -143,6 +146,11 @@ type Config struct {
// HTTPAddr is the listen address for `tapir serve` (the Stage-0 web UI). // HTTPAddr is the listen address for `tapir serve` (the Stage-0 web UI).
HTTPAddr string HTTPAddr string
// MetricsAddr is the listen address for the Prometheus /metrics endpoint
// (ADR-030). A SEPARATE port from HTTPAddr so /metrics is never exposed on the
// public app — only scraped in-cluster. Empty disables the metrics server.
MetricsAddr string
// PublicURL is the externally-reachable base URL of the deployed service, // PublicURL is the externally-reachable base URL of the deployed service,
// e.g. "https://tapir.d-ma.be". Used to build absolute links handed to humans // e.g. "https://tapir.d-ma.be". Used to build absolute links handed to humans
// (the `tapir invite` URL). No trailing slash is assumed — callers trim it. // (the `tapir invite` URL). No trailing slash is assumed — callers trim it.
@@ -178,6 +186,7 @@ const (
defaultYTConnectRedirectURL = "https://tapir.d-ma.be/oauth/youtube/callback" defaultYTConnectRedirectURL = "https://tapir.d-ma.be/oauth/youtube/callback"
defaultOAuthRedirectAddr = "localhost:8080" defaultOAuthRedirectAddr = "localhost:8080"
defaultHTTPAddr = ":8080" defaultHTTPAddr = ":8080"
defaultMetricsAddr = ":9090"
defaultFetchBackoff = time.Hour defaultFetchBackoff = time.Hour
defaultFetchRate = 2 * time.Second defaultFetchRate = 2 * time.Second
defaultPublicURL = "https://tapir.d-ma.be" defaultPublicURL = "https://tapir.d-ma.be"
@@ -209,6 +218,7 @@ func Load() (Config, error) {
SecretsFile: envOr("TAPIR_SECRETS_FILE", defaultSecretsFile()), SecretsFile: envOr("TAPIR_SECRETS_FILE", defaultSecretsFile()),
OAuthRedirectAddr: envOr("TAPIR_OAUTH_REDIRECT_ADDR", defaultOAuthRedirectAddr), OAuthRedirectAddr: envOr("TAPIR_OAUTH_REDIRECT_ADDR", defaultOAuthRedirectAddr),
HTTPAddr: envOr("TAPIR_HTTP_ADDR", defaultHTTPAddr), HTTPAddr: envOr("TAPIR_HTTP_ADDR", defaultHTTPAddr),
MetricsAddr: lookupOr("TAPIR_METRICS_ADDR", defaultMetricsAddr),
PublicURL: envOr("TAPIR_PUBLIC_URL", defaultPublicURL), PublicURL: envOr("TAPIR_PUBLIC_URL", defaultPublicURL),
OIDCIssuer: os.Getenv("TAPIR_OIDC_ISSUER"), OIDCIssuer: os.Getenv("TAPIR_OIDC_ISSUER"),
DexClientID: os.Getenv("TAPIR_DEX_CLIENT_ID"), DexClientID: os.Getenv("TAPIR_DEX_CLIENT_ID"),
@@ -313,6 +323,10 @@ func Load() (Config, error) {
} }
c.OnboardMaxVideoSeconds = onboardMax c.OnboardMaxVideoSeconds = onboardMax
if c.UploadConfig, err = loadUploadConfig(); err != nil {
return Config{}, err
}
return c, nil return c, nil
} }
+118
View File
@@ -0,0 +1,118 @@
package config
import (
"fmt"
"os"
"strconv"
"time"
)
// UploadConfig holds the settings for audio upload and private transcription
// (ADR-033). It is embedded in Config, so its fields read as c.UploadsEnabled etc.
type UploadConfig struct {
// UploadsEnabled mounts the /uploads routes and starts the worker. Off by
// default: the runtime image must contain ffmpeg and ffprobe first (ADR-033).
UploadsEnabled bool
// UploadMaxBytes caps one upload. UploadMaxDuration caps one recording; 90
// minutes is the longest length MEASURED, not a ceiling that was found, so
// raising it is a measurement, not an edit.
UploadMaxBytes int64
UploadMaxDuration time.Duration
// TranscribeModel is the primary gateway alias; TranscribeFallbackModel is the
// fallback, tried on ANY primary failure. The fallback is a cloud endpoint
// (berget): set it EMPTY to keep private audio on the homelab.
TranscribeModel string
TranscribeFallbackModel string
// TranscribeTimeout bounds one job (conversion + transcription); a 90 minute
// recording took ~7.5 minutes on iguana.
TranscribeTimeout time.Duration
// UploadTmpDir is where uploads are spooled and converted. Audio is deleted
// after transcription, so this holds at most the in-flight files.
UploadTmpDir string
FFmpegBin string
FFprobeBin string
// MaxUploadedTranscriptChars bounds the transcript sent to a chat model for an
// UPLOADED recording. The 18,000-char default for captions is ~22 minutes of
// speech; an hour of meeting is ~50,000 chars. MEASURED 2026-10-01 against the
// real models: iguana/gemma4-26b answers correctly to 90,000 chars but silently
// drops the head of a 120,000-char transcript, so 90,000 is the ceiling of the
// best model, not a round number.
MaxUploadedTranscriptChars int
// UploadChatModel is the model chat over an uploaded recording opens with. It is
// NOT the summarizer default: koala/phi4-mini has an 8,192-token context and
// returns HTTP 400 on an hour of speech (~13,000 tokens).
UploadChatModel string
}
const (
defaultUploadMaxMB = 200
defaultUploadMaxMinutes = 90
defaultTranscribeModel = "iguana/kb-whisper"
defaultTranscribeFallbackModel = "berget/kb-whisper"
defaultTranscribeTimeout = 30 * time.Minute
defaultMaxUploadedChars = 90000
defaultUploadChatModel = "iguana/gemma4-26b"
)
func loadUploadConfig() (UploadConfig, error) {
u := UploadConfig{
TranscribeModel: envOr("TAPIR_TRANSCRIBE_MODEL", defaultTranscribeModel),
TranscribeFallbackModel: lookupOr("TAPIR_TRANSCRIBE_FALLBACK_MODEL", defaultTranscribeFallbackModel),
UploadTmpDir: envOr("TAPIR_UPLOAD_TMP_DIR", os.TempDir()),
FFmpegBin: envOr("TAPIR_FFMPEG_BIN", "ffmpeg"),
FFprobeBin: envOr("TAPIR_FFPROBE_BIN", "ffprobe"),
UploadChatModel: envOr("TAPIR_UPLOAD_CHAT_MODEL", defaultUploadChatModel),
}
if v := os.Getenv("TAPIR_UPLOADS_ENABLED"); v != "" {
on, err := strconv.ParseBool(v)
if err != nil {
return UploadConfig{}, fmt.Errorf("config: TAPIR_UPLOADS_ENABLED=%q: %w", v, err)
}
u.UploadsEnabled = on
}
mb, err := positiveInt("TAPIR_UPLOAD_MAX_MB", defaultUploadMaxMB)
if err != nil {
return UploadConfig{}, err
}
u.UploadMaxBytes = int64(mb) << 20
mins, err := positiveInt("TAPIR_UPLOAD_MAX_MINUTES", defaultUploadMaxMinutes)
if err != nil {
return UploadConfig{}, err
}
u.UploadMaxDuration = time.Duration(mins) * time.Minute
timeout, err := durationOr("TAPIR_TRANSCRIBE_TIMEOUT", defaultTranscribeTimeout)
if err != nil {
return UploadConfig{}, err
}
if timeout <= 0 {
return UploadConfig{}, fmt.Errorf("config: TAPIR_TRANSCRIBE_TIMEOUT must be positive, got %v", timeout)
}
u.TranscribeTimeout = timeout
chars, err := intOr("TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS", defaultMaxUploadedChars)
if err != nil {
return UploadConfig{}, err
}
if chars < 0 {
chars = 0
}
u.MaxUploadedTranscriptChars = chars
return u, nil
}
// positiveInt is intOr that refuses zero and negatives: a zero upload limit would
// reject every file, which is a misconfiguration to fail on, not to run with.
func positiveInt(key string, fallback int) (int, error) {
n, err := intOr(key, fallback)
if err != nil {
return 0, err
}
if n <= 0 {
return 0, fmt.Errorf("config: %s must be positive, got %d", key, n)
}
return n, nil
}
+141
View File
@@ -0,0 +1,141 @@
package config
import (
"os"
"testing"
"time"
)
// clearUploadEnv unsets every upload knob so a test observes the default, not the
// host environment.
func clearUploadEnv(t *testing.T) {
t.Helper()
for _, k := range []string{
"TAPIR_UPLOADS_ENABLED", "TAPIR_UPLOAD_MAX_MB", "TAPIR_UPLOAD_MAX_MINUTES",
"TAPIR_TRANSCRIBE_MODEL", "TAPIR_TRANSCRIBE_FALLBACK_MODEL", "TAPIR_TRANSCRIBE_TIMEOUT",
"TAPIR_UPLOAD_TMP_DIR", "TAPIR_FFMPEG_BIN", "TAPIR_FFPROBE_BIN",
"TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS", "TAPIR_UPLOAD_CHAT_MODEL",
} {
unset(t, k)
}
}
func TestLoad_UploadDefaults(t *testing.T) {
clearUploadEnv(t)
c, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
if c.UploadsEnabled {
t.Error("uploads must be OFF by default: the runtime image needs ffmpeg first (ADR-033)")
}
if c.UploadMaxBytes != 200<<20 {
t.Errorf("UploadMaxBytes = %d, want 200 MB", c.UploadMaxBytes)
}
if c.UploadMaxDuration != 90*time.Minute {
t.Errorf("UploadMaxDuration = %v, want 90m (the longest length measured, not a found ceiling)", c.UploadMaxDuration)
}
if c.TranscribeModel != "iguana/kb-whisper" {
t.Errorf("TranscribeModel = %q", c.TranscribeModel)
}
if c.TranscribeFallbackModel != "berget/kb-whisper" {
t.Errorf("TranscribeFallbackModel = %q", c.TranscribeFallbackModel)
}
if c.TranscribeTimeout != 30*time.Minute {
t.Errorf("TranscribeTimeout = %v, want 30m (a 90 min recording took ~7.5 min)", c.TranscribeTimeout)
}
if c.UploadTmpDir != os.TempDir() {
t.Errorf("UploadTmpDir = %q, want os.TempDir()", c.UploadTmpDir)
}
if c.FFmpegBin != "ffmpeg" || c.FFprobeBin != "ffprobe" {
t.Errorf("bins = %q, %q", c.FFmpegBin, c.FFprobeBin)
}
if c.MaxUploadedTranscriptChars != 90000 {
t.Errorf("MaxUploadedTranscriptChars = %d, want 90000: iguana/gemma4-26b answered correctly at 90,000 chars "+
"but silently dropped the head of a 120,000-char transcript (docs/research/chat-context)", c.MaxUploadedTranscriptChars)
}
if c.UploadChatModel != "iguana/gemma4-26b" {
t.Errorf("UploadChatModel = %q: koala/phi4-mini has an 8,192-token context and answers HTTP 400 on an hour of speech", c.UploadChatModel)
}
}
func TestLoad_UploadChatModelOverride(t *testing.T) {
clearUploadEnv(t)
t.Setenv("TAPIR_UPLOAD_CHAT_MODEL", "koala/qwen36-35b-a3b")
c, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
if c.UploadChatModel != "koala/qwen36-35b-a3b" {
t.Errorf("UploadChatModel = %q", c.UploadChatModel)
}
}
func TestLoad_UploadValuesAreParsed(t *testing.T) {
clearUploadEnv(t)
setEnv(t, map[string]string{
"TAPIR_UPLOADS_ENABLED": "true",
"TAPIR_UPLOAD_MAX_MB": "50",
"TAPIR_UPLOAD_MAX_MINUTES": "30",
"TAPIR_TRANSCRIBE_MODEL": "koala/whisper",
"TAPIR_TRANSCRIBE_TIMEOUT": "10m",
"TAPIR_UPLOAD_TMP_DIR": "/var/spool/tapir",
"TAPIR_FFMPEG_BIN": "/opt/ffmpeg",
"TAPIR_FFPROBE_BIN": "/opt/ffprobe",
"TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS": "60000",
})
c, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
if !c.UploadsEnabled || c.UploadMaxBytes != 50<<20 || c.UploadMaxDuration != 30*time.Minute ||
c.TranscribeModel != "koala/whisper" || c.TranscribeTimeout != 10*time.Minute ||
c.UploadTmpDir != "/var/spool/tapir" || c.FFmpegBin != "/opt/ffmpeg" ||
c.FFprobeBin != "/opt/ffprobe" || c.MaxUploadedTranscriptChars != 60000 {
t.Errorf("values not applied: %+v", c)
}
}
func TestLoad_EmptyTranscribeFallbackDisablesIt(t *testing.T) {
clearUploadEnv(t)
t.Setenv("TAPIR_TRANSCRIBE_FALLBACK_MODEL", "")
c, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
if c.TranscribeFallbackModel != "" {
t.Errorf("TranscribeFallbackModel = %q: an explicit empty value must disable the cloud fallback, "+
"so private audio can be kept on the homelab", c.TranscribeFallbackModel)
}
}
func TestLoad_UploadRejectsInvalidValues(t *testing.T) {
tests := []struct{ key, val string }{
{"TAPIR_UPLOADS_ENABLED", "maybe"},
{"TAPIR_UPLOAD_MAX_MB", "lots"},
{"TAPIR_UPLOAD_MAX_MB", "0"},
{"TAPIR_UPLOAD_MAX_MB", "-5"},
{"TAPIR_UPLOAD_MAX_MINUTES", "0"},
{"TAPIR_UPLOAD_MAX_MINUTES", "forever"},
{"TAPIR_TRANSCRIBE_TIMEOUT", "soon"},
{"TAPIR_TRANSCRIBE_TIMEOUT", "0s"},
}
for _, tt := range tests {
t.Run(tt.key+"="+tt.val, func(t *testing.T) {
clearUploadEnv(t)
t.Setenv(tt.key, tt.val)
if _, err := Load(); err == nil {
t.Errorf("Load accepted %s=%q; a zero or negative limit would reject every upload", tt.key, tt.val)
}
})
}
}
+59
View File
@@ -111,3 +111,62 @@ type Summary struct {
FallbackUsed bool FallbackUsed bool
CreatedAt time.Time CreatedAt time.Time
} }
// Segment is one timed span of a transcription. Times are seconds from the start
// of the audio.
type Segment struct {
Start float64 `json:"start"`
End float64 `json:"end"`
Text string `json:"text"`
}
// Transcription is the result of transcribing user-supplied audio (ADR-033):
// the full text, its timed segments, which backend served it, and how long the
// audio was. Endpoint is recorded because the backends differ in timestamp
// granularity — which one served a transcript is a quality fact.
type Transcription struct {
Text string
Segments []Segment
Endpoint string
AudioSeconds float64
}
// UploadStatus is the state of an uploaded-audio transcription job.
type UploadStatus string
const (
UploadQueued UploadStatus = "queued"
UploadTranscribing UploadStatus = "transcribing"
UploadDone UploadStatus = "done"
UploadFailed UploadStatus = "failed"
)
// UploadErrorKind is the typed reason an upload failed. It is deliberately NOT a
// provider's error string: the whisper server's "failed to process audio" has had
// at least five distinct causes, so nothing may branch on it (ADR-033).
type UploadErrorKind string
const (
// UploadErrInterrupted: the process restarted mid-job. The audio is not
// retained, so the job cannot resume; the user re-uploads.
UploadErrInterrupted UploadErrorKind = "interrupted"
// UploadErrTranscriptionFailed: every configured transcription endpoint failed.
UploadErrTranscriptionFailed UploadErrorKind = "transcription_failed"
)
// UploadedTranscript is a user's PRIVATE transcript of audio they uploaded. It is
// never stored in, or merged with, the shared public-caption Transcript (ADR-021).
// Content and Segments are empty until Status is UploadDone.
type UploadedTranscript struct {
ID string
UserID string
Title string
Status UploadStatus
Content string
Segments []Segment
Endpoint string
AudioSeconds float64
ErrorKind UploadErrorKind
CreatedAt time.Time
FinishedAt time.Time // zero until the job reaches done or failed
}
+139
View File
@@ -0,0 +1,139 @@
// Package metrics is Tapir's Prometheus instrumentation (ADR-030, issue #15). It
// owns the collectors and a small typed API the rest of the app calls — adapters
// never touch prometheus types directly. Two themes:
//
// - HTTP/session: request count + latency by route (the matched pattern, so
// cardinality stays bounded), and logins.
// - AI (the priority): summarization latency by model/outcome/fallback, caption
// fetch latency by outcome, chat latency by model, and LLM token usage.
//
// Handler() is served on a dedicated port (never the public app port) so a scrape
// is in-cluster only. slog timing lines are emitted at the call sites too.
package metrics
import (
"net/http"
"strconv"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
// latencyBuckets spans sub-second UI calls up to multi-minute model calls (a cold
// local model load is tens of seconds; the cloud fallback can be longer).
var latencyBuckets = []float64{0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 20, 30, 60, 120, 300}
var (
httpRequests = promauto.NewCounterVec(prometheus.CounterOpts{
Name: "tapir_http_requests_total",
Help: "HTTP requests by method, matched route pattern, and status code.",
}, []string{"method", "route", "code"})
httpDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
Name: "tapir_http_request_duration_seconds",
Help: "HTTP request latency by method and matched route pattern.",
Buckets: []float64{0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 5},
}, []string{"method", "route"})
logins = promauto.NewCounter(prometheus.CounterOpts{
Name: "tapir_logins_total",
Help: "Successful OIDC logins (session established).",
})
summarizeDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
Name: "tapir_summarize_duration_seconds",
Help: "Per-endpoint summarization latency by model, outcome (success|parse_error|error), and whether it was a fallback.",
Buckets: latencyBuckets,
}, []string{"model", "outcome", "fallback"})
captionFetchDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
Name: "tapir_caption_fetch_duration_seconds",
Help: "Caption fetch latency by outcome (captions|none|rate_limited).",
Buckets: latencyBuckets,
}, []string{"outcome"})
chatDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
Name: "tapir_chat_duration_seconds",
Help: "Per-video Q&A answer latency by model.",
Buckets: latencyBuckets,
}, []string{"model"})
llmTokens = promauto.NewCounterVec(prometheus.CounterOpts{
Name: "tapir_llm_tokens_total",
Help: "LLM tokens consumed by model and kind (prompt|completion).",
}, []string{"model", "kind"})
)
// Handler serves the Prometheus exposition format. Mount on the dedicated metrics
// port, never the public app mux.
func Handler() http.Handler { return promhttp.Handler() }
// IncLogin records a successful login.
func IncLogin() { logins.Inc() }
// ObserveSummarize records one summarization endpoint attempt.
func ObserveSummarize(model, outcome string, fallback bool, d time.Duration) {
summarizeDuration.WithLabelValues(model, outcome, strconv.FormatBool(fallback)).Observe(d.Seconds())
}
// ObserveCaptionFetch records one caption fetch by outcome.
func ObserveCaptionFetch(outcome string, d time.Duration) {
captionFetchDuration.WithLabelValues(outcome).Observe(d.Seconds())
}
// ObserveChat records one Q&A answer latency.
func ObserveChat(model string, d time.Duration) {
chatDuration.WithLabelValues(model).Observe(d.Seconds())
}
// RecordTokens records LLM token usage from a completion's usage block. Zero
// counts are skipped so a provider that omits usage adds nothing.
func RecordTokens(model string, prompt, completion int) {
if prompt > 0 {
llmTokens.WithLabelValues(model, "prompt").Add(float64(prompt))
}
if completion > 0 {
llmTokens.WithLabelValues(model, "completion").Add(float64(completion))
}
}
// HTTPMiddleware records request count + latency. It reads r.Pattern AFTER the
// inner handler routes (Go 1.22 sets it during ServeMux matching), so the label is
// the bounded registered pattern (e.g. "GET /v/{videoId}"), never the raw path
// with its high-cardinality ids. Unmatched requests bucket as "other".
func HTTPMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
sw := &statusWriter{ResponseWriter: w, code: http.StatusOK}
next.ServeHTTP(sw, r)
route := r.Pattern
if route == "" {
route = "other"
}
httpRequests.WithLabelValues(r.Method, route, strconv.Itoa(sw.code)).Inc()
httpDuration.WithLabelValues(r.Method, route).Observe(time.Since(start).Seconds())
})
}
// statusWriter captures the response status for the request-count label.
type statusWriter struct {
http.ResponseWriter
code int
wroteHeader bool
}
func (s *statusWriter) WriteHeader(code int) {
if !s.wroteHeader {
s.code = code
s.wroteHeader = true
}
s.ResponseWriter.WriteHeader(code)
}
func (s *statusWriter) Write(b []byte) (int, error) {
s.wroteHeader = true // an implicit 200
return s.ResponseWriter.Write(b)
}
+92
View File
@@ -0,0 +1,92 @@
package metrics
import (
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/testutil"
dto "github.com/prometheus/client_model/go"
"github.com/stretchr/testify/require"
)
// histCount reads a histogram child's observation count (testutil.ToFloat64 only
// works on counters/gauges; a histogram's WithLabelValues child is an Observer).
func histCount(t *testing.T, o prometheus.Observer) uint64 {
t.Helper()
m, ok := o.(prometheus.Metric)
require.True(t, ok, "histogram child must be a prometheus.Metric")
var d dto.Metric
require.NoError(t, m.Write(&d))
return d.GetHistogram().GetSampleCount()
}
// TestObserveSummarizeRecordsModelOutcomeFallback: a success observation lands on
// the right model/outcome/fallback series.
func TestObserveSummarizeRecordsModelOutcomeFallback(t *testing.T) {
before := histCount(t, summarizeDuration.WithLabelValues("koala/phi4-mini", "success", "false"))
ObserveSummarize("koala/phi4-mini", "success", false, 1200*time.Millisecond)
after := histCount(t, summarizeDuration.WithLabelValues("koala/phi4-mini", "success", "false"))
require.Equal(t, before+1, after, "one success observation recorded for the model")
}
// TestObserveSummarizeRecordsFailureOutcomes: error and parse_error are distinct
// series so a fallback chain's failures are visible.
func TestObserveSummarizeRecordsFailureOutcomes(t *testing.T) {
e0 := histCount(t, summarizeDuration.WithLabelValues("m", "error", "false"))
p0 := histCount(t, summarizeDuration.WithLabelValues("m", "parse_error", "false"))
ObserveSummarize("m", "error", false, time.Second)
ObserveSummarize("m", "parse_error", false, time.Second)
require.Equal(t, e0+1, histCount(t, summarizeDuration.WithLabelValues("m", "error", "false")))
require.Equal(t, p0+1, histCount(t, summarizeDuration.WithLabelValues("m", "parse_error", "false")))
}
func TestObserveCaptionFetchByOutcome(t *testing.T) {
b := histCount(t, captionFetchDuration.WithLabelValues("captions"))
ObserveCaptionFetch("captions", 3*time.Second)
require.Equal(t, b+1, histCount(t, captionFetchDuration.WithLabelValues("captions")))
}
func TestChatAnswerLatencyRecorded(t *testing.T) {
b := histCount(t, chatDuration.WithLabelValues("iguana/gemma4-26b"))
ObserveChat("iguana/gemma4-26b", 2*time.Second)
require.Equal(t, b+1, histCount(t, chatDuration.WithLabelValues("iguana/gemma4-26b")))
}
// TestRecordTokens: prompt + completion land on their kind series; zero is skipped.
func TestRecordTokens(t *testing.T) {
p0 := testutil.ToFloat64(llmTokens.WithLabelValues("m", "prompt"))
c0 := testutil.ToFloat64(llmTokens.WithLabelValues("m", "completion"))
RecordTokens("m", 100, 40)
RecordTokens("m", 0, 0) // skipped, no panic
require.Equal(t, p0+100, testutil.ToFloat64(llmTokens.WithLabelValues("m", "prompt")))
require.Equal(t, c0+40, testutil.ToFloat64(llmTokens.WithLabelValues("m", "completion")))
}
func TestLoginCounted(t *testing.T) {
b := testutil.ToFloat64(logins)
IncLogin()
require.Equal(t, b+1, testutil.ToFloat64(logins))
}
// TestHTTPMiddlewareRecordsByRoutePattern: the request is counted under the bounded
// registered pattern (r.Pattern after routing), not the raw path with its ids.
func TestHTTPMiddlewareRecordsByRoutePattern(t *testing.T) {
mux := http.NewServeMux()
mux.HandleFunc("GET /v/{videoId}", func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusTeapot)
})
h := HTTPMiddleware(mux)
before := testutil.ToFloat64(httpRequests.WithLabelValues("GET", "GET /v/{videoId}", "418"))
rec := httptest.NewRecorder()
h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/v/abc-123", nil))
require.Equal(t, http.StatusTeapot, rec.Code)
after := testutil.ToFloat64(httpRequests.WithLabelValues("GET", "GET /v/{videoId}", "418"))
require.Equal(t, before+1, after, "counted under the pattern, not /v/abc-123")
require.Equal(t, float64(0), testutil.ToFloat64(httpRequests.WithLabelValues("GET", "/v/abc-123", "418")),
"raw path must never be a label value")
}
+1 -1
View File
@@ -7,7 +7,7 @@ package ports
import ( import (
"context" "context"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
) )
// VideoSource is a video platform Tapir watches (YouTube, Vimeo). // VideoSource is a video platform Tapir watches (YouTube, Vimeo).
+3 -3
View File
@@ -19,9 +19,9 @@ import (
"slices" "slices"
"time" "time"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
"gitea.d-ma.be/mathias/tapir/internal/ports" "git.d-ma.be/mathias/tapir/internal/ports"
"gitea.d-ma.be/mathias/tapir/internal/usecase" "git.d-ma.be/mathias/tapir/internal/usecase"
) )
// passCandidate is a video that passed all pre-filters (seen/manual/backoff) // passCandidate is a video that passed all pre-filters (seen/manual/backoff)
+3 -3
View File
@@ -9,9 +9,9 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
"gitea.d-ma.be/mathias/tapir/internal/runner" "git.d-ma.be/mathias/tapir/internal/runner"
"gitea.d-ma.be/mathias/tapir/internal/usecase" "git.d-ma.be/mathias/tapir/internal/usecase"
) )
const testUser = "11111111-1111-1111-1111-111111111111" const testUser = "11111111-1111-1111-1111-111111111111"
+115
View File
@@ -0,0 +1,115 @@
package uploads
import (
"context"
"errors"
"fmt"
"io"
"os"
"git.d-ma.be/mathias/tapir/internal/domain"
)
// ErrUnreadable is what a Prober returns when the FILE is the problem (not media,
// truncated, damaged), as opposed to the prober itself failing. Accept turns it
// into a user-facing rejection; any other prober error is our fault and surfaces
// as an internal error — a missing ffprobe must not read as "your file is bad".
var ErrUnreadable = errors.New("uploads: file is not readable media")
// Prober inspects a file on disk. The ffprobe adapter implements it.
type Prober interface {
Probe(ctx context.Context, path string) (Probe, error)
}
// Store is the slice of the private transcript store the intake needs.
type Store interface {
CreateUpload(ctx context.Context, userID, title string) (string, error)
FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error
}
// Job is one accepted upload awaiting transcription. Path is a temp file the
// receiver owns: it must delete it when finished (the audio is never retained).
type Job struct {
UserID string
UploadID string
Path string
}
// Submitter hands accepted jobs to the in-process worker (ADR-033).
type Submitter interface {
Submit(Job) error
}
// Intake validates an uploaded recording and, if it passes, queues it.
type Intake struct {
store Store
prober Prober
sub Submitter
limits Limits
tmpDir string
}
// NewIntake wires an Intake. tmpDir is where uploads are spooled; it must exist.
func NewIntake(s Store, p Prober, sub Submitter, l Limits, tmpDir string) *Intake {
return &Intake{store: s, prober: p, sub: sub, limits: l, tmpDir: tmpDir}
}
// Accept spools src to disk (bounded by MaxBytes), probes it, validates it, records
// a queued upload and submits the job. It returns the upload id. A file that fails
// validation yields a *RejectError and creates NO row — rejection is synchronous
// and leaves nothing behind. On success the temp file is handed to the job and the
// worker deletes it; on every failure path Accept deletes it itself.
func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reader) (string, error) {
f, err := os.CreateTemp(in.tmpDir, "upload-*")
if err != nil {
return "", fmt.Errorf("uploads: spool: %w", err)
}
path := f.Name()
handedOver := false
defer func() {
if !handedOver {
_ = os.Remove(path)
}
}()
// Read one byte past the cap so "exactly at the limit" and "over" are told apart.
n, err := io.Copy(f, io.LimitReader(src, in.limits.MaxBytes+1))
if cerr := f.Close(); err == nil {
err = cerr
}
if err != nil {
return "", fmt.Errorf("uploads: receive: %w", err)
}
if n > in.limits.MaxBytes {
return "", &RejectError{Reason: ReasonTooLarge, Message: fmt.Sprintf(
"This file is larger than the %d MB limit.", in.limits.MaxBytes>>20)}
}
probe, err := in.prober.Probe(ctx, path)
if errors.Is(err, ErrUnreadable) {
return "", &RejectError{
Reason: ReasonUnreadable,
Message: "This file could not be read as audio; it may be damaged or not a recording.",
}
}
if err != nil {
return "", fmt.Errorf("uploads: probe: %w", err)
}
if err := Validate(probe, in.limits); err != nil {
return "", err
}
id, err := in.store.CreateUpload(ctx, userID, title)
if err != nil {
return "", fmt.Errorf("uploads: create: %w", err)
}
if err := in.sub.Submit(Job{UserID: userID, UploadID: id, Path: path}); err != nil {
// A row that will never run must not sit queued: that is the stuck spinner.
if ferr := in.store.FailUpload(ctx, userID, id, domain.UploadErrInterrupted); ferr != nil {
return "", fmt.Errorf("uploads: submit: %w (and failing the row: %v)", err, ferr)
}
return "", fmt.Errorf("uploads: submit: %w", err)
}
handedOver = true
return id, nil
}
+212
View File
@@ -0,0 +1,212 @@
package uploads_test
import (
"bytes"
"context"
"errors"
"io"
"os"
"path/filepath"
"strings"
"testing"
"testing/iotest"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
type fakeStore struct {
created []string // titles
failed map[string]domain.UploadErrorKind
nextID string
}
func (f *fakeStore) CreateUpload(_ context.Context, _, title string) (string, error) {
f.created = append(f.created, title)
return f.nextID, nil
}
func (f *fakeStore) FailUpload(_ context.Context, _, id string, kind domain.UploadErrorKind) error {
if f.failed == nil {
f.failed = map[string]domain.UploadErrorKind{}
}
f.failed[id] = kind
return nil
}
type fakeProber struct {
probe uploads.Probe
err error
paths []string
seen []byte // content of the file at probe time
}
func (f *fakeProber) Probe(_ context.Context, path string) (uploads.Probe, error) {
f.paths = append(f.paths, path)
f.seen, _ = os.ReadFile(path)
return f.probe, f.err
}
type fakeSubmitter struct {
jobs []uploads.Job
err error
}
func (f *fakeSubmitter) Submit(j uploads.Job) error {
f.jobs = append(f.jobs, j)
return f.err
}
type rig struct {
in *uploads.Intake
st *fakeStore
pr *fakeProber
sub *fakeSubmitter
tmp string
}
func newRig(t *testing.T) *rig {
t.Helper()
r := &rig{
st: &fakeStore{nextID: "up-1"},
pr: &fakeProber{probe: uploads.Probe{Duration: 10 * time.Minute, HasAudio: true}},
sub: &fakeSubmitter{},
tmp: t.TempDir(),
}
r.in = uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: 90 * time.Minute}, r.tmp)
return r
}
func (r *rig) tmpFiles(t *testing.T) []string {
t.Helper()
ents, err := os.ReadDir(r.tmp)
require.NoError(t, err)
var names []string
for _, e := range ents {
names = append(names, e.Name())
}
return names
}
func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
r := newRig(t)
audio := bytes.Repeat([]byte("a"), 500)
id, err := r.in.Accept(context.Background(), "user-1", "Teams export", bytes.NewReader(audio))
require.NoError(t, err)
assert.Equal(t, "up-1", id)
assert.Equal(t, []string{"Teams export"}, r.st.created)
assert.Equal(t, audio, r.pr.seen, "the prober must see the complete uploaded bytes, on disk")
require.Len(t, r.sub.jobs, 1)
job := r.sub.jobs[0]
assert.Equal(t, uploads.Job{UserID: "user-1", UploadID: "up-1", Path: job.Path}, job)
assert.Equal(t, filepath.Dir(job.Path), r.tmp, "the temp file lives in the configured directory")
got, err := os.ReadFile(job.Path)
require.NoError(t, err)
assert.Equal(t, audio, got, "the handed-over file still exists: the worker, not Accept, deletes it")
}
func TestAccept_OversizeBodyIsRejectedAndLeavesNothingBehind(t *testing.T) {
r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "big", strings.NewReader(strings.Repeat("a", 1025)))
var rej *uploads.RejectError
require.True(t, errors.As(err, &rej))
assert.Equal(t, uploads.ReasonTooLarge, rej.Reason)
assert.Contains(t, rej.Message, "limit")
assert.Empty(t, r.pr.paths, "a file already over the size cap must not be probed")
assert.Empty(t, r.st.created, "a rejected upload creates no row")
assert.Empty(t, r.sub.jobs)
assert.Empty(t, r.tmpFiles(t), "the partial temp file must be removed")
}
func TestAccept_BodyExactlyAtTheLimitIsAccepted(t *testing.T) {
r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "edge", strings.NewReader(strings.Repeat("a", 1024)))
require.NoError(t, err)
}
func TestAccept_ProbeVerdictRejectsWithoutRowAndRemovesTheFile(t *testing.T) {
tests := []struct {
name string
probe uploads.Probe
reason uploads.Reason
}{
{"video", uploads.Probe{Duration: time.Minute, HasAudio: true, HasVideo: true}, uploads.ReasonVideo},
{"too long", uploads.Probe{Duration: 2 * time.Hour, HasAudio: true}, uploads.ReasonTooLong},
{"no audio", uploads.Probe{Duration: time.Minute}, uploads.ReasonNoAudio},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
r := newRig(t)
r.pr.probe = tt.probe
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
var rej *uploads.RejectError
require.True(t, errors.As(err, &rej))
assert.Equal(t, tt.reason, rej.Reason)
assert.Empty(t, r.st.created)
assert.Empty(t, r.sub.jobs)
assert.Empty(t, r.tmpFiles(t))
})
}
}
func TestAccept_UnreadableFileIsARejectionButAProberFaultIsAnError(t *testing.T) {
t.Run("ffprobe says the file is not media", func(t *testing.T) {
r := newRig(t)
r.pr.err = uploads.ErrUnreadable
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("not media"))
var rej *uploads.RejectError
require.True(t, errors.As(err, &rej), "the user's file is the problem: tell them")
assert.Equal(t, uploads.ReasonUnreadable, rej.Reason)
assert.Empty(t, r.tmpFiles(t))
})
t.Run("the prober itself broke", func(t *testing.T) {
r := newRig(t)
r.pr.err = errors.New("exec: ffprobe: executable file not found")
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
require.Error(t, err)
var rej *uploads.RejectError
assert.False(t, errors.As(err, &rej), "our fault must not be reported as the user's bad file")
assert.Empty(t, r.st.created)
assert.Empty(t, r.tmpFiles(t))
})
}
func TestAccept_ClientDisconnectMidUploadLeavesNothingBehind(t *testing.T) {
r := newRig(t)
body := io.MultiReader(strings.NewReader("partial"), iotest.ErrReader(errors.New("client went away")))
_, err := r.in.Accept(context.Background(), "user-1", "t", body)
require.Error(t, err)
assert.Empty(t, r.pr.paths)
assert.Empty(t, r.st.created)
assert.Empty(t, r.tmpFiles(t))
}
func TestAccept_SubmitFailureFailsTheRowHonestlyAndRemovesTheFile(t *testing.T) {
r := newRig(t)
r.sub.err = errors.New("worker stopped")
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
require.Error(t, err)
assert.Equal(t, domain.UploadErrInterrupted, r.st.failed["up-1"],
"a row that will never run must not sit queued: that is the stuck spinner ADR-025 forbids")
assert.Empty(t, r.tmpFiles(t))
}
+82
View File
@@ -0,0 +1,82 @@
// Package uploads is the use case behind "upload an audio recording" (ADR-033):
// validate the file by PROBING it, keep the transcript private, never retain the
// audio. It depends only on ports (a Prober, a store, a job Submitter) — the
// concrete ffprobe adapter, the web handler and the transcription worker are wired
// from outside.
package uploads
import (
"fmt"
"time"
)
// Limits bound what an upload may be. Both are rejected synchronously, with a
// specific message, rather than by a timeout (ADR-033).
type Limits struct {
MaxBytes int64
MaxDuration time.Duration
}
// Probe is what inspecting an uploaded file revealed. It is the input to Validate
// because a name or MIME type cannot say whether a container carries video: a
// Teams voice-only export is typically an .mp4 with no video stream.
type Probe struct {
Duration time.Duration
HasAudio bool
HasVideo bool
}
// Reason is the machine-checkable class of a rejection.
type Reason string
const (
ReasonVideo Reason = "video"
ReasonNoAudio Reason = "no_audio"
ReasonTooLong Reason = "too_long"
ReasonTooLarge Reason = "too_large"
ReasonUnreadable Reason = "unreadable"
)
// RejectError is a synchronous, user-presentable refusal. Message is written for
// the person who uploaded the file; Reason is for code and tests.
type RejectError struct {
Reason Reason
Message string
}
func (e *RejectError) Error() string { return string(e.Reason) + ": " + e.Message }
// Validate decides whether a probed file may be transcribed. Video is checked
// first so a video file is always refused as video, whatever its length.
func Validate(p Probe, l Limits) error {
switch {
case p.HasVideo:
return &RejectError{Reason: ReasonVideo, Message: "This file contains a video track. " +
"Only audio recordings are supported for now; video upload is tracked in issue #28."}
case !p.HasAudio:
return &RejectError{Reason: ReasonNoAudio, Message: "This file has no audio track to transcribe."}
case p.Duration <= 0:
return &RejectError{Reason: ReasonUnreadable, Message: "Could not read a length from this file; it may be damaged."}
case p.Duration > l.MaxDuration:
return &RejectError{Reason: ReasonTooLong, Message: fmt.Sprintf(
"This recording is %s long; the limit is %s.", FormatDuration(p.Duration), FormatDuration(l.MaxDuration))}
}
return nil
}
// FormatDuration renders a length for people: "1 h 42 min", "45 min", "30 s".
func FormatDuration(d time.Duration) string {
d = d.Round(time.Second)
h := int(d / time.Hour)
m := int(d % time.Hour / time.Minute)
switch {
case h > 0 && m > 0:
return fmt.Sprintf("%d h %d min", h, m)
case h > 0:
return fmt.Sprintf("%d h", h)
case m > 0:
return fmt.Sprintf("%d min", m)
default:
return fmt.Sprintf("%d s", int(d/time.Second))
}
}
+95
View File
@@ -0,0 +1,95 @@
package uploads_test
import (
"errors"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
var testLimits = uploads.Limits{MaxBytes: 200 << 20, MaxDuration: 90 * time.Minute}
func TestValidate(t *testing.T) {
tests := []struct {
name string
probe uploads.Probe
wantReason uploads.Reason // "" = accepted
wantInMsg []string
}{
{
name: "audio-only file within the duration limit is accepted",
probe: uploads.Probe{Duration: 45 * time.Minute, HasAudio: true},
},
{
name: "a recording exactly at the limit is accepted",
probe: uploads.Probe{Duration: 90 * time.Minute, HasAudio: true},
},
{
name: "a file with a video stream is rejected and names issue 28",
probe: uploads.Probe{Duration: 4 * time.Minute, HasAudio: true, HasVideo: true},
wantReason: uploads.ReasonVideo,
wantInMsg: []string{"video", "#28"},
},
{
name: "video with no audio is still rejected as video, not as missing audio",
probe: uploads.Probe{Duration: 4 * time.Minute, HasVideo: true},
wantReason: uploads.ReasonVideo,
wantInMsg: []string{"#28"},
},
{
name: "a file with no audio stream is rejected",
probe: uploads.Probe{Duration: 4 * time.Minute},
wantReason: uploads.ReasonNoAudio,
wantInMsg: []string{"no audio"},
},
{
name: "a recording over the limit is rejected, stating both lengths",
probe: uploads.Probe{Duration: 102 * time.Minute, HasAudio: true},
wantReason: uploads.ReasonTooLong,
wantInMsg: []string{"1 h 42 min", "1 h 30 min"},
},
{
name: "zero duration is unreadable, not accepted as an empty recording",
probe: uploads.Probe{HasAudio: true},
wantReason: uploads.ReasonUnreadable,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := uploads.Validate(tt.probe, testLimits)
if tt.wantReason == "" {
require.NoError(t, err)
return
}
var rej *uploads.RejectError
require.True(t, errors.As(err, &rej), "want a *RejectError, got %v", err)
assert.Equal(t, tt.wantReason, rej.Reason)
for _, s := range tt.wantInMsg {
assert.Contains(t, rej.Message, s)
}
})
}
}
func TestFormatDuration(t *testing.T) {
tests := []struct {
d time.Duration
want string
}{
{90 * time.Minute, "1 h 30 min"},
{102 * time.Minute, "1 h 42 min"},
{45 * time.Minute, "45 min"},
{2 * time.Hour, "2 h"},
{30 * time.Second, "30 s"},
}
for _, tt := range tests {
t.Run(tt.want, func(t *testing.T) {
assert.Equal(t, tt.want, uploads.FormatDuration(tt.d))
})
}
}
+191
View File
@@ -0,0 +1,191 @@
package uploads
import (
"context"
"errors"
"fmt"
"log/slog"
"os"
"sync"
"time"
"git.d-ma.be/mathias/tapir/internal/domain"
)
// ErrStopped is returned by Submit once the worker has stopped.
var ErrStopped = errors.New("uploads: worker stopped")
// Converter turns the uploaded file into the 16 kHz mono PCM WAV whisper wants.
type Converter interface {
ToWAV(ctx context.Context, src, dst string) error
}
// Transcriber turns a WAV into a timed transcription. The gateway chain
// (iguana primary, berget fallback) implements it.
type Transcriber interface {
Transcribe(ctx context.Context, wavPath string) (domain.Transcription, error)
}
// WorkerStore is the slice of the private transcript store the worker drives.
type WorkerStore interface {
StartTranscribing(ctx context.Context, userID, id string) error
CompleteUpload(ctx context.Context, userID, id string, t domain.Transcription) error
FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error
}
// Worker transcribes accepted uploads in-process, ONE AT A TIME (ADR-033). It is
// not a second deployable: it shares the web process's lifetime, which is why it
// inherits ADR-018's single-replica assumption — the queue is process-local. The
// concurrency of one is deliberate: two simultaneous 90-minute jobs on iguana were
// never measured, and a Metal OOM there latches the whisper backend (infra#273).
type Worker struct {
store WorkerStore
conv Converter
tr Transcriber
tmpDir string
timeout time.Duration
mu sync.Mutex
queue []Job
stopped bool
wake chan struct{}
}
// NewWorker wires a Worker. timeout bounds each job (conversion + transcription);
// a job that outlives it is failed rather than left spinning.
func NewWorker(s WorkerStore, c Converter, t Transcriber, tmpDir string, timeout time.Duration) *Worker {
return &Worker{store: s, conv: c, tr: t, tmpDir: tmpDir, timeout: timeout, wake: make(chan struct{}, 1)}
}
// Submit queues a job. It never blocks. It satisfies Submitter.
func (w *Worker) Submit(j Job) error {
w.mu.Lock()
defer w.mu.Unlock()
if w.stopped {
return ErrStopped
}
w.queue = append(w.queue, j)
select {
case w.wake <- struct{}{}:
default:
}
return nil
}
// Run processes queued jobs until ctx ends, then deletes the audio of anything
// still queued and refuses new jobs. The rows of those jobs stay queued or
// transcribing and are failed as interrupted by the start-up sweep
// (store.FailInterruptedUploads): the audio is not retained, so they cannot resume.
func (w *Worker) Run(ctx context.Context) {
defer w.stop()
for {
job, ok := w.next(ctx)
if !ok {
return
}
w.Process(ctx, job)
}
}
// next blocks for the next job. Cancellation wins over a non-empty queue: a job
// still queued at shutdown must not be STARTED (it would begin a long conversion
// with an already-cancelled context); stop() deletes its audio instead.
func (w *Worker) next(ctx context.Context) (Job, bool) {
for {
if ctx.Err() != nil {
return Job{}, false
}
w.mu.Lock()
if len(w.queue) > 0 {
j := w.queue[0]
w.queue = w.queue[1:]
w.mu.Unlock()
return j, true
}
w.mu.Unlock()
select {
case <-w.wake:
case <-ctx.Done():
return Job{}, false
}
}
}
func (w *Worker) stop() {
w.mu.Lock()
defer w.mu.Unlock()
w.stopped = true
for _, j := range w.queue {
_ = os.Remove(j.Path)
}
w.queue = nil
}
// Process runs one job: row -> transcribing, convert, transcribe, store. The
// uploaded file and the converted WAV are deleted on every path — the audio is
// never retained (ADR-033). A failure is recorded with a TYPED kind; a shutdown
// mid-job is not a failure and leaves the row for the start-up sweep.
func (w *Worker) Process(ctx context.Context, job Job) {
defer func() { _ = os.Remove(job.Path) }()
log := slog.Default().With("upload", job.UploadID)
start := time.Now()
if err := w.store.StartTranscribing(ctx, job.UserID, job.UploadID); err != nil {
log.Error("upload: start transcribing", "err", err)
w.fail(ctx, job, log)
return
}
wav, err := w.tempWAV()
if err != nil {
log.Error("upload: temp wav", "err", err)
w.fail(ctx, job, log)
return
}
defer func() { _ = os.Remove(wav) }()
jobCtx, cancel := context.WithTimeout(ctx, w.timeout)
defer cancel()
if err := w.conv.ToWAV(jobCtx, job.Path, wav); err != nil {
log.Error("upload: convert", "err", err)
w.fail(ctx, job, log)
return
}
t, err := w.tr.Transcribe(jobCtx, wav)
if err != nil {
log.Error("upload: transcribe", "err", err, "elapsed", time.Since(start).Round(time.Second))
w.fail(ctx, job, log)
return
}
if err := w.store.CompleteUpload(ctx, job.UserID, job.UploadID, t); err != nil {
log.Error("upload: store transcript", "err", err)
w.fail(ctx, job, log)
return
}
log.Info("upload transcribed", "endpoint", t.Endpoint, "audio_s", t.AudioSeconds,
"segments", len(t.Segments), "elapsed", time.Since(start).Round(time.Second))
}
// fail records a transcription failure — unless the PARENT context ended, which
// is a shutdown, not a failure: the row is left for the start-up sweep.
func (w *Worker) fail(ctx context.Context, job Job, log *slog.Logger) {
if ctx.Err() != nil {
return
}
if err := w.store.FailUpload(ctx, job.UserID, job.UploadID, domain.UploadErrTranscriptionFailed); err != nil {
log.Error("upload: record failure", "err", err)
}
}
func (w *Worker) tempWAV() (string, error) {
f, err := os.CreateTemp(w.tmpDir, "wav-*")
if err != nil {
return "", fmt.Errorf("uploads: temp wav: %w", err)
}
name := f.Name()
if err := f.Close(); err != nil {
_ = os.Remove(name)
return "", fmt.Errorf("uploads: temp wav: %w", err)
}
return name, nil
}
+282
View File
@@ -0,0 +1,282 @@
package uploads_test
import (
"context"
"errors"
"os"
"path/filepath"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/uploads"
)
type wStore struct {
mu sync.Mutex
started []string
completed map[string]domain.Transcription
failed map[string]domain.UploadErrorKind
startErr error
}
func newWStore() *wStore {
return &wStore{completed: map[string]domain.Transcription{}, failed: map[string]domain.UploadErrorKind{}}
}
func (s *wStore) StartTranscribing(_ context.Context, _, id string) error {
s.mu.Lock()
defer s.mu.Unlock()
s.started = append(s.started, id)
return s.startErr
}
func (s *wStore) CompleteUpload(_ context.Context, _, id string, t domain.Transcription) error {
s.mu.Lock()
defer s.mu.Unlock()
s.completed[id] = t
return nil
}
func (s *wStore) FailUpload(_ context.Context, _, id string, k domain.UploadErrorKind) error {
s.mu.Lock()
defer s.mu.Unlock()
s.failed[id] = k
return nil
}
func (s *wStore) snapshot() (map[string]domain.Transcription, map[string]domain.UploadErrorKind) {
s.mu.Lock()
defer s.mu.Unlock()
c := map[string]domain.Transcription{}
f := map[string]domain.UploadErrorKind{}
for k, v := range s.completed {
c[k] = v
}
for k, v := range s.failed {
f[k] = v
}
return c, f
}
type fakeConverter struct {
err error
srcs []string
dsts []string
}
func (c *fakeConverter) ToWAV(_ context.Context, src, dst string) error {
c.srcs = append(c.srcs, src)
c.dsts = append(c.dsts, dst)
if c.err != nil {
return c.err
}
return os.WriteFile(dst, []byte("wav"), 0o600)
}
type fakeTranscriber struct {
result domain.Transcription
err error
paths []string
// block, when non-nil, holds every call until it is closed or ctx ends.
block chan struct{}
running atomic.Int32
maxSeen atomic.Int32
}
func (f *fakeTranscriber) Transcribe(ctx context.Context, wav string) (domain.Transcription, error) {
f.paths = append(f.paths, wav)
n := f.running.Add(1)
defer f.running.Add(-1)
for {
m := f.maxSeen.Load()
if n <= m || f.maxSeen.CompareAndSwap(m, n) {
break
}
}
if f.block != nil {
select {
case <-f.block:
case <-ctx.Done():
return domain.Transcription{}, ctx.Err()
}
}
return f.result, f.err
}
type wrig struct {
w *uploads.Worker
st *wStore
conv *fakeConverter
tr *fakeTranscriber
tmp string
}
func newWRig(t *testing.T, timeout time.Duration) *wrig {
t.Helper()
r := &wrig{
st: newWStore(),
conv: &fakeConverter{},
tr: &fakeTranscriber{result: domain.Transcription{Text: "synthetic text", Endpoint: "iguana/kb-whisper", AudioSeconds: 3}},
tmp: t.TempDir(),
}
r.w = uploads.NewWorker(r.st, r.conv, r.tr, r.tmp, timeout)
return r
}
// spool writes a fake uploaded file (what Intake would hand over) and returns a Job.
func (r *wrig) spool(t *testing.T, id string) uploads.Job {
t.Helper()
p := filepath.Join(r.tmp, "upload-"+id)
require.NoError(t, os.WriteFile(p, []byte("audio"), 0o600))
return uploads.Job{UserID: "u1", UploadID: id, Path: p}
}
func (r *wrig) leftovers(t *testing.T) []string {
t.Helper()
ents, err := os.ReadDir(r.tmp)
require.NoError(t, err)
var n []string
for _, e := range ents {
n = append(n, e.Name())
}
return n
}
func TestWorkerProcess_SuccessConvertsTranscribesStoresAndDeletesTheAudio(t *testing.T) {
r := newWRig(t, time.Minute)
job := r.spool(t, "a")
r.w.Process(context.Background(), job)
c, f := r.st.snapshot()
assert.Equal(t, []string{"a"}, r.st.started)
assert.Equal(t, "synthetic text", c["a"].Text)
assert.Equal(t, "iguana/kb-whisper", c["a"].Endpoint)
assert.Empty(t, f)
require.Len(t, r.conv.srcs, 1)
assert.Equal(t, job.Path, r.conv.srcs[0], "the converter reads the uploaded file")
assert.Equal(t, r.conv.dsts, r.tr.paths, "whisper is sent the converted WAV, not the original")
assert.Equal(t, r.tmp, filepath.Dir(r.conv.dsts[0]))
assert.Empty(t, r.leftovers(t), "the audio is never retained: both the upload and the WAV are deleted")
}
func TestWorkerProcess_FailuresAreTypedAndStillDeleteTheAudio(t *testing.T) {
tests := []struct {
name string
setup func(*wrig)
}{
{"conversion fails", func(r *wrig) { r.conv.err = errors.New("moov atom not found") }},
{"every transcription endpoint fails", func(r *wrig) { r.tr.err = errors.New("iguana down; berget down") }},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
r := newWRig(t, time.Minute)
tt.setup(r)
job := r.spool(t, "a")
r.w.Process(context.Background(), job)
c, f := r.st.snapshot()
assert.Empty(t, c)
assert.Equal(t, domain.UploadErrTranscriptionFailed, f["a"])
assert.Empty(t, r.leftovers(t))
})
}
t.Run("conversion failure never reaches whisper", func(t *testing.T) {
r := newWRig(t, time.Minute)
r.conv.err = errors.New("bad")
r.w.Process(context.Background(), r.spool(t, "a"))
assert.Empty(t, r.tr.paths)
})
}
func TestWorkerProcess_ShutdownMidJobLeavesTheRowForTheStartupSweep(t *testing.T) {
r := newWRig(t, time.Minute)
r.tr.block = make(chan struct{})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
job := r.spool(t, "a")
go func() { r.w.Process(ctx, job); close(done) }()
require.Eventually(t, func() bool { return r.tr.running.Load() == 1 }, 2*time.Second, 5*time.Millisecond)
cancel()
<-done
_, f := r.st.snapshot()
assert.Empty(t, f, "a shutdown is not a transcription failure: the row stays for FailInterruptedUploads to mark interrupted")
assert.Empty(t, r.leftovers(t), "but the audio is still deleted")
}
func TestWorkerProcess_AJobThatOutlivesItsTimeoutIsFailedNotStuck(t *testing.T) {
r := newWRig(t, 50*time.Millisecond)
r.tr.block = make(chan struct{}) // never released
done := make(chan struct{})
go func() { r.w.Process(context.Background(), r.spool(t, "a")); close(done) }()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("Process did not return: a hung endpoint was not bounded by the job timeout")
}
_, f := r.st.snapshot()
assert.Equal(t, domain.UploadErrTranscriptionFailed, f["a"], "a hung endpoint must end in failed, not an eternal spinner (ADR-025)")
assert.Empty(t, r.leftovers(t))
}
func TestWorkerProcess_StartFailureDropsTheJobAndDeletesTheAudio(t *testing.T) {
r := newWRig(t, time.Minute)
r.st.startErr = errors.New("db down")
r.w.Process(context.Background(), r.spool(t, "a"))
assert.Empty(t, r.tr.paths, "never transcribe a job whose row could not be moved to transcribing")
_, f := r.st.snapshot()
assert.Contains(t, f, "a")
assert.Empty(t, r.leftovers(t))
}
func TestWorkerRun_ProcessesOneJobAtATime(t *testing.T) {
r := newWRig(t, time.Minute)
r.tr.block = make(chan struct{})
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go r.w.Run(ctx)
for _, id := range []string{"a", "b", "c"} {
require.NoError(t, r.w.Submit(r.spool(t, id)))
}
require.Eventually(t, func() bool { return r.tr.running.Load() == 1 }, 2*time.Second, 5*time.Millisecond)
time.Sleep(50 * time.Millisecond) // give a (wrong) second job time to start
assert.Equal(t, int32(1), r.tr.running.Load(), "two simultaneous long jobs on iguana were never measured: concurrency is 1")
close(r.tr.block)
require.Eventually(t, func() bool { c, _ := r.st.snapshot(); return len(c) == 3 }, 3*time.Second, 5*time.Millisecond)
assert.Equal(t, int32(1), r.tr.maxSeen.Load())
}
func TestWorkerRun_StoppingDeletesQueuedAudioAndRefusesNewJobs(t *testing.T) {
r := newWRig(t, time.Minute)
r.tr.block = make(chan struct{})
ctx, cancel := context.WithCancel(context.Background())
stopped := make(chan struct{})
go func() { r.w.Run(ctx); close(stopped) }()
require.NoError(t, r.w.Submit(r.spool(t, "a")))
require.Eventually(t, func() bool { return r.tr.running.Load() == 1 }, 2*time.Second, 5*time.Millisecond)
require.NoError(t, r.w.Submit(r.spool(t, "b"))) // queued behind the running job
cancel()
<-stopped
assert.Len(t, r.conv.srcs, 1, "a job still queued at shutdown must never be started: only the running job was converted")
assert.Empty(t, r.leftovers(t), "audio of queued jobs is deleted on stop: the rows are swept to interrupted at next start")
assert.ErrorIs(t, r.w.Submit(r.spool(t, "c")), uploads.ErrStopped)
}
+2 -2
View File
@@ -14,8 +14,8 @@ import (
"fmt" "fmt"
"sync" "sync"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
"gitea.d-ma.be/mathias/tapir/internal/ports" "git.d-ma.be/mathias/tapir/internal/ports"
) )
// ErrNotImplemented marks scaffold methods awaiting implementation. // ErrNotImplemented marks scaffold methods awaiting implementation.
+1 -1
View File
@@ -4,7 +4,7 @@ import (
"context" "context"
"testing" "testing"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
) )
// These tests pin the ADR-021 read-stored-first behaviour at the engine core: // These tests pin the ADR-021 read-stored-first behaviour at the engine core:
+1 -1
View File
@@ -3,7 +3,7 @@ package web
import ( import (
"net/http" "net/http"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
// handleAccount renders the account page: the user's display name, the // handleAccount renders the account page: the user's display name, the
+2 -2
View File
@@ -9,8 +9,8 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
"gitea.d-ma.be/mathias/tapir/internal/web" "git.d-ma.be/mathias/tapir/internal/web"
) )
// fakeSecrets is a SecretRemover that records the refs it was asked to delete, so // fakeSecrets is a SecretRemover that records the refs it was asked to delete, so
+26 -2
View File
@@ -6,8 +6,10 @@ import (
"net/http" "net/http"
"strings" "strings"
"gitea.d-ma.be/mathias/tapir/internal/adapters/chat" "github.com/a-h/templ"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store"
"git.d-ma.be/mathias/tapir/internal/adapters/chat"
"git.d-ma.be/mathias/tapir/internal/adapters/store"
) )
// Chatter is the per-video chat backend (ADR-027). *chat.Service satisfies it; // Chatter is the per-video chat backend (ADR-027). *chat.Service satisfies it;
@@ -43,6 +45,28 @@ type chatView struct {
History []chat.Turn History []chat.Turn
Truncated bool Truncated bool
Error string Error string
// PostURL and Subject let the same chat panel serve an UPLOADED recording
// (ADR-033): where the form posts, and what the transcript is called. Empty
// means a YouTube video (/v/{VideoID}/chat, "video"), so existing callers are
// unchanged.
PostURL templ.SafeURL
Subject string
}
// postURL is where the chat form posts.
func (v chatView) postURL() templ.SafeURL {
if v.PostURL != "" {
return v.PostURL
}
return chatURL(v.VideoID)
}
// subject is what the transcript is called in the panel's wording.
func (v chatView) subject() string {
if v.Subject != "" {
return v.Subject
}
return "video"
} }
// handleChat renders the chat page for a summarized video (GET). Entry is scoped // handleChat renders the chat page for a summarized video (GET). Entry is scoped
+4 -4
View File
@@ -12,10 +12,10 @@ import (
"github.com/jackc/pgx/v5/pgxpool" "github.com/jackc/pgx/v5/pgxpool"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"gitea.d-ma.be/mathias/tapir/internal/adapters/chat" "git.d-ma.be/mathias/tapir/internal/adapters/chat"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
"gitea.d-ma.be/mathias/tapir/internal/domain" "git.d-ma.be/mathias/tapir/internal/domain"
"gitea.d-ma.be/mathias/tapir/internal/web" "git.d-ma.be/mathias/tapir/internal/web"
) )
// videoZ is a video id used by the isolation test for a DIFFERENT user's video. // videoZ is a video id used by the isolation test for a DIFFERENT user's video.
+2 -2
View File
@@ -12,8 +12,8 @@ import (
"sync" "sync"
"time" "time"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
"gitea.d-ma.be/mathias/tapir/internal/auth" "git.d-ma.be/mathias/tapir/internal/auth"
) )
// Connections is the narrow write port the connect flow depends on (Clean // Connections is the narrow write port the connect flow depends on (Clean
+3 -3
View File
@@ -11,9 +11,9 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"golang.org/x/oauth2" "golang.org/x/oauth2"
"gitea.d-ma.be/mathias/tapir/internal/adapters/store" "git.d-ma.be/mathias/tapir/internal/adapters/store"
"gitea.d-ma.be/mathias/tapir/internal/auth" "git.d-ma.be/mathias/tapir/internal/auth"
"gitea.d-ma.be/mathias/tapir/internal/web" "git.d-ma.be/mathias/tapir/internal/web"
) )
// fakeWriter is a TokenWriter capturing the persisted (ref, value). // fakeWriter is a TokenWriter capturing the persisted (ref, value).

Some files were not shown because too many files have changed in this diff Show More