Compare commits
24
Commits
813192e5ba
...
v0.32.1
+8
-2
@@ -1551,8 +1551,14 @@ deferred. The scope cut — chat first, no rich analysis, no video — is what k
|
||||
disabled by default, a failed iguana job fails honestly and the user retries. *Blocks:*
|
||||
the production default only; the adapter ships either way.
|
||||
- ~~Verify Tapir's ingress `proxy-body-size` against the 200 MB cap (`infra`).~~ **Resolved 2026-10-03:** real unauthenticated bodies of 2, 30, 120 and 210 MiB all pass NPM (TLS, piguard) and Traefik to tapir (a 302 to login); there is no body limit in that range. Uploads were enabled in production the same day, local engines only (infra#520).
|
||||
- Per-transcript deletion is not built; account deletion cascades. A private-content store
|
||||
without a delete button is a gap worth a ticket, not a silent omission.
|
||||
- ~~Per-transcript deletion is not built; account deletion cascades.~~ **Built 2026-10-03
|
||||
(#37):** `POST /uploads/{id}/delete`, owner-scoped (a non-owner gets 404 by the same RLS
|
||||
mechanism), confirm-then-delete on the recording's page. A recording that is **queued or
|
||||
transcribing is refused (409) until it ends**, because the worker would otherwise write to a
|
||||
row that is gone; a job whose service was lost stays undeletable until the next start's sweep
|
||||
marks it failed. The refusal reflects the row's state when the delete was attempted: a
|
||||
recording that finishes in the same instant still gets a 409 once, and a retry deletes it.
|
||||
The home list now links to `/uploads` when uploads are mounted.
|
||||
|
||||
**Reversibility.** Additive: one table (down-migration drops it), one adapter, one upload
|
||||
surface, one image change. Removing the upload route removes the feature; `transcripts`, `videos`
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
# ADR-034 second blind check: protocol (ready to run; tapir#33, D9, D10)
|
||||
|
||||
This repo is public and the recordings are private, so this file holds the method only. The
|
||||
sheet, the key and the filled-in verdicts stay outside the repo, and the trial data is deleted
|
||||
after the check. Results go back here as aggregate numbers, as in
|
||||
`reconcile-blind-check-2026-10-02.md`.
|
||||
|
||||
## What the check decides
|
||||
|
||||
ADR-034 (two engines, a local model reconciles their disagreements) is `Proposed` and off in
|
||||
production. Two conditions were set for keeping it. The first run answered the first one on one
|
||||
reading only and did not answer the second:
|
||||
|
||||
1. **The reconciled reading is right at least 85% of the time.** Open flaw in the bar: it did not
|
||||
say how ties count. The first run gave 95% (decided items only) or 70% (all 30), so it was met on
|
||||
one reading and missed on the other.
|
||||
2. **It is no worse than the better single engine.** Not answered: before ADR-035 the spots the
|
||||
model *kept* as the base were not stored, so kb-whisper alone could not be compared.
|
||||
|
||||
ADR-035 now stores every judged spot (`reconciliation.contested`: both readings, which one was
|
||||
used, unsure or not, a few words of left context, a time). That makes condition 2 measurable.
|
||||
|
||||
## Rules to fix BEFORE looking at any result (the owner confirms or changes them)
|
||||
|
||||
| Rule | Proposed |
|
||||
|---|---|
|
||||
| Who judges | The speaker, hearing the audio where the text cannot settle it (D9). Not Opus: it cannot hear, and its agreement with the speaker was only shown on items picked as decidable. |
|
||||
| Sample | 30 spots drawn at random with a fixed seed, from ALL judged spots (changed and kept-base alike), not only the changed ones. Insertions and deletions that cannot be placed are listed separately and not sampled. |
|
||||
| Verdict per spot | `x`, `y` (the two readings, shown in random order, key kept apart), `neither`, or `cannot tell`. |
|
||||
| Condition 1 | Reconciled is right on at least 85% of the spots where the judge could tell (`cannot tell` excluded, reported, and `neither` counts as not right). Also report all-30. Pass needs the decided-only figure at 85% or more AND the all-30 figure above 70%. |
|
||||
| Condition 2 | Over the same spots, the reconciled text is right at least as often as the better single engine (base engine alone is right where the base reading is right; the other engine alone where the alt reading is right). Ties favour the single engine. |
|
||||
| If it fails | Switch reconcile off by configuration; the single-engine path is unchanged (ADR-034 reversibility). |
|
||||
|
||||
Why a fixed all-30 floor: the literal 70% last time sat well under the bar, and "decided only" can
|
||||
hide a model that mostly lands on spelling differences.
|
||||
|
||||
## What a run needs (D10: the owner is present)
|
||||
|
||||
- The same real recording as before (about 60 minutes). Only the owner has it.
|
||||
- A **throwaway** tapir instance with its own Postgres, not the production pod, so production's
|
||||
key pin and deployment are untouched. Settings: `TAPIR_RECONCILE_BASE_MODEL=iguana/pianissimo`,
|
||||
`TAPIR_TRANSCRIBE_MODEL=iguana/kb-whisper`, `TAPIR_RECONCILE_MODEL=iguana/gemma4-31b`, cloud
|
||||
fallbacks empty. Job time was 21 min 49 s per 63.6 min last run.
|
||||
- iguana with Low Power Mode off and gemma4-31b able to load (the production-iguana model load
|
||||
that needs a quiet window; D6).
|
||||
|
||||
## Steps
|
||||
|
||||
1. Start the throwaway instance, upload the recording, wait for `done` (about 22 minutes).
|
||||
2. Read `reconciliation.contested` from the throwaway database. From it build, **outside the repo**:
|
||||
`sheet` (spot number, time, left context, reading x, reading y, random order) and `key` (which
|
||||
of x/y is the base, which is the other engine, which one the reconciler used).
|
||||
3. The speaker fills in the sheet (about 20 to 30 minutes for 30 spots, audio open at the times).
|
||||
The agent does not see the key until the sheet is returned.
|
||||
4. Score with the rules above. Write the aggregate numbers into a dated note next to this one,
|
||||
and amend ADR-034's validation section (kept, or switched off, and the tie rule used).
|
||||
5. Delete the throwaway instance, its database and the sheet, as last time.
|
||||
6. Only if the check passes and the owner wants it in production: enabling reconcile there is
|
||||
the separate D10 step (it needs `iguana/pianissimo` and the reconciler added to the tapir key
|
||||
pin in the infra repo plus the env vars on the deployment, `risk/medium`, owner present).
|
||||
|
||||
## Open and owner-only
|
||||
|
||||
- Confirm or change the rules table above before the run.
|
||||
- The **tool that builds the sheet and key** is the one piece not in the repo (last time it was a
|
||||
one-off script). It can be built test-first on request; until then the agent builds it ad hoc
|
||||
outside the repo.
|
||||
- A Berget adjudicator (D8, tapir#45) is a different question with its own measurement and is
|
||||
not part of this check: this check judges the local setup that ADR-034 describes.
|
||||
@@ -1,7 +1,8 @@
|
||||
# Sprint: audio upload follow-ups (Sweep Sprint, 2026-10)
|
||||
|
||||
Method: Sweep Sprint (phase 1 parallel read-only research per issue: **done**; phase 2 sequential
|
||||
execution, one issue per commit: **not started, waits for the owner's go**). Closing oath below.
|
||||
execution, one issue per commit: **v0.31.1 and v0.32.0 shipped 2026-10-03/04, the owner-held items
|
||||
below remain open**). Closing oath below.
|
||||
|
||||
Milestones: [v0.31.1 - security and robustness](https://git.d-ma.be/mathias/tapir/milestone/10),
|
||||
[v0.32.0 - audio upload follow-ups](https://git.d-ma.be/mathias/tapir/milestone/11).
|
||||
@@ -63,8 +64,8 @@ Release v0.31.1: one adversarial review round over 2-5, then tag, push, observe.
|
||||
| D5 | #38: Berget length measurement needs a throwaway key | Yes, throwaway key | #38 |
|
||||
| D6 | #38: load large models on production iguana | Yes, in a quiet window with the owner present | #38 |
|
||||
| D7 | #46: where the ffmpeg tarball lives | Do nothing now; file an issue | none |
|
||||
| D8 | #45: evaluate Berget as the reconciler | No; stay local-only | #45 |
|
||||
| D9 | Who judges ADR-034's second blind check | The owner; do not reuse the one-off Opus authorisation | ADR-034 |
|
||||
| D8 | #45: evaluate Berget as the reconciler | **Decided 2026-10-04 by the owner: yes, Berget may be used, and each user must be able to keep a recording on local models only** (reverses the recommendation). Opt-out first: #60; #45 stays as the model measurement | #60, #45 |
|
||||
| D9 | Who judges ADR-034's second blind check | The owner; do not reuse the one-off Opus authorisation. Protocol: `docs/research/reconcile-second-check-protocol.md` | ADR-034 |
|
||||
| D10 | When to enable reconcile in production | Needs `iguana/pianissimo` in the key pin and env vars; owner present | ADR-034 |
|
||||
|
||||
## Dropped or reduced, with reasons
|
||||
@@ -86,3 +87,29 @@ Release v0.31.1: one adversarial review round over 2-5, then tag, push, observe.
|
||||
SemVer: v0.31.1 patch (fixes, no new behaviour); v0.32.0 minor (nav link, delete, cloud labelling).
|
||||
Every push to tapir main redeploys the pod and fails any job in flight: push when no upload is
|
||||
running (check the pod log first).
|
||||
|
||||
## Status (2026-10-04)
|
||||
|
||||
**Shipped and deployed:** v0.31.1 (PR #55, merge e037c6b) and v0.32.0 (PR #58, merge 85f4cc3; pod
|
||||
tapir-597dbfccc-ng78q 1/1, 0 restarts, image 85f4cc3). Each went through an adversarial review
|
||||
round by agent with the verdicts recorded on the PR (v0.31.1: BLOCK then APPROVE; v0.32.0: BLOCK,
|
||||
BLOCK, APPROVE, APPROVE; every block was a real defect, fixed test-first).
|
||||
|
||||
**Live evidence:** a foreign-origin `POST` to `/account/delete` and to `/uploads/{id}/delete` is
|
||||
refused (403) on production, a same-origin one reaches the login redirect (302).
|
||||
|
||||
**Not observed live (needs an authenticated session, so it is the owner's look; closed on word or screenshot):**
|
||||
|
||||
- #52, cloud side: a transcript served by the Berget fallback names `berget`; a reconciled all-local
|
||||
one shows no note. The local side was observed on 2026-10-04 (a new clip served by iguana: chip
|
||||
`iguana/…`, no cloud note). The cloud side is carried as an acceptance item in #60.
|
||||
|
||||
**Observed live since:** #37 (owner, 2026-10-04: link, confirm-then-delete, no control while transcribing; closed). #50: at the 2026-10-04 rollout the old pod went `Terminating` then `Completed`
|
||||
(not `Error`), 0 restarts; #50 closed. No job was in flight, so a drained job is still unobserved
|
||||
(a recorded limitation, D2).
|
||||
|
||||
**Still open (owner's call, milestone 11):** #36, #38, #45 (now the Berget adjudicator measurement), #60 (per-recording local-only choice, 4 sub-decisions), #46 remainder, #56 (slow-drip bodies hold
|
||||
a receive slot), #57 (shutdown logging, second signal). Decisions D4-D6, D9, D10 above are unchanged; D8 is decided (above).
|
||||
|
||||
**Deferred and filed from the reviews:** #56, #57, #59 (start-up exits when the identity provider's OIDC
|
||||
discovery returns 503 (seen once at 19:40 on 2026-10-03; the pod restarted itself and recovered)).
|
||||
|
||||
@@ -245,6 +245,19 @@ Feature: Upload an audio recording, get a private Swedish transcript, discuss it
|
||||
Then its page says in plain words that the audio left the homelab and names the provider
|
||||
And a locally transcribed recording carries no such note
|
||||
|
||||
Scenario: I can delete a finished recording
|
||||
When I confirm deleting a done or failed recording
|
||||
Then the recording and its transcript are gone and I am back on my list
|
||||
And another user's recording is a 404 and is untouched
|
||||
|
||||
Scenario: A recording still being transcribed cannot be deleted
|
||||
When I try to delete a recording that is queued or transcribing
|
||||
Then I am told it is still being transcribed and nothing is removed
|
||||
|
||||
Scenario: The home list links to my recordings when uploads are on
|
||||
Given audio upload is enabled
|
||||
Then the home list links to the recordings page
|
||||
|
||||
Scenario: A full queue is a retry, not an error
|
||||
When too many recordings are already waiting
|
||||
Then the upload is answered as busy, its row is failed honestly and its audio is deleted
|
||||
|
||||
@@ -33,9 +33,9 @@ require (
|
||||
github.com/rogpeppe/go-internal v1.15.0 // indirect
|
||||
github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 // indirect
|
||||
go.yaml.in/yaml/v2 v2.4.2 // indirect
|
||||
golang.org/x/sync v0.18.0 // indirect
|
||||
golang.org/x/sync v0.21.0 // indirect
|
||||
golang.org/x/sys v0.41.0 // indirect
|
||||
golang.org/x/text v0.31.0 // indirect
|
||||
golang.org/x/text v0.39.0 // indirect
|
||||
google.golang.org/protobuf v1.36.8 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
)
|
||||
|
||||
@@ -113,12 +113,12 @@ go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=
|
||||
go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
|
||||
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
|
||||
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
|
||||
golang.org/x/sync v0.18.0 h1:kr88TuHDroi+UVf+0hZnirlk8o8T+4MrK6mr60WkH/I=
|
||||
golang.org/x/sync v0.18.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
|
||||
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
|
||||
golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
|
||||
golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
|
||||
golang.org/x/text v0.31.0 h1:aC8ghyu4JhP8VojJ2lEHBnochRno1sgL6nEi9WGFGMM=
|
||||
golang.org/x/text v0.31.0/go.mod h1:tKRAlv61yKIjGGHX/4tP1LTbc13YSec1pxVEWXzfoeM=
|
||||
golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus=
|
||||
golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM=
|
||||
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
|
||||
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
|
||||
google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc=
|
||||
|
||||
@@ -79,3 +79,78 @@ func TestClient_AFailingBodyIsReadOnlyUpToTheErrorBound(t *testing.T) {
|
||||
t.Fatalf("the client kept reading an endless error body: took %s", time.Since(start))
|
||||
}
|
||||
}
|
||||
|
||||
// sizedGateway answers 200 with a valid completion padded with trailing whitespace to
|
||||
// exactly size bytes, so the boundary of the cap can be probed to the byte.
|
||||
func sizedGateway(t *testing.T, size int) *httptest.Server {
|
||||
t.Helper()
|
||||
const base = `{"choices":[{"message":{"content":"ok"}}]}`
|
||||
payload := base + strings.Repeat(" ", size-len(base))
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(payload))
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return srv
|
||||
}
|
||||
|
||||
func TestClient_ABodyOfExactlyTheCapIsReadAndOneByteMoreIsRefused(t *testing.T) {
|
||||
const capBytes = 4096
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
size int
|
||||
wantErr bool
|
||||
}{
|
||||
{"one under", capBytes - 1, false},
|
||||
{"exactly the cap", capBytes, false},
|
||||
{"one over", capBytes + 1, true},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
c := New(sizedGateway(t, tt.size).URL, "", "m", 5*time.Second, WithMaxResponseBytes(capBytes))
|
||||
|
||||
got, err := c.Complete(context.Background(), "s", "u")
|
||||
|
||||
if tt.wantErr {
|
||||
if err == nil || !strings.Contains(err.Error(), "exceeds") {
|
||||
t.Fatalf("want a response-too-large error, got %q, %v", got, err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil || got != "ok" {
|
||||
t.Fatalf("want ok, got %q, %v", got, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// New is what production calls: no option. The default is therefore the live bound.
|
||||
func TestClient_TheDefaultCapIsEightMiBWhenNoOptionIsGiven(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
size int
|
||||
wantErr bool
|
||||
}{
|
||||
{"exactly 8 MiB", 8 << 20, false},
|
||||
{"8 MiB plus one", 8<<20 + 1, true},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
c := New(sizedGateway(t, tt.size).URL, "", "m", 10*time.Second)
|
||||
|
||||
_, err := c.Complete(context.Background(), "s", "u")
|
||||
|
||||
if (err != nil) != tt.wantErr {
|
||||
t.Fatalf("size %d: err = %v, wantErr %v", tt.size, err, tt.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestClient_ANonPositiveCapOptionKeepsTheDefault(t *testing.T) {
|
||||
for _, n := range []int64{0, -1} {
|
||||
c := New(sizedGateway(t, 1024).URL, "", "m", 5*time.Second, WithMaxResponseBytes(n))
|
||||
|
||||
if got, err := c.Complete(context.Background(), "s", "u"); err != nil || got != "ok" {
|
||||
t.Fatalf("WithMaxResponseBytes(%d): want ok from the default cap, got %q, %v", n, got, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -202,6 +202,7 @@ func (c *Client) Complete(ctx context.Context, system, user string) (string, err
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
// Only the bounded head is ever used (HTTPError.Error truncates to it), so read no more.
|
||||
// A read error here only shortens a diagnostic; the status code below is what callers act on.
|
||||
head, _ := io.ReadAll(io.LimitReader(resp.Body, maxErrorBody+1))
|
||||
return "", &HTTPError{Code: resp.StatusCode, Body: string(head)}
|
||||
}
|
||||
|
||||
@@ -17,6 +17,10 @@ import (
|
||||
// nothing about whether someone else's upload exists (ADR-033).
|
||||
var ErrUploadNotFound = errors.New("store: uploaded transcript not found")
|
||||
|
||||
// ErrUploadInProgress is returned by DeleteUpload for an upload that is still queued or
|
||||
// transcribing: the worker would write its result to a row that no longer exists.
|
||||
var ErrUploadInProgress = errors.New("store: uploaded transcript is still being processed")
|
||||
|
||||
// Uploaded transcripts are PRIVATE per-user content, stored in their own
|
||||
// RLS-scoped table (migration 017) — never in the shared `transcripts` (ADR-021).
|
||||
// Every method goes through withUser, so row-level security confines it to the
|
||||
@@ -80,12 +84,14 @@ func (s *Store) SaveProgress(ctx context.Context, userID, id string, steps []dom
|
||||
WHERE id = $1 AND user_id = $2`, b, at)
|
||||
}
|
||||
|
||||
// Heartbeat stamps the upload as alive now. It does not touch progress_at: alive says
|
||||
// the service is still reporting, not that work is moving (ADR-035).
|
||||
func (s *Store) Heartbeat(ctx context.Context, userID, id string) error {
|
||||
// Heartbeat stamps the upload as alive at the given time. It does not touch progress_at:
|
||||
// alive says the service is still reporting, not that work is moving (ADR-035). The time
|
||||
// comes from the application clock, the one progress_at and the page's health check use,
|
||||
// so a database clock that disagrees cannot make a healthy job look lost (#51).
|
||||
func (s *Store) Heartbeat(ctx context.Context, userID, id string, at time.Time) error {
|
||||
return s.updateUpload(ctx, userID, id, "heartbeat",
|
||||
`UPDATE uploaded_transcripts SET alive_at = NOW()
|
||||
WHERE id = $1 AND user_id = $2`)
|
||||
`UPDATE uploaded_transcripts SET alive_at = $3
|
||||
WHERE id = $1 AND user_id = $2`, at)
|
||||
}
|
||||
|
||||
// FailUpload moves a queued or transcribing upload to failed with a typed reason. A
|
||||
@@ -101,6 +107,33 @@ func (s *Store) FailUpload(ctx context.Context, userID, id string, kind domain.U
|
||||
string(kind))
|
||||
}
|
||||
|
||||
// DeleteUpload removes one of the caller's finished (done or failed) uploads. An upload
|
||||
// still queued or transcribing is refused with ErrUploadInProgress and left untouched; one
|
||||
// that is not the caller's is ErrUploadNotFound, as for every other write (#37).
|
||||
func (s *Store) DeleteUpload(ctx context.Context, userID, id string) error {
|
||||
return s.withUser(ctx, userID, func(tx pgx.Tx) error {
|
||||
ct, err := tx.Exec(ctx,
|
||||
`DELETE FROM uploaded_transcripts
|
||||
WHERE id = $1 AND user_id = $2 AND status IN ('done', 'failed')`, id, userID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("store: delete upload: %w", err)
|
||||
}
|
||||
if ct.RowsAffected() > 0 {
|
||||
return nil
|
||||
}
|
||||
var exists bool
|
||||
if err := tx.QueryRow(ctx,
|
||||
`SELECT EXISTS (SELECT 1 FROM uploaded_transcripts WHERE id = $1 AND user_id = $2)`,
|
||||
id, userID).Scan(&exists); err != nil {
|
||||
return fmt.Errorf("store: delete upload: %w", err)
|
||||
}
|
||||
if exists {
|
||||
return ErrUploadInProgress
|
||||
}
|
||||
return ErrUploadNotFound
|
||||
})
|
||||
}
|
||||
|
||||
// updateUpload runs one UPDATE keyed on ($1 = id, $2 = userID) plus extra args,
|
||||
// and maps "no row touched" to ErrUploadNotFound — which is what a write to another
|
||||
// user's upload looks like under RLS (the row is invisible, not forbidden).
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
package store_test
|
||||
|
||||
// Deleting one recording (#37). A finished or failed upload can be removed by its owner;
|
||||
// one that is queued or transcribing is refused until it ends, because the worker would
|
||||
// otherwise write its result to a row that is gone (D3 in the sprint plan).
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"git.d-ma.be/mathias/tapir/internal/adapters/store"
|
||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||
)
|
||||
|
||||
func TestDeleteUpload_RemovesAFinishedOrFailedUploadOfTheOwner(t *testing.T) {
|
||||
for name, finish := range map[string]func(*store.Store, string) error{
|
||||
"done": func(s *store.Store, id string) error {
|
||||
return s.CompleteUpload(context.Background(), userA, id, domain.Transcription{Text: "synthetic", Endpoint: "iguana/kb-whisper"})
|
||||
},
|
||||
"failed": func(s *store.Store, id string) error {
|
||||
return s.FailUpload(context.Background(), userA, id, domain.UploadErrTranscriptionFailed)
|
||||
},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
s := newStore(t)
|
||||
id := startedUpload(t, s)
|
||||
require.NoError(t, s.StartTranscribing(context.Background(), userA, id))
|
||||
require.NoError(t, finish(s, id))
|
||||
|
||||
require.NoError(t, s.DeleteUpload(context.Background(), userA, id))
|
||||
|
||||
_, err := s.GetUpload(context.Background(), userA, id)
|
||||
assert.ErrorIs(t, err, store.ErrUploadNotFound)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeleteUpload_RefusesAnUploadThatIsStillQueuedOrTranscribingAndKeepsTheRow(t *testing.T) {
|
||||
s := newStore(t)
|
||||
ctx := context.Background()
|
||||
queued := startedUpload(t, s)
|
||||
running, err := s.CreateUpload(ctx, userA, "running", false)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, s.StartTranscribing(ctx, userA, running))
|
||||
|
||||
for _, id := range []string{queued, running} {
|
||||
assert.ErrorIs(t, s.DeleteUpload(ctx, userA, id), store.ErrUploadInProgress)
|
||||
_, err := s.GetUpload(ctx, userA, id)
|
||||
assert.NoError(t, err, "a refused delete must leave the row")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeleteUpload_AnotherUsersUploadIsNotFoundAndSurvives(t *testing.T) {
|
||||
s := newStore(t)
|
||||
ctx := context.Background()
|
||||
id := startedUpload(t, s)
|
||||
require.NoError(t, s.StartTranscribing(ctx, userA, id))
|
||||
require.NoError(t, s.FailUpload(ctx, userA, id, domain.UploadErrTranscriptionFailed))
|
||||
|
||||
assert.ErrorIs(t, s.DeleteUpload(ctx, userB, id), store.ErrUploadNotFound,
|
||||
"existence is not disclosed to a non-owner")
|
||||
|
||||
_, err := s.GetUpload(ctx, userA, id)
|
||||
assert.NoError(t, err, "B's delete must not have removed A's row")
|
||||
}
|
||||
@@ -37,7 +37,7 @@ type Reporter interface {
|
||||
// is for people, so a failed write is logged and never stops the job.
|
||||
type Sink interface {
|
||||
SaveProgress(ctx context.Context, steps []domain.ProgressStep, at time.Time) error
|
||||
Beat(ctx context.Context) error
|
||||
Beat(ctx context.Context, at time.Time) error
|
||||
}
|
||||
|
||||
const (
|
||||
@@ -57,8 +57,11 @@ func WithClock(now func() time.Time) Option { return func(t *Tracker) { t.now =
|
||||
func WithMinSave(d time.Duration) Option { return func(t *Tracker) { t.minSave = d } }
|
||||
|
||||
// WithSaveTimeout bounds one save or heartbeat write. Progress is cosmetic, so a database
|
||||
// that stops answering costs a dropped write, never a stalled job: saves run under the
|
||||
// tracker's lock, which Begin and Advance (called by the engines) need (#51).
|
||||
// that stops answering costs a dropped write rather than an unbounded stall. It is still a
|
||||
// delay: saves run under the tracker's lock, which Begin and Advance (called by the engines)
|
||||
// need, so each transition can wait up to this long, and a hung database slows a job by
|
||||
// that much per call, out of the job's own time budget (#51). A dropped final save is not
|
||||
// retried; the row keeps its last saved step.
|
||||
func WithSaveTimeout(d time.Duration) Option { return func(t *Tracker) { t.saveTimeout = d } }
|
||||
|
||||
// Tracker implements Reporter for one job.
|
||||
@@ -221,7 +224,7 @@ func (t *Tracker) Heartbeat(ctx context.Context, beatEvery, stillEvery time.Dura
|
||||
return
|
||||
case <-beat.C:
|
||||
beatCtx, cancel := context.WithTimeout(ctx, t.saveTimeout)
|
||||
err := t.sink.Beat(beatCtx)
|
||||
err := t.sink.Beat(beatCtx, t.now())
|
||||
cancel()
|
||||
if err != nil && ctx.Err() == nil {
|
||||
t.log.Warn("progress: heartbeat not saved", "err", err)
|
||||
|
||||
@@ -96,3 +96,59 @@ func TestTracker_TheStillRunningLineIsNotHeldUpBehindAHungSave(t *testing.T) {
|
||||
|
||||
require.Eventually(t, func() bool { return strings.Contains(buf.String(), "still running") }, 3*time.Second, 5*time.Millisecond)
|
||||
}
|
||||
|
||||
// deadlineSink records how long each write was given, and answers at once.
|
||||
type deadlineSink struct {
|
||||
mu sync.Mutex
|
||||
saves []time.Duration
|
||||
beats []time.Duration
|
||||
noDLOK bool
|
||||
}
|
||||
|
||||
func (d *deadlineSink) record(ctx context.Context, into *[]time.Duration) {
|
||||
d.mu.Lock()
|
||||
defer d.mu.Unlock()
|
||||
dl, ok := ctx.Deadline()
|
||||
if !ok {
|
||||
d.noDLOK = true
|
||||
return
|
||||
}
|
||||
*into = append(*into, time.Until(dl))
|
||||
}
|
||||
|
||||
func (d *deadlineSink) SaveProgress(ctx context.Context, _ []domain.ProgressStep, _ time.Time) error {
|
||||
d.record(ctx, &d.saves)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (d *deadlineSink) Beat(ctx context.Context, _ time.Time) error {
|
||||
d.record(ctx, &d.beats)
|
||||
return nil
|
||||
}
|
||||
|
||||
// The production value is the one the worker gets: it builds a Tracker without options.
|
||||
// A zero or tiny default would drop every write, a huge one would bound nothing.
|
||||
func TestTracker_TheDefaultSaveAndBeatDeadlineIsAFewSecondsAway(t *testing.T) {
|
||||
sink := &deadlineSink{}
|
||||
tr := progress.NewTracker(context.Background(), sink, slog.New(slog.NewTextHandler(&lockedBuf{}, nil)),
|
||||
[]progress.Planned{{Key: "a", Label: "A"}})
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
go tr.Heartbeat(ctx, 5*time.Millisecond, time.Hour)
|
||||
|
||||
tr.Begin("a", 0)
|
||||
require.Eventually(t, func() bool {
|
||||
sink.mu.Lock()
|
||||
defer sink.mu.Unlock()
|
||||
return len(sink.beats) > 0
|
||||
}, 3*time.Second, 5*time.Millisecond)
|
||||
|
||||
sink.mu.Lock()
|
||||
defer sink.mu.Unlock()
|
||||
require.False(t, sink.noDLOK, "a write was given no deadline at all")
|
||||
require.NotEmpty(t, sink.saves)
|
||||
for _, d := range append(append([]time.Duration{}, sink.saves...), sink.beats...) {
|
||||
require.Greater(t, d, time.Second, "the deadline is already (nearly) spent when the write starts")
|
||||
require.LessOrEqual(t, d, 5*time.Second, "the default bound is 5 s")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
package uploads
|
||||
|
||||
// Observes what jobSink hands the store: the heartbeat's timestamp is the one the tracker
|
||||
// stamped (its application clock), not a second reading taken here (#51).
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.d-ma.be/mathias/tapir/internal/domain"
|
||||
)
|
||||
|
||||
type atStore struct{ beatAt, saveAt time.Time }
|
||||
|
||||
func (*atStore) StartTranscribing(context.Context, string, string) error { return nil }
|
||||
func (*atStore) CompleteUpload(context.Context, string, string, domain.Transcription) error {
|
||||
return nil
|
||||
}
|
||||
func (*atStore) FailUpload(context.Context, string, string, domain.UploadErrorKind) error { return nil }
|
||||
func (s *atStore) SaveProgress(_ context.Context, _, _ string, _ []domain.ProgressStep, at time.Time) error {
|
||||
s.saveAt = at
|
||||
return nil
|
||||
}
|
||||
func (s *atStore) Heartbeat(_ context.Context, _, _ string, at time.Time) error {
|
||||
s.beatAt = at
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestJobSink_ForwardsTheTrackersTimestampToTheStoreUnchanged(t *testing.T) {
|
||||
st := &atStore{}
|
||||
sink := jobSink{st: st, userID: "u", id: "i"}
|
||||
stamp := time.Date(2020, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||
|
||||
if err := sink.Beat(context.Background(), stamp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := sink.SaveProgress(context.Background(), nil, stamp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if !st.beatAt.Equal(stamp) {
|
||||
t.Errorf("heartbeat reached the store stamped %v, want the tracker's %v", st.beatAt, stamp)
|
||||
}
|
||||
if !st.saveAt.Equal(stamp) {
|
||||
t.Errorf("progress save reached the store stamped %v, want the tracker's %v", st.saveAt, stamp)
|
||||
}
|
||||
}
|
||||
@@ -47,7 +47,7 @@ type WorkerStore interface {
|
||||
CompleteUpload(ctx context.Context, userID, id string, t domain.Transcription) error
|
||||
FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error
|
||||
SaveProgress(ctx context.Context, userID, id string, steps []domain.ProgressStep, at time.Time) error
|
||||
Heartbeat(ctx context.Context, userID, id string) error
|
||||
Heartbeat(ctx context.Context, userID, id string, at time.Time) error
|
||||
}
|
||||
|
||||
// WorkerOption tunes a Worker.
|
||||
@@ -91,7 +91,9 @@ func (s jobSink) SaveProgress(ctx context.Context, steps []domain.ProgressStep,
|
||||
return s.st.SaveProgress(ctx, s.userID, s.id, steps, at)
|
||||
}
|
||||
|
||||
func (s jobSink) Beat(ctx context.Context) error { return s.st.Heartbeat(ctx, s.userID, s.id) }
|
||||
func (s jobSink) Beat(ctx context.Context, at time.Time) error {
|
||||
return s.st.Heartbeat(ctx, s.userID, s.id, at)
|
||||
}
|
||||
|
||||
// Worker transcribes accepted uploads in-process, ONE AT A TIME (ADR-033). It is
|
||||
// not a second deployable: it shares the web process's lifetime, which is why it
|
||||
|
||||
@@ -125,6 +125,20 @@ func TestCrossOrigin_AForeignPostIsRefusedOnEachListedStateChangingPath(t *testi
|
||||
}
|
||||
}
|
||||
|
||||
func TestCrossOrigin_ARefusedRecordingDeletePostLeavesTheRecordingInPlace(t *testing.T) {
|
||||
for name, h := range foreign {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
||||
|
||||
rec := do(t, app, withHeaders(httptest.NewRequest(http.MethodPost, "/uploads/"+id+"/delete", nil), h))
|
||||
|
||||
assert.Equal(t, http.StatusForbidden, rec.Code)
|
||||
assert.True(t, uploadExists(t, id), "a refused post must not delete the recording")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A login redirect comes back to /auth/callback as a cross-site GET navigation, and
|
||||
// health probes carry no headers: safe methods must never be refused on origin grounds.
|
||||
func TestCrossOrigin_ACrossSiteGETIsNotRefused(t *testing.T) {
|
||||
|
||||
@@ -59,6 +59,9 @@ type Store interface {
|
||||
// non-owner's id is ErrUploadNotFound.
|
||||
GetUpload(ctx context.Context, userID, id string) (*domain.UploadedTranscript, error)
|
||||
ListUploads(ctx context.Context, userID string) ([]domain.UploadedTranscript, error)
|
||||
// DeleteUpload removes a finished upload; ErrUploadInProgress while it is still being
|
||||
// processed, ErrUploadNotFound for a non-owner.
|
||||
DeleteUpload(ctx context.Context, userID, id string) error
|
||||
|
||||
// SetTranscriptStatus clears or updates a video's transcript backoff state.
|
||||
// Used by handleRetryNow to clear rate_limited_at before immediate processing.
|
||||
@@ -200,6 +203,7 @@ func (a *App) Router() http.Handler {
|
||||
app.HandleFunc("POST /uploads", a.handleUploadPost)
|
||||
app.HandleFunc("GET /uploads/{id}", a.handleUploadDetail)
|
||||
app.HandleFunc("GET /uploads/{id}/status", a.handleUploadStatus)
|
||||
app.HandleFunc("POST /uploads/{id}/delete", a.handleUploadDelete)
|
||||
// Chat over the uploaded transcript reuses the ADR-027 Chatter; mounted only
|
||||
// when one is wired.
|
||||
if a.Chat != nil {
|
||||
@@ -309,7 +313,7 @@ func (a *App) handleList(w http.ResponseWriter, r *http.Request) {
|
||||
a.render(w, r, summaryList(buckets, hasConnected, autoSummarize))
|
||||
return
|
||||
}
|
||||
a.render(w, r, ListPage(buckets, f, stats, takeFlash(w, r), hasConnected, channels, autoSummarize))
|
||||
a.render(w, r, ListPage(buckets, f, stats, takeFlash(w, r), hasConnected, channels, autoSummarize, a.Uploads != nil))
|
||||
}
|
||||
|
||||
// handleDetail renders one summary in full (highlights, takeaways, action group).
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
package oidc_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"git.d-ma.be/mathias/tapir/internal/web/oidc"
|
||||
)
|
||||
|
||||
func testConfig(issuer string) oidc.Config {
|
||||
return oidc.Config{
|
||||
Issuer: issuer,
|
||||
ClientID: testClientID,
|
||||
ClientSecret: "test-client-secret",
|
||||
RedirectURL: "http://tapir.test/auth/callback",
|
||||
SessionSecret: "test-session-secret-please-change",
|
||||
}
|
||||
}
|
||||
|
||||
// #59: a discovery 503 at start-up is retried instead of failing New.
|
||||
func TestNewRetriesDiscoveryAfter503UntilIssuerAnswers(t *testing.T) {
|
||||
f := newFakeIssuer(t)
|
||||
f.unavailableFor = 2
|
||||
|
||||
auth, err := oidc.New(context.Background(), testConfig(f.server.URL),
|
||||
oidc.WithDiscoveryRetry(5, time.Millisecond))
|
||||
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, auth)
|
||||
require.Equal(t, int32(3), f.discoveryHits.Load())
|
||||
}
|
||||
|
||||
func TestNewReturnsDiscoveryErrorAfterAttemptsExhausted(t *testing.T) {
|
||||
f := newFakeIssuer(t)
|
||||
f.unavailableFor = 1000
|
||||
|
||||
_, err := oidc.New(context.Background(), testConfig(f.server.URL),
|
||||
oidc.WithDiscoveryRetry(3, time.Millisecond))
|
||||
|
||||
require.ErrorContains(t, err, "503")
|
||||
require.Equal(t, int32(3), f.discoveryHits.Load())
|
||||
}
|
||||
|
||||
func TestNewStopsDiscoveryBackoffWhenContextCancelled(t *testing.T) {
|
||||
f := newFakeIssuer(t)
|
||||
f.unavailableFor = 1000
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
start := time.Now()
|
||||
_, err := oidc.New(ctx, testConfig(f.server.URL),
|
||||
oidc.WithDiscoveryRetry(5, time.Hour))
|
||||
|
||||
require.True(t, errors.Is(err, context.DeadlineExceeded), "err = %v", err)
|
||||
require.Less(t, time.Since(start), 5*time.Second)
|
||||
require.Equal(t, int32(1), f.discoveryHits.Load())
|
||||
}
|
||||
+70
-20
@@ -57,6 +57,10 @@ const (
|
||||
pendingTTL = 10 * time.Minute
|
||||
sessionCookie = "tapir_session"
|
||||
loginPath = "/auth/login"
|
||||
|
||||
// Discovery retry (#59): 5 tries, 2+4+8+16 s back-off ≈ 30 s, then fail.
|
||||
defaultDiscoveryAttempts = 5
|
||||
defaultDiscoveryDelay = 2 * time.Second
|
||||
)
|
||||
|
||||
// DexAuth implements web.Auth against Dex.
|
||||
@@ -70,6 +74,9 @@ type DexAuth struct {
|
||||
sessionTTL time.Duration
|
||||
now func() time.Time
|
||||
insecure bool // tests only: drop the Secure attribute so cookies flow over http
|
||||
|
||||
discoveryAttempts int
|
||||
discoveryDelay time.Duration
|
||||
}
|
||||
|
||||
var _ web.Auth = (*DexAuth)(nil)
|
||||
@@ -96,14 +103,55 @@ func WithSessionTTL(ttl time.Duration) Option {
|
||||
}
|
||||
}
|
||||
|
||||
// WithDiscoveryRetry overrides the discovery attempt count and the initial
|
||||
// back-off delay (doubled after each failed attempt).
|
||||
func WithDiscoveryRetry(attempts int, delay time.Duration) Option {
|
||||
return func(d *DexAuth) {
|
||||
if attempts > 0 {
|
||||
d.discoveryAttempts = attempts
|
||||
}
|
||||
if delay > 0 {
|
||||
d.discoveryDelay = delay
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// WithInsecureCookies drops the Secure cookie attribute. TEST ONLY — it lets
|
||||
// the session cookie travel over plain http; never use it in production.
|
||||
func WithInsecureCookies() Option {
|
||||
return func(d *DexAuth) { d.insecure = true }
|
||||
}
|
||||
|
||||
// New discovers the issuer and returns a configured DexAuth. ctx bounds the
|
||||
// discovery request only.
|
||||
// discover runs OIDC discovery up to attempts times, doubling delay between
|
||||
// tries (#59: an issuer briefly unavailable behind its proxy answered 503 and
|
||||
// serve exited). A cancelled ctx ends the back-off immediately with ctx's error.
|
||||
func discover(ctx context.Context, issuer string, attempts int, delay time.Duration) (*oidc.Provider, error) {
|
||||
var lastErr error
|
||||
for i := range attempts {
|
||||
if i > 0 {
|
||||
t := time.NewTimer(delay)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
t.Stop()
|
||||
return nil, fmt.Errorf("after %d attempts (last: %v): %w", i, lastErr, ctx.Err())
|
||||
case <-t.C:
|
||||
}
|
||||
delay *= 2
|
||||
}
|
||||
p, err := oidc.NewProvider(ctx, issuer)
|
||||
if err == nil {
|
||||
return p, nil
|
||||
}
|
||||
lastErr = err
|
||||
if ctx.Err() != nil {
|
||||
return nil, fmt.Errorf("after %d attempts: %w", i+1, ctx.Err())
|
||||
}
|
||||
}
|
||||
return nil, fmt.Errorf("after %d attempts: %w", attempts, lastErr)
|
||||
}
|
||||
|
||||
// New discovers the issuer and returns a configured DexAuth. ctx bounds discovery,
|
||||
// including its retry back-off.
|
||||
func New(ctx context.Context, cfg Config, opts ...Option) (*DexAuth, error) {
|
||||
for name, val := range map[string]string{
|
||||
"issuer": cfg.Issuer,
|
||||
@@ -117,29 +165,31 @@ func New(ctx context.Context, cfg Config, opts ...Option) (*DexAuth, error) {
|
||||
}
|
||||
}
|
||||
|
||||
provider, err := oidc.NewProvider(ctx, cfg.Issuer)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("oidc: discover issuer: %w", err)
|
||||
}
|
||||
|
||||
d := &DexAuth{
|
||||
cfg: cfg,
|
||||
verifier: provider.Verifier(&oidc.Config{ClientID: cfg.ClientID}),
|
||||
oauth: &oauth2.Config{
|
||||
ClientID: cfg.ClientID,
|
||||
ClientSecret: cfg.ClientSecret,
|
||||
Endpoint: provider.Endpoint(),
|
||||
RedirectURL: cfg.RedirectURL,
|
||||
Scopes: []string{oidc.ScopeOpenID, "profile", "email"},
|
||||
},
|
||||
pending: newPendingStore(),
|
||||
secret: []byte(cfg.SessionSecret),
|
||||
sessionTTL: defaultSessionTTL,
|
||||
now: time.Now,
|
||||
cfg: cfg,
|
||||
pending: newPendingStore(),
|
||||
secret: []byte(cfg.SessionSecret),
|
||||
sessionTTL: defaultSessionTTL,
|
||||
now: time.Now,
|
||||
discoveryAttempts: defaultDiscoveryAttempts,
|
||||
discoveryDelay: defaultDiscoveryDelay,
|
||||
}
|
||||
for _, o := range opts {
|
||||
o(d)
|
||||
}
|
||||
|
||||
provider, err := discover(ctx, cfg.Issuer, d.discoveryAttempts, d.discoveryDelay)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("oidc: discover issuer: %w", err)
|
||||
}
|
||||
d.verifier = provider.Verifier(&oidc.Config{ClientID: cfg.ClientID})
|
||||
d.oauth = &oauth2.Config{
|
||||
ClientID: cfg.ClientID,
|
||||
ClientSecret: cfg.ClientSecret,
|
||||
Endpoint: provider.Endpoint(),
|
||||
RedirectURL: cfg.RedirectURL,
|
||||
Scopes: []string{oidc.ScopeOpenID, "profile", "email"},
|
||||
}
|
||||
return d, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -34,6 +35,11 @@ type fakeIssuer struct {
|
||||
sub string
|
||||
email string
|
||||
nonce string
|
||||
|
||||
// discoveryHits counts requests to the discovery document; the first
|
||||
// unavailableFor of them answer 503 (issuer briefly down behind its proxy).
|
||||
discoveryHits atomic.Int32
|
||||
unavailableFor int32
|
||||
}
|
||||
|
||||
const testKID = "test-key"
|
||||
@@ -47,6 +53,10 @@ func newFakeIssuer(t *testing.T) *fakeIssuer {
|
||||
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, _ *http.Request) {
|
||||
if f.discoveryHits.Add(1) <= f.unavailableFor {
|
||||
http.Error(w, "no available server", http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
writeJSON(t, w, map[string]any{
|
||||
"issuer": f.server.URL,
|
||||
"authorization_endpoint": f.server.URL + "/authorize",
|
||||
|
||||
@@ -66,7 +66,7 @@ func TestParseYouTubeVideoID(t *testing.T) {
|
||||
func TestListPageShowsPasteFormOnlyWhenConnected(t *testing.T) {
|
||||
render := func(connected bool) string {
|
||||
var buf bytes.Buffer
|
||||
if err := ListPage(listBuckets{}, Filter{}, PipelineStats{}, "", connected, nil, true).Render(context.Background(), &buf); err != nil {
|
||||
if err := ListPage(listBuckets{}, Filter{}, PipelineStats{}, "", connected, nil, true, false).Render(context.Background(), &buf); err != nil {
|
||||
t.Fatalf("render: %v", err)
|
||||
}
|
||||
return buf.String()
|
||||
|
||||
+48
-3
@@ -210,6 +210,32 @@ func (a *App) handleUploadStatus(w http.ResponseWriter, r *http.Request) {
|
||||
a.render(w, r, uploadStatus(*u, a.Chat != nil, a.now()))
|
||||
}
|
||||
|
||||
// handleUploadDelete removes one of the caller's finished recordings and returns to the
|
||||
// list. Another user's id is a 404 like every other upload route; a recording still being
|
||||
// transcribed is refused (409) rather than deleted under the worker (#37).
|
||||
func (a *App) handleUploadDelete(w http.ResponseWriter, r *http.Request) {
|
||||
userID, ok := a.currentUserID(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
id := r.PathValue("id")
|
||||
if !uuidShape.MatchString(id) {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
switch err := a.Store.DeleteUpload(r.Context(), userID, id); {
|
||||
case err == nil:
|
||||
http.Redirect(w, r, "/uploads", http.StatusSeeOther)
|
||||
case errors.Is(err, store.ErrUploadNotFound):
|
||||
http.NotFound(w, r)
|
||||
case errors.Is(err, store.ErrUploadInProgress):
|
||||
a.renderUploadsPage(w, r, userID, http.StatusConflict,
|
||||
"This recording is still being transcribed. You can delete it once it has finished.")
|
||||
default:
|
||||
a.serverError(w, r, "delete upload", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (a *App) loadOwnedUpload(w http.ResponseWriter, r *http.Request) (*domain.UploadedTranscript, bool) {
|
||||
userID, ok := a.currentUserID(w, r)
|
||||
if !ok {
|
||||
@@ -247,12 +273,31 @@ func titleFromFilename(name string) string {
|
||||
// cloudProvider names the cloud service that produced a transcript, or false when it was
|
||||
// served locally (or by an unknown endpoint). "Local" is config.IsLocalModel: the same
|
||||
// definition the start-up validation uses, not a second string match here (#52).
|
||||
//
|
||||
// The endpoint is one alias, except for the two-engine path (ADR-034), which writes
|
||||
// "<base> + <alt>, reconciled by <model>" (ensemble.go). That shape is taken apart exactly
|
||||
// and each alias is judged whole, so a composite of local models is local, and anything
|
||||
// else (a cloud alias, one with no provider prefix, one with a space or a connector word in
|
||||
// it) is not. A string that is not that shape is one alias.
|
||||
func cloudProvider(endpoint string) (string, bool) {
|
||||
if endpoint == "" || config.IsLocalModel(endpoint) {
|
||||
endpoint = strings.TrimSpace(endpoint)
|
||||
if endpoint == "" {
|
||||
return "", false
|
||||
}
|
||||
provider, _, _ := strings.Cut(endpoint, "/")
|
||||
return provider, true
|
||||
aliases := []string{endpoint}
|
||||
if engines, reconciler, ok := strings.Cut(endpoint, ", reconciled by "); ok {
|
||||
aliases = append(strings.Split(engines, " + "), reconciler)
|
||||
}
|
||||
for _, a := range aliases {
|
||||
if a = strings.TrimSpace(a); !config.IsLocalModel(a) {
|
||||
provider, _, _ := strings.Cut(a, "/")
|
||||
if provider = strings.TrimSpace(provider); provider == "" {
|
||||
provider = a
|
||||
}
|
||||
return provider, true
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// hasNote reports whether any phase carries a note. A note is the only warning for a
|
||||
|
||||
@@ -86,6 +86,7 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
|
||||
<summary>Transcript</summary>
|
||||
<pre class="transcript">{ u.Content }</pre>
|
||||
</details>
|
||||
@deleteRecording(u.ID)
|
||||
</div>
|
||||
case domain.UploadFailed:
|
||||
<div class="upload-status upload-failed flash flash-error" role="alert">
|
||||
@@ -97,6 +98,7 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
|
||||
@progressSteps(u.Progress, now, false)
|
||||
</details>
|
||||
}
|
||||
@deleteRecording(u.ID)
|
||||
</div>
|
||||
default:
|
||||
<div
|
||||
@@ -122,6 +124,20 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
|
||||
}
|
||||
}
|
||||
|
||||
// deleteRecording is the confirm-then-delete control, shown only once a recording has
|
||||
// finished (done or failed): deleting one that is still being processed is refused.
|
||||
templ deleteRecording(id string) {
|
||||
<details class="confirm-delete">
|
||||
<summary class="btn-danger">Delete this recording…</summary>
|
||||
<div class="confirm-body">
|
||||
<p>This permanently deletes the recording and its transcript. Are you sure?</p>
|
||||
<form method="post" action={ templ.SafeURL("/uploads/" + id + "/delete") }>
|
||||
<button type="submit" class="btn-danger">Yes, permanently delete it</button>
|
||||
</form>
|
||||
</div>
|
||||
</details>
|
||||
}
|
||||
|
||||
// UploadChatPage is the no-JS full page for chatting with an uploaded transcript:
|
||||
// the same chat panel as a video's (ADR-027), scoped to this recording.
|
||||
templ UploadChatPage(u domain.UploadedTranscript, v chatView) {
|
||||
|
||||
@@ -53,3 +53,88 @@ func TestUploadStatus_AnEndpointThatOnlyLooksLocalIsStillNamedAsCloud(t *testing
|
||||
|
||||
assert.Contains(t, b, "left the homelab")
|
||||
}
|
||||
|
||||
// The two-engine path (ADR-034) writes one composite endpoint string. Every name in it is
|
||||
// a local model, so the audio never left the homelab and the note must not say it did.
|
||||
func TestUploadStatus_AReconciledAllLocalEndpointCarriesNoCloudNote(t *testing.T) {
|
||||
for _, endpoint := range []string{
|
||||
"iguana/kb-whisper + koala/whisper, reconciled by iguana/gemma4-26b",
|
||||
"iguana/kb-whisper + koala/whisper, reconciled by iguana/gemma4-26b\n",
|
||||
} {
|
||||
t.Run(endpoint, func(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
||||
setEndpoint(t, id, endpoint)
|
||||
|
||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||
|
||||
assert.NotContains(t, b, "left the homelab")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadStatus_AReconciledEndpointNamingACloudEngineStillNamesTheCloudProvider(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
||||
setEndpoint(t, id, "iguana/kb-whisper + berget/kb-whisper, reconciled by iguana/gemma4-26b")
|
||||
|
||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||
|
||||
assert.Contains(t, b, "cloud service (berget)")
|
||||
}
|
||||
|
||||
// An alias without a provider prefix is not a local model, wherever it sits in the
|
||||
// composite: it must never be dropped from the judgement (PR #58 re-review).
|
||||
func TestUploadStatus_AReconciledEndpointWithAnUnprefixedAliasIsNotCalledLocal(t *testing.T) {
|
||||
for _, endpoint := range []string{
|
||||
"iguana/kb-whisper + whisper-1, reconciled by iguana/gemma4-26b",
|
||||
"iguana/kb-whisper + whisper 1, reconciled by iguana/gemma4-26b",
|
||||
"iguana/kb-whisper + berget / x, reconciled by iguana/gemma4-26b",
|
||||
} {
|
||||
t.Run(endpoint, func(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
||||
setEndpoint(t, id, endpoint)
|
||||
|
||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||
|
||||
assert.Contains(t, b, "left the homelab")
|
||||
assert.NotContains(t, b, "cloud service ()", "the note names a provider")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// "by", "reconciled" and "+" are connectors only inside the composite format. As a whole
|
||||
// endpoint, or as an alias within the composite, they are model names and are judged.
|
||||
func TestUploadStatus_AConnectorWordUsedAsAModelNameIsStillJudged(t *testing.T) {
|
||||
for _, endpoint := range []string{
|
||||
"by",
|
||||
"reconciled",
|
||||
"+",
|
||||
"iguana/kb-whisper + by, reconciled by iguana/gemma4-26b",
|
||||
"iguana/kb-whisper + reconciled, reconciled by iguana/gemma4-26b",
|
||||
"iguana/kb-whisper + +, reconciled by iguana/gemma4-26b",
|
||||
"/x", // no provider at all: the note must still name something
|
||||
} {
|
||||
t.Run(endpoint, func(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
||||
setEndpoint(t, id, endpoint)
|
||||
|
||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||
|
||||
assert.Contains(t, b, "left the homelab")
|
||||
assert.NotContains(t, b, "cloud service ()")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadStatus_AReconcilerThatIsNotLocalNamesTheCloudProvider(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
|
||||
setEndpoint(t, id, "iguana/kb-whisper + koala/whisper, reconciled by berget/mistral-large")
|
||||
|
||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||
|
||||
assert.Contains(t, b, "cloud service (berget)")
|
||||
}
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
package web_test
|
||||
|
||||
// Deleting one recording (#37). Owner-scoped like every other upload route: another user's
|
||||
// id is a 404 and the row survives; an upload still being processed is refused.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func deleteReq(id string) *http.Request {
|
||||
return httptest.NewRequest(http.MethodPost, "/uploads/"+id+"/delete", nil)
|
||||
}
|
||||
|
||||
func uploadExists(t *testing.T, id string) bool {
|
||||
t.Helper()
|
||||
var n int
|
||||
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT count(*) FROM uploaded_transcripts WHERE id = $1`, id).Scan(&n))
|
||||
return n == 1
|
||||
}
|
||||
|
||||
func TestUploadDelete_AnOwnerRemovesAFinishedOrFailedRecordingAndLandsOnTheList(t *testing.T) {
|
||||
for _, status := range []string{"done", "failed"} {
|
||||
t.Run(status, func(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "Roadmap sync", status, "transcription_failed")
|
||||
|
||||
rec := do(t, app, deleteReq(id))
|
||||
|
||||
assert.Equal(t, http.StatusSeeOther, rec.Code)
|
||||
assert.Equal(t, "/uploads", rec.Header().Get("Location"))
|
||||
assert.False(t, uploadExists(t, id))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadDelete_AnotherUsersRecordingIs404AndSurvives(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
theirs := seedUpload(t, otherUserID, "Their private meeting", "done", meetingText)
|
||||
|
||||
rec := do(t, app, deleteReq(theirs))
|
||||
|
||||
assert.Equal(t, http.StatusNotFound, rec.Code)
|
||||
assert.True(t, uploadExists(t, theirs), "a non-owner's delete must not remove the row")
|
||||
}
|
||||
|
||||
func TestUploadDelete_ARecordingStillBeingTranscribedIsRefusedWith409AndKept(t *testing.T) {
|
||||
for _, status := range []string{"queued", "transcribing"} {
|
||||
t.Run(status, func(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", status, "")
|
||||
|
||||
rec := do(t, app, deleteReq(id))
|
||||
|
||||
assert.Equal(t, http.StatusConflict, rec.Code)
|
||||
assert.Contains(t, body(t, rec), "still being transcribed")
|
||||
assert.True(t, uploadExists(t, id))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUploadDelete_ANonIDIsA404NotAServerError(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
|
||||
assert.Equal(t, http.StatusNotFound, do(t, app, deleteReq("not-a-uuid")).Code)
|
||||
}
|
||||
|
||||
func TestUploadPage_OffersDeleteOnlyForAFinishedOrFailedRecording(t *testing.T) {
|
||||
for status, want := range map[string]bool{"done": true, "failed": true, "queued": false, "transcribing": false} {
|
||||
t.Run(status, func(t *testing.T) {
|
||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
id := seedUpload(t, userID, "m", status, "transcription_failed")
|
||||
|
||||
b := body(t, do(t, app, get("/uploads/"+id)))
|
||||
|
||||
assert.Equal(t, want, strings.Contains(b, `action="/uploads/`+id+`/delete"`))
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
package web_test
|
||||
|
||||
// /uploads was mounted but nothing linked to it (#37). The shared Layout has no access to
|
||||
// the App, so the entry point is the home list, shown only when uploads are mounted.
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
func TestHomeList_LinksToRecordingsOnlyWhenUploadsAreMounted(t *testing.T) {
|
||||
with, _ := newUploadApp(t, audioOK, 200<<20)
|
||||
without := newApp(t)
|
||||
|
||||
assert.Contains(t, body(t, do(t, with, get("/"))), `href="/uploads"`)
|
||||
assert.NotContains(t, body(t, do(t, without, get("/"))), `href="/uploads"`,
|
||||
"a link to a route that is not mounted would be a dead end")
|
||||
}
|
||||
+367
-311
File diff suppressed because it is too large
Load Diff
@@ -107,9 +107,12 @@ templ flashBanner(code string) {
|
||||
// #summary-list region; a non-HTMX request renders the whole page. flash carries
|
||||
// a one-shot notification (e.g. "connected", "registered") surfaced on arrival
|
||||
// after a POST→redirect.
|
||||
templ ListPage(b listBuckets, f Filter, stats PipelineStats, flash string, hasConnected bool, channels []string, autoSummarize bool) {
|
||||
templ ListPage(b listBuckets, f Filter, stats PipelineStats, flash string, hasConnected bool, channels []string, autoSummarize bool, uploadsEnabled bool) {
|
||||
@Layout("Tapir — Summaries") {
|
||||
@flashBanner(flash)
|
||||
if uploadsEnabled {
|
||||
<p class="muted"><a href="/uploads">Recordings</a>: upload audio and get a private transcript.</p>
|
||||
}
|
||||
if hasConnected {
|
||||
@pasteForm()
|
||||
}
|
||||
|
||||
+358
-348
File diff suppressed because it is too large
Load Diff
@@ -124,6 +124,10 @@ var scenarioCoverage = map[string]string{
|
||||
|
||||
"A transcript made by the cloud fallback says the audio left the homelab": "TestUploadStatus_ACloudServedTranscriptSaysTheAudioLeftTheHomelab",
|
||||
|
||||
"I can delete a finished recording": "TestUploadDelete_AnOwnerRemovesAFinishedOrFailedRecordingAndLandsOnTheList",
|
||||
"A recording still being transcribed cannot be deleted": "TestUploadDelete_ARecordingStillBeingTranscribedIsRefusedWith409AndKept",
|
||||
"The home list links to my recordings when uploads are on": "TestHomeList_LinksToRecordingsOnlyWhenUploadsAreMounted",
|
||||
|
||||
// audio_upload.feature, ADR-034
|
||||
"Two engines transcribe the audio and their disagreements are reconciled": "TestEnsemble_BothEnginesRunInOrderAndTheirDisagreementIsReconciled",
|
||||
"Trivial disagreements never reach the model": "TestReconcile_TrivialDisputesAreResolvedWithoutAskingTheModel",
|
||||
|
||||
Reference in New Issue
Block a user