feat(uploads): choose the language of a recording; record it, show it, carry it to the engines (#61)

Form field (Swedish preselected, Other = auto) read before the file part and refused
before anything is spooled if outside the closed set; stored on the row (migration 021,
CHECK constraint, default sv for history); carried on the Job and the worker's context;
shown on the status page. Feature scenarios and the coverage map are updated, and
ADR-033/034 amended.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-04 22:29:58 +02:00
co-authored by Claude Sonnet 5.5
parent ed9d76b0cf
commit 0338b18a17
26 changed files with 718 additions and 371 deletions
+10 -1
View File
@@ -1434,7 +1434,12 @@ an hour of Swedish *was* an unknown here; it is now measured, see decision 8.)
empty disables), recorded per row, and called out in Open items rather than defaulted empty disables), recorded per row, and called out in Open items rather than defaulted
silently. silently.
- Swedish only (`language=sv`). Language detection and routing (#28 decision 2) are not - Swedish only (`language=sv`). Language detection and routing (#28 decision 2) are not
touched. touched. **Amended 2026-10-04 (#61, interim):** the language is now a per-upload choice on
the form, `sv` (preselected, and what an upload that names none gets) or `auto` ("Other":
`language` is omitted so the engine detects it), a closed set held by the form, the request
code and a CHECK constraint on `uploaded_transcripts.language` (migration 021). Automatic
detection and a general-purpose engine for other languages are still not built (#61); an
`auto` upload goes to the same configured engine, which is not tuned for it.
- `-H 'Expect:'`, multipart field `file` — encoded in the adapter and its tests, not rediscovered. - `-H 'Expect:'`, multipart field `file` — encoded in the adapter and its tests, not rediscovered.
7. **Jobs run in-process, one at a time (ADR-018 stands).** Transcription is minutes (~7.5 min 7. **Jobs run in-process, one at a time (ADR-018 stands).** Transcription is minutes (~7.5 min
@@ -1645,6 +1650,10 @@ on the slow numbers and is reopened: 31B scored 97% against 92%.
job timeout rises from 30 to 60 minutes: ~4 min of kb-whisper and ~40 s of pianissimo per hour of job timeout rises from 30 to 60 minutes: ~4 min of kb-whisper and ~40 s of pianissimo per hour of
audio, plus the adjudication. audio, plus the adjudication.
**Amended 2026-10-04 (#61, interim):** reconcile is for Swedish. A recording whose language is
`auto` is never sent to the base engine (it is Swedish-only) and is not reconciled: the other
engine transcribes alone, then the cloud fallback if that fails and the job is live, and the
page does not claim a second opinion.
8. **Not decided here.** A cloud adjudicator (rejected: private content); tuning the prompt; making 8. **Not decided here.** A cloud adjudicator (rejected: private content); tuning the prompt; making
`gemma4-31b` the default; a "reconciling" progress state (the page says "transcribing" for the whole `gemma4-31b` the default; a "reconciling" progress state (the page says "transcribing" for the whole
job); feeding the uncertain spots to chat; whether pianissimo should also be a middle tier in the job); feeding the uncertain spots to chat; whether pianissimo should also be a middle tier in the
+1 -1
View File
@@ -25,7 +25,7 @@ type fakeUploadsStore struct {
failedKinds []domain.UploadErrorKind failedKinds []domain.UploadErrorKind
} }
func (f *fakeUploadsStore) CreateUpload(context.Context, string, string, bool) (string, error) { func (f *fakeUploadsStore) CreateUpload(context.Context, string, string, domain.Language, bool) (string, error) {
return "id", nil return "id", nil
} }
+16
View File
@@ -274,3 +274,19 @@ Feature: Upload an audio recording, get a private Swedish transcript, discuss it
Scenario: Input that cannot be right is a client error Scenario: Input that cannot be right is a client error
When the id is not an id, or a header or title holds control bytes When the id is not an id, or a header or title holds control bytes
Then the answer is a 4xx or the title is cleaned, never a server error Then the answer is a 4xx or the title is cleaned, never a server error
Scenario: I choose the language of a recording
When I upload a recording and choose Swedish or Other on the form
Then the recording is transcribed as that language and its page says which
Scenario: Swedish stays the default
When an upload names no language
Then it is transcribed as Swedish, as before the choice existed
Scenario: A language outside the list is refused
When an upload names a language the form does not offer
Then it is answered 400 before anything is stored or sent to an engine
Scenario: A recording in another language never reaches the Swedish-only engine
When I choose Other for a recording
Then the language is left for the engine to detect and no Swedish-only engine or reconcile is used
@@ -0,0 +1,2 @@
ALTER TABLE uploaded_transcripts
DROP COLUMN language;
@@ -0,0 +1,13 @@
-- Migration 021: the language an upload is transcribed as (tapir#61).
--
-- 'sv' Swedish: the engines are sent language=sv. Every row before this migration was
-- transcribed that way (the request hard-coded it), so the default is true history.
-- 'auto' Other: the engine detects the language and the Swedish-only engine is not used.
--
-- A closed set held by the table as well as by the form and the request code, so a second
-- writer or a bug cannot store a value the engines are never meant to receive. The column
-- sits on the already-FORCE'd, RLS-scoped uploaded_transcripts, so the exact-set isolation
-- test is unaffected.
ALTER TABLE uploaded_transcripts
ADD COLUMN language TEXT NOT NULL DEFAULT 'sv'
CONSTRAINT uploaded_transcripts_language_check CHECK (language IN ('sv', 'auto'));
@@ -31,12 +31,12 @@ var ErrUploadInProgress = errors.New("store: uploaded transcript is still being
// CreateUpload records a new upload in the queued state and returns its id. // CreateUpload records a new upload in the queued state and returns its id.
// hadVideo records that the file carried a video track (only its audio is used); // hadVideo records that the file carried a video track (only its audio is used);
// it is kept for the life of the row so the page can say so. // it is kept for the life of the row so the page can say so.
func (s *Store) CreateUpload(ctx context.Context, userID, title string, hadVideo bool) (string, error) { func (s *Store) CreateUpload(ctx context.Context, userID, title string, lang domain.Language, hadVideo bool) (string, error) {
var id string var id string
if err := s.withUser(ctx, userID, func(tx pgx.Tx) error { if err := s.withUser(ctx, userID, func(tx pgx.Tx) error {
return tx.QueryRow(ctx, return tx.QueryRow(ctx,
`INSERT INTO uploaded_transcripts (user_id, title, had_video) VALUES ($1, $2, $3) RETURNING id`, `INSERT INTO uploaded_transcripts (user_id, title, language, had_video) VALUES ($1, $2, $3, $4) RETURNING id`,
userID, title, hadVideo).Scan(&id) userID, title, string(lang), hadVideo).Scan(&id)
}); err != nil { }); err != nil {
return "", fmt.Errorf("store: create upload: %w", err) return "", fmt.Errorf("store: create upload: %w", err)
} }
@@ -221,7 +221,7 @@ func (s *Store) FailInterruptedUploads(ctx context.Context) (int, error) {
return total, nil return total, nil
} }
const uploadSelect = `SELECT id, user_id, title, status, content, segments, endpoint, const uploadSelect = `SELECT id, user_id, title, language, status, content, segments, endpoint,
audio_seconds, error_kind, had_video, reconciliation, progress, progress_at, alive_at, audio_seconds, error_kind, had_video, reconciliation, progress, progress_at, alive_at,
created_at, finished_at FROM uploaded_transcripts` created_at, finished_at FROM uploaded_transcripts`
@@ -234,11 +234,13 @@ func scanUpload(row pgx.Row, u *domain.UploadedTranscript, withText bool) error
seconds *float64 seconds *float64
finished, progressAt, aliveAt *time.Time finished, progressAt, aliveAt *time.Time
) )
if err := row.Scan(&u.ID, &u.UserID, &u.Title, &status, &content, &segments, &endpoint, var language string
if err := row.Scan(&u.ID, &u.UserID, &u.Title, &language, &status, &content, &segments, &endpoint,
&seconds, &errKind, &u.HadVideo, &reconciliation, &progress, &progressAt, &aliveAt, &seconds, &errKind, &u.HadVideo, &reconciliation, &progress, &progressAt, &aliveAt,
&u.CreatedAt, &finished); err != nil { &u.CreatedAt, &finished); err != nil {
return err return err
} }
u.Language = domain.Language(language)
u.Status = domain.UploadStatus(deref(status)) u.Status = domain.UploadStatus(deref(status))
u.Endpoint = deref(endpoint) u.Endpoint = deref(endpoint)
u.ErrorKind = domain.UploadErrorKind(deref(errKind)) u.ErrorKind = domain.UploadErrorKind(deref(errKind))
@@ -42,7 +42,7 @@ func TestDeleteUpload_RefusesAnUploadThatIsStillQueuedOrTranscribingAndKeepsTheR
s := newStore(t) s := newStore(t)
ctx := context.Background() ctx := context.Background()
queued := startedUpload(t, s) queued := startedUpload(t, s)
running, err := s.CreateUpload(ctx, userA, "running", false) running, err := s.CreateUpload(ctx, userA, "running", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, running)) require.NoError(t, s.StartTranscribing(ctx, userA, running))
@@ -34,7 +34,7 @@ func TestFailUpload_StillFailsAQueuedOrTranscribingUpload(t *testing.T) {
s := newStore(t) s := newStore(t)
ctx := context.Background() ctx := context.Background()
queued := startedUpload(t, s) queued := startedUpload(t, s)
running, err := s.CreateUpload(ctx, userA, "r", false) running, err := s.CreateUpload(ctx, userA, "r", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, running)) require.NoError(t, s.StartTranscribing(ctx, userA, running))
@@ -20,7 +20,7 @@ import (
func startedUpload(t *testing.T, s *store.Store) string { func startedUpload(t *testing.T, s *store.Store) string {
t.Helper() t.Helper()
seedUsers(t, userA, userB) seedUsers(t, userA, userB)
id, err := s.CreateUpload(context.Background(), userA, "m", false) id, err := s.CreateUpload(context.Background(), userA, "m", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
return id return id
} }
@@ -33,7 +33,7 @@ func TestCreateUpload_StartsQueuedAndIsOwnedByTheCreator(t *testing.T) {
seedUsers(t, userA) seedUsers(t, userA)
ctx := context.Background() ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "Teams export", false) id, err := s.CreateUpload(ctx, userA, "Teams export", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
got, err := s.GetUpload(ctx, userA, id) got, err := s.GetUpload(ctx, userA, id)
@@ -50,7 +50,7 @@ func TestUploadLifecycle_CompletedKeepsTextSegmentsEndpointAndDuration(t *testin
s := newStore(t) s := newStore(t)
seedUsers(t, userA) seedUsers(t, userA)
ctx := context.Background() ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", false) id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id)) require.NoError(t, s.StartTranscribing(ctx, userA, id))
@@ -84,7 +84,7 @@ func TestUploadLifecycle_FailedRecordsTypedKindAndNoContent(t *testing.T) {
s := newStore(t) s := newStore(t)
seedUsers(t, userA) seedUsers(t, userA)
ctx := context.Background() ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", false) id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id)) require.NoError(t, s.StartTranscribing(ctx, userA, id))
@@ -102,7 +102,7 @@ func TestGetUpload_AnotherUsersIDIsNotFound(t *testing.T) {
s := newStore(t) s := newStore(t)
seedUsers(t, userA, userB) seedUsers(t, userA, userB)
ctx := context.Background() ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "A's meeting", false) id, err := s.CreateUpload(ctx, userA, "A's meeting", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
_, err = s.GetUpload(ctx, userB, id) _, err = s.GetUpload(ctx, userB, id)
@@ -115,7 +115,7 @@ func TestUploadWrites_ByAnotherUserChangeNothing(t *testing.T) {
s := newStore(t) s := newStore(t)
seedUsers(t, userA, userB) seedUsers(t, userA, userB)
ctx := context.Background() ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "A's meeting", false) id, err := s.CreateUpload(ctx, userA, "A's meeting", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
assert.ErrorIs(t, s.StartTranscribing(ctx, userB, id), store.ErrUploadNotFound) assert.ErrorIs(t, s.StartTranscribing(ctx, userB, id), store.ErrUploadNotFound)
@@ -132,11 +132,11 @@ func TestListUploads_ReturnsOnlyTheCallersNewestFirstWithoutTranscriptText(t *te
s := newStore(t) s := newStore(t)
seedUsers(t, userA, userB) seedUsers(t, userA, userB)
ctx := context.Background() ctx := context.Background()
first, err := s.CreateUpload(ctx, userA, "first", false) first, err := s.CreateUpload(ctx, userA, "first", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
second, err := s.CreateUpload(ctx, userA, "second", false) second, err := s.CreateUpload(ctx, userA, "second", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
_, err = s.CreateUpload(ctx, userB, "B's", false) _, err = s.CreateUpload(ctx, userB, "B's", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, first)) require.NoError(t, s.StartTranscribing(ctx, userA, first))
require.NoError(t, s.CompleteUpload(ctx, userA, first, domain.Transcription{Text: "long text body", Endpoint: "e"})) require.NoError(t, s.CompleteUpload(ctx, userA, first, domain.Transcription{Text: "long text body", Endpoint: "e"}))
@@ -160,12 +160,12 @@ func TestFailInterruptedUploads_FailsOnlyUnfinishedRowsForEveryUser(t *testing.T
_, err := super.Exec(ctx, `INSERT INTO user_identities (dex_subject, user_id) VALUES ($1, $2)`, subject, uid) _, err := super.Exec(ctx, `INSERT INTO user_identities (dex_subject, user_id) VALUES ($1, $2)`, subject, uid)
require.NoError(t, err) require.NoError(t, err)
} }
queued, err := s.CreateUpload(ctx, userA, "queued", false) queued, err := s.CreateUpload(ctx, userA, "queued", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
running, err := s.CreateUpload(ctx, userB, "running", false) running, err := s.CreateUpload(ctx, userB, "running", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userB, running)) require.NoError(t, s.StartTranscribing(ctx, userB, running))
done, err := s.CreateUpload(ctx, userA, "done", false) done, err := s.CreateUpload(ctx, userA, "done", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, done)) require.NoError(t, s.StartTranscribing(ctx, userA, done))
require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "kept", Endpoint: "e"})) require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "kept", Endpoint: "e"}))
@@ -191,9 +191,9 @@ func TestCreateUpload_RecordsWhetherTheFileHadAVideoTrack(t *testing.T) {
s := newStore(t) s := newStore(t)
seedUsers(t, userA) seedUsers(t, userA)
ctx := context.Background() ctx := context.Background()
withVideo, err := s.CreateUpload(ctx, userA, "camera off", true) withVideo, err := s.CreateUpload(ctx, userA, "camera off", domain.LangSwedish, true)
require.NoError(t, err) require.NoError(t, err)
audioOnly, err := s.CreateUpload(ctx, userA, "voice only", false) audioOnly, err := s.CreateUpload(ctx, userA, "voice only", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
gotVideo, err := s.GetUpload(ctx, userA, withVideo) gotVideo, err := s.GetUpload(ctx, userA, withVideo)
@@ -216,9 +216,9 @@ func TestUploadHadVideo_SurvivesCompletionAndFailure(t *testing.T) {
s := newStore(t) s := newStore(t)
seedUsers(t, userA) seedUsers(t, userA)
ctx := context.Background() ctx := context.Background()
done, err := s.CreateUpload(ctx, userA, "d", true) done, err := s.CreateUpload(ctx, userA, "d", domain.LangSwedish, true)
require.NoError(t, err) require.NoError(t, err)
failed, err := s.CreateUpload(ctx, userA, "f", true) failed, err := s.CreateUpload(ctx, userA, "f", domain.LangSwedish, true)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, done)) require.NoError(t, s.StartTranscribing(ctx, userA, done))
require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "x", Endpoint: "e"})) require.NoError(t, s.CompleteUpload(ctx, userA, done, domain.Transcription{Text: "x", Endpoint: "e"}))
@@ -235,7 +235,7 @@ func TestUploadLifecycle_ReconciliationRoundTripsWithItsContestedSpots(t *testin
s := newStore(t) s := newStore(t)
seedUsers(t, userA) seedUsers(t, userA)
ctx := context.Background() ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", false) id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id)) require.NoError(t, s.StartTranscribing(ctx, userA, id))
want := &domain.Reconciliation{ want := &domain.Reconciliation{
@@ -258,7 +258,7 @@ func TestUploadLifecycle_ASingleEngineTranscriptHasNoReconciliation(t *testing.T
s := newStore(t) s := newStore(t)
seedUsers(t, userA) seedUsers(t, userA)
ctx := context.Background() ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", false) id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id)) require.NoError(t, s.StartTranscribing(ctx, userA, id))
require.NoError(t, s.CompleteUpload(ctx, userA, id, domain.Transcription{Text: "x", Endpoint: "e"})) require.NoError(t, s.CompleteUpload(ctx, userA, id, domain.Transcription{Text: "x", Endpoint: "e"}))
@@ -273,7 +273,7 @@ func TestUploadLifecycle_ASkippedReconciliationKeepsItsReason(t *testing.T) {
s := newStore(t) s := newStore(t)
seedUsers(t, userA) seedUsers(t, userA)
ctx := context.Background() ctx := context.Background()
id, err := s.CreateUpload(ctx, userA, "m", false) id, err := s.CreateUpload(ctx, userA, "m", domain.LangSwedish, false)
require.NoError(t, err) require.NoError(t, err)
require.NoError(t, s.StartTranscribing(ctx, userA, id)) require.NoError(t, s.StartTranscribing(ctx, userA, id))
rec := &domain.Reconciliation{Status: domain.ReconcileSkipped, Reason: "the model failed 3 times in a row: gateway down", Errors: 3} rec := &domain.Reconciliation{Status: domain.ReconcileSkipped, Reason: "the model failed 3 times in a row: gateway down", Errors: 3}
@@ -284,3 +284,55 @@ func TestUploadLifecycle_ASkippedReconciliationKeepsItsReason(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, rec, got.Reconciliation) assert.Equal(t, rec, got.Reconciliation)
} }
func TestCreateUpload_RecordsTheLanguageTheUserChoseAndTheListCarriesIt(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
ctx := context.Background()
swedish, err := s.CreateUpload(ctx, userA, "sv", domain.LangSwedish, false)
require.NoError(t, err)
auto, err := s.CreateUpload(ctx, userA, "auto", domain.LangAuto, false)
require.NoError(t, err)
gotSv, err := s.GetUpload(ctx, userA, swedish)
require.NoError(t, err)
gotAuto, err := s.GetUpload(ctx, userA, auto)
require.NoError(t, err)
listed, err := s.ListUploads(ctx, userA)
require.NoError(t, err)
assert.Equal(t, domain.LangSwedish, gotSv.Language)
assert.Equal(t, domain.LangAuto, gotAuto.Language)
byID := map[string]domain.Language{}
for _, u := range listed {
byID[u.ID] = u.Language
}
assert.Equal(t, map[string]domain.Language{swedish: domain.LangSwedish, auto: domain.LangAuto}, byID)
}
// The closed set is also held by the table, so a bug or a second writer cannot store a
// value the engines are never meant to receive (tapir#61).
func TestUploadedTranscripts_TheTableRefusesALanguageOutsideTheClosedSet(t *testing.T) {
newStore(t)
seedUsers(t, userA)
_, err := rawPool(t).Exec(context.Background(),
`INSERT INTO uploaded_transcripts (user_id, title, language) VALUES ($1, 'x', 'en')`, userA)
require.Error(t, err)
assert.Contains(t, err.Error(), "language", "the CHECK constraint names the column")
}
// A row from before the column existed was transcribed as Swedish; the default says so.
func TestUploadedTranscripts_ARowInsertedWithoutALanguageIsSwedish(t *testing.T) {
s := newStore(t)
seedUsers(t, userA)
var id string
require.NoError(t, rawPool(t).QueryRow(context.Background(),
`INSERT INTO uploaded_transcripts (user_id, title) VALUES ($1, 'old') RETURNING id`, userA).Scan(&id))
got, err := s.GetUpload(context.Background(), userA, id)
require.NoError(t, err)
assert.Equal(t, domain.LangSwedish, got.Language)
}
+3
View File
@@ -246,6 +246,9 @@ type UploadedTranscript struct {
ID string ID string
UserID string UserID string
Title string Title string
// Language is what the upload was (or will be) transcribed as: sv, or auto for
// "Other" (tapir#61).
Language Language
Status UploadStatus Status UploadStatus
Content string Content string
Segments []Segment Segments []Segment
+7 -4
View File
@@ -25,7 +25,7 @@ type Prober interface {
// Store is the slice of the private transcript store the intake needs. // Store is the slice of the private transcript store the intake needs.
type Store interface { type Store interface {
CreateUpload(ctx context.Context, userID, title string, hadVideo bool) (string, error) CreateUpload(ctx context.Context, userID, title string, lang domain.Language, hadVideo bool) (string, error)
FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error FailUpload(ctx context.Context, userID, id string, kind domain.UploadErrorKind) error
} }
@@ -35,6 +35,9 @@ type Job struct {
UserID string UserID string
UploadID string UploadID string
Path string Path string
// Language is what the user chose for this recording; empty means Swedish (a job
// queued before the choice existed).
Language domain.Language
// Duration is what the file CLAIMED when probed. The worker compares it with what // Duration is what the file CLAIMED when probed. The worker compares it with what
// was really converted; zero means unknown (ADR-035 review). // was really converted; zero means unknown (ADR-035 review).
Duration time.Duration Duration time.Duration
@@ -135,7 +138,7 @@ const busyMessage = "Too many recordings are waiting to be transcribed. Please t
// validation yields a *RejectError and creates NO row — rejection is synchronous // validation yields a *RejectError and creates NO row — rejection is synchronous
// and leaves nothing behind. On success the temp file is handed to the job and the // and leaves nothing behind. On success the temp file is handed to the job and the
// worker deletes it; on every failure path Accept deletes it itself. // worker deletes it; on every failure path Accept deletes it itself.
func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reader) (string, error) { func (in *Intake) Accept(ctx context.Context, userID, title string, lang domain.Language, src io.Reader) (string, error) {
// Refuse before spooling when the queue has no room, if the submitter can say so: a full // Refuse before spooling when the queue has no room, if the submitter can say so: a full
// queue otherwise cost up to the upload cap on disk, an ffprobe run and a failed row for // queue otherwise cost up to the upload cap on disk, an ffprobe run and a failed row for
// every refused attempt. // every refused attempt.
@@ -190,11 +193,11 @@ func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reade
return "", err return "", err
} }
id, err := in.store.CreateUpload(ctx, userID, title, probe.HasVideo) id, err := in.store.CreateUpload(ctx, userID, title, lang, probe.HasVideo)
if err != nil { if err != nil {
return "", fmt.Errorf("uploads: create: %w", err) return "", fmt.Errorf("uploads: create: %w", err)
} }
if err := in.sub.Submit(Job{UserID: userID, UploadID: id, Path: path, Duration: probe.Duration}); err != nil { if err := in.sub.Submit(Job{UserID: userID, UploadID: id, Path: path, Language: lang, Duration: probe.Duration}); err != nil {
if errors.Is(err, ErrQueueFull) { if errors.Is(err, ErrQueueFull) {
// The user's to retry, not a server fault: refuse it as such, and do not leave // The user's to retry, not a server fault: refuse it as such, and do not leave
// the row queued -- that is the stuck spinner. // the row queued -- that is the stuck spinner.
+5 -5
View File
@@ -23,7 +23,7 @@ func TestAccept_AFullQueueIsABusyRejectionNotAServerErrorAndLeavesNothingBehind(
r := newRig(t) r := newRig(t)
r.sub.err = uploads.ErrQueueFull r.sub.err = uploads.ErrQueueFull
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data")) _, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
var rej *uploads.RejectError var rej *uploads.RejectError
require.True(t, errors.As(err, &rej), "a full queue must be a rejection the web layer can answer with 429: %v", err) require.True(t, errors.As(err, &rej), "a full queue must be a rejection the web layer can answer with 429: %v", err)
@@ -48,7 +48,7 @@ func TestAccept_ANoRoomQueueIsRefusedBeforeAnythingIsSpooledProbedOrStored(t *te
full := &roomSubmitter{room: false} full := &roomSubmitter{room: false}
in := uploads.NewIntake(r.st, r.pr, full, uploads.Limits{MaxBytes: 1024, MaxDuration: 90 * time.Minute}, r.tmp) in := uploads.NewIntake(r.st, r.pr, full, uploads.Limits{MaxBytes: 1024, MaxDuration: 90 * time.Minute}, r.tmp)
_, err := in.Accept(context.Background(), "user-1", "t", strings.NewReader("data")) _, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
var rej *uploads.RejectError var rej *uploads.RejectError
require.True(t, errors.As(err, &rej)) require.True(t, errors.As(err, &rej))
@@ -62,7 +62,7 @@ func TestAccept_ANoRoomQueueIsRefusedBeforeAnythingIsSpooledProbedOrStored(t *te
func TestAccept_ASubmitterThatCannotSayStillWorksAsBefore(t *testing.T) { func TestAccept_ASubmitterThatCannotSayStillWorksAsBefore(t *testing.T) {
r := newRig(t) r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data")) _, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
require.NoError(t, err, "a Submitter without HasRoom is accepted: the check at Submit remains") require.NoError(t, err, "a Submitter without HasRoom is accepted: the check at Submit remains")
} }
@@ -81,7 +81,7 @@ func (b *brokenBody) Read(p []byte) (int, error) {
func TestAccept_AnErrorReadingTheClientsBodyIsMarkedAsTheClientsAndAWrappedSubmitErrorIsNot(t *testing.T) { func TestAccept_AnErrorReadingTheClientsBodyIsMarkedAsTheClientsAndAWrappedSubmitErrorIsNot(t *testing.T) {
r := newRig(t) r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "t", &brokenBody{n: 1}) _, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, &brokenBody{n: 1})
require.Error(t, err) require.Error(t, err)
assert.ErrorIs(t, err, uploads.ErrClientRead, "a dropped client body is the sender's to retry") assert.ErrorIs(t, err, uploads.ErrClientRead, "a dropped client body is the sender's to retry")
@@ -90,7 +90,7 @@ func TestAccept_AnErrorReadingTheClientsBodyIsMarkedAsTheClientsAndAWrappedSubmi
// A fault that is ours must NOT carry the marker, even if it wraps the same sentinel. // A fault that is ours must NOT carry the marker, even if it wraps the same sentinel.
r2 := newRig(t) r2 := newRig(t)
r2.sub.err = fmt.Errorf("db: %w", io.ErrUnexpectedEOF) r2.sub.err = fmt.Errorf("db: %w", io.ErrUnexpectedEOF)
_, err = r2.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data")) _, err = r2.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
require.Error(t, err) require.Error(t, err)
assert.NotErrorIs(t, err, uploads.ErrClientRead) assert.NotErrorIs(t, err, uploads.ErrClientRead)
} }
+7 -6
View File
@@ -15,6 +15,7 @@ import (
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/uploads" "git.d-ma.be/mathias/tapir/internal/uploads"
) )
@@ -31,7 +32,7 @@ func startSlowReceive(t *testing.T, r *rig, in *uploads.Intake, user string, wan
pr, pw := io.Pipe() pr, pw := io.Pipe()
s := &slowReceive{w: pw, done: make(chan error, 1)} s := &slowReceive{w: pw, done: make(chan error, 1)}
go func() { go func() {
_, err := in.Accept(context.Background(), user, "t", pr) _, err := in.Accept(context.Background(), user, "t", domain.LangSwedish, pr)
s.done <- err s.done <- err
}() }()
require.Eventually(t, func() bool { return len(r.tmpFiles(t)) == wantFiles }, 2*time.Second, 5*time.Millisecond, require.Eventually(t, func() bool { return len(r.tmpFiles(t)) == wantFiles }, 2*time.Second, 5*time.Millisecond,
@@ -80,7 +81,7 @@ func TestAccept_APerUserLimitOnConcurrentReceivesRefusesTheNextBodyBeforeSpoolin
startSlowReceive(t, r, in, "user-1", 1) startSlowReceive(t, r, in, "user-1", 1)
startSlowReceive(t, r, in, "user-1", 2) startSlowReceive(t, r, in, "user-1", 2)
_, err := in.Accept(context.Background(), "user-1", "t", strings.NewReader("data")) _, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
requireBusy(t, err) requireBusy(t, err)
assert.Len(t, r.tmpFiles(t), 2, "the refused body was never spooled") assert.Len(t, r.tmpFiles(t), 2, "the refused body was never spooled")
@@ -98,7 +99,7 @@ func TestAccept_AnOverallLimitOnConcurrentReceivesRefusesTheNextUser(t *testing.
startSlowReceive(t, r, in, u, i+1) startSlowReceive(t, r, in, u, i+1)
} }
_, err := in.Accept(context.Background(), "d", "t", strings.NewReader("data")) _, err := in.Accept(context.Background(), "d", "t", domain.LangSwedish, strings.NewReader("data"))
requireBusy(t, err) requireBusy(t, err)
assert.Len(t, r.tmpFiles(t), 3) assert.Len(t, r.tmpFiles(t), 3)
@@ -112,7 +113,7 @@ func TestAccept_AFinishedReceiveFreesItsSlot(t *testing.T) {
first := startSlowReceive(t, r, in, "user-1", 1) first := startSlowReceive(t, r, in, "user-1", 1)
require.NoError(t, first.finish(t)) require.NoError(t, first.finish(t))
_, err := in.Accept(context.Background(), "user-1", "t", strings.NewReader("data")) _, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
require.NoError(t, err, "the slot is released when Accept returns") require.NoError(t, err, "the slot is released when Accept returns")
} }
@@ -122,10 +123,10 @@ func TestAccept_AFailedReceiveFreesItsSlot(t *testing.T) {
in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp) in := uploads.NewIntake(r.st, r.pr, r.sub, uploads.Limits{MaxBytes: 1024, MaxDuration: time.Hour}, r.tmp)
in.MaxReceivesPerUser = 1 in.MaxReceivesPerUser = 1
in.MaxReceives = 5 in.MaxReceives = 5
_, err := in.Accept(context.Background(), "user-1", "t", &brokenBody{n: 1}) _, err := in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, &brokenBody{n: 1})
require.ErrorIs(t, err, uploads.ErrClientRead) require.ErrorIs(t, err, uploads.ErrClientRead)
_, err = in.Accept(context.Background(), "user-1", "t", strings.NewReader("data")) _, err = in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
require.NoError(t, err, "a dropped client must not leak a slot") require.NoError(t, err, "a dropped client must not leak a slot")
} }
+3 -2
View File
@@ -12,6 +12,7 @@ import (
"testing" "testing"
"time" "time"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/uploads" "git.d-ma.be/mathias/tapir/internal/uploads"
) )
@@ -21,11 +22,11 @@ func TestNewIntake_DefaultsAllowTwoReceivesPerUserAndFourOverall(t *testing.T) {
startSlowReceive(t, r, in, "a", 1) startSlowReceive(t, r, in, "a", 1)
startSlowReceive(t, r, in, "a", 2) startSlowReceive(t, r, in, "a", 2)
_, err := in.Accept(context.Background(), "a", "t", strings.NewReader("x")) _, err := in.Accept(context.Background(), "a", "t", domain.LangSwedish, strings.NewReader("x"))
requireBusy(t, err) // a third body from one user requireBusy(t, err) // a third body from one user
startSlowReceive(t, r, in, "b", 3) startSlowReceive(t, r, in, "b", 3)
startSlowReceive(t, r, in, "b", 4) startSlowReceive(t, r, in, "b", 4)
_, err = in.Accept(context.Background(), "c", "t", strings.NewReader("x")) _, err = in.Accept(context.Background(), "c", "t", domain.LangSwedish, strings.NewReader("x"))
requireBusy(t, err) // a fifth body overall requireBusy(t, err) // a fifth body overall
} }
+24 -11
View File
@@ -21,13 +21,15 @@ import (
type fakeStore struct { type fakeStore struct {
created []string // titles created []string // titles
langs []domain.Language
hadVideo []bool hadVideo []bool
failed map[string]domain.UploadErrorKind failed map[string]domain.UploadErrorKind
nextID string nextID string
} }
func (f *fakeStore) CreateUpload(_ context.Context, _, title string, hadVideo bool) (string, error) { func (f *fakeStore) CreateUpload(_ context.Context, _, title string, lang domain.Language, hadVideo bool) (string, error) {
f.created = append(f.created, title) f.created = append(f.created, title)
f.langs = append(f.langs, lang)
f.hadVideo = append(f.hadVideo, hadVideo) f.hadVideo = append(f.hadVideo, hadVideo)
return f.nextID, nil return f.nextID, nil
} }
@@ -98,7 +100,7 @@ func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
r := newRig(t) r := newRig(t)
audio := bytes.Repeat([]byte("a"), 500) audio := bytes.Repeat([]byte("a"), 500)
id, err := r.in.Accept(context.Background(), "user-1", "Teams export", bytes.NewReader(audio)) id, err := r.in.Accept(context.Background(), "user-1", "Teams export", domain.LangSwedish, bytes.NewReader(audio))
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, "up-1", id) assert.Equal(t, "up-1", id)
@@ -106,7 +108,7 @@ func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
assert.Equal(t, audio, r.pr.seen, "the prober must see the complete uploaded bytes, on disk") assert.Equal(t, audio, r.pr.seen, "the prober must see the complete uploaded bytes, on disk")
require.Len(t, r.sub.jobs, 1) require.Len(t, r.sub.jobs, 1)
job := r.sub.jobs[0] job := r.sub.jobs[0]
assert.Equal(t, uploads.Job{UserID: "user-1", UploadID: "up-1", Path: job.Path, Duration: 10 * time.Minute}, job, assert.Equal(t, uploads.Job{UserID: "user-1", UploadID: "up-1", Path: job.Path, Duration: 10 * time.Minute, Language: domain.LangSwedish}, job,
"the job carries what the file CLAIMED, so the worker can compare it with what was converted") "the job carries what the file CLAIMED, so the worker can compare it with what was converted")
assert.Equal(t, filepath.Dir(job.Path), r.tmp, "the temp file lives in the configured directory") assert.Equal(t, filepath.Dir(job.Path), r.tmp, "the temp file lives in the configured directory")
got, err := os.ReadFile(job.Path) got, err := os.ReadFile(job.Path)
@@ -117,7 +119,7 @@ func TestAccept_ValidAudioIsStoredProbedQueuedAndSubmitted(t *testing.T) {
func TestAccept_OversizeBodyIsRejectedAndLeavesNothingBehind(t *testing.T) { func TestAccept_OversizeBodyIsRejectedAndLeavesNothingBehind(t *testing.T) {
r := newRig(t) r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "big", strings.NewReader(strings.Repeat("a", 1025))) _, err := r.in.Accept(context.Background(), "user-1", "big", domain.LangSwedish, strings.NewReader(strings.Repeat("a", 1025)))
var rej *uploads.RejectError var rej *uploads.RejectError
require.True(t, errors.As(err, &rej)) require.True(t, errors.As(err, &rej))
@@ -132,7 +134,7 @@ func TestAccept_OversizeBodyIsRejectedAndLeavesNothingBehind(t *testing.T) {
func TestAccept_BodyExactlyAtTheLimitIsAccepted(t *testing.T) { func TestAccept_BodyExactlyAtTheLimitIsAccepted(t *testing.T) {
r := newRig(t) r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "edge", strings.NewReader(strings.Repeat("a", 1024))) _, err := r.in.Accept(context.Background(), "user-1", "edge", domain.LangSwedish, strings.NewReader(strings.Repeat("a", 1024)))
require.NoError(t, err) require.NoError(t, err)
} }
@@ -152,7 +154,7 @@ func TestAccept_ProbeVerdictRejectsWithoutRowAndRemovesTheFile(t *testing.T) {
r := newRig(t) r := newRig(t)
r.pr.probe = tt.probe r.pr.probe = tt.probe
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data")) _, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
var rej *uploads.RejectError var rej *uploads.RejectError
require.True(t, errors.As(err, &rej)) require.True(t, errors.As(err, &rej))
@@ -169,7 +171,7 @@ func TestAccept_UnreadableFileIsARejectionButAProberFaultIsAnError(t *testing.T)
r := newRig(t) r := newRig(t)
r.pr.err = uploads.ErrUnreadable r.pr.err = uploads.ErrUnreadable
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("not media")) _, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("not media"))
var rej *uploads.RejectError var rej *uploads.RejectError
require.True(t, errors.As(err, &rej), "the user's file is the problem: tell them") require.True(t, errors.As(err, &rej), "the user's file is the problem: tell them")
@@ -180,7 +182,7 @@ func TestAccept_UnreadableFileIsARejectionButAProberFaultIsAnError(t *testing.T)
r := newRig(t) r := newRig(t)
r.pr.err = errors.New("exec: ffprobe: executable file not found") r.pr.err = errors.New("exec: ffprobe: executable file not found")
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data")) _, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
require.Error(t, err) require.Error(t, err)
var rej *uploads.RejectError var rej *uploads.RejectError
@@ -194,7 +196,7 @@ func TestAccept_ClientDisconnectMidUploadLeavesNothingBehind(t *testing.T) {
r := newRig(t) r := newRig(t)
body := io.MultiReader(strings.NewReader("partial"), iotest.ErrReader(errors.New("client went away"))) body := io.MultiReader(strings.NewReader("partial"), iotest.ErrReader(errors.New("client went away")))
_, err := r.in.Accept(context.Background(), "user-1", "t", body) _, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, body)
require.Error(t, err) require.Error(t, err)
assert.Empty(t, r.pr.paths) assert.Empty(t, r.pr.paths)
@@ -206,7 +208,7 @@ func TestAccept_SubmitFailureFailsTheRowHonestlyAndRemovesTheFile(t *testing.T)
r := newRig(t) r := newRig(t)
r.sub.err = errors.New("worker stopped") r.sub.err = errors.New("worker stopped")
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data")) _, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
require.Error(t, err) require.Error(t, err)
assert.Equal(t, domain.UploadErrInterrupted, r.st.failed["up-1"], assert.Equal(t, domain.UploadErrInterrupted, r.st.failed["up-1"],
@@ -228,7 +230,7 @@ func TestAccept_AFileWithAVideoTrackIsAcceptedAndRecordsThatItHadOne(t *testing.
r := newRig(t) r := newRig(t)
r.pr.probe = tt.probe r.pr.probe = tt.probe
id, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data")) id, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangSwedish, strings.NewReader("data"))
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, "up-1", id) assert.Equal(t, "up-1", id)
@@ -238,3 +240,14 @@ func TestAccept_AFileWithAVideoTrackIsAcceptedAndRecordsThatItHadOne(t *testing.
}) })
} }
} }
func TestAccept_TheChosenLanguageIsStoredAndHandedToTheJob(t *testing.T) {
r := newRig(t)
_, err := r.in.Accept(context.Background(), "user-1", "t", domain.LangAuto, strings.NewReader("data"))
require.NoError(t, err)
assert.Equal(t, []domain.Language{domain.LangAuto}, r.st.langs, "the row records what the user chose")
require.Len(t, r.sub.jobs, 1)
assert.Equal(t, domain.LangAuto, r.sub.jobs[0].Language, "the worker is told, so the engines are")
}
+1 -1
View File
@@ -313,7 +313,7 @@ func (w *Worker) Process(ctx context.Context, job Job) {
if !planned { if !planned {
tr.Begin("transcribe", 0) tr.Begin("transcribe", 0)
} }
t, err := w.tr.Transcribe(progress.With(jobCtx, tr), wav) t, err := w.tr.Transcribe(domain.WithLanguage(progress.With(jobCtx, tr), job.Language), wav)
if err != nil { if err != nil {
log.Error("upload: transcribe", "kind", errkind.Of(err), "elapsed", time.Since(start).Round(time.Second)) log.Error("upload: transcribe", "kind", errkind.Of(err), "elapsed", time.Since(start).Round(time.Second))
abort(domain.UploadErrTranscriptionFailed) abort(domain.UploadErrTranscriptionFailed)
+23
View File
@@ -126,6 +126,7 @@ type fakeTranscriber struct {
result domain.Transcription result domain.Transcription
err error err error
paths []string paths []string
langs []domain.Language // what each call's context carried
// block, when non-nil, holds every call until it is closed or ctx ends. // block, when non-nil, holds every call until it is closed or ctx ends.
block chan struct{} block chan struct{}
running atomic.Int32 running atomic.Int32
@@ -134,6 +135,7 @@ type fakeTranscriber struct {
func (f *fakeTranscriber) Transcribe(ctx context.Context, wav string) (domain.Transcription, error) { func (f *fakeTranscriber) Transcribe(ctx context.Context, wav string) (domain.Transcription, error) {
f.paths = append(f.paths, wav) f.paths = append(f.paths, wav)
f.langs = append(f.langs, domain.LanguageFrom(ctx))
n := f.running.Add(1) n := f.running.Add(1)
defer f.running.Add(-1) defer f.running.Add(-1)
for { for {
@@ -337,3 +339,24 @@ func (dbErr) Error() string {
return `ERROR: invalid input syntax for type json: "zebrakaka secret words"` return `ERROR: invalid input syntax for type json: "zebrakaka secret words"`
} }
func (dbErr) SQLState() string { return "22P02" } func (dbErr) SQLState() string { return "22P02" }
func TestWorker_TheTranscriberSeesTheLanguageOfTheJobOnItsContext(t *testing.T) {
for name, tt := range map[string]struct {
job domain.Language
want domain.Language
}{
"auto": {domain.LangAuto, domain.LangAuto},
"swedish": {domain.LangSwedish, domain.LangSwedish},
"a job with no language": {"", domain.LangSwedish}, // queued before the field existed
} {
t.Run(name, func(t *testing.T) {
r := newWRig(t, time.Minute)
job := r.spool(t, "a")
job.Language = tt.job
r.w.Process(context.Background(), job)
assert.Equal(t, []domain.Language{tt.want}, r.tr.langs)
})
}
}
+31 -2
View File
@@ -28,7 +28,7 @@ import (
// *uploads.Intake satisfies it. It is the whole of what the handlers know about // *uploads.Intake satisfies it. It is the whole of what the handlers know about
// validation, spooling and queueing. // validation, spooling and queueing.
type Uploader interface { type Uploader interface {
Accept(ctx context.Context, userID, title string, src io.Reader) (id string, err error) Accept(ctx context.Context, userID, title string, lang domain.Language, src io.Reader) (id string, err error)
} }
const ( const (
@@ -36,6 +36,8 @@ const (
// and the title field; the cap itself is enforced on the file part by Intake. // and the title field; the cap itself is enforced on the file part by Intake.
uploadOverhead = 1 << 20 uploadOverhead = 1 << 20
maxTitleRunes = 200 maxTitleRunes = 200
// maxLanguageBytes bounds the language field: the longest value is four bytes.
maxLanguageBytes = 16
// uploadPollEvery is how often an unfinished upload re-asks for its status. // uploadPollEvery is how often an unfinished upload re-asks for its status.
uploadPollEvery = "2s" uploadPollEvery = "2s"
// serviceSilentAfter is how long a job may go without any signal (heartbeat or // serviceSilentAfter is how long a job may go without any signal (heartbeat or
@@ -89,6 +91,7 @@ func (a *App) handleUploadPost(w http.ResponseWriter, r *http.Request) {
return return
} }
title := "" title := ""
lang := domain.LangSwedish
for { for {
part, err := mr.NextPart() part, err := mr.NextPart()
if errors.Is(err, io.EOF) { if errors.Is(err, io.EOF) {
@@ -121,11 +124,29 @@ func (a *App) handleUploadPost(w http.ResponseWriter, r *http.Request) {
return return
} }
title = cleanTitle(string(b)) title = cleanTitle(string(b))
case "language":
// A closed set read before the file: anything else is refused before the file
// is spooled, so a crafted value never reaches an engine (tapir#61).
b, err := io.ReadAll(io.LimitReader(part, maxLanguageBytes+1))
if err != nil {
if a.abandoned(w, r) {
return
}
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "That upload could not be read. Please try again.")
return
}
l, ok := domain.ParseLanguage(string(b))
if !ok || len(b) > maxLanguageBytes {
a.logger().Info("upload rejected", "reason", "language")
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "Choose a language from the list.")
return
}
lang = l
case "file": case "file":
if title == "" { if title == "" {
title = titleFromFilename(part.FileName()) title = titleFromFilename(part.FileName())
} }
id, err := a.Uploads.Accept(r.Context(), userID, title, part) id, err := a.Uploads.Accept(r.Context(), userID, title, lang, part)
if err != nil { if err != nil {
a.uploadFailure(w, r, userID, err) a.uploadFailure(w, r, userID, err)
return return
@@ -300,6 +321,14 @@ func cloudProvider(endpoint string) (string, bool) {
return "", false return "", false
} }
// languageLabel is how the page names the language a recording was transcribed as.
func languageLabel(l domain.Language) string {
if l == domain.LangAuto {
return "Other (detected by the engine)"
}
return "Swedish"
}
// hasNote reports whether any phase carries a note. A note is the only warning for a // hasNote reports whether any phase carries a note. A note is the only warning for a
// recording that is shorter than its file claimed, so a record that has one is shown open. // recording that is shorter than its file claimed, so a record that has one is shown open.
func hasNote(steps []domain.ProgressStep) bool { func hasNote(steps []domain.ProgressStep) bool {
+9
View File
@@ -24,6 +24,14 @@ templ UploadsPage(v uploadsView) {
} }
<form method="post" action="/uploads" enctype="multipart/form-data"> <form method="post" action="/uploads" enctype="multipart/form-data">
<p><label>Title (optional) <input type="text" name="title" maxlength="200"/></label></p> <p><label>Title (optional) <input type="text" name="title" maxlength="200"/></label></p>
<p>
<label>Language
<select name="language">
<option value="sv" selected>Swedish</option>
<option value="auto">Other (detected by the engine)</option>
</select>
</label>
</p>
<p><input type="file" name="file" accept="audio/*,.m4a,.mp3,.wav,.opus,.ogg,.mp4" required/></p> <p><input type="file" name="file" accept="audio/*,.m4a,.mp3,.wav,.opus,.ogg,.mp4" required/></p>
<p><button type="submit">Upload and transcribe</button></p> <p><button type="submit">Upload and transcribe</button></p>
</form> </form>
@@ -66,6 +74,7 @@ templ uploadStatus(u domain.UploadedTranscript, chatEnabled bool, now time.Time)
Transcribed { uploads.FormatDuration(time.Duration(u.AudioSeconds * float64(time.Second))) } Transcribed { uploads.FormatDuration(time.Duration(u.AudioSeconds * float64(time.Second))) }
of audio by <span class="chip">{ u.Endpoint }</span>. of audio by <span class="chip">{ u.Endpoint }</span>.
</p> </p>
<p>Language: { languageLabel(u.Language) }</p>
if provider, cloud := cloudProvider(u.Endpoint); cloud { if provider, cloud := cloudProvider(u.Endpoint); cloud {
<p class="cloud-note"> <p class="cloud-note">
This recording's audio left the homelab: it was transcribed by a cloud service ({ provider }) This recording's audio left the homelab: it was transcribed by a cloud service ({ provider })
@@ -14,6 +14,8 @@ import (
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
) )
// A client that hangs up in the middle of an upload is routine, not a server fault. The // A client that hangs up in the middle of an upload is routine, not a server fault. The
@@ -85,7 +87,7 @@ func TestUploadPost_AClientThatHangsUpAnywhereInTheBodyIsLoggedAtInfoAndNeverAsA
// context is cancelled, then Accept fails the way a read from a dead connection does. // context is cancelled, then Accept fails the way a read from a dead connection does.
type goneMidAccept struct{ cancel context.CancelFunc } type goneMidAccept struct{ cancel context.CancelFunc }
func (g goneMidAccept) Accept(context.Context, string, string, io.Reader) (string, error) { func (g goneMidAccept) Accept(context.Context, string, string, domain.Language, io.Reader) (string, error) {
g.cancel() g.cancel()
return "", fmt.Errorf("receive: %w", io.ErrClosedPipe) return "", fmt.Errorf("receive: %w", io.ErrClosedPipe)
} }
+2 -1
View File
@@ -16,6 +16,7 @@ import (
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
"git.d-ma.be/mathias/tapir/internal/uploads" "git.d-ma.be/mathias/tapir/internal/uploads"
"git.d-ma.be/mathias/tapir/internal/web" "git.d-ma.be/mathias/tapir/internal/web"
) )
@@ -132,7 +133,7 @@ func TestUploadPost_AcceptedAudioQueuesARowAndRedirectsToIt(t *testing.T) {
assert.Equal(t, "Sprint review", title) assert.Equal(t, "Sprint review", title)
assert.Equal(t, "/uploads/"+id, rec.Header().Get("Location")) assert.Equal(t, "/uploads/"+id, rec.Header().Get("Location"))
require.Len(t, sub.jobs, 1) require.Len(t, sub.jobs, 1)
assert.Equal(t, uploads.Job{UserID: userID, UploadID: id, Path: sub.jobs[0].Path, Duration: audioOK.Duration}, sub.jobs[0]) assert.Equal(t, uploads.Job{UserID: userID, UploadID: id, Path: sub.jobs[0].Path, Language: domain.LangSwedish, Duration: audioOK.Duration}, sub.jobs[0])
spooled, err := os.ReadFile(sub.jobs[0].Path) spooled, err := os.ReadFile(sub.jobs[0].Path)
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, "synthetic audio bytes", string(spooled)) assert.Equal(t, "synthetic audio bytes", string(spooled))
@@ -0,0 +1,149 @@
package web_test
// The language of a recording is chosen on the upload form (tapir#61). These drive the real
// Router, Intake and Postgres store; the submitter is the process boundary and records the
// job the worker would receive.
import (
"bytes"
"context"
"mime/multipart"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
)
// langReq builds an upload whose parts come in the order given: "title", "language" and
// "file" (the file part always carries the same synthetic bytes).
func langReq(t *testing.T, order []string, language string) *http.Request {
t.Helper()
var buf bytes.Buffer
mw := multipart.NewWriter(&buf)
for _, part := range order {
switch part {
case "title":
require.NoError(t, mw.WriteField("title", "m"))
case "language":
require.NoError(t, mw.WriteField("language", language))
case "file":
fw, err := mw.CreateFormFile("file", "m.wav")
require.NoError(t, err)
_, err = fw.Write([]byte("RIFF....WAVE"))
require.NoError(t, err)
}
}
require.NoError(t, mw.Close())
req := httptest.NewRequest(http.MethodPost, "/uploads", &buf)
req.Header.Set("Content-Type", mw.FormDataContentType())
return req
}
func rowLanguage(t *testing.T, location string) domain.Language {
t.Helper()
id := strings.TrimPrefix(location, "/uploads/")
got, err := newStore(t).GetUpload(context.Background(), userID, id)
require.NoError(t, err)
return got.Language
}
func TestUploadPost_TheLanguageFieldReachesTheJobAndTheRow(t *testing.T) {
for _, tt := range []struct {
name string
language string
want domain.Language
}{
{"other", "auto", domain.LangAuto},
{"swedish", "sv", domain.LangSwedish},
} {
t.Run(tt.name, func(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
rec := do(t, app, langReq(t, []string{"title", "language", "file"}, tt.language))
require.Equal(t, http.StatusSeeOther, rec.Code)
require.Len(t, sub.jobs, 1)
assert.Equal(t, tt.want, sub.jobs[0].Language)
assert.Equal(t, tt.want, rowLanguage(t, rec.Header().Get("Location")))
})
}
}
func TestUploadPost_NoLanguageFieldIsSwedishAsBefore(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
rec := do(t, app, langReq(t, []string{"title", "file"}, ""))
require.Equal(t, http.StatusSeeOther, rec.Code)
require.Len(t, sub.jobs, 1)
assert.Equal(t, domain.LangSwedish, sub.jobs[0].Language)
assert.Equal(t, domain.LangSwedish, rowLanguage(t, rec.Header().Get("Location")))
}
// A part after the file is never read (the file part ends the stream), so a client that
// orders its fields wrongly gets Swedish, the old behaviour, not a guess.
func TestUploadPost_ALanguageFieldAfterTheFileIsNotReadAndTheUploadIsSwedish(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
rec := do(t, app, langReq(t, []string{"title", "file", "language"}, "auto"))
require.Equal(t, http.StatusSeeOther, rec.Code)
require.Len(t, sub.jobs, 1)
assert.Equal(t, domain.LangSwedish, sub.jobs[0].Language)
}
func TestUploadPost_ALanguageOutsideTheClosedSetIsRefusedBeforeAnythingIsSpooledOrStored(t *testing.T) {
for _, bad := range []string{"en", "SV", " sv", "sv,auto", "../auto", strings.Repeat("a", 5000)} {
t.Run(bad[:min(len(bad), 12)], func(t *testing.T) {
app, sub := newUploadApp(t, audioOK, 200<<20)
rec := do(t, app, langReq(t, []string{"title", "language", "file"}, bad))
assert.Equal(t, http.StatusBadRequest, rec.Code)
assert.Empty(t, sub.jobs, "a refused post must not enqueue transcription")
rows, err := newStore(t).ListUploads(context.Background(), userID)
require.NoError(t, err)
assert.Empty(t, rows, "and must leave no row")
assert.Contains(t, body(t, rec), "language")
})
}
}
func TestUploadPage_TheFormOffersSwedishPreselectedAndOtherBeforeTheFilePart(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
b := body(t, do(t, app, get("/uploads")))
sel := strings.Index(b, `name="language"`)
file := strings.Index(b, `name="file"`)
require.True(t, sel >= 0, "the form has a language field")
assert.True(t, sel < file, "the field must come before the file part: the stream ends at the file")
assert.Regexp(t, `<option value="sv" selected`, b, "Swedish is preselected: today's behaviour")
assert.Contains(t, b, `<option value="auto"`)
}
func TestUploadStatus_NamesTheLanguageTheRecordingWasTranscribedAs(t *testing.T) {
for _, tt := range []struct {
language domain.Language
want string
}{
{domain.LangSwedish, "Swedish"},
{domain.LangAuto, "Other (detected by the engine)"},
} {
t.Run(string(tt.language), func(t *testing.T) {
app, _ := newUploadApp(t, audioOK, 200<<20)
id := seedUpload(t, userID, "m", "done", "Synthetic transcript text.")
_, err := rawPool(t).Exec(context.Background(), `UPDATE uploaded_transcripts SET language = $2 WHERE id = $1`, id, string(tt.language))
require.NoError(t, err)
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
assert.Contains(t, b, "Language: "+tt.want)
})
}
}
+3 -1
View File
@@ -13,6 +13,8 @@ import (
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"git.d-ma.be/mathias/tapir/internal/domain"
) )
// Error paths of the upload routes log a kind, never the error's text, which can carry // Error paths of the upload routes log a kind, never the error's text, which can carry
@@ -37,7 +39,7 @@ func (l *lockedBuf) String() string {
type failingUploader struct{ err error } type failingUploader struct{ err error }
func (f failingUploader) Accept(context.Context, string, string, io.Reader) (string, error) { func (f failingUploader) Accept(context.Context, string, string, domain.Language, io.Reader) (string, error) {
return "", f.err return "", f.err
} }
File diff suppressed because it is too large Load Diff
@@ -123,6 +123,10 @@ var scenarioCoverage = map[string]string{
"A long recording is not cut to its first third": "TestAnswerMaxCharsOverridesTheServiceCap", "A long recording is not cut to its first third": "TestAnswerMaxCharsOverridesTheServiceCap",
"A transcript made by the cloud fallback says the audio left the homelab": "TestUploadStatus_ACloudServedTranscriptSaysTheAudioLeftTheHomelab", "A transcript made by the cloud fallback says the audio left the homelab": "TestUploadStatus_ACloudServedTranscriptSaysTheAudioLeftTheHomelab",
"I choose the language of a recording": "TestUploadPost_TheLanguageFieldReachesTheJobAndTheRow",
"Swedish stays the default": "TestUploadPost_NoLanguageFieldIsSwedishAsBefore",
"A language outside the list is refused": "TestUploadPost_ALanguageOutsideTheClosedSetIsRefusedBeforeAnythingIsSpooledOrStored",
"A recording in another language never reaches the Swedish-only engine": "TestEnsemble_AnAutoLanguageUploadNeverReachesTheBaseEngineAndIsNotReconciled",
"I can delete a finished recording": "TestUploadDelete_AnOwnerRemovesAFinishedOrFailedRecordingAndLandsOnTheList", "I can delete a finished recording": "TestUploadDelete_AnOwnerRemovesAFinishedOrFailedRecordingAndLandsOnTheList",
"A recording still being transcribed cannot be deleted": "TestUploadDelete_ARecordingStillBeingTranscribedIsRefusedWith409AndKept", "A recording still being transcribed cannot be deleted": "TestUploadDelete_ARecordingStillBeingTranscribedIsRefusedWith409AndKept",