Compare commits
33
Commits
aa54089230
...
08ffc2ff62
+21
-8
@@ -1735,17 +1735,30 @@ wording while only `content` is merged.
|
|||||||
`network`, `error`), never its text, which can hold a provider's response body, the spool path or
|
`network`, `error`), never its text, which can hold a provider's response body, the spool path or
|
||||||
a word from the transcript (`internal/errkind`). The stored reconcile reason and an engine's
|
a word from the transcript (`internal/errkind`). The stored reconcile reason and an engine's
|
||||||
failure reason are kinds; the page no longer shows provider text.
|
failure reason are kinds; the page no longer shows provider text.
|
||||||
- **Every local engine call has its own deadline** (`TAPIR_TRANSCRIBE_ENGINE_TIMEOUT`, default 25
|
- **Every local engine call has its own deadline** (`TAPIR_TRANSCRIBE_ENGINE_TIMEOUT`, default 20
|
||||||
minutes), so a wedged engine fails while the job lives and the fallback can run; the job's own
|
minutes: two wedged engines in sequence then leave 20 of the job's 60 minutes for a fallback),
|
||||||
|
so a wedged engine fails while the job lives and the fallback can run; the job's own
|
||||||
deadline or a cancel still never reaches the cloud.
|
deadline or a cancel still never reaches the cloud.
|
||||||
- **The duration limit is checked against what is converted**, not the file's header: ffmpeg is
|
- **The duration limit is checked against what is converted**, not the file's header: ffmpeg is
|
||||||
capped at the limit plus a minute, a longer result fails the job (`audio_too_long`), and a result
|
capped at the limit plus a minute, a longer result fails the job (`audio_too_long`), and a result
|
||||||
under 80% of the claimed length and more than two minutes short fails it
|
under 80% of the claimed length and more than two minutes short is **transcribed and noted** on
|
||||||
(`audio_incomplete`) instead of returning a partial transcript with no warning.
|
the audio step ("the audio is 20 min long; the file claimed 1 h 19 min"), not refused. The second
|
||||||
- **The queue is bounded**, 20 waiting in total and 3 per user; a full queue is a 429 with a typed
|
adversarial review showed a complete variable-bitrate MP3 whose header over-estimated its length
|
||||||
reason, never a stuck row or a server error.
|
by 4x was wrongly refused by the first version of this check; a note keeps a truncated file
|
||||||
- **Bad input is a 4xx**: a non-UUID id is a 404, a malformed multipart header is a 400 whose text is
|
visible without turning away real recordings.
|
||||||
not logged, and a title or file name with control bytes or invalid UTF-8 is cleaned, not a 500.
|
- **The queue is bounded**, 20 waiting in total and 3 per user, and its room is checked before the
|
||||||
|
upload is spooled (not after 200 MB and an ffprobe run); a full queue is a 429 with a typed reason,
|
||||||
|
never a stuck row or a server error.
|
||||||
|
- **The configured upload chat model is callable.** It is added to the shared chat service's model
|
||||||
|
list when uploads are on and it is local, because the service re-maps a model it does not offer
|
||||||
|
(a valid `iguana/gemma4-31b` was being silently swapped for `koala/phi4-mini`).
|
||||||
|
- **Bad input is a 4xx**: a non-UUID id is a 404, a malformed or cut-off multipart body is a 400
|
||||||
|
whose text is not logged, and a title with control bytes, bidirectional controls or invalid UTF-8 is
|
||||||
|
cleaned (emoji joiners, Swedish letters and variation selectors are kept), not a 500.
|
||||||
|
- **The image was built and inspected** (buildah on koala, 2026-10-02): the first build FAILED
|
||||||
|
(`tar: xz: Cannot exec`, the golang image has no xz); fixed, rebuilt at 180 MB, ffmpeg and ffprobe
|
||||||
|
run as `nonroot` in distroless. CI does not build the image on a pull request
|
||||||
|
(`if: github.event_name != 'pull_request'`), so the first CI build of it is the merge to main.
|
||||||
Not done, filed as follow-ups: a vendored ffmpeg tarball, CSRF cookie review, ffprobe CPU and
|
Not done, filed as follow-ups: a vendored ffmpeg tarball, CSRF cookie review, ffprobe CPU and
|
||||||
stream-count limits, mov external references, `alive_at` clock skew, a hung database blocking the
|
stream-count limits, mov external references, `alive_at` clock skew, a hung database blocking the
|
||||||
tracker, an unbounded gateway response body, a user-facing notice for a recording that is partial but
|
tracker, an unbounded gateway response body, a user-facing notice for a recording that is partial but
|
||||||
|
|||||||
@@ -36,7 +36,11 @@ FROM golang:1.26 AS ffmpeg
|
|||||||
ARG FFMPEG_VERSION=7.0.2
|
ARG FFMPEG_VERSION=7.0.2
|
||||||
ARG FFMPEG_SHA256=abda8d77ce8309141f83ab8edf0596834087c52467f6badf376a6a2a4c87cf67
|
ARG FFMPEG_SHA256=abda8d77ce8309141f83ab8edf0596834087c52467f6badf376a6a2a4c87cf67
|
||||||
|
|
||||||
|
# xz is not in the golang image (found by building it: `tar: xz: Cannot exec`).
|
||||||
RUN set -eu; \
|
RUN set -eu; \
|
||||||
|
apt-get update; \
|
||||||
|
apt-get install -y --no-install-recommends xz-utils; \
|
||||||
|
rm -rf /var/lib/apt/lists/*; \
|
||||||
mkdir -p /out; \
|
mkdir -p /out; \
|
||||||
for dir in releases old-releases; do \
|
for dir in releases old-releases; do \
|
||||||
curl -fsSL -o /out/ffmpeg.tar.xz "https://johnvansickle.com/ffmpeg/${dir}/ffmpeg-${FFMPEG_VERSION}-amd64-static.tar.xz" && break; \
|
curl -fsSL -o /out/ffmpeg.tar.xz "https://johnvansickle.com/ffmpeg/${dir}/ffmpeg-${FFMPEG_VERSION}-amd64-static.tar.xz" && break; \
|
||||||
|
|||||||
@@ -134,6 +134,13 @@ func chatModels(cfg config.Config) []string {
|
|||||||
add(cfg.SummarizerModel)
|
add(cfg.SummarizerModel)
|
||||||
add(cfg.FallbackModel)
|
add(cfg.FallbackModel)
|
||||||
add(cfg.CloudFallbackModel)
|
add(cfg.CloudFallbackModel)
|
||||||
|
// The model chat over an uploaded recording opens on must be one the shared service can
|
||||||
|
// call: it re-maps a model it does not offer, which silently swapped a configured 31B for
|
||||||
|
// phi4-mini, whose context cannot hold an hour of speech. Local only, and only when
|
||||||
|
// uploads are on, so the video chat is unchanged otherwise (second adversarial review).
|
||||||
|
if cfg.UploadsEnabled && config.IsLocalModel(cfg.UploadChatModel) {
|
||||||
|
add(cfg.UploadChatModel)
|
||||||
|
}
|
||||||
return models
|
return models
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -44,3 +44,42 @@ func TestUploadChatModels_NoLocalModelMeansNoneOfferedNotACloudDefault(t *testin
|
|||||||
t.Errorf("uploadChatModels = %v, want none", got)
|
t.Errorf("uploadChatModels = %v, want none", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A valid local TAPIR_UPLOAD_CHAT_MODEL that is not in the video chat's chain used to be
|
||||||
|
// accepted by config and then silently ignored: the shared chat service re-maps a model it
|
||||||
|
// does not offer, so the user got koala/phi4-mini, whose 8,192-token context cannot hold an
|
||||||
|
// hour of speech (second adversarial review).
|
||||||
|
|
||||||
|
func TestChatModels_AnUploadChatModelOutsideTheChainIsOfferedWhenUploadsAreOn(t *testing.T) {
|
||||||
|
var cfg config.Config
|
||||||
|
cfg.SummarizerModel = "koala/phi4-mini"
|
||||||
|
cfg.FallbackModel = "iguana/gemma4-26b"
|
||||||
|
cfg.UploadsEnabled = true
|
||||||
|
cfg.UploadChatModel = "iguana/gemma4-31b"
|
||||||
|
|
||||||
|
service := chatModels(cfg)
|
||||||
|
upload := uploadChatModels(cfg)
|
||||||
|
|
||||||
|
if !reflect.DeepEqual(service, []string{"koala/phi4-mini", "iguana/gemma4-26b", "iguana/gemma4-31b"}) {
|
||||||
|
t.Errorf("chatModels = %v: the shared service must be able to call the configured upload model", service)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(upload, service) {
|
||||||
|
t.Errorf("uploadChatModels = %v, want %v", upload, service)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestChatModels_NothingIsAddedWhenUploadsAreOffOrTheModelIsNotLocal(t *testing.T) {
|
||||||
|
var cfg config.Config
|
||||||
|
cfg.SummarizerModel = "koala/phi4-mini"
|
||||||
|
cfg.UploadChatModel = "iguana/gemma4-31b"
|
||||||
|
|
||||||
|
if got := chatModels(cfg); !reflect.DeepEqual(got, []string{"koala/phi4-mini"}) {
|
||||||
|
t.Errorf("uploads off: chatModels = %v, want the video chain unchanged", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg.UploadsEnabled = true
|
||||||
|
cfg.UploadChatModel = "berget/mistral-small"
|
||||||
|
if got := chatModels(cfg); !reflect.DeepEqual(got, []string{"koala/phi4-mini"}) {
|
||||||
|
t.Errorf("a cloud upload chat model was added to the service: %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -43,3 +43,36 @@ func TestBuildTranscriber_AWedgedPrimaryReachesTheFallbackBeforeTheJobDeadline(t
|
|||||||
require.NoError(t, err, "the wedged primary used the whole job and the fallback never ran")
|
require.NoError(t, err, "the wedged primary used the whole job and the fallback never ran")
|
||||||
assert.Equal(t, "berget/kb-whisper", got.Endpoint)
|
assert.Equal(t, "berget/kb-whisper", got.Endpoint)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestBuildTranscriber_AWedgedBaseEngineIsASoftFailureInTheEnsemble(t *testing.T) {
|
||||||
|
// The same deadline must guard the ensemble's BASE engine, not only the chain's primary
|
||||||
|
// (second adversarial review: the wiring was tested for the chain alone).
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
_ = r.ParseMultipartForm(1 << 20)
|
||||||
|
if r.FormValue("model") == "iguana/pianissimo" {
|
||||||
|
<-r.Context().Done()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_, _ = w.Write([]byte(`{"text":"hej","duration":1,"segments":[{"start":0,"end":1,"text":"hej"}]}`))
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
cfg := uploadCfg(t, true)
|
||||||
|
cfg.GatewayURL = srv.URL
|
||||||
|
cfg.TranscribeModel = "iguana/kb-whisper"
|
||||||
|
cfg.ReconcileBaseModel = "iguana/pianissimo"
|
||||||
|
cfg.ReconcileModel = "iguana/gemma4-31b"
|
||||||
|
cfg.TranscribeEngineTimeout = 150 * time.Millisecond
|
||||||
|
wav := filepath.Join(t.TempDir(), "a.wav")
|
||||||
|
require.NoError(t, os.WriteFile(wav, []byte("RIFF....WAVEfmt "), 0o600))
|
||||||
|
job, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
got, err := buildTranscriber(cfg).Transcribe(job, wav)
|
||||||
|
|
||||||
|
require.NoError(t, err, "a wedged base engine must not use up the job")
|
||||||
|
assert.Equal(t, "hej", got.Text)
|
||||||
|
require.NotNil(t, got.Reconciliation)
|
||||||
|
assert.Equal(t, "single_engine", string(got.Reconciliation.Status))
|
||||||
|
assert.Contains(t, got.Reconciliation.Reason, "timeout")
|
||||||
|
}
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ it** — endpoints and aliases drift, and this file is a snapshot (2026-06-06),
|
|||||||
`{"error":"failed to process audio"}` string — it has had at least five causes. Tapir's knobs:
|
`{"error":"failed to process audio"}` string — it has had at least five causes. Tapir's knobs:
|
||||||
`TAPIR_UPLOADS_ENABLED` (off by default), `TAPIR_UPLOAD_MAX_MB` (200), `TAPIR_UPLOAD_MAX_MINUTES`
|
`TAPIR_UPLOADS_ENABLED` (off by default), `TAPIR_UPLOAD_MAX_MB` (200), `TAPIR_UPLOAD_MAX_MINUTES`
|
||||||
(90), `TAPIR_TRANSCRIBE_MODEL`, `TAPIR_TRANSCRIBE_FALLBACK_MODEL` (empty disables the cloud
|
(90), `TAPIR_TRANSCRIBE_MODEL`, `TAPIR_TRANSCRIBE_FALLBACK_MODEL` (empty disables the cloud
|
||||||
fallback), `TAPIR_TRANSCRIBE_TIMEOUT` (60m, the whole job), `TAPIR_TRANSCRIBE_ENGINE_TIMEOUT` (25m, one engine call: a wedged engine then fails while the job lives and the fallback can run), `TAPIR_UPLOAD_TMP_DIR`, `TAPIR_FFMPEG_BIN`,
|
fallback), `TAPIR_TRANSCRIBE_TIMEOUT` (60m, the whole job), `TAPIR_TRANSCRIBE_ENGINE_TIMEOUT` (20m, one engine call: a wedged engine then fails while the job lives and the fallback can run), `TAPIR_UPLOAD_TMP_DIR`, `TAPIR_FFMPEG_BIN`,
|
||||||
`TAPIR_FFPROBE_BIN`, `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS` (90000 — the measured ceiling of
|
`TAPIR_FFPROBE_BIN`, `TAPIR_MAX_UPLOADED_TRANSCRIPT_CHARS` (90000 — the measured ceiling of
|
||||||
`iguana/gemma4-26b`, which silently drops the head above it), `TAPIR_UPLOAD_CHAT_MODEL`
|
`iguana/gemma4-26b`, which silently drops the head above it), `TAPIR_UPLOAD_CHAT_MODEL`
|
||||||
(`iguana/gemma4-26b`; must be a local `koala/` or `iguana/` alias, since the whole private transcript is sent to it; `koala/phi4-mini`'s 8,192-token context 400s on an hour of speech). `ffmpeg` + `ffprobe` are in the runtime image (static, pinned by sha256; #34 decided A on 2026-10-02); Tapir's own ingress body limit for a 200 MB upload is unverified (infra).
|
(`iguana/gemma4-26b`; must be a local `koala/` or `iguana/` alias, since the whole private transcript is sent to it; `koala/phi4-mini`'s 8,192-token context 400s on an hour of speech). `ffmpeg` + `ffprobe` are in the runtime image (static, pinned by sha256; #34 decided A on 2026-10-02); Tapir's own ingress body limit for a 200 MB upload is unverified (infra).
|
||||||
|
|||||||
@@ -235,9 +235,10 @@ Feature: Upload an audio recording, get a private Swedish transcript, discuss it
|
|||||||
When a file claims to be short but converts to more than the limit
|
When a file claims to be short but converts to more than the limit
|
||||||
Then the job fails with a plain explanation and no engine receives the audio
|
Then the job fails with a plain explanation and no engine receives the audio
|
||||||
|
|
||||||
Scenario: A truncated recording is refused, not half transcribed
|
Scenario: A recording much shorter than its file claimed is transcribed and noted
|
||||||
When a file converts to far less audio than it claimed
|
When a file converts to under 80 percent of the length it claimed and more than two minutes short
|
||||||
Then the job fails and asks for the complete recording
|
Then it is transcribed anyway and its audio step says how long the audio really is
|
||||||
|
And a complete recording whose header over-estimated its length is not refused
|
||||||
|
|
||||||
Scenario: A full queue is a retry, not an error
|
Scenario: A full queue is a retry, not an error
|
||||||
When too many recordings are already waiting
|
When too many recordings are already waiting
|
||||||
|
|||||||
@@ -78,7 +78,7 @@ printf data > "$last"`)
|
|||||||
assert.Less(t, fw, in)
|
assert.Less(t, fw, in)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestFFmpegToWAV_CapsTheOutputLengthWhenAMaximumIsSet(t *testing.T) {
|
func TestFFmpegToWAV_PassesAnOutputLengthCapAfterTheInputWhenAMaximumIsSet(t *testing.T) {
|
||||||
// The duration limit is a CLAIM in the file's header; a file that lies about it can
|
// The duration limit is a CLAIM in the file's header; a file that lies about it can
|
||||||
// convert to hours of audio (adversarial review F2: a 2 h file claiming 10 s became a
|
// convert to hours of audio (adversarial review F2: a 2 h file claiming 10 s became a
|
||||||
// 120-minute WAV). The cap is an OUTPUT option, after -i, with a minute of slack.
|
// 120-minute WAV). The cap is an OUTPUT option, after -i, with a minute of slack.
|
||||||
@@ -97,7 +97,7 @@ printf data > "$last"`)
|
|||||||
assert.Less(t, strings.Index(joined, " -i /spool/in "), strings.Index(joined, " -t 5460 "), "-t after -i is an output cap; before it, an input one")
|
assert.Less(t, strings.Index(joined, " -i /spool/in "), strings.Index(joined, " -t 5460 "), "-t after -i is an output cap; before it, an input one")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestFFmpegToWAV_WithNoMaximumThereIsNoLengthCap(t *testing.T) {
|
func TestFFmpegToWAV_PassesNoLengthCapWhenNoMaximumIsSet(t *testing.T) {
|
||||||
rec := filepath.Join(t.TempDir(), "argv")
|
rec := filepath.Join(t.TempDir(), "argv")
|
||||||
bin := stub(t, `printf '%s\n' "$@" > `+rec+`
|
bin := stub(t, `printf '%s\n' "$@" > `+rec+`
|
||||||
for a; do last="$a"; done
|
for a; do last="$a"; done
|
||||||
|
|||||||
@@ -66,7 +66,7 @@ const (
|
|||||||
defaultTranscribeModel = "iguana/kb-whisper"
|
defaultTranscribeModel = "iguana/kb-whisper"
|
||||||
defaultTranscribeFallbackModel = "berget/kb-whisper"
|
defaultTranscribeFallbackModel = "berget/kb-whisper"
|
||||||
defaultTranscribeTimeout = 60 * time.Minute
|
defaultTranscribeTimeout = 60 * time.Minute
|
||||||
defaultTranscribeEngineTimeout = 25 * time.Minute
|
defaultTranscribeEngineTimeout = 20 * time.Minute
|
||||||
defaultMaxUploadedChars = 90000
|
defaultMaxUploadedChars = 90000
|
||||||
defaultUploadChatModel = "iguana/gemma4-26b"
|
defaultUploadChatModel = "iguana/gemma4-26b"
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -76,14 +76,14 @@ func TestLoad_ALocalUploadChatModelIsAccepted(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestLoad_TheEngineTimeoutDefaultsToTwentyFiveMinutesAndMustBePositive(t *testing.T) {
|
func TestLoad_TheEngineTimeoutDefaultsToTwentyMinutesAndMustBePositive(t *testing.T) {
|
||||||
clearUploadEnv(t)
|
clearUploadEnv(t)
|
||||||
cfg, err := Load()
|
cfg, err := Load()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Load: %v", err)
|
t.Fatalf("Load: %v", err)
|
||||||
}
|
}
|
||||||
if cfg.TranscribeEngineTimeout != 25*time.Minute {
|
if cfg.TranscribeEngineTimeout != 20*time.Minute {
|
||||||
t.Errorf("TranscribeEngineTimeout = %v, want 25m: a 90-minute recording takes about 8 minutes on kb-whisper", cfg.TranscribeEngineTimeout)
|
t.Errorf("TranscribeEngineTimeout = %v, want 20m: a 90-minute recording takes about 8 minutes on kb-whisper, and two wedged engines in sequence must leave the job time to reach a fallback", cfg.TranscribeEngineTimeout)
|
||||||
}
|
}
|
||||||
|
|
||||||
setEnv(t, map[string]string{"TAPIR_TRANSCRIBE_ENGINE_TIMEOUT": "7m"})
|
setEnv(t, map[string]string{"TAPIR_TRANSCRIBE_ENGINE_TIMEOUT": "7m"})
|
||||||
|
|||||||
@@ -203,6 +203,10 @@ type ProgressStep struct {
|
|||||||
FinishedAt time.Time `json:"finished_at,omitzero"`
|
FinishedAt time.Time `json:"finished_at,omitzero"`
|
||||||
Done int `json:"done,omitempty"`
|
Done int `json:"done,omitempty"`
|
||||||
Total int `json:"total,omitempty"`
|
Total int `json:"total,omitempty"`
|
||||||
|
// Note is a short plain remark about the phase that the reader should see even though
|
||||||
|
// it did not fail, for example that the audio is much shorter than the file claimed.
|
||||||
|
// Durations and counts only: never transcript text, a name or a path.
|
||||||
|
Note string `json:"note,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// UploadStatus is the state of an uploaded-audio transcription job.
|
// UploadStatus is the state of an uploaded-audio transcription job.
|
||||||
@@ -233,10 +237,6 @@ const (
|
|||||||
// UploadErrBusy: too many recordings were already waiting, so this one was not
|
// UploadErrBusy: too many recordings were already waiting, so this one was not
|
||||||
// queued. The user's to retry later (ADR-035 review).
|
// queued. The user's to retry later (ADR-035 review).
|
||||||
UploadErrBusy UploadErrorKind = "queue_full"
|
UploadErrBusy UploadErrorKind = "queue_full"
|
||||||
// UploadErrIncomplete: the audio, once converted, ends far earlier than the file
|
|
||||||
// claimed: a truncated recording that would otherwise give a partial transcript with
|
|
||||||
// no warning.
|
|
||||||
UploadErrIncomplete UploadErrorKind = "audio_incomplete"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// UploadedTranscript is a user's PRIVATE transcript of audio they uploaded. It is
|
// UploadedTranscript is a user's PRIVATE transcript of audio they uploaded. It is
|
||||||
|
|||||||
@@ -186,6 +186,20 @@ func (t *Tracker) Finish() {
|
|||||||
t.save()
|
t.save()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Annotate attaches a short plain note to a phase and saves it at once. It is for a
|
||||||
|
// remark the reader should see although nothing failed. Durations and counts only: a note
|
||||||
|
// is shown on a page about a private recording.
|
||||||
|
func (t *Tracker) Annotate(key, note string) {
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
s := t.find(key)
|
||||||
|
if s == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.Note = note
|
||||||
|
t.save()
|
||||||
|
}
|
||||||
|
|
||||||
// Heartbeat stamps the job's row every beatEvery and logs which phase is still running
|
// Heartbeat stamps the job's row every beatEvery and logs which phase is still running
|
||||||
// every stillEvery, until ctx ends. It runs on its own, independent of any engine call:
|
// every stillEvery, until ctx ends. It runs on its own, independent of any engine call:
|
||||||
// a heartbeat that stopped means the process is gone, not that a request is slow.
|
// a heartbeat that stopped means the process is gone, not that a request is slow.
|
||||||
|
|||||||
@@ -387,3 +387,16 @@ func TestWithAndFrom_ACarriedReporterIsTheOneReturned(t *testing.T) {
|
|||||||
|
|
||||||
require.Equal(t, domain.StepRunning, stateOf(sink.last(t), "audio"))
|
require.Equal(t, domain.StepRunning, stateOf(sink.last(t), "audio"))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestTracker_AnnotateAttachesANoteToAStepAndSavesItAtOnce(t *testing.T) {
|
||||||
|
tr, sink, _, _ := newTracker(t)
|
||||||
|
tr.Start()
|
||||||
|
tr.Begin("audio", 0)
|
||||||
|
before := sink.saveCount()
|
||||||
|
|
||||||
|
tr.Annotate("audio", "20 min of the 79 the file claimed")
|
||||||
|
|
||||||
|
require.Equal(t, before+1, sink.saveCount())
|
||||||
|
require.Equal(t, "20 min of the 79 the file claimed", sink.last(t)[0].Note)
|
||||||
|
require.NotPanics(t, func() { tr.Annotate("no-such-step", "x") })
|
||||||
|
}
|
||||||
|
|||||||
@@ -37,10 +37,10 @@ func TestReconcileSpots_AutoResolvedDisputesAreNotStored(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestReconcileSpots_ASpotCarriesTheWordsJustBeforeIt(t *testing.T) {
|
func TestReconcileSpots_ASpotCarriesTheWordsJustBeforeIt(t *testing.T) {
|
||||||
out := merge(t, tr("vi skickar in ett vanligt meddelande till kassasystemet"), tr("vi skickar in ett KORT meddelande till kassasystemet"), always(reconcile.VerdictAlt))
|
out := merge(t, tr("grannen målar ett vanligt staket mot gatan"), tr("grannen målar ett KORT staket mot gatan"), always(reconcile.VerdictAlt))
|
||||||
|
|
||||||
require.Len(t, out.Reconciliation.Contested, 1)
|
require.Len(t, out.Reconciliation.Contested, 1)
|
||||||
assert.Equal(t, "vi skickar in ett", out.Reconciliation.Contested[0].Before)
|
assert.Equal(t, "grannen målar ett", out.Reconciliation.Contested[0].Before)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestReconcileSpots_TheContextIsAFewWordsNotTheWholeMeeting(t *testing.T) {
|
func TestReconcileSpots_TheContextIsAFewWordsNotTheWholeMeeting(t *testing.T) {
|
||||||
|
|||||||
@@ -58,12 +58,21 @@ func NewIntake(s Store, p Prober, sub Submitter, l Limits, tmpDir string) *Intak
|
|||||||
return &Intake{store: s, prober: p, sub: sub, limits: l, tmpDir: tmpDir}
|
return &Intake{store: s, prober: p, sub: sub, limits: l, tmpDir: tmpDir}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// busyMessage is what a user is told when the queue is full.
|
||||||
|
const busyMessage = "Too many recordings are waiting to be transcribed. Please try again in a few minutes."
|
||||||
|
|
||||||
// Accept spools src to disk (bounded by MaxBytes), probes it, validates it, records
|
// Accept spools src to disk (bounded by MaxBytes), probes it, validates it, records
|
||||||
// a queued upload and submits the job. It returns the upload id. A file that fails
|
// a queued upload and submits the job. It returns the upload id. A file that fails
|
||||||
// validation yields a *RejectError and creates NO row — rejection is synchronous
|
// validation yields a *RejectError and creates NO row — rejection is synchronous
|
||||||
// and leaves nothing behind. On success the temp file is handed to the job and the
|
// and leaves nothing behind. On success the temp file is handed to the job and the
|
||||||
// worker deletes it; on every failure path Accept deletes it itself.
|
// worker deletes it; on every failure path Accept deletes it itself.
|
||||||
func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reader) (string, error) {
|
func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reader) (string, error) {
|
||||||
|
// Refuse before spooling when the queue has no room, if the submitter can say so: a full
|
||||||
|
// queue otherwise cost up to the upload cap on disk, an ffprobe run and a failed row for
|
||||||
|
// every refused attempt.
|
||||||
|
if rc, ok := in.sub.(interface{ HasRoom(userID string) bool }); ok && !rc.HasRoom(userID) {
|
||||||
|
return "", &RejectError{Reason: ReasonBusy, Message: busyMessage}
|
||||||
|
}
|
||||||
f, err := os.CreateTemp(in.tmpDir, "upload-*")
|
f, err := os.CreateTemp(in.tmpDir, "upload-*")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", fmt.Errorf("uploads: spool: %w", err)
|
return "", fmt.Errorf("uploads: spool: %w", err)
|
||||||
@@ -114,7 +123,7 @@ func (in *Intake) Accept(ctx context.Context, userID, title string, src io.Reade
|
|||||||
if ferr := in.store.FailUpload(ctx, userID, id, domain.UploadErrBusy); ferr != nil {
|
if ferr := in.store.FailUpload(ctx, userID, id, domain.UploadErrBusy); ferr != nil {
|
||||||
return "", fmt.Errorf("uploads: submit: %w (and failing the row: %v)", err, ferr)
|
return "", fmt.Errorf("uploads: submit: %w (and failing the row: %v)", err, ferr)
|
||||||
}
|
}
|
||||||
return "", &RejectError{Reason: ReasonBusy, Message: "Too many recordings are waiting to be transcribed. Please try again in a few minutes."}
|
return "", &RejectError{Reason: ReasonBusy, Message: busyMessage}
|
||||||
}
|
}
|
||||||
// A row that will never run must not sit queued: that is the stuck spinner.
|
// A row that will never run must not sit queued: that is the stuck spinner.
|
||||||
if ferr := in.store.FailUpload(ctx, userID, id, domain.UploadErrInterrupted); ferr != nil {
|
if ferr := in.store.FailUpload(ctx, userID, id, domain.UploadErrInterrupted); ferr != nil {
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
@@ -29,3 +30,37 @@ func TestAccept_AFullQueueIsABusyRejectionNotAServerErrorAndLeavesNothingBehind(
|
|||||||
assert.Equal(t, domain.UploadErrBusy, r.st.failed["up-1"], "the row must not sit queued: that is the stuck spinner")
|
assert.Equal(t, domain.UploadErrBusy, r.st.failed["up-1"], "the row must not sit queued: that is the stuck spinner")
|
||||||
assert.Empty(t, r.tmpFiles(t), "the audio is deleted")
|
assert.Empty(t, r.tmpFiles(t), "the audio is deleted")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// roomSubmitter is a Submitter that can say, before anything is spooled, that there is no room.
|
||||||
|
type roomSubmitter struct {
|
||||||
|
fakeSubmitter
|
||||||
|
room bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *roomSubmitter) HasRoom(string) bool { return r.room }
|
||||||
|
|
||||||
|
func TestAccept_ANoRoomQueueIsRefusedBeforeAnythingIsSpooledProbedOrStored(t *testing.T) {
|
||||||
|
// Checking only at Submit meant 200 MB was spooled and ffprobe run for every refused
|
||||||
|
// attempt, and a failed row was left behind each time (second adversarial review).
|
||||||
|
r := newRig(t)
|
||||||
|
full := &roomSubmitter{room: false}
|
||||||
|
in := uploads.NewIntake(r.st, r.pr, full, uploads.Limits{MaxBytes: 1024, MaxDuration: 90 * time.Minute}, r.tmp)
|
||||||
|
|
||||||
|
_, err := in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||||
|
|
||||||
|
var rej *uploads.RejectError
|
||||||
|
require.True(t, errors.As(err, &rej))
|
||||||
|
assert.Equal(t, uploads.ReasonBusy, rej.Reason)
|
||||||
|
assert.Empty(t, r.st.created, "no row for an upload that was never accepted")
|
||||||
|
assert.Empty(t, r.pr.paths, "ffprobe was never run on it")
|
||||||
|
assert.Empty(t, r.tmpFiles(t), "nothing was spooled")
|
||||||
|
assert.Empty(t, full.jobs)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAccept_ASubmitterThatCannotSayStillWorksAsBefore(t *testing.T) {
|
||||||
|
r := newRig(t)
|
||||||
|
|
||||||
|
_, err := r.in.Accept(context.Background(), "user-1", "t", strings.NewReader("data"))
|
||||||
|
|
||||||
|
require.NoError(t, err, "a Submitter without HasRoom is accepted: the check at Submit remains")
|
||||||
|
}
|
||||||
|
|||||||
+47
-14
@@ -130,6 +130,24 @@ func NewWorker(s WorkerStore, c Converter, t Transcriber, tmpDir string, timeout
|
|||||||
return w
|
return w
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// HasRoom reports whether a job from this user would be accepted right now. It lets the
|
||||||
|
// intake refuse BEFORE spooling up to the upload cap and running ffprobe; Submit still
|
||||||
|
// enforces the same limits atomically, so a race costs at worst one refused submit.
|
||||||
|
func (w *Worker) HasRoom(userID string) bool {
|
||||||
|
w.mu.Lock()
|
||||||
|
defer w.mu.Unlock()
|
||||||
|
if w.stopped || len(w.queue) >= w.maxQueued {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
mine := 0
|
||||||
|
for _, q := range w.queue {
|
||||||
|
if q.UserID == userID {
|
||||||
|
mine++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return mine < w.maxPerUser
|
||||||
|
}
|
||||||
|
|
||||||
// Submit queues a job. It never blocks. It satisfies Submitter.
|
// Submit queues a job. It never blocks. It satisfies Submitter.
|
||||||
func (w *Worker) Submit(j Job) error {
|
func (w *Worker) Submit(j Job) error {
|
||||||
w.mu.Lock()
|
w.mu.Lock()
|
||||||
@@ -275,11 +293,18 @@ func (w *Worker) Process(ctx context.Context, job Job) {
|
|||||||
abort(domain.UploadErrTranscriptionFailed)
|
abort(domain.UploadErrTranscriptionFailed)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if kind, bad := w.audioLengthProblem(wav, job.Duration); bad {
|
got, tooLong := w.audioLengthProblem(wav)
|
||||||
abort(kind)
|
if tooLong {
|
||||||
log.Error("upload: converted audio length rejected", "kind", string(kind))
|
abort(domain.UploadErrTooLong)
|
||||||
|
log.Error("upload: converted audio is over the limit", "audio_s", int(got.Seconds()), "limit_s", int(w.maxDuration.Seconds()))
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
if w.maxDuration > 0 && job.Duration > 0 {
|
||||||
|
if note := shortfallNote(got, job.Duration); note != "" {
|
||||||
|
log.Warn("upload: audio is much shorter than the file claimed", "audio_s", int(got.Seconds()), "claimed_s", int(job.Duration.Seconds()))
|
||||||
|
tr.Annotate("audio", note)
|
||||||
|
}
|
||||||
|
}
|
||||||
tr.Done("audio")
|
tr.Done("audio")
|
||||||
|
|
||||||
if !planned {
|
if !planned {
|
||||||
@@ -343,21 +368,29 @@ const (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// audioLengthProblem measures the converted WAV (16 kHz mono 16-bit PCM is 32,000 bytes a
|
// audioLengthProblem measures the converted WAV (16 kHz mono 16-bit PCM is 32,000 bytes a
|
||||||
// second after a 44-byte header) against the limit and against what the file claimed.
|
// second after a 44-byte header) against the limit. It returns the real length and whether
|
||||||
func (w *Worker) audioLengthProblem(wav string, claimed time.Duration) (domain.UploadErrorKind, bool) {
|
// that is over the limit by more than the tolerance.
|
||||||
|
func (w *Worker) audioLengthProblem(wav string) (got time.Duration, tooLong bool) {
|
||||||
if w.maxDuration <= 0 {
|
if w.maxDuration <= 0 {
|
||||||
return "", false
|
return 0, false
|
||||||
}
|
}
|
||||||
info, err := os.Stat(wav)
|
info, err := os.Stat(wav)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", false
|
return 0, false
|
||||||
}
|
}
|
||||||
got := time.Duration(float64(max(info.Size()-44, 0)) / 32000 * float64(time.Second))
|
got = time.Duration(float64(max(info.Size()-44, 0)) / 32000 * float64(time.Second))
|
||||||
switch {
|
return got, got > w.maxDuration+tooLongSlack
|
||||||
case got > w.maxDuration+tooLongSlack:
|
}
|
||||||
return domain.UploadErrTooLong, true
|
|
||||||
case float64(got) < incompleteBelow*float64(claimed) && claimed-got > incompleteShortfall:
|
// shortfallNote is a remark for the audio step when the converted audio is much shorter
|
||||||
return domain.UploadErrIncomplete, true
|
// than the file claimed, or "" when it is not. It is a NOTE and never a failure: a
|
||||||
|
// container header can over-estimate a complete file (a variable-bitrate MP3 without a Xing
|
||||||
|
// header claimed 79 minutes for 20 real ones), so refusing would turn away real
|
||||||
|
// recordings, while saying nothing about a truncated one would hide a partial transcript.
|
||||||
|
func shortfallNote(got, claimed time.Duration) string {
|
||||||
|
if float64(got) >= incompleteBelow*float64(claimed) || claimed-got <= incompleteShortfall {
|
||||||
|
return ""
|
||||||
}
|
}
|
||||||
return "", false
|
return fmt.Sprintf("The audio is %s long; the file claimed %s. If the recording was cut off, upload the complete one.",
|
||||||
|
FormatDuration(got), FormatDuration(claimed))
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -69,15 +69,84 @@ func TestWorkerDuration_JustOverTheLimitWithinTheSlackStillTranscribes(t *testin
|
|||||||
assert.Contains(t, completed, "a", "ffmpeg rounds: 20 s over a 90-minute limit is not a lie")
|
assert.Contains(t, completed, "a", "ffmpeg rounds: 20 s over a 90-minute limit is not a lie")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestWorkerDuration_AFileThatEndsFarEarlierThanItClaimedIsRefusedNotHalfTranscribed(t *testing.T) {
|
func audioStepNote(r *wrig) string {
|
||||||
|
for _, st := range r.st.lastSteps() {
|
||||||
|
if st.Key == "audio" {
|
||||||
|
return st.Note
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerDuration_AFileThatEndsFarEarlierThanItClaimedIsStillTranscribedAndSaysSo(t *testing.T) {
|
||||||
|
// A container header can over-estimate a complete file (VBR MP3 without a Xing header
|
||||||
|
// claimed 79 minutes for 20 real ones). Refusing it was wrong; staying silent about a
|
||||||
|
// truncated one is wrong too. So it is transcribed, and the audio step carries a note.
|
||||||
r := newWRig(t, time.Minute)
|
r := newWRig(t, time.Minute)
|
||||||
w := durationWorker(t, r, 20*60, 90*time.Minute)
|
w := durationWorker(t, r, 20*60, 90*time.Minute)
|
||||||
|
|
||||||
w.Process(context.Background(), jobWith(r, t, 60*time.Minute))
|
w.Process(context.Background(), jobWith(r, t, 79*time.Minute))
|
||||||
|
|
||||||
_, failed := r.st.snapshot()
|
completed, failed := r.st.snapshot()
|
||||||
assert.Equal(t, domain.UploadErrIncomplete, failed["a"], "a recording truncated at a third must not return a third of a transcript with no warning")
|
assert.Empty(t, failed, "a complete recording with a bad header estimate must not be refused")
|
||||||
assert.Empty(t, r.tr.paths)
|
assert.Contains(t, completed, "a")
|
||||||
|
note := audioStepNote(r)
|
||||||
|
assert.Contains(t, note, "20 min", "what was really transcribed")
|
||||||
|
assert.Contains(t, note, "1 h 19 min", "what the file claimed")
|
||||||
|
assert.Contains(t, note, "cut off", "and what to do if that is not a surprise")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerDuration_TheShortfallNoteIsPinnedAtEightyPercentAndTwoMinutes(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
claimed time.Duration
|
||||||
|
wav float64
|
||||||
|
noted bool
|
||||||
|
}{
|
||||||
|
{"79 percent of an hour", 60 * time.Minute, 0.79 * 3600, true},
|
||||||
|
{"81 percent of an hour", 60 * time.Minute, 0.81 * 3600, false},
|
||||||
|
{"exactly two minutes short and under 80 percent", 6 * time.Minute, 4 * 60, false},
|
||||||
|
{"two minutes and a second short and under 80 percent", 6 * time.Minute, 4*60 - 1, true},
|
||||||
|
{"unknown claimed length", 0, 20 * 60, false},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
w := durationWorker(t, r, tt.wav, 90*time.Minute)
|
||||||
|
|
||||||
|
w.Process(context.Background(), jobWith(r, t, tt.claimed))
|
||||||
|
|
||||||
|
assert.Equal(t, tt.noted, audioStepNote(r) != "")
|
||||||
|
completed, _ := r.st.snapshot()
|
||||||
|
assert.Contains(t, completed, "a", "a note never stops the job")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWorkerDuration_TheTooLongToleranceIsExactlyThirtySeconds(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
wav float64
|
||||||
|
fail bool
|
||||||
|
}{
|
||||||
|
{"at the limit plus 30 s", 90*60 + 30, false},
|
||||||
|
{"at the limit plus 31 s", 90*60 + 31, true},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
r := newWRig(t, time.Minute)
|
||||||
|
w := durationWorker(t, r, tt.wav, 90*time.Minute)
|
||||||
|
|
||||||
|
w.Process(context.Background(), jobWith(r, t, 90*time.Minute))
|
||||||
|
|
||||||
|
_, failed := r.st.snapshot()
|
||||||
|
if tt.fail {
|
||||||
|
assert.Equal(t, domain.UploadErrTooLong, failed["a"])
|
||||||
|
} else {
|
||||||
|
assert.Empty(t, failed)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestWorkerDuration_AnEstimateThatIsMerelyALittleOffIsNotTruncation(t *testing.T) {
|
func TestWorkerDuration_AnEstimateThatIsMerelyALittleOffIsNotTruncation(t *testing.T) {
|
||||||
|
|||||||
@@ -78,3 +78,15 @@ func TestWorkerSubmit_AJobThatHasLeftTheQueueFreesItsPlace(t *testing.T) {
|
|||||||
|
|
||||||
assert.NoError(t, w.Submit(r.spool(t, "second")), "the cap counts jobs waiting, not jobs ever submitted")
|
assert.NoError(t, w.Submit(r.spool(t, "second")), "the cap counts jobs waiting, not jobs ever submitted")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestWorkerHasRoom_SaysWhetherASubmitWouldBeAccepted(t *testing.T) {
|
||||||
|
w := queueWorker(t, 3, 2)
|
||||||
|
assert.True(t, w.HasRoom("alice"))
|
||||||
|
require.NoError(t, submit(w, "alice", 1))
|
||||||
|
require.NoError(t, submit(w, "alice", 2))
|
||||||
|
|
||||||
|
assert.False(t, w.HasRoom("alice"), "alice is at her share")
|
||||||
|
assert.True(t, w.HasRoom("bob"), "bob is not")
|
||||||
|
require.NoError(t, submit(w, "bob", 1))
|
||||||
|
assert.False(t, w.HasRoom("carol"), "the queue as a whole is at its cap")
|
||||||
|
}
|
||||||
|
|||||||
+14
-3
@@ -152,6 +152,10 @@ func (a *App) uploadFailure(w http.ResponseWriter, r *http.Request, userID strin
|
|||||||
a.logger().Info("upload rejected", "reason", string(uploads.ReasonTooLarge))
|
a.logger().Info("upload rejected", "reason", string(uploads.ReasonTooLarge))
|
||||||
a.renderUploadsPage(w, r, userID, http.StatusRequestEntityTooLarge,
|
a.renderUploadsPage(w, r, userID, http.StatusRequestEntityTooLarge,
|
||||||
"This file is larger than the upload limit.")
|
"This file is larger than the upload limit.")
|
||||||
|
case errors.Is(err, io.ErrUnexpectedEOF):
|
||||||
|
// The sender hung up mid-file: theirs to retry, not a server fault.
|
||||||
|
a.logger().Info("upload rejected", "reason", "truncated")
|
||||||
|
a.renderUploadsPage(w, r, userID, http.StatusBadRequest, "The upload was cut off before it finished. Please try again.")
|
||||||
default:
|
default:
|
||||||
// Not err: it can carry the file's name, a spool path or a database value.
|
// Not err: it can carry the file's name, a spool path or a database value.
|
||||||
a.logger().Error("upload failed", "kind", errkind.Of(err))
|
a.logger().Error("upload failed", "kind", errkind.Of(err))
|
||||||
@@ -218,6 +222,12 @@ func titleFromFilename(name string) string {
|
|||||||
// uuidShape is the form of an upload id. Anything else cannot be one, so it is a 404.
|
// uuidShape is the form of an upload id. Anything else cannot be one, so it is a 404.
|
||||||
var uuidShape = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
|
var uuidShape = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
|
||||||
|
|
||||||
|
// isBidiControl reports the invisible characters that change text direction. Emoji joiners
|
||||||
|
// (ZWJ, ZWNJ) and variation selectors are not among them and are kept.
|
||||||
|
func isBidiControl(r rune) bool {
|
||||||
|
return r >= 0x202A && r <= 0x202E || r >= 0x2066 && r <= 0x2069 || r == 0x200E || r == 0x200F || r == 0x061C
|
||||||
|
}
|
||||||
|
|
||||||
// cleanTitle makes a title safe to store and show: invalid UTF-8 dropped, control
|
// cleanTitle makes a title safe to store and show: invalid UTF-8 dropped, control
|
||||||
// characters (NUL, newlines, escapes) turned into spaces, runs of space collapsed, trimmed
|
// characters (NUL, newlines, escapes) turned into spaces, runs of space collapsed, trimmed
|
||||||
// and clipped. A title is cosmetic; bad bytes in it must never fail an upload, and Postgres
|
// and clipped. A title is cosmetic; bad bytes in it must never fail an upload, and Postgres
|
||||||
@@ -225,7 +235,10 @@ var uuidShape = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4
|
|||||||
func cleanTitle(s string) string {
|
func cleanTitle(s string) string {
|
||||||
s = strings.ToValidUTF8(s, "")
|
s = strings.ToValidUTF8(s, "")
|
||||||
s = strings.Map(func(r rune) rune {
|
s = strings.Map(func(r rune) rune {
|
||||||
if unicode.IsControl(r) {
|
switch {
|
||||||
|
case isBidiControl(r):
|
||||||
|
return -1 // dropped: U+202E would reverse the rest of the title on its owner's page
|
||||||
|
case unicode.IsControl(r):
|
||||||
return ' '
|
return ' '
|
||||||
}
|
}
|
||||||
return r
|
return r
|
||||||
@@ -250,8 +263,6 @@ func uploadFailureMessage(k domain.UploadErrorKind) string {
|
|||||||
return "Too many recordings were waiting to be transcribed when this one was uploaded. Please upload it again in a few minutes."
|
return "Too many recordings were waiting to be transcribed when this one was uploaded. Please upload it again in a few minutes."
|
||||||
case domain.UploadErrTooLong:
|
case domain.UploadErrTooLong:
|
||||||
return "This recording is longer than the limit once its audio is read (the file's header said otherwise). Please upload a shorter recording."
|
return "This recording is longer than the limit once its audio is read (the file's header said otherwise). Please upload a shorter recording."
|
||||||
case domain.UploadErrIncomplete:
|
|
||||||
return "This recording ends much earlier than the file says it should; it may be cut off or damaged. Please upload the complete recording."
|
|
||||||
case domain.UploadErrTranscriptionFailed:
|
case domain.UploadErrTranscriptionFailed:
|
||||||
return "Transcription failed on every available service. Please upload it again later."
|
return "Transcription failed on every available service. Please upload it again later."
|
||||||
default:
|
default:
|
||||||
|
|||||||
@@ -218,6 +218,9 @@ templ progressSteps(steps []domain.ProgressStep, now time.Time, live bool) {
|
|||||||
if t := stepTime(s, now); t != "" {
|
if t := stepTime(s, now); t != "" {
|
||||||
<span class="step-time">{ t }</span>
|
<span class="step-time">{ t }</span>
|
||||||
}
|
}
|
||||||
|
if s.Note != "" {
|
||||||
|
<span class="step-note">{ s.Note }</span>
|
||||||
|
}
|
||||||
if s.State == domain.StepRunning && s.Total > 0 {
|
if s.State == domain.StepRunning && s.Total > 0 {
|
||||||
<progress class="step-bar" value={ strconv.Itoa(s.Done) } max={ strconv.Itoa(s.Total) }></progress>
|
<progress class="step-bar" value={ strconv.Itoa(s.Done) } max={ strconv.Itoa(s.Total) }></progress>
|
||||||
<span class="step-count">{ strconv.Itoa(s.Done) } of { strconv.Itoa(s.Total) }</span>
|
<span class="step-count">{ strconv.Itoa(s.Done) } of { strconv.Itoa(s.Total) }</span>
|
||||||
|
|||||||
@@ -1,7 +1,9 @@
|
|||||||
package web_test
|
package web_test
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
|
"io"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -60,3 +62,48 @@ func TestUploadPost_AMalformedMultipartHeaderIsA400AndItsTextIsNotLogged(t *test
|
|||||||
assert.Empty(t, sub.jobs)
|
assert.Empty(t, sub.jobs)
|
||||||
assert.NotContains(t, buf.String(), "zebrakaka", "a file name is private content")
|
assert.NotContains(t, buf.String(), "zebrakaka", "a file name is private content")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestUploadPost_ABodyCutOffInsideTheFilePartIsA400NotA500(t *testing.T) {
|
||||||
|
app, sub := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
req := uploadReq(t, "t", "a.m4a", bytes.Repeat([]byte("a"), 4096))
|
||||||
|
raw, err := io.ReadAll(req.Body)
|
||||||
|
require.NoError(t, err)
|
||||||
|
req.Body = io.NopCloser(bytes.NewReader(raw[:len(raw)-2000])) // the sender hung up mid-file
|
||||||
|
buf := &lockedBuf{}
|
||||||
|
app.Log = slog.New(slog.NewTextHandler(buf, nil))
|
||||||
|
|
||||||
|
rec := do(t, app, req)
|
||||||
|
|
||||||
|
assert.Equal(t, http.StatusBadRequest, rec.Code, "a connection that drops mid-upload is the sender's to retry, not a server fault")
|
||||||
|
assert.Empty(t, sub.jobs)
|
||||||
|
assert.NotContains(t, buf.String(), "level=ERROR")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadPost_BidirectionalControlCharactersAreStrippedFromATitle(t *testing.T) {
|
||||||
|
// U+202E reverses the text that follows it, so a title could render reversed on its
|
||||||
|
// owner's own page (second adversarial review).
|
||||||
|
app, sub := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
|
||||||
|
rec := do(t, app, uploadReq(t, "Sprint \u202Ereview \u2066x\u2069 \u200Fok", "a.m4a", []byte("x")))
|
||||||
|
|
||||||
|
require.Equal(t, http.StatusSeeOther, rec.Code)
|
||||||
|
var title string
|
||||||
|
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT title FROM uploaded_transcripts WHERE id = $1`, sub.jobs[0].UploadID).Scan(&title))
|
||||||
|
for _, r := range []string{"\u202E", "\u2066", "\u2069", "\u200F"} {
|
||||||
|
assert.NotContains(t, title, r)
|
||||||
|
}
|
||||||
|
assert.Contains(t, title, "Sprint")
|
||||||
|
assert.Contains(t, title, "review")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUploadPost_SwedishLettersAndEmojiSurviveTheTitleCleaning(t *testing.T) {
|
||||||
|
app, sub := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
|
||||||
|
// Includes a ZWJ sequence (woman + ZWJ + laptop) and a ZWNJ: joiners are not direction controls.
|
||||||
|
rec := do(t, app, uploadReq(t, "Möte åäö 🎙️ 👩\u200d💻 a\u200cb", "a.m4a", []byte("x")))
|
||||||
|
|
||||||
|
require.Equal(t, http.StatusSeeOther, rec.Code)
|
||||||
|
var title string
|
||||||
|
require.NoError(t, rawPool(t).QueryRow(context.Background(), `SELECT title FROM uploaded_transcripts WHERE id = $1`, sub.jobs[0].UploadID).Scan(&title))
|
||||||
|
assert.Equal(t, "Möte åäö 🎙️ 👩\u200d💻 a\u200cb", title, "joiners and variation selectors belong to the text")
|
||||||
|
}
|
||||||
|
|||||||
@@ -465,7 +465,6 @@ func TestUploadStatus_ALengthFailureIsExplainedInWordsAndNotAsAGenericError(t *t
|
|||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
tests := []struct{ kind, want string }{
|
tests := []struct{ kind, want string }{
|
||||||
{"audio_too_long", "longer than the limit"},
|
{"audio_too_long", "longer than the limit"},
|
||||||
{"audio_incomplete", "ends much earlier than"},
|
|
||||||
}
|
}
|
||||||
for _, tt := range tests {
|
for _, tt := range tests {
|
||||||
t.Run(tt.kind, func(t *testing.T) {
|
t.Run(tt.kind, func(t *testing.T) {
|
||||||
|
|||||||
@@ -251,3 +251,18 @@ func TestUploadProgress_TheFullPageCarriesTheVerySameProgressListAsTheFragment(t
|
|||||||
assert.Equal(t, stepsBlock(t, fragment), stepsBlock(t, full),
|
assert.Equal(t, stepsBlock(t, fragment), stepsBlock(t, full),
|
||||||
"the page a user opens and the fragment HTMX swaps in must show the same list, or the display jumps on the first poll")
|
"the page a user opens and the fragment HTMX swaps in must show the same list, or the display jumps on the first poll")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestUploadProgress_AStepNoteIsShownUnderItsStepAndEscaped(t *testing.T) {
|
||||||
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
|
fixedClock(app)
|
||||||
|
id := seedUpload(t, userID, "m", "transcribing", "")
|
||||||
|
steps := twoEngineRun()
|
||||||
|
steps[1].Note = "The audio is 20 min long; the file claimed 1 h 19 min. <b>cut off</b>?"
|
||||||
|
seedProgress(t, id, steps, progressNow.Add(-10*time.Second), progressNow.Add(-4*time.Second))
|
||||||
|
|
||||||
|
b := statusBody(t, app, id)
|
||||||
|
|
||||||
|
assert.Contains(t, b, "The audio is 20 min long")
|
||||||
|
assert.Contains(t, b, "step-note")
|
||||||
|
assert.NotContains(t, b, "<b>cut off</b>", "a note is words, never markup")
|
||||||
|
}
|
||||||
|
|||||||
@@ -47,6 +47,9 @@ func TestRenderUploadProgressStatesForReview(t *testing.T) {
|
|||||||
step("save", "Saving the transcript", domain.StepDone, time.Minute, time.Second, 0, 0),
|
step("save", "Saving the transcript", domain.StepDone, time.Minute, time.Second, 0, 0),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
noted := twoEngineRun()
|
||||||
|
noted[1].Note = "The audio is 20 min long; the file claimed 1 h 19 min. If the recording was cut off, upload the complete one."
|
||||||
|
|
||||||
cases := []struct {
|
cases := []struct {
|
||||||
name, status, content string
|
name, status, content string
|
||||||
steps []domain.ProgressStep
|
steps []domain.ProgressStep
|
||||||
@@ -58,6 +61,7 @@ func TestRenderUploadProgressStatesForReview(t *testing.T) {
|
|||||||
{"lost", "transcribing", "", twoEngineRun(), 3 * time.Minute, 3 * time.Minute},
|
{"lost", "transcribing", "", twoEngineRun(), 3 * time.Minute, 3 * time.Minute},
|
||||||
{"failed", "failed", "transcription_failed", failed, 2 * time.Minute, 2 * time.Minute},
|
{"failed", "failed", "transcription_failed", failed, 2 * time.Minute, 2 * time.Minute},
|
||||||
{"done", "done", "Synthetic transcript text for the review.", done, time.Minute, time.Minute},
|
{"done", "done", "Synthetic transcript text for the review.", done, time.Minute, time.Minute},
|
||||||
|
{"noted", "transcribing", "", noted, 10 * time.Second, 4 * time.Second},
|
||||||
{"queued", "queued", "", nil, 0, 0},
|
{"queued", "queued", "", nil, 0, 0},
|
||||||
}
|
}
|
||||||
for _, c := range cases {
|
for _, c := range cases {
|
||||||
|
|||||||
@@ -62,11 +62,11 @@ func TestUploadStatus_ASpotShowsTheWordsJustBeforeIt(t *testing.T) {
|
|||||||
app, _ := newUploadApp(t, audioOK, 200<<20)
|
app, _ := newUploadApp(t, audioOK, 200<<20)
|
||||||
id := seedUpload(t, userID, "m", "done", "Synthetic transcript.")
|
id := seedUpload(t, userID, "m", "done", "Synthetic transcript.")
|
||||||
setReconciliation(t, id, `{"status":"reconciled","base":"a","alt":"b","reconciler":"c","disputes":1,"changed":1,
|
setReconciliation(t, id, `{"status":"reconciled","base":"a","alt":"b","reconciler":"c","disputes":1,"changed":1,
|
||||||
"contested":[{"seconds":2640,"base":"vanligt","alt":"KORT","used":"alt","unsure":false,"before":"vi skickar in ett"}]}`)
|
"contested":[{"seconds":2640,"base":"vanligt","alt":"KORT","used":"alt","unsure":false,"before":"grannen målar ett"}]}`)
|
||||||
|
|
||||||
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
b := body(t, do(t, app, get("/uploads/"+id+"/status")))
|
||||||
|
|
||||||
assert.Contains(t, b, "vi skickar in ett", "the words before the spot are how a reader finds it in the audio")
|
assert.Contains(t, b, "grannen målar ett", "the words before the spot are how a reader finds it in the audio")
|
||||||
assert.Contains(t, b, "vanligt")
|
assert.Contains(t, b, "vanligt")
|
||||||
assert.Contains(t, b, "KORT")
|
assert.Contains(t, b, "KORT")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1034,70 +1034,89 @@ func progressSteps(steps []domain.ProgressStep, now time.Time, live bool) templ.
|
|||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if s.State == domain.StepRunning && s.Total > 0 {
|
if s.Note != "" {
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 86, "<progress class=\"step-bar\" value=\"")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 86, "<span class=\"step-note\">")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var49 string
|
var templ_7745c5c3_Var49 string
|
||||||
templ_7745c5c3_Var49, templ_7745c5c3_Err = templ.ResolveAttributeValue(strconv.Itoa(s.Done))
|
templ_7745c5c3_Var49, templ_7745c5c3_Err = templ.JoinStringErrs(s.Note)
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 222, Col: 60}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 222, Col: 37}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var49)
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var49))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 87, "\" max=\"")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 87, "</span> ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if s.State == domain.StepRunning && s.Total > 0 {
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 88, "<progress class=\"step-bar\" value=\"")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var50 string
|
var templ_7745c5c3_Var50 string
|
||||||
templ_7745c5c3_Var50, templ_7745c5c3_Err = templ.ResolveAttributeValue(strconv.Itoa(s.Total))
|
templ_7745c5c3_Var50, templ_7745c5c3_Err = templ.ResolveAttributeValue(strconv.Itoa(s.Done))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 222, Col: 90}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 225, Col: 60}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var50)
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var50)
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 88, "\"></progress> <span class=\"step-count\">")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 89, "\" max=\"")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var51 string
|
var templ_7745c5c3_Var51 string
|
||||||
templ_7745c5c3_Var51, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(s.Done))
|
templ_7745c5c3_Var51, templ_7745c5c3_Err = templ.ResolveAttributeValue(strconv.Itoa(s.Total))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 223, Col: 52}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 225, Col: 90}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var51))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ_7745c5c3_Var51)
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 89, " of ")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 90, "\"></progress> <span class=\"step-count\">")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var52 string
|
var templ_7745c5c3_Var52 string
|
||||||
templ_7745c5c3_Var52, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(s.Total))
|
templ_7745c5c3_Var52, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(s.Done))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 223, Col: 81}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 226, Col: 52}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var52))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var52))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 90, "</span>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 91, " of ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
var templ_7745c5c3_Var53 string
|
||||||
|
templ_7745c5c3_Var53, templ_7745c5c3_Err = templ.JoinStringErrs(strconv.Itoa(s.Total))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 226, Col: 81}
|
||||||
|
}
|
||||||
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var53))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 92, "</span>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 91, "</li>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 93, "</li>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 92, "</ol>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 94, "</ol>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
@@ -1123,45 +1142,45 @@ func healthNote(h jobHealth, silent time.Duration) templ.Component {
|
|||||||
}()
|
}()
|
||||||
}
|
}
|
||||||
ctx = templ.InitializeContext(ctx)
|
ctx = templ.InitializeContext(ctx)
|
||||||
templ_7745c5c3_Var53 := templ.GetChildren(ctx)
|
templ_7745c5c3_Var54 := templ.GetChildren(ctx)
|
||||||
if templ_7745c5c3_Var53 == nil {
|
if templ_7745c5c3_Var54 == nil {
|
||||||
templ_7745c5c3_Var53 = templ.NopComponent
|
templ_7745c5c3_Var54 = templ.NopComponent
|
||||||
}
|
}
|
||||||
ctx = templ.ClearChildren(ctx)
|
ctx = templ.ClearChildren(ctx)
|
||||||
switch h {
|
switch h {
|
||||||
case healthLost:
|
case healthLost:
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 93, "<p class=\"step-health step-health-lost\" role=\"alert\">The service stopped reporting ")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 95, "<p class=\"step-health step-health-lost\" role=\"alert\">The service stopped reporting ")
|
||||||
if templ_7745c5c3_Err != nil {
|
|
||||||
return templ_7745c5c3_Err
|
|
||||||
}
|
|
||||||
var templ_7745c5c3_Var54 string
|
|
||||||
templ_7745c5c3_Var54, templ_7745c5c3_Err = templ.JoinStringErrs(minutesAgo(silent))
|
|
||||||
if templ_7745c5c3_Err != nil {
|
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 236, Col: 54}
|
|
||||||
}
|
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var54))
|
|
||||||
if templ_7745c5c3_Err != nil {
|
|
||||||
return templ_7745c5c3_Err
|
|
||||||
}
|
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 94, " ago. It may have restarted or crashed. This page keeps checking; if nothing changes, upload the recording again.</p>")
|
|
||||||
if templ_7745c5c3_Err != nil {
|
|
||||||
return templ_7745c5c3_Err
|
|
||||||
}
|
|
||||||
case healthQuiet:
|
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 95, "<p class=\"step-health step-health-quiet\">Nothing new for ")
|
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
var templ_7745c5c3_Var55 string
|
var templ_7745c5c3_Var55 string
|
||||||
templ_7745c5c3_Var55, templ_7745c5c3_Err = templ.JoinStringErrs(minutesAgo(silent))
|
templ_7745c5c3_Var55, templ_7745c5c3_Err = templ.JoinStringErrs(minutesAgo(silent))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 241, Col: 40}
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 239, Col: 54}
|
||||||
}
|
}
|
||||||
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var55))
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var55))
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 96, ": the model may be slow. The service is still running, and the job gives up on its own if it takes too long.</p>")
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 96, " ago. It may have restarted or crashed. This page keeps checking; if nothing changes, upload the recording again.</p>")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
case healthQuiet:
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 97, "<p class=\"step-health step-health-quiet\">Nothing new for ")
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
var templ_7745c5c3_Var56 string
|
||||||
|
templ_7745c5c3_Var56, templ_7745c5c3_Err = templ.JoinStringErrs(minutesAgo(silent))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ.Error{Err: templ_7745c5c3_Err, FileName: `internal/web/uploads.templ`, Line: 244, Col: 40}
|
||||||
|
}
|
||||||
|
_, templ_7745c5c3_Err = templ_7745c5c3_Buffer.WriteString(templ.EscapeString(templ_7745c5c3_Var56))
|
||||||
|
if templ_7745c5c3_Err != nil {
|
||||||
|
return templ_7745c5c3_Err
|
||||||
|
}
|
||||||
|
templ_7745c5c3_Err = templruntime.WriteString(templ_7745c5c3_Buffer, 98, ": the model may be slow. The service is still running, and the job gives up on its own if it takes too long.</p>")
|
||||||
if templ_7745c5c3_Err != nil {
|
if templ_7745c5c3_Err != nil {
|
||||||
return templ_7745c5c3_Err
|
return templ_7745c5c3_Err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -780,6 +780,7 @@ a.btn, a.btn:visited { color: var(--accent-fg); }
|
|||||||
.step-skipped { opacity: .7; }
|
.step-skipped { opacity: .7; }
|
||||||
.step-pending { opacity: .75; }
|
.step-pending { opacity: .75; }
|
||||||
.step-bar { flex: none; width: calc(100% - 1.6rem); margin: 0 0 0 1.6rem; height: .5rem; accent-color: var(--accent); }
|
.step-bar { flex: none; width: calc(100% - 1.6rem); margin: 0 0 0 1.6rem; height: .5rem; accent-color: var(--accent); }
|
||||||
|
.step-note { flex: none; width: calc(100% - 1.6rem); margin-left: 1.6rem; font-weight: 400; color: var(--muted); font-family: system-ui, -apple-system, sans-serif; font-size: .82rem; }
|
||||||
.step-count { flex: none; width: calc(100% - 1.6rem); margin-left: 1.6rem; font-weight: 400; color: var(--muted); }
|
.step-count { flex: none; width: calc(100% - 1.6rem); margin-left: 1.6rem; font-weight: 400; color: var(--muted); }
|
||||||
.step-running .step-mark-warn { color: var(--err-fg); }
|
.step-running .step-mark-warn { color: var(--err-fg); }
|
||||||
.spinner { display: inline-block; vertical-align: -.1em; width: .8rem; height: .8rem; box-sizing: border-box; border: 2px solid var(--line); border-top-color: var(--accent); border-radius: 50%; animation: tapir-spin 1s linear infinite; }
|
.spinner { display: inline-block; vertical-align: -.1em; width: .8rem; height: .8rem; box-sizing: border-box; border: 2px solid var(--line); border-top-color: var(--accent); border-radius: 50%; animation: tapir-spin 1s linear infinite; }
|
||||||
|
|||||||
@@ -135,26 +135,26 @@ var scenarioCoverage = map[string]string{
|
|||||||
"A single-engine transcript claims no second opinion": "TestUploadStatus_ASingleEngineTranscriptDoesNotClaimASecondOpinion",
|
"A single-engine transcript claims no second opinion": "TestUploadStatus_ASingleEngineTranscriptDoesNotClaimASecondOpinion",
|
||||||
|
|
||||||
// audio_upload.feature, ADR-035
|
// audio_upload.feature, ADR-035
|
||||||
"I can see which phase my upload is in": "TestUploadProgress_ARunningJobShowsEveryPhaseInOrderWithTheRunningOneMarked",
|
"I can see which phase my upload is in": "TestUploadProgress_ARunningJobShowsEveryPhaseInOrderWithTheRunningOneMarked",
|
||||||
"A bar appears only where the count is real": "TestUploadProgress_OnlyACountedPhaseGetsABarAndItShowsRealCounts",
|
"A bar appears only where the count is real": "TestUploadProgress_OnlyACountedPhaseGetsABarAndItShowsRealCounts",
|
||||||
"A slow engine call is not mistaken for a dead service": "TestUploadProgress_ALongEngineCallWithAFreshHeartbeatIsStillJustRunning",
|
"A slow engine call is not mistaken for a dead service": "TestUploadProgress_ALongEngineCallWithAFreshHeartbeatIsStillJustRunning",
|
||||||
"A service that stopped reporting is called lost": "TestUploadProgress_AHeartbeatThatStoppedMeansTheServiceIsLostAndTheJobIsNotCalledRunning",
|
"A service that stopped reporting is called lost": "TestUploadProgress_AHeartbeatThatStoppedMeansTheServiceIsLostAndTheJobIsNotCalledRunning",
|
||||||
"A model that stops advancing is called quiet": "TestUploadProgress_ACountedPhaseThatStopsAdvancingGoesQuietNotLost",
|
"A model that stops advancing is called quiet": "TestUploadProgress_ACountedPhaseThatStopsAdvancingGoesQuietNotLost",
|
||||||
"A failed job shows where it stopped": "TestUploadProgress_AFailedJobShowsWhichPhaseFailedAndWhatNeverStarted",
|
"A failed job shows where it stopped": "TestUploadProgress_AFailedJobShowsWhichPhaseFailedAndWhatNeverStarted",
|
||||||
"A finished job keeps its record of what was done": "TestUploadProgress_ADoneJobKeepsItsRecordOfWhatWasDoneAndHowLongEachTook",
|
"A finished job keeps its record of what was done": "TestUploadProgress_ADoneJobKeepsItsRecordOfWhatWasDoneAndHowLongEachTook",
|
||||||
"The heartbeat runs on its own while an engine call is stuck": "TestWorkerProgress_TheHeartbeatBeatsWhileAnEngineCallIsStuckAndStopsWhenTheJobEnds",
|
"The heartbeat runs on its own while an engine call is stuck": "TestWorkerProgress_TheHeartbeatBeatsWhileAnEngineCallIsStuckAndStopsWhenTheJobEnds",
|
||||||
"The log names every phase and never the transcript": "TestWorkerProgress_OnASuccessfulJobTheLogNamesEveryPhaseInOrderAndNeverTheTranscriptOrThePath",
|
"The log names every phase and never the transcript": "TestWorkerProgress_OnASuccessfulJobTheLogNamesEveryPhaseInOrderAndNeverTheTranscriptOrThePath",
|
||||||
"Every judged spot is stored but only the ones to check are listed": "TestUploadStatus_OnlyChangedAndUncertainSpotsAreListedNotTheKeptOnes",
|
"Every judged spot is stored but only the ones to check are listed": "TestUploadStatus_OnlyChangedAndUncertainSpotsAreListedNotTheKeptOnes",
|
||||||
"Chat about a recording is offered with local models only": "TestUploadChatPage_DoesNotOfferACloudModelEvenThoughTheServiceKnowsOne",
|
"Chat about a recording is offered with local models only": "TestUploadChatPage_DoesNotOfferACloudModelEvenThoughTheServiceKnowsOne",
|
||||||
"A forged cloud model is never sent the transcript": "TestUploadChatPost_AForgedCloudModelIsReplacedByALocalOneBeforeTheTranscriptIsSent",
|
"A forged cloud model is never sent the transcript": "TestUploadChatPost_AForgedCloudModelIsReplacedByALocalOneBeforeTheTranscriptIsSent",
|
||||||
"A wedged engine does not use up the whole job": "TestChain_AHungPrimaryReachesTheFallbackWhenItHasItsOwnDeadline",
|
"A wedged engine does not use up the whole job": "TestChain_AHungPrimaryReachesTheFallbackWhenItHasItsOwnDeadline",
|
||||||
"A file that lies about its length is refused": "TestWorkerDuration_AConvertedLengthOverTheLimitFailsTheJobBeforeAnyEngineSeesIt",
|
"A file that lies about its length is refused": "TestWorkerDuration_AConvertedLengthOverTheLimitFailsTheJobBeforeAnyEngineSeesIt",
|
||||||
"A truncated recording is refused, not half transcribed": "TestWorkerDuration_AFileThatEndsFarEarlierThanItClaimedIsRefusedNotHalfTranscribed",
|
"A recording much shorter than its file claimed is transcribed and noted": "TestWorkerDuration_AFileThatEndsFarEarlierThanItClaimedIsStillTranscribedAndSaysSo",
|
||||||
"A full queue is a retry, not an error": "TestAccept_AFullQueueIsABusyRejectionNotAServerErrorAndLeavesNothingBehind",
|
"A full queue is a retry, not an error": "TestAccept_AFullQueueIsABusyRejectionNotAServerErrorAndLeavesNothingBehind",
|
||||||
"One user cannot fill the queue for everyone": "TestWorkerSubmit_OneUserCannotFillTheQueueForEveryoneElse",
|
"One user cannot fill the queue for everyone": "TestWorkerSubmit_OneUserCannotFillTheQueueForEveryoneElse",
|
||||||
"Errors are shown as kinds, never their text": "TestReconcile_TheBreakerReasonCarriesAKindNotTheModelsText",
|
"Errors are shown as kinds, never their text": "TestReconcile_TheBreakerReasonCarriesAKindNotTheModelsText",
|
||||||
"Input that cannot be right is a client error": "TestUploadRoutes_ANonUUIDIdIs404NotA500",
|
"Input that cannot be right is a client error": "TestUploadRoutes_ANonUUIDIdIs404NotA500",
|
||||||
"The saved segments carry the merged words": "TestReconcileSegments_AChangedWordReplacesTheBaseWordInItsOwnSegmentOnly",
|
"The saved segments carry the merged words": "TestReconcileSegments_AChangedWordReplacesTheBaseWordInItsOwnSegmentOnly",
|
||||||
|
|
||||||
// summarize_new_video.feature
|
// summarize_new_video.feature
|
||||||
"A subscribed channel posts a video that has captions": "TestSubscribedVideoWithCaptionsIsSummarizedAndDelivered",
|
"A subscribed channel posts a video that has captions": "TestSubscribedVideoWithCaptionsIsSummarizedAndDelivered",
|
||||||
|
|||||||
Reference in New Issue
Block a user