Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9db06d8a63 | ||
|
|
1aa8a97f95 | ||
|
|
cc69a912f4 | ||
|
|
f4a0544903 | ||
|
|
e2a52789b9 | ||
|
|
e696b6405b |
@@ -86,7 +86,7 @@ Skills live in the canonical library `mathias/skills` and are wired into this re
|
||||
|
||||
## Current build state (start here for the first task)
|
||||
|
||||
The repo is **green and shipping** — last tag `v0.14.0`. `task check` passes (fmt, vet, lint,
|
||||
The repo is **green and shipping** — last tag `v0.15.0`. `task check` passes (fmt, vet, lint,
|
||||
`go test -p 1 ./...`). Go is `1.26.1` (see `go.mod`).
|
||||
|
||||
- Clean Architecture core is implemented: `internal/domain` (entities), `internal/ports`
|
||||
@@ -95,7 +95,8 @@ The repo is **green and shipping** — last tag `v0.14.0`. `task check` passes (
|
||||
green against it.
|
||||
- Adapters present under `internal/adapters/`: `youtube` (captions-first `VideoSource`,
|
||||
timedtext/InnerTube acquisition per ADR-010), `summarizer` + `llm` (the copied AI router,
|
||||
Primary→Fallback per ADR-004), `store` (Postgres, golang-migrate migrations 001–015),
|
||||
now a resilient endpoint chain — local primary → local fallback → external worst-case,
|
||||
parse-failure-aware, ADR-004 + ADR-022), `store` (Postgres, golang-migrate migrations 001–015),
|
||||
`secrets` (file-backed `SecretStore`). The brain HTTP sink (ADR-005) is the remaining
|
||||
optional sink.
|
||||
- Stage 1 is open (ADR-012): multi-user with **DB-enforced** isolation — Postgres RLS `FORCE`d
|
||||
|
||||
@@ -855,6 +855,42 @@ reply still parses; a transcript within budget is unchanged).
|
||||
|
||||
---
|
||||
|
||||
## ADR-023 — Drop Shorts/livestreams at discovery to protect the caption budget
|
||||
|
||||
**Status:** Accepted (2026-06-10). **Builds on ADR-014** (per-IP caption rate limit is the
|
||||
binding constraint) and the ADR-022 live-run findings.
|
||||
|
||||
**Context.** The scarce resource is the unofficial timedtext caption fetch (per-egress-IP
|
||||
429, ~3 successful/pass). The first multi-user run showed the candidate set was mostly noise —
|
||||
Shorts, sub-minute clips, and live broadcasts — each of which still consumes a caption-fetch
|
||||
attempt (and a "none" result is a *completed* fetch, so it costs budget even when it yields
|
||||
nothing). Spending the rate-limited budget on content the user will not read is the waste to
|
||||
cut first; it is cheaper and lower-risk than raising the ceiling (multi-IP, Whisper).
|
||||
|
||||
Duration and live status are NOT in the playlistItems discovery response, but they ARE in the
|
||||
Data API `videos.list` (contentDetails.duration + snippet.liveBroadcastContent) — the official
|
||||
**quota-based** API (1 unit/call, 50 ids/call), which is a *different* limit from the timedtext
|
||||
429. So one cheap quota call buys a filter that saves many expensive throttled fetches.
|
||||
|
||||
**Decision.**
|
||||
1. `NewVideos` enriches its candidates with a single `videos.list` call and drops, before
|
||||
returning: videos shorter than `TAPIR_MIN_VIDEO_SECONDS` (default 60) and any `live`/
|
||||
`upcoming` broadcast. Dropped videos are never persisted, so they also declutter the list.
|
||||
2. The filter is **degrade-open**: `MinVideoSeconds=0` disables it (no quota call), and a
|
||||
`videos.list` error returns the candidates unfiltered — discovery must never break because a
|
||||
metadata call hiccuped (worst case = pre-ADR-023 behaviour).
|
||||
3. The paste-a-URL path (`VideoByID`) is **not** filtered — an explicit user request for a
|
||||
specific video (even a Short) is honoured.
|
||||
|
||||
**Reversibility.** Pure discovery-time filter + config. `TAPIR_MIN_VIDEO_SECONDS=0` restores
|
||||
the old behaviour. No schema change, no effect on already-stored videos.
|
||||
|
||||
**Quota note.** Per-channel enrichment adds ~1 unit/channel/pass. At pilot scale (≤3 users)
|
||||
this is well under the 10k/day cap; at larger scale, batch `videos.list` across channels
|
||||
(50 ids/call) by collecting all discovered ids per pass before enriching.
|
||||
|
||||
---
|
||||
|
||||
## Rejected alternatives
|
||||
|
||||
Approaches considered during the 2026-06-02 planning + grill session and **deliberately not
|
||||
|
||||
@@ -88,6 +88,7 @@ func buildProcessor(cfg config.Config, st *store.Store) (*usecase.Engine, error)
|
||||
ClientSecret: cfg.YTClientSecret,
|
||||
TokenSecretRef: cfg.YTTokenRef,
|
||||
PreferredLanguages: []string{"en"},
|
||||
MinVideoSeconds: cfg.MinVideoSeconds,
|
||||
}, secretStore)
|
||||
|
||||
sum := buildSummarizer(cfg)
|
||||
|
||||
+55
-8
@@ -33,9 +33,15 @@ func buildUserRunner(cfg config.Config, st *store.Store, secretStore ports.Secre
|
||||
ClientSecret: cfg.YTClientSecret,
|
||||
TokenSecretRef: web.YouTubeTokenRef(userID),
|
||||
PreferredLanguages: []string{"en"},
|
||||
MinVideoSeconds: cfg.MinVideoSeconds,
|
||||
}, secretStore)
|
||||
|
||||
engine := usecase.NewEngine(src, buildSummarizer(cfg), st)
|
||||
// Share the transcript cache (ADR-021) on the scheduler path too — without
|
||||
// this every scheduled pass re-fetches transcripts it already had, burning the
|
||||
// scarce per-IP caption budget (ADR-014) and starving other users. The web
|
||||
// "Summarize now" path already sets this; the scheduler omitting it was a bug.
|
||||
engine.Transcripts = st
|
||||
|
||||
return runner.New(src, st, engine, userID, log,
|
||||
runner.WithBackoff(cfg.FetchBackoff),
|
||||
@@ -57,6 +63,7 @@ type userLister interface {
|
||||
// isolation). Returns the stats summed across users.
|
||||
func runDiscoveryPass(
|
||||
ctx context.Context,
|
||||
pass int,
|
||||
lister userLister,
|
||||
runUser func(context.Context, string) (runner.Stats, error),
|
||||
log *slog.Logger,
|
||||
@@ -67,15 +74,17 @@ func runDiscoveryPass(
|
||||
return runner.Stats{}
|
||||
}
|
||||
|
||||
log.Info("scheduler: starting discovery pass", "users", len(users))
|
||||
var total runner.Stats
|
||||
// Keep only users with a video connection. A pass for a connectionless user
|
||||
// (e.g. a stale Dex-era orphan identity) only tries to resolve a token that
|
||||
// was never minted, logging a spurious "ref not found" every tick. Filtering
|
||||
// here — BEFORE rotation — also keeps fairness honest: rotation is over the
|
||||
// users that actually consume the caption budget, so a dead identity can't eat
|
||||
// a rotation slot and skew the lead share.
|
||||
var connected []store.UserIdentity
|
||||
for _, u := range users {
|
||||
if ctx.Err() != nil {
|
||||
break // shutting down: stop enumerating
|
||||
return runner.Stats{} // shutting down
|
||||
}
|
||||
// Skip users with no video connection. A discovery pass for them only
|
||||
// attempts to resolve a token that was never minted, logging a spurious
|
||||
// "ref not found" every tick (e.g. stale Dex-era orphan identities).
|
||||
conns, err := lister.ConnectionsForUser(ctx, u.UserID)
|
||||
if err != nil {
|
||||
log.Warn("scheduler: list connections failed", "user", u.UserID, "err", err)
|
||||
@@ -85,6 +94,22 @@ func runDiscoveryPass(
|
||||
log.Debug("scheduler: skipping user with no video connections", "user", u.UserID)
|
||||
continue
|
||||
}
|
||||
connected = append(connected, u)
|
||||
}
|
||||
|
||||
// Rotate who goes first each pass. Caption fetches share one per-egress-IP
|
||||
// rate budget (ADR-014); whoever runs first each pass spends the pre-throttle
|
||||
// window, so a FIXED order permanently starves whoever is last (a new pilot
|
||||
// user got 0 fetches for 12h while the first-listed user got all of them).
|
||||
// Rotation over the connected set gives each real user the lead in turn.
|
||||
connected = rotateUsers(connected, pass)
|
||||
|
||||
log.Info("scheduler: starting discovery pass", "users", len(connected))
|
||||
var total runner.Stats
|
||||
for _, u := range connected {
|
||||
if ctx.Err() != nil {
|
||||
break // shutting down: stop enumerating
|
||||
}
|
||||
stats, err := runUser(ctx, u.UserID)
|
||||
total = sumStats(total, stats)
|
||||
if err != nil {
|
||||
@@ -116,7 +141,8 @@ func runScheduler(
|
||||
return // disabled
|
||||
}
|
||||
|
||||
runDiscoveryPass(ctx, lister, runUser, log)
|
||||
pass := 0
|
||||
runDiscoveryPass(ctx, pass, lister, runUser, log)
|
||||
|
||||
ticker := time.NewTicker(interval)
|
||||
defer ticker.Stop()
|
||||
@@ -125,11 +151,32 @@ func runScheduler(
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
runDiscoveryPass(ctx, lister, runUser, log)
|
||||
pass++
|
||||
runDiscoveryPass(ctx, pass, lister, runUser, log)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// rotateUsers left-rotates users by pass positions so a different user leads each
|
||||
// pass. With n users, user i leads on every pass where pass ≡ i (mod n). A pass
|
||||
// offset that is negative or exceeds n is normalised. Order within the rotation
|
||||
// is otherwise preserved, so the set of users run is unchanged — only who is
|
||||
// first (and thus wins the scarce caption-fetch budget) rotates.
|
||||
func rotateUsers(users []store.UserIdentity, pass int) []store.UserIdentity {
|
||||
n := len(users)
|
||||
if n <= 1 {
|
||||
return users
|
||||
}
|
||||
off := ((pass % n) + n) % n
|
||||
if off == 0 {
|
||||
return users
|
||||
}
|
||||
out := make([]store.UserIdentity, 0, n)
|
||||
out = append(out, users[off:]...)
|
||||
out = append(out, users[:off]...)
|
||||
return out
|
||||
}
|
||||
|
||||
// sumStats adds two passes' stats field-wise, so runDiscoveryPass can report a
|
||||
// per-tick aggregate across all users.
|
||||
func sumStats(a, b runner.Stats) runner.Stats {
|
||||
|
||||
@@ -45,6 +45,7 @@ func (f fakeLister) ConnectionsForUser(_ context.Context, userID string) ([]stor
|
||||
type countingRunUser struct {
|
||||
mu sync.Mutex
|
||||
calls map[string]int
|
||||
order []string // userIDs in the order they were run, across all passes
|
||||
failFor map[string]bool
|
||||
}
|
||||
|
||||
@@ -60,12 +61,19 @@ func (c *countingRunUser) run(_ context.Context, userID string) (runner.Stats, e
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
c.calls[userID]++
|
||||
c.order = append(c.order, userID)
|
||||
if c.failFor[userID] {
|
||||
return runner.Stats{Errors: 1}, errors.New("boom")
|
||||
}
|
||||
return runner.Stats{Summarized: 1}, nil
|
||||
}
|
||||
|
||||
func (c *countingRunUser) runOrder() []string {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return append([]string(nil), c.order...)
|
||||
}
|
||||
|
||||
func (c *countingRunUser) count(userID string) int {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
@@ -94,7 +102,7 @@ func TestDiscoveryPassRunsEveryUserOnce(t *testing.T) {
|
||||
lister := fakeLister{users: usersN("a", "b", "c")}
|
||||
rc := newCountingRunUser()
|
||||
|
||||
stats := runDiscoveryPass(context.Background(), lister, rc.run, quietLog())
|
||||
stats := runDiscoveryPass(context.Background(), 0, lister, rc.run, quietLog())
|
||||
|
||||
require.Equal(t, 1, rc.count("a"))
|
||||
require.Equal(t, 1, rc.count("b"))
|
||||
@@ -102,13 +110,46 @@ func TestDiscoveryPassRunsEveryUserOnce(t *testing.T) {
|
||||
require.Equal(t, 3, stats.Summarized, "stats are summed across users")
|
||||
}
|
||||
|
||||
// Caption fetches share one per-IP budget; a fixed user order starves whoever is
|
||||
// last. Each pass must rotate which user leads so the lead slot is shared.
|
||||
func TestDiscoveryPassRotatesLeadUser(t *testing.T) {
|
||||
lister := fakeLister{users: usersN("a", "b", "c")}
|
||||
rc := newCountingRunUser()
|
||||
|
||||
runDiscoveryPass(context.Background(), 0, lister, rc.run, quietLog())
|
||||
runDiscoveryPass(context.Background(), 1, lister, rc.run, quietLog())
|
||||
runDiscoveryPass(context.Background(), 2, lister, rc.run, quietLog())
|
||||
|
||||
require.Equal(t, []string{"a", "b", "c", "b", "c", "a", "c", "a", "b"}, rc.runOrder(),
|
||||
"each pass left-rotates the user order so every user leads in turn")
|
||||
// Fairness: over a full rotation cycle every user ran the same number of times.
|
||||
require.Equal(t, 3, rc.count("a"))
|
||||
require.Equal(t, 3, rc.count("b"))
|
||||
require.Equal(t, 3, rc.count("c"))
|
||||
}
|
||||
|
||||
// A connectionless orphan must not consume a rotation slot: rotation is over the
|
||||
// connected users only, so two real users alternate the lead 50/50 even with a
|
||||
// dead identity listed between them.
|
||||
func TestDiscoveryPassRotationIgnoresConnectionlessUsers(t *testing.T) {
|
||||
lister := fakeLister{users: usersN("a", "orphan", "c"), noConn: map[string]bool{"orphan": true}}
|
||||
rc := newCountingRunUser()
|
||||
|
||||
runDiscoveryPass(context.Background(), 0, lister, rc.run, quietLog())
|
||||
runDiscoveryPass(context.Background(), 1, lister, rc.run, quietLog())
|
||||
|
||||
require.Equal(t, []string{"a", "c", "c", "a"}, rc.runOrder(),
|
||||
"only connected users rotate; the orphan never runs and never holds a slot")
|
||||
require.Equal(t, 0, rc.count("orphan"))
|
||||
}
|
||||
|
||||
func TestDiscoveryPassSkipsUsersWithoutConnections(t *testing.T) {
|
||||
// b never connected a video source (e.g. a stale Dex-era orphan identity).
|
||||
// It must be skipped silently — not run and logged as a token error every pass.
|
||||
lister := fakeLister{users: usersN("a", "b", "c"), noConn: map[string]bool{"b": true}}
|
||||
rc := newCountingRunUser()
|
||||
|
||||
stats := runDiscoveryPass(context.Background(), lister, rc.run, quietLog())
|
||||
stats := runDiscoveryPass(context.Background(), 0, lister, rc.run, quietLog())
|
||||
|
||||
require.Equal(t, 1, rc.count("a"))
|
||||
require.Equal(t, 0, rc.count("b"), "a user with no connection must be skipped, not run")
|
||||
@@ -121,7 +162,7 @@ func TestDiscoveryPassOneUserFailureDoesNotStopOthers(t *testing.T) {
|
||||
lister := fakeLister{users: usersN("a", "b", "c")}
|
||||
rc := newCountingRunUser("b") // user b's pass errors
|
||||
|
||||
stats := runDiscoveryPass(context.Background(), lister, rc.run, quietLog())
|
||||
stats := runDiscoveryPass(context.Background(), 0, lister, rc.run, quietLog())
|
||||
|
||||
require.Equal(t, 1, rc.count("a"))
|
||||
require.Equal(t, 1, rc.count("b"))
|
||||
@@ -134,7 +175,7 @@ func TestDiscoveryPassListerErrorIsContained(t *testing.T) {
|
||||
lister := fakeLister{err: errors.New("db down")}
|
||||
rc := newCountingRunUser()
|
||||
|
||||
stats := runDiscoveryPass(context.Background(), lister, rc.run, quietLog())
|
||||
stats := runDiscoveryPass(context.Background(), 0, lister, rc.run, quietLog())
|
||||
|
||||
require.Equal(t, 0, rc.total(), "no users enumerated → no passes")
|
||||
require.Equal(t, runner.Stats{}, stats)
|
||||
|
||||
@@ -39,6 +39,11 @@ it** — endpoints and aliases drift, and this file is a snapshot (2026-06-06),
|
||||
- `TAPIR_MAX_TRANSCRIPT_CHARS` — transcript truncation budget sent to the model. **Default
|
||||
`18000`** (~fits an 8k-context model). `0` disables truncation. Prevents the context-overflow
|
||||
HTTP 400 a long transcript caused on `phi4-mini`.
|
||||
- **Discovery low-value filter (ADR-023).** `TAPIR_MIN_VIDEO_SECONDS` — **default `60`**. At
|
||||
discovery, `NewVideos` enriches candidates with one cheap `videos.list` call (quota API, NOT
|
||||
the timedtext 429 path) and drops videos shorter than this plus any live/upcoming broadcast,
|
||||
so the scarce caption-fetch budget isn't spent on Shorts. `0` disables the filter. The
|
||||
paste-a-URL path is never filtered.
|
||||
- **Thinking models need an explicit `max_tokens`.** qwen3 / deepseek-r1 spend the budget on
|
||||
reasoning and return **empty content** if `max_tokens` is too low (or unset). The summarizer's
|
||||
parser treats an empty summary as an error for exactly this reason. **Done (2026-06-02, Worker F):**
|
||||
|
||||
@@ -66,6 +66,13 @@ type Config struct {
|
||||
// poll. Zero means defaultMaxVideos.
|
||||
MaxVideosPerSubscription int
|
||||
|
||||
// MinVideoSeconds drops videos shorter than this from discovery (Shorts/clips,
|
||||
// ADR-023). NewVideos enriches candidates with a single cheap videos.list call
|
||||
// (contentDetails.duration + snippet.liveBroadcastContent) and filters before
|
||||
// returning, so the scarce caption-fetch budget is never spent on them. Live
|
||||
// and upcoming broadcasts are dropped too. Zero disables the filter.
|
||||
MinVideoSeconds int
|
||||
|
||||
// BaseURL overrides the Data API root. Empty means defaultBaseURL.
|
||||
BaseURL string
|
||||
|
||||
@@ -242,7 +249,97 @@ func (a *Adapter) NewVideos(ctx context.Context, sub domain.Subscription) ([]dom
|
||||
break
|
||||
}
|
||||
}
|
||||
return videos, nil
|
||||
|
||||
// Drop Shorts/sub-minute clips and live/upcoming broadcasts before they ever
|
||||
// reach the rate-limited caption path (ADR-023). One cheap videos.list call
|
||||
// (quota API, not the timedtext throttle) supplies duration + live status.
|
||||
return a.filterLowValue(ctx, client, videos), nil
|
||||
}
|
||||
|
||||
// filterLowValue removes videos shorter than cfg.MinVideoSeconds and any live or
|
||||
// upcoming broadcast, using a single videos.list lookup for duration +
|
||||
// liveBroadcastContent. The filter is best-effort: if MinVideoSeconds is 0 (off)
|
||||
// or the lookup fails, the input is returned unfiltered — discovery must not break
|
||||
// because a metadata call hiccuped; the worst case is the pre-ADR-023 behaviour.
|
||||
func (a *Adapter) filterLowValue(ctx context.Context, client *http.Client, videos []domain.Video) []domain.Video {
|
||||
if a.cfg.MinVideoSeconds <= 0 || len(videos) == 0 {
|
||||
return videos
|
||||
}
|
||||
|
||||
ids := make([]string, 0, len(videos))
|
||||
for _, v := range videos {
|
||||
ids = append(ids, v.ProviderVideoID)
|
||||
}
|
||||
q := url.Values{
|
||||
"part": {"contentDetails,snippet"},
|
||||
"id": {strings.Join(ids, ",")},
|
||||
}
|
||||
var resp videoListResponse
|
||||
if err := a.getJSON(ctx, client, "/videos", q, &resp); err != nil {
|
||||
// Degrade open: keep the candidates rather than lose discovery.
|
||||
return videos
|
||||
}
|
||||
|
||||
type meta struct {
|
||||
seconds int
|
||||
live string
|
||||
}
|
||||
byID := make(map[string]meta, len(resp.Items))
|
||||
for _, it := range resp.Items {
|
||||
byID[it.ID] = meta{seconds: parseISO8601Seconds(it.ContentDetails.Duration), live: it.Snippet.LiveBroadcastContent}
|
||||
}
|
||||
|
||||
kept := videos[:0]
|
||||
for _, v := range videos {
|
||||
m, ok := byID[v.ProviderVideoID]
|
||||
if !ok {
|
||||
kept = append(kept, v) // unknown metadata: keep, let the fetch decide
|
||||
continue
|
||||
}
|
||||
if m.live != "" && m.live != "none" {
|
||||
continue // live or upcoming broadcast
|
||||
}
|
||||
if m.seconds > 0 && m.seconds < a.cfg.MinVideoSeconds {
|
||||
continue // Short / sub-threshold clip
|
||||
}
|
||||
kept = append(kept, v)
|
||||
}
|
||||
return kept
|
||||
}
|
||||
|
||||
// parseISO8601Seconds parses an ISO 8601 duration as returned by the YouTube Data
|
||||
// API (e.g. "PT1H2M3S", "PT45S", "PT3M") into seconds. Only the hour/minute/second
|
||||
// components YouTube emits are handled; an unparseable or zero value returns 0,
|
||||
// which the caller treats as "unknown" (not filtered on duration).
|
||||
func parseISO8601Seconds(d string) int {
|
||||
if !strings.HasPrefix(d, "PT") {
|
||||
return 0
|
||||
}
|
||||
d = d[2:]
|
||||
total, num := 0, 0
|
||||
seen := false
|
||||
for _, r := range d {
|
||||
switch {
|
||||
case r >= '0' && r <= '9':
|
||||
num = num*10 + int(r-'0')
|
||||
seen = true
|
||||
case r == 'H':
|
||||
total += num * 3600
|
||||
num, seen = 0, false
|
||||
case r == 'M':
|
||||
total += num * 60
|
||||
num, seen = 0, false
|
||||
case r == 'S':
|
||||
total += num
|
||||
num, seen = 0, false
|
||||
default:
|
||||
return 0 // unexpected component (days/weeks) — treat as unknown
|
||||
}
|
||||
}
|
||||
if seen {
|
||||
return 0 // trailing digits without a unit: malformed
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
// VideoByID fetches a single video's metadata (videos.list, snippet) for an
|
||||
@@ -377,11 +474,16 @@ type playlistItemListResponse struct {
|
||||
|
||||
type videoListResponse struct {
|
||||
Items []struct {
|
||||
ID string `json:"id"`
|
||||
Snippet struct {
|
||||
Title string `json:"title"`
|
||||
ChannelTitle string `json:"channelTitle"`
|
||||
PublishedAt time.Time `json:"publishedAt"`
|
||||
LiveBroadcastContent string `json:"liveBroadcastContent"`
|
||||
} `json:"snippet"`
|
||||
ContentDetails struct {
|
||||
Duration string `json:"duration"` // ISO 8601, e.g. "PT1M30S"
|
||||
} `json:"contentDetails"`
|
||||
} `json:"items"`
|
||||
}
|
||||
|
||||
|
||||
@@ -186,6 +186,91 @@ func TestNewVideosCapsAtMax(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewVideosFiltersShortsAndLive: with MinVideoSeconds set, discovery enriches
|
||||
// candidates via videos.list and drops sub-threshold clips (Shorts) and
|
||||
// live/upcoming broadcasts before they reach the rate-limited caption path.
|
||||
func TestNewVideosFiltersShortsAndLive(t *testing.T) {
|
||||
a, _ := newTestAdapter(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/playlistItems":
|
||||
_, _ = w.Write([]byte(`{
|
||||
"items": [
|
||||
{"snippet": {"title": "Real Talk", "publishedAt": "2026-06-03T10:00:00Z", "resourceId": {"videoId": "long1"}}},
|
||||
{"snippet": {"title": "A Short", "publishedAt": "2026-06-03T09:00:00Z", "resourceId": {"videoId": "short1"}}},
|
||||
{"snippet": {"title": "Live Now", "publishedAt": "2026-06-03T08:00:00Z", "resourceId": {"videoId": "live1"}}}
|
||||
]
|
||||
}`))
|
||||
case "/videos":
|
||||
if got := r.URL.Query().Get("part"); got != "contentDetails,snippet" {
|
||||
t.Errorf("videos.list part=%q, want contentDetails,snippet", got)
|
||||
}
|
||||
_, _ = w.Write([]byte(`{
|
||||
"items": [
|
||||
{"id": "long1", "contentDetails": {"duration": "PT12M30S"}, "snippet": {"liveBroadcastContent": "none"}},
|
||||
{"id": "short1", "contentDetails": {"duration": "PT45S"}, "snippet": {"liveBroadcastContent": "none"}},
|
||||
{"id": "live1", "contentDetails": {"duration": "PT0S"}, "snippet": {"liveBroadcastContent": "live"}}
|
||||
]
|
||||
}`))
|
||||
default:
|
||||
t.Errorf("unexpected path %q", r.URL.Path)
|
||||
}
|
||||
})
|
||||
a.cfg.MinVideoSeconds = 60
|
||||
|
||||
vids, err := a.NewVideos(context.Background(), domain.Subscription{ID: "s1", UserID: "u1", ChannelID: "UC_acme"})
|
||||
if err != nil {
|
||||
t.Fatalf("NewVideos: %v", err)
|
||||
}
|
||||
if len(vids) != 1 || vids[0].ProviderVideoID != "long1" {
|
||||
t.Fatalf("expected only long1 to survive the filter, got %+v", vids)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewVideosNoFilterWhenDisabled: MinVideoSeconds=0 keeps the pre-ADR-023
|
||||
// behaviour — no videos.list call, no filtering.
|
||||
func TestNewVideosNoFilterWhenDisabled(t *testing.T) {
|
||||
a, _ := newTestAdapter(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/videos" {
|
||||
t.Errorf("videos.list must not be called when MinVideoSeconds is 0")
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"items": [
|
||||
{"snippet": {"title": "A Short", "publishedAt": "2026-06-03T09:00:00Z", "resourceId": {"videoId": "short1"}}}
|
||||
]}`))
|
||||
})
|
||||
a.cfg.MinVideoSeconds = 0
|
||||
|
||||
vids, err := a.NewVideos(context.Background(), domain.Subscription{ID: "s1", UserID: "u1", ChannelID: "UC_acme"})
|
||||
if err != nil {
|
||||
t.Fatalf("NewVideos: %v", err)
|
||||
}
|
||||
if len(vids) != 1 {
|
||||
t.Fatalf("filter disabled must keep all videos, got %d", len(vids))
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseISO8601Seconds(t *testing.T) {
|
||||
cases := []struct {
|
||||
in string
|
||||
want int
|
||||
}{
|
||||
{"PT45S", 45},
|
||||
{"PT1M30S", 90},
|
||||
{"PT3M", 180},
|
||||
{"PT1H2M3S", 3723},
|
||||
{"PT2H", 7200},
|
||||
{"PT0S", 0},
|
||||
{"", 0},
|
||||
{"garbage", 0},
|
||||
{"P1D", 0}, // days component not handled → unknown
|
||||
{"PT10", 0}, // trailing digits without a unit → malformed
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := parseISO8601Seconds(c.in); got != c.want {
|
||||
t.Errorf("parseISO8601Seconds(%q) = %d, want %d", c.in, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestUploadsPlaylistID covers the zero-cost UC->UU derivation, including
|
||||
// non-standard ids that must fall through unchanged (handled via fallback).
|
||||
func TestUploadsPlaylistID(t *testing.T) {
|
||||
|
||||
@@ -46,6 +46,12 @@ type Config struct {
|
||||
// MaxTranscriptChars bounds the transcript text sent to the model so a long
|
||||
// transcript does not overflow a small-context primary. 0 disables truncation.
|
||||
MaxTranscriptChars int
|
||||
|
||||
// MinVideoSeconds drops videos shorter than this from discovery (Shorts and
|
||||
// other sub-minute clips that are noise and waste the scarce caption-fetch
|
||||
// budget, ADR-014/ADR-023). Enforced via a cheap Data API videos.list lookup at
|
||||
// discovery, never the rate-limited caption path. 0 disables the filter.
|
||||
MinVideoSeconds int
|
||||
// SummarizerTimeout bounds a single completion call. Thinking models are
|
||||
// slow, so the default is generous.
|
||||
SummarizerTimeout time.Duration
|
||||
@@ -138,6 +144,7 @@ const (
|
||||
defaultCloudFallbackModel = "berget/mistral-small"
|
||||
defaultSummaryMaxTokens = 1500
|
||||
defaultMaxTranscriptChars = 18000
|
||||
defaultMinVideoSeconds = 60
|
||||
defaultSummarizerTimeout = 5 * time.Minute
|
||||
defaultYTTokenRef = "youtube/refresh_token"
|
||||
defaultYTConnectRedirectURL = "https://tapir.d-ma.be/oauth/youtube/callback"
|
||||
@@ -230,6 +237,15 @@ func Load() (Config, error) {
|
||||
}
|
||||
c.MaxTranscriptChars = maxChars
|
||||
|
||||
minVideo, err := intOr("TAPIR_MIN_VIDEO_SECONDS", defaultMinVideoSeconds)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if minVideo < 0 {
|
||||
minVideo = 0
|
||||
}
|
||||
c.MinVideoSeconds = minVideo
|
||||
|
||||
onboard, err := intOr("TAPIR_ONBOARD_SUMMARIZE_COUNT", defaultOnboardSummarizeCount)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
|
||||
@@ -233,11 +233,20 @@ func (a *App) handleList(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
hasConnected := len(conns) > 0
|
||||
|
||||
if isHTMX(r) {
|
||||
a.render(w, r, summaryList(buckets, hasConnected))
|
||||
// Summarization mode drives the backlog copy: an auto user is told summaries
|
||||
// land gradually; a manual user is told to click Summarize (the first pilot
|
||||
// user sat in manual mode reading "land automatically" and waited forever).
|
||||
autoSummarize, err := a.Store.GetAutoSummarize(r.Context(), userID)
|
||||
if err != nil {
|
||||
a.serverError(w, r, "summarize mode", err)
|
||||
return
|
||||
}
|
||||
a.render(w, r, ListPage(buckets, f, stats, takeFlash(w, r), hasConnected, channels))
|
||||
|
||||
if isHTMX(r) {
|
||||
a.render(w, r, summaryList(buckets, hasConnected, autoSummarize))
|
||||
return
|
||||
}
|
||||
a.render(w, r, ListPage(buckets, f, stats, takeFlash(w, r), hasConnected, channels, autoSummarize))
|
||||
}
|
||||
|
||||
// handleDetail renders one summary in full (highlights, takeaways, action group).
|
||||
|
||||
@@ -477,3 +477,36 @@ func postAction(t *testing.T, app *web.App, videoID, action string, htmx bool) *
|
||||
}
|
||||
return do(t, app, req)
|
||||
}
|
||||
|
||||
// TestListManualModeBannerCopy: a manual-mode user with un-summarized videos
|
||||
// sees the manual prompt (click Summarize), NOT the "summaries land
|
||||
// automatically" copy that misled the first pilot user into waiting forever.
|
||||
func TestListManualModeBannerCopy(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
app := newApp(t)
|
||||
p := rawPool(t)
|
||||
resetDB(t, p)
|
||||
seedVideo(t, p, videoX, "X Title", "https://x", time.Time{}) // pending, un-summarized
|
||||
require.NoError(t, app.Store.SetAutoSummarize(ctx, userID, false))
|
||||
|
||||
html := body(t, do(t, app, httptest.NewRequest(http.MethodGet, "/", nil)))
|
||||
require.Contains(t, html, "Manual mode")
|
||||
require.Contains(t, html, "are not summarized automatically")
|
||||
require.NotContains(t, html, "land gradually",
|
||||
"manual-mode user must not be told summaries arrive automatically")
|
||||
}
|
||||
|
||||
// TestListAutoModeBannerCopy: an auto-mode user with a backlog sees the
|
||||
// gradual-delivery copy, not the manual prompt.
|
||||
func TestListAutoModeBannerCopy(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
app := newApp(t)
|
||||
p := rawPool(t)
|
||||
resetDB(t, p)
|
||||
seedVideo(t, p, videoX, "X Title", "https://x", time.Time{})
|
||||
require.NoError(t, app.Store.SetAutoSummarize(ctx, userID, true))
|
||||
|
||||
html := body(t, do(t, app, httptest.NewRequest(http.MethodGet, "/", nil)))
|
||||
require.Contains(t, html, "land gradually")
|
||||
require.NotContains(t, html, "are not summarized automatically")
|
||||
}
|
||||
|
||||
@@ -65,7 +65,7 @@ func TestParseYouTubeVideoID(t *testing.T) {
|
||||
func TestListPageShowsPasteFormOnlyWhenConnected(t *testing.T) {
|
||||
render := func(connected bool) string {
|
||||
var buf bytes.Buffer
|
||||
if err := ListPage(listBuckets{}, Filter{}, PipelineStats{}, "", connected, nil).Render(context.Background(), &buf); err != nil {
|
||||
if err := ListPage(listBuckets{}, Filter{}, PipelineStats{}, "", connected, nil, true).Render(context.Background(), &buf); err != nil {
|
||||
t.Fatalf("render: %v", err)
|
||||
}
|
||||
return buf.String()
|
||||
|
||||
@@ -104,7 +104,7 @@ templ flashBanner(code string) {
|
||||
// #summary-list region; a non-HTMX request renders the whole page. flash carries
|
||||
// a one-shot notification (e.g. "connected", "registered") surfaced on arrival
|
||||
// after a POST→redirect.
|
||||
templ ListPage(b listBuckets, f Filter, stats PipelineStats, flash string, hasConnected bool, channels []string) {
|
||||
templ ListPage(b listBuckets, f Filter, stats PipelineStats, flash string, hasConnected bool, channels []string, autoSummarize bool) {
|
||||
@Layout("Tapir — Summaries") {
|
||||
@flashBanner(flash)
|
||||
if hasConnected {
|
||||
@@ -116,14 +116,23 @@ templ ListPage(b listBuckets, f Filter, stats PipelineStats, flash string, hasCo
|
||||
if stats.RateLimited > 0 || stats.Pending > 0 || stats.NoText > 0 {
|
||||
@pipelineBar(stats)
|
||||
}
|
||||
if stats.RateLimited+stats.Pending > 0 {
|
||||
if (stats.RateLimited+stats.Pending) > 0 && autoSummarize {
|
||||
<p class="pipeline-note muted">
|
||||
Tapir fetches captions slowly on purpose, to respect YouTube's limits —
|
||||
new summaries land gradually. Check back tomorrow.
|
||||
</p>
|
||||
}
|
||||
if (stats.RateLimited+stats.Pending) > 0 && !autoSummarize {
|
||||
<p class="pipeline-note muted">
|
||||
You are in Manual mode: new videos appear here but are not summarized
|
||||
automatically. Use the Summarize button on the ones you want.
|
||||
</p>
|
||||
<p class="pipeline-note muted">
|
||||
<a href="/account">Switch to Automatic</a> to have new videos summarized for you.
|
||||
</p>
|
||||
}
|
||||
<div id="summary-list">
|
||||
@summaryList(b, hasConnected)
|
||||
@summaryList(b, hasConnected, autoSummarize)
|
||||
</div>
|
||||
}
|
||||
}
|
||||
@@ -203,12 +212,16 @@ templ filterForm(f Filter, channels []string) {
|
||||
// and a single disclosure holding the older un-summarized back-catalogue. Cards
|
||||
// reflow to a single column on mobile; an empty list shows a friendly first-run
|
||||
// state instead of a blank table.
|
||||
templ summaryList(b listBuckets, hasConnected bool) {
|
||||
templ summaryList(b listBuckets, hasConnected bool, autoSummarize bool) {
|
||||
if b.empty() {
|
||||
if hasConnected {
|
||||
<div class="empty empty-connected">
|
||||
<strong>Your account is connected</strong>
|
||||
if autoSummarize {
|
||||
<span>Tapir is finding your subscriptions and fetching captions — summaries appear here gradually. Check back later.</span>
|
||||
} else {
|
||||
<span>Tapir is finding your subscriptions. You are in Manual mode, so videos appear here with a Summarize button — pick the ones you want, or switch to Automatic in your account.</span>
|
||||
}
|
||||
</div>
|
||||
} else {
|
||||
<div class="empty">
|
||||
|
||||
+274
-249
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user