feat(summarizer): upgrade to Anthropic Haiku (primary) + Sonnet (chat) via LiteLLM after infra#154 #22

Open
opened 2026-06-24 10:04:09 +00:00 by mathias · 0 comments
Owner

Context

Tapir's summarizer chain is currently: koala/phi4-miniiguana/gemma4-26b → berget fallback (ADR-027, decision: tapir-iguana-summarizer-fallback-choice). Once infra#154 adds direct Anthropic/Gemini/Mistral keys, tapir can route to frontier models for higher-quality summaries without going through berget.

Chat-with-transcript model is unknown — likely the same chain. Confirm by reading internal/summarizer/ config.

Changes

1. Read current model config

# On koala
grep -r "model\|phi4\|gemma\|litellm" ~/dev/AGENTS/tapir/internal/summarizer/ \
  ~/dev/AGENTS/tapir/internal/chat/ \
  ~/dev/AGENTS/tapir/config/ 2>/dev/null | head -40

Identify where model names are configured (env var, config file, hardcoded constant).

2. Add frontier model options

Add the following to tapir's model config (env vars or config file — match existing pattern):

# Primary summarizer (high quality, cost-aware)
TAPIR_SUMMARIZER_MODEL=anthropic/claude-haiku-4-5

# Fallback chain (in order)
TAPIR_SUMMARIZER_FALLBACK_1=iguana/gemma4-31b      # was gemma4-26b — upgrade
TAPIR_SUMMARIZER_FALLBACK_2=koala/phi4-mini         # local fast fallback
# Remove berget from fallback chain — direct Anthropic is cheaper

# Chat-with-transcript model
TAPIR_CHAT_MODEL=anthropic/claude-sonnet-4-6        # best for interactive Q&A
TAPIR_CHAT_FALLBACK=iguana/gemma4-31b

Rationale:

  • Haiku for summarization: fast, cheap (~$0.001/1K tokens), better quality than phi4-mini or gemma4-26b on long YouTube transcripts. Context window (200K) handles full transcripts without chunking.
  • Sonnet for chat: interactive quality matters more than cost here; Sonnet handles nuanced questions about transcript content better than any local model.
  • Drop berget from fallback: direct Anthropic is cheaper per token than berget relay + we control the key.
  • Upgrade iguana fallback from gemma4-26b → gemma4-31b: same host, better quality, confirmed working.

3. Update k3s deployment / ESO secret reference

Add TAPIR_SUMMARIZER_MODEL, TAPIR_CHAT_MODEL to tapir's k3s deployment env if they aren't already env-driven. Tapir should be reading its LiteLLM endpoint from TAPIR_LITELLM_URL (or equivalent) pointing at http://koala:30401/v1 — confirm this is already the case.

No new secrets needed — tapir authenticates to LiteLLM via LITELLM_MASTER_KEY which should already be in tapir's env.

4. Test

# Submit a known video and verify summary quality with new model
# Compare to the same video summary produced by phi4-mini

Acceptance criteria

  • Current model config location identified and documented in comment
  • TAPIR_SUMMARIZER_MODEL=anthropic/claude-haiku-4-5 wired (env or config)
  • TAPIR_CHAT_MODEL=anthropic/claude-sonnet-4-6 wired
  • Fallback chain updated: gemma4-31b → phi4-mini (berget removed)
  • Tapir deployment updated in k3s if env-driven
  • At least one real YouTube summary produced with new model — quality noticeably better than phi4-mini baseline
  • Comment with: where model config lived, what changed, sample summary excerpt (first 3 sentences)

Risk: LOW — model config change only. Fallback chain means tapir still works if Anthropic API is down.

Dependency

infra#154 — Anthropic key must be in LiteLLM. Confirm smoke test passed before changing tapir's primary model.

## Context Tapir's summarizer chain is currently: `koala/phi4-mini` → `iguana/gemma4-26b` → berget fallback (ADR-027, decision: tapir-iguana-summarizer-fallback-choice). Once infra#154 adds direct Anthropic/Gemini/Mistral keys, tapir can route to frontier models for higher-quality summaries without going through berget. Chat-with-transcript model is unknown — likely the same chain. Confirm by reading `internal/summarizer/` config. ## Changes ### 1. Read current model config ```bash # On koala grep -r "model\|phi4\|gemma\|litellm" ~/dev/AGENTS/tapir/internal/summarizer/ \ ~/dev/AGENTS/tapir/internal/chat/ \ ~/dev/AGENTS/tapir/config/ 2>/dev/null | head -40 ``` Identify where model names are configured (env var, config file, hardcoded constant). ### 2. Add frontier model options Add the following to tapir's model config (env vars or config file — match existing pattern): ```bash # Primary summarizer (high quality, cost-aware) TAPIR_SUMMARIZER_MODEL=anthropic/claude-haiku-4-5 # Fallback chain (in order) TAPIR_SUMMARIZER_FALLBACK_1=iguana/gemma4-31b # was gemma4-26b — upgrade TAPIR_SUMMARIZER_FALLBACK_2=koala/phi4-mini # local fast fallback # Remove berget from fallback chain — direct Anthropic is cheaper # Chat-with-transcript model TAPIR_CHAT_MODEL=anthropic/claude-sonnet-4-6 # best for interactive Q&A TAPIR_CHAT_FALLBACK=iguana/gemma4-31b ``` Rationale: - Haiku for summarization: fast, cheap (~$0.001/1K tokens), better quality than phi4-mini or gemma4-26b on long YouTube transcripts. Context window (200K) handles full transcripts without chunking. - Sonnet for chat: interactive quality matters more than cost here; Sonnet handles nuanced questions about transcript content better than any local model. - Drop berget from fallback: direct Anthropic is cheaper per token than berget relay + we control the key. - Upgrade iguana fallback from gemma4-26b → gemma4-31b: same host, better quality, confirmed working. ### 3. Update k3s deployment / ESO secret reference Add `TAPIR_SUMMARIZER_MODEL`, `TAPIR_CHAT_MODEL` to tapir's k3s deployment env if they aren't already env-driven. Tapir should be reading its LiteLLM endpoint from `TAPIR_LITELLM_URL` (or equivalent) pointing at `http://koala:30401/v1` — confirm this is already the case. No new secrets needed — tapir authenticates to LiteLLM via `LITELLM_MASTER_KEY` which should already be in tapir's env. ### 4. Test ```bash # Submit a known video and verify summary quality with new model # Compare to the same video summary produced by phi4-mini ``` ## Acceptance criteria - [ ] Current model config location identified and documented in comment - [ ] `TAPIR_SUMMARIZER_MODEL=anthropic/claude-haiku-4-5` wired (env or config) - [ ] `TAPIR_CHAT_MODEL=anthropic/claude-sonnet-4-6` wired - [ ] Fallback chain updated: gemma4-31b → phi4-mini (berget removed) - [ ] Tapir deployment updated in k3s if env-driven - [ ] At least one real YouTube summary produced with new model — quality noticeably better than phi4-mini baseline - [ ] Comment with: where model config lived, what changed, sample summary excerpt (first 3 sentences) **Risk:** LOW — model config change only. Fallback chain means tapir still works if Anthropic API is down. ## Dependency infra#154 — Anthropic key must be in LiteLLM. Confirm smoke test passed before changing tapir's primary model.
Sign in to join this conversation.
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/tapir#22