feat(summarizer): move local fallback off koala to iguana/gemma4-26b
koala now carries other GPU loads, so the first fallback should not run there. Change the default chain to koala/phi4-mini → iguana/gemma4-26b → berget/mistral-small: the local fallback now runs on iguana (M2 Ultra headroom, different host = different egress IP for the rare fallback fetch). gemma4-26b is the brain-validated homelab general-purpose model (agentsquad H2/H3 executor) and returned valid summary JSON on the real prompt in a smoke test (~37s incl. cold-load — fine for a fallback path). Pure config default (TAPIR_FALLBACK_MODEL); chain mechanism (ADR-022) unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
+12
-4
@@ -827,10 +827,18 @@ The fix is resilience around it, not replacing it.
|
|||||||
**Decision.**
|
**Decision.**
|
||||||
1. **Ordered endpoint chain (`summarizer.NewChain`).** Endpoints are tried in order; the first to
|
1. **Ordered endpoint chain (`summarizer.NewChain`).** Endpoints are tried in order; the first to
|
||||||
return a *parseable* summary wins. Default chain:
|
return a *parseable* summary wins. Default chain:
|
||||||
`koala/phi4-mini` (primary, local) → `koala/phi4-14b` (fallback, local) →
|
`koala/phi4-mini` (primary, local) → `iguana/gemma4-26b` (fallback, local on a
|
||||||
`berget/mistral-small` (worst-case, external). All three are reached through the **one** LiteLLM
|
*different host*) → `berget/mistral-small` (worst-case, external). All three are reached through
|
||||||
gateway by alias — the gateway already fronts both llama-swap and berget — so a fallback is a
|
the **one** LiteLLM gateway by alias — the gateway already fronts both llama-swap and berget — so
|
||||||
different alias, not a second client config.
|
a fallback is a different alias, not a second client config.
|
||||||
|
|
||||||
|
**Update 2026-06-11:** the local fallback moved from `koala/phi4-14b` to `iguana/gemma4-26b`.
|
||||||
|
koala now carries other GPU loads, so keeping the fallback on koala competed with them; iguana
|
||||||
|
(M2 Ultra) has the headroom, and a different host is also a different egress IP for the rare
|
||||||
|
fallback fetch. `gemma4-26b` is the brain-validated homelab general-purpose model (agentsquad
|
||||||
|
H2/H3 executor) and returned valid summary JSON on the real prompt in a smoke test
|
||||||
|
(~37s incl. cold-load — fine for a path hit only when the fast primary fails). Pure config:
|
||||||
|
`TAPIR_FALLBACK_MODEL`.
|
||||||
2. **A parse failure advances the chain, same as a transport error.** "Reliably summarized" means
|
2. **A parse failure advances the chain, same as a transport error.** "Reliably summarized" means
|
||||||
*parseable summary returned*, not *HTTP 200*. This is the behaviour the old Primary→Fallback
|
*parseable summary returned*, not *HTTP 200*. This is the behaviour the old Primary→Fallback
|
||||||
shape missed.
|
shape missed.
|
||||||
|
|||||||
@@ -30,7 +30,9 @@ it** — endpoints and aliases drift, and this file is a snapshot (2026-06-06),
|
|||||||
- **Summarizer fallback chain (ADR-022).** The primary alias is the *first* of an ordered chain;
|
- **Summarizer fallback chain (ADR-022).** The primary alias is the *first* of an ordered chain;
|
||||||
on failure or unparseable output the summarizer advances to the next model. All reached through
|
on failure or unparseable output the summarizer advances to the next model. All reached through
|
||||||
the same gateway by alias.
|
the same gateway by alias.
|
||||||
- `TAPIR_FALLBACK_MODEL` — local fallback. **Default `koala/phi4-14b`.** Empty disables it.
|
- `TAPIR_FALLBACK_MODEL` — local fallback. **Default `iguana/gemma4-26b`** — on iguana, NOT
|
||||||
|
koala, so the fallback does not compete with koala's other GPU loads (and runs from a different
|
||||||
|
egress IP). Empty disables it.
|
||||||
- `TAPIR_CLOUD_FALLBACK_MODEL` — worst-case EXTERNAL fallback. **Default `berget/mistral-small`.**
|
- `TAPIR_CLOUD_FALLBACK_MODEL` — worst-case EXTERNAL fallback. **Default `berget/mistral-small`.**
|
||||||
**Set this empty (`""`) for any client/NDA deployment** so content never leaves the local
|
**Set this empty (`""`) for any client/NDA deployment** so content never leaves the local
|
||||||
stack — the chain then contains only local endpoints.
|
stack — the chain then contains only local endpoints.
|
||||||
|
|||||||
@@ -33,7 +33,10 @@ type Config struct {
|
|||||||
SummarizerModel string
|
SummarizerModel string
|
||||||
// FallbackModel is the LOCAL fallback alias tried when the primary fails or
|
// FallbackModel is the LOCAL fallback alias tried when the primary fails or
|
||||||
// returns unparseable output (ADR-022). Kept local so content stays on the
|
// returns unparseable output (ADR-022). Kept local so content stays on the
|
||||||
// homelab stack. Empty disables it. Default a bigger-context local model.
|
// homelab stack. Default is an IGUANA model (not koala) so the fallback runs
|
||||||
|
// on a different host than the koala primary — koala carries other loads, and
|
||||||
|
// a different host also means a different egress IP for the (rare) fallback.
|
||||||
|
// Empty disables it.
|
||||||
FallbackModel string
|
FallbackModel string
|
||||||
// CloudFallbackModel is the worst-case EXTERNAL fallback alias, tried only
|
// CloudFallbackModel is the worst-case EXTERNAL fallback alias, tried only
|
||||||
// after every local endpoint has failed (ADR-022). For client deployments set
|
// after every local endpoint has failed (ADR-022). For client deployments set
|
||||||
@@ -148,7 +151,7 @@ func (c Config) DexConfigured() bool { return strings.TrimSpace(c.OIDCIssuer) !=
|
|||||||
const (
|
const (
|
||||||
defaultGatewayURL = "http://koala:30401/v1"
|
defaultGatewayURL = "http://koala:30401/v1"
|
||||||
defaultSummarizerModel = "koala/phi4-mini"
|
defaultSummarizerModel = "koala/phi4-mini"
|
||||||
defaultFallbackModel = "koala/phi4-14b"
|
defaultFallbackModel = "iguana/gemma4-26b"
|
||||||
defaultCloudFallbackModel = "berget/mistral-small"
|
defaultCloudFallbackModel = "berget/mistral-small"
|
||||||
defaultSummaryMaxTokens = 1500
|
defaultSummaryMaxTokens = 1500
|
||||||
defaultMaxTranscriptChars = 18000
|
defaultMaxTranscriptChars = 18000
|
||||||
|
|||||||
Reference in New Issue
Block a user