diff --git a/DECISIONS.md b/DECISIONS.md index 9e03cf1..73759d4 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -827,10 +827,18 @@ The fix is resilience around it, not replacing it. **Decision.** 1. **Ordered endpoint chain (`summarizer.NewChain`).** Endpoints are tried in order; the first to return a *parseable* summary wins. Default chain: - `koala/phi4-mini` (primary, local) → `koala/phi4-14b` (fallback, local) → - `berget/mistral-small` (worst-case, external). All three are reached through the **one** LiteLLM - gateway by alias — the gateway already fronts both llama-swap and berget — so a fallback is a - different alias, not a second client config. + `koala/phi4-mini` (primary, local) → `iguana/gemma4-26b` (fallback, local on a + *different host*) → `berget/mistral-small` (worst-case, external). All three are reached through + the **one** LiteLLM gateway by alias — the gateway already fronts both llama-swap and berget — so + a fallback is a different alias, not a second client config. + + **Update 2026-06-11:** the local fallback moved from `koala/phi4-14b` to `iguana/gemma4-26b`. + koala now carries other GPU loads, so keeping the fallback on koala competed with them; iguana + (M2 Ultra) has the headroom, and a different host is also a different egress IP for the rare + fallback fetch. `gemma4-26b` is the brain-validated homelab general-purpose model (agentsquad + H2/H3 executor) and returned valid summary JSON on the real prompt in a smoke test + (~37s incl. cold-load — fine for a path hit only when the fast primary fails). Pure config: + `TAPIR_FALLBACK_MODEL`. 2. **A parse failure advances the chain, same as a transport error.** "Reliably summarized" means *parseable summary returned*, not *HTTP 200*. This is the behaviour the old Primary→Fallback shape missed. diff --git a/docs/homelab-integration.md b/docs/homelab-integration.md index 69c51ba..1f452cc 100644 --- a/docs/homelab-integration.md +++ b/docs/homelab-integration.md @@ -30,7 +30,9 @@ it** — endpoints and aliases drift, and this file is a snapshot (2026-06-06), - **Summarizer fallback chain (ADR-022).** The primary alias is the *first* of an ordered chain; on failure or unparseable output the summarizer advances to the next model. All reached through the same gateway by alias. - - `TAPIR_FALLBACK_MODEL` — local fallback. **Default `koala/phi4-14b`.** Empty disables it. + - `TAPIR_FALLBACK_MODEL` — local fallback. **Default `iguana/gemma4-26b`** — on iguana, NOT + koala, so the fallback does not compete with koala's other GPU loads (and runs from a different + egress IP). Empty disables it. - `TAPIR_CLOUD_FALLBACK_MODEL` — worst-case EXTERNAL fallback. **Default `berget/mistral-small`.** **Set this empty (`""`) for any client/NDA deployment** so content never leaves the local stack — the chain then contains only local endpoints. diff --git a/internal/config/config.go b/internal/config/config.go index 79e2dfb..4ffa0e4 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -33,7 +33,10 @@ type Config struct { SummarizerModel string // FallbackModel is the LOCAL fallback alias tried when the primary fails or // returns unparseable output (ADR-022). Kept local so content stays on the - // homelab stack. Empty disables it. Default a bigger-context local model. + // homelab stack. Default is an IGUANA model (not koala) so the fallback runs + // on a different host than the koala primary — koala carries other loads, and + // a different host also means a different egress IP for the (rare) fallback. + // Empty disables it. FallbackModel string // CloudFallbackModel is the worst-case EXTERNAL fallback alias, tried only // after every local endpoint has failed (ADR-022). For client deployments set @@ -148,7 +151,7 @@ func (c Config) DexConfigured() bool { return strings.TrimSpace(c.OIDCIssuer) != const ( defaultGatewayURL = "http://koala:30401/v1" defaultSummarizerModel = "koala/phi4-mini" - defaultFallbackModel = "koala/phi4-14b" + defaultFallbackModel = "iguana/gemma4-26b" defaultCloudFallbackModel = "berget/mistral-small" defaultSummaryMaxTokens = 1500 defaultMaxTranscriptChars = 18000