Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a9be5f285b |
+12
-4
@@ -827,10 +827,18 @@ The fix is resilience around it, not replacing it.
|
||||
**Decision.**
|
||||
1. **Ordered endpoint chain (`summarizer.NewChain`).** Endpoints are tried in order; the first to
|
||||
return a *parseable* summary wins. Default chain:
|
||||
`koala/phi4-mini` (primary, local) → `koala/phi4-14b` (fallback, local) →
|
||||
`berget/mistral-small` (worst-case, external). All three are reached through the **one** LiteLLM
|
||||
gateway by alias — the gateway already fronts both llama-swap and berget — so a fallback is a
|
||||
different alias, not a second client config.
|
||||
`koala/phi4-mini` (primary, local) → `iguana/gemma4-26b` (fallback, local on a
|
||||
*different host*) → `berget/mistral-small` (worst-case, external). All three are reached through
|
||||
the **one** LiteLLM gateway by alias — the gateway already fronts both llama-swap and berget — so
|
||||
a fallback is a different alias, not a second client config.
|
||||
|
||||
**Update 2026-06-11:** the local fallback moved from `koala/phi4-14b` to `iguana/gemma4-26b`.
|
||||
koala now carries other GPU loads, so keeping the fallback on koala competed with them; iguana
|
||||
(M2 Ultra) has the headroom, and a different host is also a different egress IP for the rare
|
||||
fallback fetch. `gemma4-26b` is the brain-validated homelab general-purpose model (agentsquad
|
||||
H2/H3 executor) and returned valid summary JSON on the real prompt in a smoke test
|
||||
(~37s incl. cold-load — fine for a path hit only when the fast primary fails). Pure config:
|
||||
`TAPIR_FALLBACK_MODEL`.
|
||||
2. **A parse failure advances the chain, same as a transport error.** "Reliably summarized" means
|
||||
*parseable summary returned*, not *HTTP 200*. This is the behaviour the old Primary→Fallback
|
||||
shape missed.
|
||||
|
||||
@@ -30,7 +30,9 @@ it** — endpoints and aliases drift, and this file is a snapshot (2026-06-06),
|
||||
- **Summarizer fallback chain (ADR-022).** The primary alias is the *first* of an ordered chain;
|
||||
on failure or unparseable output the summarizer advances to the next model. All reached through
|
||||
the same gateway by alias.
|
||||
- `TAPIR_FALLBACK_MODEL` — local fallback. **Default `koala/phi4-14b`.** Empty disables it.
|
||||
- `TAPIR_FALLBACK_MODEL` — local fallback. **Default `iguana/gemma4-26b`** — on iguana, NOT
|
||||
koala, so the fallback does not compete with koala's other GPU loads (and runs from a different
|
||||
egress IP). Empty disables it.
|
||||
- `TAPIR_CLOUD_FALLBACK_MODEL` — worst-case EXTERNAL fallback. **Default `berget/mistral-small`.**
|
||||
**Set this empty (`""`) for any client/NDA deployment** so content never leaves the local
|
||||
stack — the chain then contains only local endpoints.
|
||||
|
||||
@@ -33,7 +33,10 @@ type Config struct {
|
||||
SummarizerModel string
|
||||
// FallbackModel is the LOCAL fallback alias tried when the primary fails or
|
||||
// returns unparseable output (ADR-022). Kept local so content stays on the
|
||||
// homelab stack. Empty disables it. Default a bigger-context local model.
|
||||
// homelab stack. Default is an IGUANA model (not koala) so the fallback runs
|
||||
// on a different host than the koala primary — koala carries other loads, and
|
||||
// a different host also means a different egress IP for the (rare) fallback.
|
||||
// Empty disables it.
|
||||
FallbackModel string
|
||||
// CloudFallbackModel is the worst-case EXTERNAL fallback alias, tried only
|
||||
// after every local endpoint has failed (ADR-022). For client deployments set
|
||||
@@ -148,7 +151,7 @@ func (c Config) DexConfigured() bool { return strings.TrimSpace(c.OIDCIssuer) !=
|
||||
const (
|
||||
defaultGatewayURL = "http://koala:30401/v1"
|
||||
defaultSummarizerModel = "koala/phi4-mini"
|
||||
defaultFallbackModel = "koala/phi4-14b"
|
||||
defaultFallbackModel = "iguana/gemma4-26b"
|
||||
defaultCloudFallbackModel = "berget/mistral-small"
|
||||
defaultSummaryMaxTokens = 1500
|
||||
defaultMaxTranscriptChars = 18000
|
||||
|
||||
Reference in New Issue
Block a user