feat(summarizer): resilient endpoint chain with local→cloud fallback (ADR-022)
The first friendly-pilot live run produced zero summaries: koala/phi4-mini hit three silent failure modes — 8k context overflow on long transcripts (HTTP 400), intermittent malformed JSON (highlights as a bare string), and no fallback wired at all (summarizer.New(primary, nil)). Keep phi4-mini as the fast primary and add resilience around it: - Ordered endpoint chain (summarizer.NewChain): phi4-mini → koala/phi4-14b (local) → berget/mistral-small (worst-case external). All reached through the one LiteLLM gateway by alias. - A parse failure now advances the chain like a transport error — the old Primary→Fallback shape returned the parse error without trying anyone else. - Tolerant parse: highlights/takeaways coerce string→[]string, absorbing the common small-model quirk without spending a fallback round-trip. - Transcript truncation (TAPIR_MAX_TRANSCRIPT_CHARS=18000) prevents the overflow rather than recovering from it; validated to fit phi4-mini's 8k window. - Bounded completion budget (TAPIR_SUMMARY_MAX_TOKENS=1500) — the old 8192 budget itself contributed to the overflow. Local-first guarantee preserved by ordering: external endpoint is tried only after every local one fails. TAPIR_CLOUD_FALLBACK_MODEL="" disables it entirely for client/NDA deployments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -31,5 +31,17 @@ Feature: Local-first AI with optional BYO fallback
|
||||
When any transcript is summarized
|
||||
Then my content is only ever sent to the local AI stack
|
||||
|
||||
# "Reliably" is operationalized as: Primary returned without error within timeout.
|
||||
Scenario: A model returns unparseable output and the next endpoint succeeds
|
||||
Given the local AI stack is available
|
||||
But the primary model returns output that cannot be parsed into a summary
|
||||
And a fallback model is configured
|
||||
When a transcript is summarized
|
||||
Then Tapir falls back to the next model in the chain
|
||||
And the summary records fallback_used as true
|
||||
|
||||
# "Reliably" is operationalized as: an endpoint returned a PARSEABLE summary
|
||||
# within timeout. A 200 with malformed JSON (or highlights emitted as a bare
|
||||
# string) counts as a failure and advances the chain (ADR-022). Endpoints are
|
||||
# tried in order, locals first, so the external worst-case model only ever sees
|
||||
# content after every local endpoint has failed.
|
||||
# Quality scoring may be added later without changing these scenarios.
|
||||
|
||||
Reference in New Issue
Block a user