mathiasandClaude Opus 4.8
e9b5a3f3e7
The first friendly-pilot live run produced zero summaries: koala/phi4-mini hit three silent failure modes — 8k context overflow on long transcripts (HTTP 400), intermittent malformed JSON (highlights as a bare string), and no fallback wired at all (summarizer.New(primary, nil)). Keep phi4-mini as the fast primary and add resilience around it: - Ordered endpoint chain (summarizer.NewChain): phi4-mini → koala/phi4-14b (local) → berget/mistral-small (worst-case external). All reached through the one LiteLLM gateway by alias. - A parse failure now advances the chain like a transport error — the old Primary→Fallback shape returned the parse error without trying anyone else. - Tolerant parse: highlights/takeaways coerce string→[]string, absorbing the common small-model quirk without spending a fallback round-trip. - Transcript truncation (TAPIR_MAX_TRANSCRIPT_CHARS=18000) prevents the overflow rather than recovering from it; validated to fit phi4-mini's 8k window. - Bounded completion budget (TAPIR_SUMMARY_MAX_TOKENS=1500) — the old 8192 budget itself contributed to the overflow. Local-first guarantee preserved by ordering: external endpoint is tried only after every local one fails. TAPIR_CLOUD_FALLBACK_MODEL="" disables it entirely for client/NDA deployments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
48 lines
2.1 KiB
Gherkin
48 lines
2.1 KiB
Gherkin
Feature: Local-first AI with optional BYO fallback
|
|
As a user who values keeping my content on my own stack
|
|
I want summaries produced by the local AI first
|
|
So that my content only reaches a third-party model when I have explicitly opted in
|
|
|
|
Scenario: Local AI produces the summary
|
|
Given the local AI stack is available
|
|
When a transcript is summarized
|
|
Then the local AI produces the summary
|
|
And the summary records ai_provider "local"
|
|
And the summary records fallback_used as false
|
|
|
|
Scenario: Local AI fails and the user has a BYO provider configured
|
|
Given the local AI stack fails
|
|
And I have configured a BYO provider "anthropic"
|
|
When a transcript is summarized
|
|
Then Tapir falls back to my "anthropic" provider
|
|
And the summary records ai_provider "anthropic"
|
|
And the summary records fallback_used as true
|
|
|
|
Scenario: Local AI fails and the user has no BYO provider
|
|
Given the local AI stack fails
|
|
And I have not configured any BYO provider
|
|
When a transcript is summarized
|
|
Then no summary is produced
|
|
And the work is queued for retry
|
|
And my content is not sent to any third-party model
|
|
|
|
Scenario: A user without BYO never has content sent externally
|
|
Given I have not configured any BYO provider
|
|
When any transcript is summarized
|
|
Then my content is only ever sent to the local AI stack
|
|
|
|
Scenario: A model returns unparseable output and the next endpoint succeeds
|
|
Given the local AI stack is available
|
|
But the primary model returns output that cannot be parsed into a summary
|
|
And a fallback model is configured
|
|
When a transcript is summarized
|
|
Then Tapir falls back to the next model in the chain
|
|
And the summary records fallback_used as true
|
|
|
|
# "Reliably" is operationalized as: an endpoint returned a PARSEABLE summary
|
|
# within timeout. A 200 with malformed JSON (or highlights emitted as a bare
|
|
# string) counts as a failure and advances the chain (ADR-022). Endpoints are
|
|
# tried in order, locals first, so the external worst-case model only ever sees
|
|
# content after every local endpoint has failed.
|
|
# Quality scoring may be added later without changing these scenarios.
|