The last of the four mismatch clusters in #43/#44 with no issue of its own. Filed for the same reason as the retrieval one: everything else from #43 is now shipped or tracked, and this would quietly disappear.
The finding
From #44's agent column: synthesized_answer2/5 mismatch plus one partial. Three brain_answer calls were followed by three same-topic brain_query calls within a minute; one query+answer pair fired 8 seconds apart, hedging a single need.
The behaviour is unambiguous: agents do not treat a synthesised answer as terminal. They ask, then check. #44's own characterisation — "works for episodic recall, fails for how-do-I."
Why it is filed separately from retrieval
Two different failures that look alike. Retrieval is "the right note was not found"; this is "the note was found and the synthesis over it was not trusted." A fix to one does not fix the other, and conflating them produces a project rather than a fix.
Why it is not fixed by ADR-0009
I explicitly refused to claim it was. Folding brain_answer behind recall without improving synthesis would hide a trust problem behind a better name — which is why recall's strategy hint is required to stay escapable, so a caller that does not trust synthesis can still ask for raw retrieval.
Note also brain#11 (closed): brain_answer returned "no relevant content" for every telos-wing query. That was a scoping bug, fixed. This is different — the answers arrive and are not believed.
Scope, roughly
Characterise the failure honestly: is it hallucination, omission, staleness, or verbosity? "Not trusted" is a symptom, and the four have different fixes
Sample real answers against their source notes and score them — same discipline as the retrieval eval, and probably the same harness
Decide whether the answer path should cite the notes it synthesised from. A caller that can see the sources may not need to re-query, which would fix the observed behaviour without improving synthesis at all — worth testing before touching the model
Only then consider model, prompt, or context-window changes
That third box is the cheap hypothesis and should be tried first.
Watch for
brain_answer's LLM routes to berget.ai primary with an iguana ollama fallback. Any evaluation has to record which backend answered, or the result measures the router rather than the approach.
Risk: LOW to investigate.
Refs: #43, #44, brain#11 (closed, different bug), ADR-0009 §3.
The last of the four mismatch clusters in #43/#44 with no issue of its own. Filed for the same reason as the retrieval one: everything else from #43 is now shipped or tracked, and this would quietly disappear.
## The finding
From #44's agent column: `synthesized_answer` **2/5 mismatch plus one partial**. Three `brain_answer` calls were followed by three same-topic `brain_query` calls within a minute; one query+answer pair fired **8 seconds apart**, hedging a single need.
The behaviour is unambiguous: agents do not treat a synthesised answer as terminal. They ask, then check. #44's own characterisation — *"works for episodic recall, fails for how-do-I."*
## Why it is filed separately from retrieval
Two different failures that look alike. Retrieval is *"the right note was not found"*; this is *"the note was found and the synthesis over it was not trusted."* A fix to one does not fix the other, and conflating them produces a project rather than a fix.
## Why it is not fixed by ADR-0009
I explicitly refused to claim it was. Folding `brain_answer` behind `recall` without improving synthesis would hide a trust problem behind a better name — which is why `recall`'s `strategy` hint is required to stay escapable, so a caller that does not trust synthesis can still ask for raw retrieval.
Note also brain#11 (closed): `brain_answer` returned "no relevant content" for every telos-wing query. That was a scoping bug, fixed. This is different — the answers arrive and are not believed.
## Scope, roughly
- [ ] Characterise the failure honestly: is it hallucination, omission, staleness, or verbosity? "Not trusted" is a symptom, and the four have different fixes
- [ ] Sample real answers against their source notes and score them — same discipline as the retrieval eval, and probably the same harness
- [ ] Decide whether the answer path should cite the notes it synthesised from. A caller that can see the sources may not need to re-query, which would fix the observed behaviour without improving synthesis at all — worth testing before touching the model
- [ ] Only then consider model, prompt, or context-window changes
That third box is the cheap hypothesis and should be tried first.
## Watch for
`brain_answer`'s LLM routes to berget.ai primary with an iguana ollama fallback. Any evaluation has to record which backend answered, or the result measures the router rather than the approach.
**Risk:** LOW to investigate.
Refs: #43, #44, brain#11 (closed, different bug), ADR-0009 §3.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The last of the four mismatch clusters in #43/#44 with no issue of its own. Filed for the same reason as the retrieval one: everything else from #43 is now shipped or tracked, and this would quietly disappear.
The finding
From #44's agent column:
synthesized_answer2/5 mismatch plus one partial. Threebrain_answercalls were followed by three same-topicbrain_querycalls within a minute; one query+answer pair fired 8 seconds apart, hedging a single need.The behaviour is unambiguous: agents do not treat a synthesised answer as terminal. They ask, then check. #44's own characterisation — "works for episodic recall, fails for how-do-I."
Why it is filed separately from retrieval
Two different failures that look alike. Retrieval is "the right note was not found"; this is "the note was found and the synthesis over it was not trusted." A fix to one does not fix the other, and conflating them produces a project rather than a fix.
Why it is not fixed by ADR-0009
I explicitly refused to claim it was. Folding
brain_answerbehindrecallwithout improving synthesis would hide a trust problem behind a better name — which is whyrecall'sstrategyhint is required to stay escapable, so a caller that does not trust synthesis can still ask for raw retrieval.Note also brain#11 (closed):
brain_answerreturned "no relevant content" for every telos-wing query. That was a scoping bug, fixed. This is different — the answers arrive and are not believed.Scope, roughly
That third box is the cheap hypothesis and should be tried first.
Watch for
brain_answer's LLM routes to berget.ai primary with an iguana ollama fallback. Any evaluation has to record which backend answered, or the result measures the router rather than the approach.Risk: LOW to investigate.
Refs: #43, #44, brain#11 (closed, different bug), ADR-0009 §3.