mathias/hyperguild
1
0
Fork 0
Code Issues 30 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

Semantic vs lexical retrieval — the gap ADR-0009 structured but did not close #113

New Issue
Open
opened 2026-08-23 05:38:31 +00:00 by mathias · 0 comments
mathias commented 2026-08-23 05:38:31 +00:00
Owner
Copy Link
Copy Source

One of the four mismatch clusters in #43/#44 that nothing currently tracks. Filed so it does not fall off the map now that the rest of #43 is shipped or tracked.

The finding, from the original evidence

Agent column: 3/3 mismatch on semantic_retrieval — the caller knows the meaning it wants and not the indexed words, so it keyword-stuffs BM25 and reformulates. #44 records a four-step reformulation chain over a single Go bug. The human column bends the same lexical tool toward meaning-based intent.

The brain is "append + keyword-search … serving semantic intent with a lexical tool" — #43's own summary.

What has and has not been done

Structured, not solved. ADR-0009 answers the interface half: one recall verb with an optional strategy hint defaulting to server choice (#80). That decides which door the caller knocks on.

It does not improve what is behind the door. Renaming a lexical tool does not make it semantic, and I said so explicitly in the ADR rather than letting the redesign appear to have fixed it.

Relevant existing capability: pgvector is in the stack and internal/vectorstore exists, so the substrate is present — this is not a from-scratch build. Worth establishing first what is actually wired and what is dormant.

Scope, roughly

  • Establish the baseline: what does brain_query do today, and is the vector path live, partial, or dormant?
  • A reproducible retrieval eval — a handful of real queries from #44's transcripts with known-correct targets, scored. Without it "better retrieval" is unfalsifiable.
  • Then decide: hybrid BM25 + vector, reranking, or something narrower
  • recall's strategy hint stays escapable either way (ADR-0009 §3), so a caller that wants raw lexical can always have it

Deliberately not a redesign

The interface question is settled. This is a ranking-quality problem and should be measured, not architected — the eval comes before the mechanism, or this turns into an infrastructure project competing with the work that actually surfaces value (the same trap brain-improvement-brainstorm-scoped-out-2026-08-20 rejected "scaling ingestion" for).

Risk: LOW to investigate. MEDIUM if it becomes a build without an eval first.

Refs: #43, #44, #80, ADR-0009, internal/vectorstore.

One of the four mismatch clusters in #43/#44 that nothing currently tracks. Filed so it does not fall off the map now that the rest of #43 is shipped or tracked. ## The finding, from the original evidence Agent column: **3/3 mismatch on `semantic_retrieval`** — the caller knows the meaning it wants and not the indexed words, so it keyword-stuffs BM25 and reformulates. #44 records a four-step reformulation chain over a single Go bug. The human column bends the same lexical tool toward meaning-based intent. The brain is *"append + keyword-search … serving semantic intent with a lexical tool"* — #43's own summary. ## What has and has not been done **Structured, not solved.** ADR-0009 answers the *interface* half: one `recall` verb with an optional `strategy` hint defaulting to server choice (#80). That decides which door the caller knocks on. It does not improve what is behind the door. Renaming a lexical tool does not make it semantic, and I said so explicitly in the ADR rather than letting the redesign appear to have fixed it. Relevant existing capability: pgvector is in the stack and `internal/vectorstore` exists, so the substrate is present — this is not a from-scratch build. Worth establishing first what is actually wired and what is dormant. ## Scope, roughly - [ ] Establish the baseline: what does `brain_query` do today, and is the vector path live, partial, or dormant? - [ ] A reproducible retrieval eval — a handful of real queries from #44's transcripts with known-correct targets, scored. Without it "better retrieval" is unfalsifiable. - [ ] Then decide: hybrid BM25 + vector, reranking, or something narrower - [ ] `recall`'s `strategy` hint stays escapable either way (ADR-0009 §3), so a caller that wants raw lexical can always have it ## Deliberately not a redesign The interface question is settled. This is a ranking-quality problem and should be measured, not architected — the eval comes before the mechanism, or this turns into an infrastructure project competing with the work that actually surfaces value (the same trap `brain-improvement-brainstorm-scoped-out-2026-08-20` rejected "scaling ingestion" for). **Risk:** LOW to investigate. MEDIUM if it becomes a build without an eval first. Refs: #43, #44, #80, ADR-0009, `internal/vectorstore`.
mathias added the risk/lowsize/ltype/research labels 2026-08-27 05:25:42 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/ready
Meets the agent-ready contract; authorizes unattended execution
 blocked/client
Waiting on the client
 blocked/supplier
Waiting on a supplier
 blocked/upstream
Waiting on an upstream project or vendor
 risk/high
Irreversible, security-relevant, or control plane
 risk/low
Reversible, local, test-covered
 risk/medium
Touches shared state, deploys or data
 size/l
Too big - split it
 size/m
Under two days
 size/s
Under half a day
 size/xs
Under an hour
 status:dispatch
status:dispatched
status:review
status:reviewing
type/chore
Maintenance, no behaviour change
 type/decision
Human decision required; never agent/ready
 type/docs
Documentation only
 type/feat
New capability
 type/fix
Corrects broken behaviour
 type/ops
Infrastructure, deploy, runtime
 type/research
Investigate; output is a finding, not code
 type/test
Test coverage only
No labels risk/low size/l type/research
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate-bot mathias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#113