mathias/hyperguild
1
0
Fork 0
Code Issues 30 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

brain_query's "BM25" is raw substring counting: no IDF, no length normalisation, one-letter terms count every letter #203

New Issue
Open
opened 2026-10-12 13:05:24 +00:00 by mathias · 0 comments
mathias commented 2026-10-12 13:05:24 +00:00
Owner
Copy Link
Copy Source

Observed (2026-10-12)

The full-sentence query "Ska jag köpa aktier i företaget jag jobbar på?" missed the 2 KB note that answers it exactly (wiki/rikatillsammans/decisions/aktier-i-arbetsgivaren.md). The top 5 were 20–30 KB guides on other topics. The keyword query aktiesparprogram arbetsgivaren ranked the note first.

Cause

ingestion/internal/search/search.go scores each file as sum over terms of strings.Count(lower, term), with terms split on whitespace:

  • No length normalisation: long notes win any query with common words.
  • No IDF: jag, i and på weigh as much as aktiesparprogram.
  • Substring matching: the term i counts every letter i, på? keeps its punctuation, and ränta matches inside räntan but also inside unrelated words.

This is then RRF-fused with vector search (k=60), which hides part of the problem when embeddings exist. Notes newer than the embed-sync interval (300 s) get BM25 only.

Effect

Sentence-style questions (how humans and agents ask) systematically surface the longest notes in a wing. Measured on 24 questions: the right guide made the top 5 in 23, but often at ranks 3–5 behind longer notes.

Proposal

Real BM25: tokenise on word boundaries (unicode letters and digits), lowercase, drop a small Swedish/English stopword list, apply IDF over the corpus, and use the k1/b length normalisation. Cache the corpus statistics and invalidate on write. Add table-driven tests with a short exact-match note against a long tangential note, and a Swedish query containing stopwords.

Evidence: mathias/rt-kunskap eval/results-2026-10-12.md section 3.

Risk: MEDIUM (ranking changes for every wing; compare before and after on a fixed query set)

Filed by the rt-kunskap session (Claude Code)

## Observed (2026-10-12) The full-sentence query "Ska jag köpa aktier i företaget jag jobbar på?" missed the 2 KB note that answers it exactly (`wiki/rikatillsammans/decisions/aktier-i-arbetsgivaren.md`). The top 5 were 20–30 KB guides on other topics. The keyword query `aktiesparprogram arbetsgivaren` ranked the note first. ## Cause `ingestion/internal/search/search.go` scores each file as `sum over terms of strings.Count(lower, term)`, with terms split on whitespace: - **No length normalisation:** long notes win any query with common words. - **No IDF:** `jag`, `i` and `på` weigh as much as `aktiesparprogram`. - **Substring matching:** the term `i` counts every letter i, `på?` keeps its punctuation, and `ränta` matches inside `räntan` but also inside unrelated words. This is then RRF-fused with vector search (k=60), which hides part of the problem when embeddings exist. Notes newer than the embed-sync interval (300 s) get BM25 only. ## Effect Sentence-style questions (how humans and agents ask) systematically surface the longest notes in a wing. Measured on 24 questions: the right guide made the top 5 in 23, but often at ranks 3–5 behind longer notes. ## Proposal Real BM25: tokenise on word boundaries (unicode letters and digits), lowercase, drop a small Swedish/English stopword list, apply IDF over the corpus, and use the k1/b length normalisation. Cache the corpus statistics and invalidate on write. Add table-driven tests with a short exact-match note against a long tangential note, and a Swedish query containing stopwords. Evidence: `mathias/rt-kunskap` `eval/results-2026-10-12.md` section 3. **Risk:** MEDIUM (ranking changes for every wing; compare before and after on a fixed query set) --- _Filed by the rt-kunskap session (Claude Code)_
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/ready
Meets the agent-ready contract; authorizes unattended execution
 blocked/client
Waiting on the client
 blocked/supplier
Waiting on a supplier
 blocked/upstream
Waiting on an upstream project or vendor
 risk/high
Irreversible, security-relevant, or control plane
 risk/low
Reversible, local, test-covered
 risk/medium
Touches shared state, deploys or data
 size/l
Too big - split it
 size/m
Under two days
 size/s
Under half a day
 size/xs
Under an hour
 status:dispatch
status:dispatched
status:review
status:reviewing
type/chore
Maintenance, no behaviour change
 type/decision
Human decision required; never agent/ready
 type/docs
Documentation only
 type/feat
New capability
 type/fix
Corrects broken behaviour
 type/ops
Infrastructure, deploy, runtime
 type/research
Investigate; output is a finding, not code
 type/test
Test coverage only
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate-bot mathias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#203