mathias/hyperguild
1
0
Fork 0
Code Issues 30 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

chore(brain): backfill pgvector from current wiki/ #18

New Issue
Closed
opened 2026-05-19 11:10:30 +00:00 by mathias · 1 comment
mathias commented 2026-05-19 11:10:30 +00:00
Owner
Copy Link
Copy Source

Context

Hybrid retrieval (hyperguild#8) shipped and is active in the live pod (image 4af10364). The brain DB is bootstrapped, the embedder is wired, the sync loop runs every 300s — but the existing wiki/ corpus has never been embedded. Until backfill runs, brain_query cosine results stay empty and Reciprocal Rank Fusion silently reduces to BM25-only.

Action

curl -s -X POST \
  -H "Authorization: Bearer $BRAIN_MCP_TOKEN" \
  https://brain-mcp.d-ma.be/backfill-embeddings | jq

Expected response:

{"added": N, "deleted": 0, "errors": []}

Run from anywhere reachable — endpoint sits behind the static Bearer.

Verification

# Row count should match `find brain/wiki -name '*.md' -not -name '_index.md' | wc -l`
kubectl exec -n databases postgres18-0 -- \
  psql -U postgres -d brain -c "SELECT count(*) FROM brain_embeddings;"

# Smoke a hybrid query
curl -s -X POST -H "Authorization: Bearer $BRAIN_MCP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query":"static Bearer","limit":5}' \
  https://brain-mcp.d-ma.be/query | jq

Acceptance criteria

  • /backfill-embeddings returns {added>0, errors:[]} (or non-empty errors documented)
  • brain_embeddings row count matches the wiki/ markdown file count (minus _index.md)
  • A query whose phrasing only matches semantically (no shared keywords) surfaces the right note

Out of scope

Re-embedding on file edit — tracked separately.

## Context Hybrid retrieval (hyperguild#8) shipped and is active in the live pod (image `4af10364`). The brain DB is bootstrapped, the embedder is wired, the sync loop runs every 300s — but the existing wiki/ corpus has never been embedded. Until backfill runs, `brain_query` cosine results stay empty and Reciprocal Rank Fusion silently reduces to BM25-only. ## Action ```bash curl -s -X POST \ -H "Authorization: Bearer $BRAIN_MCP_TOKEN" \ https://brain-mcp.d-ma.be/backfill-embeddings | jq ``` Expected response: ```json {"added": N, "deleted": 0, "errors": []} ``` Run from anywhere reachable — endpoint sits behind the static Bearer. ## Verification ```bash # Row count should match `find brain/wiki -name '*.md' -not -name '_index.md' | wc -l` kubectl exec -n databases postgres18-0 -- \ psql -U postgres -d brain -c "SELECT count(*) FROM brain_embeddings;" # Smoke a hybrid query curl -s -X POST -H "Authorization: Bearer $BRAIN_MCP_TOKEN" \ -H "Content-Type: application/json" \ -d '{"query":"static Bearer","limit":5}' \ https://brain-mcp.d-ma.be/query | jq ``` ## Acceptance criteria - [ ] `/backfill-embeddings` returns `{added>0, errors:[]}` (or non-empty errors documented) - [ ] `brain_embeddings` row count matches the wiki/ markdown file count (minus `_index.md`) - [ ] A query whose phrasing only matches semantically (no shared keywords) surfaces the right note ## Out of scope Re-embedding on file edit — tracked separately.
mathias commented 2026-05-20 06:25:39 +00:00
Author
Owner
Copy Link
Copy Source

Superseded. Backfill is no longer needed:

  • Initial state at issue filing: 0 wiki/* rows in brain_embeddings.
  • After infra#37 part 1 (commit 078ec02, hyperguild v0.7.0): sync walks wiki/ + knowledge/ → 141 rows embedded.
  • After infra#38 (commit 37fdd33, hyperguild v0.8.0): chunk-before-embed covers the remaining 3 oversized files → 173 rows total, errors=0.

The /backfill-embeddings endpoint still exists for emergencies; not currently needed.

$ kubectl exec -n databases statefulset/postgres18 -- \
    psql -U brain_app -d brain -c "SELECT count(*) FROM brain_embeddings;"
 count
-------
   173

Closes.

Superseded. Backfill is no longer needed: - Initial state at issue filing: 0 wiki/* rows in `brain_embeddings`. - After infra#37 part 1 (commit `078ec02`, hyperguild v0.7.0): sync walks `wiki/` + `knowledge/` → **141 rows** embedded. - After infra#38 (commit `37fdd33`, hyperguild v0.8.0): chunk-before-embed covers the remaining 3 oversized files → **173 rows total, errors=0**. The `/backfill-embeddings` endpoint still exists for emergencies; not currently needed. ``` $ kubectl exec -n databases statefulset/postgres18 -- \ psql -U brain_app -d brain -c "SELECT count(*) FROM brain_embeddings;" count ------- 173 ``` Closes.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/ready
Meets the agent-ready contract; authorizes unattended execution
 blocked/client
Waiting on the client
 blocked/supplier
Waiting on a supplier
 blocked/upstream
Waiting on an upstream project or vendor
 risk/high
Irreversible, security-relevant, or control plane
 risk/low
Reversible, local, test-covered
 risk/medium
Touches shared state, deploys or data
 size/l
Too big - split it
 size/m
Under two days
 size/s
Under half a day
 size/xs
Under an hour
 status:dispatch
status:dispatched
status:review
status:reviewing
type/chore
Maintenance, no behaviour change
 type/decision
Human decision required; never agent/ready
 type/docs
Documentation only
 type/feat
New capability
 type/fix
Corrects broken behaviour
 type/ops
Infrastructure, deploy, runtime
 type/research
Investigate; output is a finding, not code
 type/test
Test coverage only
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
renovate-bot mathias
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mathias/hyperguild#18