Compare commits

...
Author SHA1 Message Date
mathiasandClaude Opus 4.8 f04b03e07e chore(context): re-sync derived adapters after root rule-0 update
CI / Lint / Test / Vet (pull_request) Successful in 13s
CI / Mirror to GitHub (pull_request) Has been skipped
context-sync regenerated the adapters from the updated root AGENT.md
(rule 0 pre-task ritual + TDD constraint). The committed adapters had
drifted; this is the documented `task check` remedy, not a content
change in this repo.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 08:25:16 +02:00
mathiasandClaude Opus 4.8 6c61f93146 feat(mcp): register brain_update + brain_get, extend brain_write handle
Wires the #45 verbs into the MCP surface (all three sites: tools()
descriptors, handleCall dispatch, package doc comment).

- brain_update: supersede-by-slug or full path; rebuilds wing _index and
  re-tunnels cross-wing matches against the new body (idempotent,
  best-effort), re-indexes the graph, returns {id, path, content_hash,
  superseded}.
- brain_get: fetch by id or path (both are the brain-relative handle);
  returns {id, path, content_hash, frontmatter, body}.
- brain_write: return contract extended from {path} to {id, path,
  content_hash} — path kept for backward compat — so the create path
  also yields a stable handle.

id == relPath; content_hash == sha256 of the file bytes. Tests cover the
supersede happy path, missing-target error + no-create, get by id/path,
write handle, and an end-to-end re-embed test that drives the real
vectorstore.Sync re-index after an update.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 08:25:10 +02:00
mathiasandClaude Opus 4.8 95a69fc2c1 feat(brain): add UpdateNote/ReadNote supersede primitives + frontmatter editor
Implements the api-layer half of #45. UpdateNote supersedes a note in
place (whole-note body replace, frontmatter re-stamp: updated_at,
supersedes=prior content hash, supersede_reason), preserving created_at,
wing, hall, and any custom fields. Never creates — a missing target is
an error so callers fall back to brain_write. ReadNote is the read-after-
write primitive (frontmatter + body + content_hash). ContentHash is the
sha256 handle that round-trips write/update → get.

Frontmatter is edited via a line-preserving ordered editor rather than a
yaml.v3 round-trip, which would reorder keys and strip comments — the
brain writes flat key:value frontmatter by hand.

Embeddings are not refreshed here: the rewritten file's mtime advances,
which the out-of-band vectorstore.Sync ticker uses to re-embed it — the
same mechanism brain_write relies on.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 08:25:00 +02:00
mathiasandClaude Opus 4.8 bb8bc0478c chore(cd): use git.d-ma.be for SSH alias + INFRA_REPO (post-rename consistency)
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 4s
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 00:19:42 +02:00
mathiasandClaude Opus 4.8 a961a3c064 fix(vectorstore): hard-split oversized heading-less chunks
A doc with no headings and no blank-line paragraphs (JSON-lines, e.g.
wiki/telos/decisions/human-intent-column.md) survived both chunk passes whole
and was sent to nomic-embed over its context window → 'input length exceeds
the context length' (400, the steady embed errors=1). Add a final hard-split
pass (line then UTF-8 rune boundaries) so no chunk exceeds maxBytes. TDD.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 00:19:42 +02:00
mathiasandClaude Opus 4.8 bec28f9014 fix(cd): point registry/patch/verify refs at git.d-ma.be after host rename
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 3s
The gitea.d-ma.be→git.d-ma.be rename moved the infra deployment manifests, but
cd.yml still sed-patched 'gitea.d-ma.be/mathias/ingestion:' — which no longer
matches, so the patch produced no change and 'git commit' failed under set -e,
wedging all deploys (image stuck at e8dbcf6). Repoint the registry image refs,
the infra-patch sed patterns, and the rollout-verify EXPECTED values to
git.d-ma.be (same registry backend). SSH alias + INFRA_REPO left as-is.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 23:22:07 +02:00
mathiasandClaude Opus 4.8 b62ac57382 fix(brain_answer): reranker is a filter, not a gate — fall back to BM25 when it keeps nothing
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 3s
The Qwen3-Reranker is a web-search cross-encoder. Against conversational /
personal-intent queries (e.g. 'what am I optimizing toward?') it scores every
candidate as 'no', so brain_answer collapsed to 'No relevant content found'
even though BM25 had retrieved on-topic notes (incl. the telos wing). Treat
the reranker as a filter: when it keeps zero results, fall back to the
BM25/vector ordering (capped to the no-reranker depth of 10). Refs brain#11.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 22:53:02 +02:00
mathiasandClaude Opus 4.8 aa918388b9 docs(brain): add two-column intent merge scaffold
CI / Lint / Test / Vet (push) Successful in 14s
CI / Mirror to GitHub (push) Successful in 4s
Drop-in unified findings doc for the brain-MCP intent study. Locks the
shared row schema + closed intent vocab both columns must conform to.
Agent column filled from agent-intent-column.jsonl (46 acts, 37%
mismatch); human column left as PENDING cells + <<SYNTH>> blocks so the
Claude.ai-history analysis merges in without re-deriving structure.

Pre-seeds the cross-consumer divergence questions: agent mismatch is
write-side-heavy (supersede + verify-landed); hypothesis is human
mismatch is read-side-heavy (semantic + answer) — interface may fail the
two consumers at opposite ends.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 20:45:24 +02:00
mathiasandClaude Opus 4.8 e8dbcf6eef docs(brain): add agent-consumer brain-MCP intent analysis
CI / Lint / Test / Vet (push) Successful in 14s
CI / Mirror to GitHub (push) Successful in 4s
Agent-consumer column of the two-part brain intent↔interface study.
Reconstructs the knowledge-act behind every brain MCP call in the
Claude Code agent transcripts on koala (the only corpus with brain
calls; brain/sessions and agentsquad eval logs carry none).

46 distinct knowledge-acts, 37% interface mismatch, 0% intent_unclear.
Headline gap: no update/supersede verb → agents blind re-write same
slug (5x); no read-after-write → lexical re-query of own note (4x);
lexical-only reads → semantic-as-keyword-stuffing chains (3x);
brain_answer hedged with parallel brain_query.

Canonical schema brain-intent-extraction.md absent on host; vocab
reconstructed, every row tagged schema_source=reconstructed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 19:44:59 +02:00
mathiasandClaude Opus 4.8 0eeb1df4a2 fix(claudewatcher): scrub bare 1Password service-account tokens (ops_)
CI / Lint / Test / Vet (push) Successful in 17s
CI / Mirror to GitHub (push) Successful in 4s
A ~/.zshrc read surfaced OP_SERVICE_ACCOUNT_TOKEN into a transcript. The
_TOKEN= form was already caught by homelab-env-token, but a bare ops_<b64>
value was not. Add an op-service-account rule (ordered early). Tests cover
both env-assigned and bare forms.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 00:28:47 +02:00
mathiasandClaude Opus 4.8 9febb1bba1 fix(claudewatcher): harden secret scrubber against \b evasion + add JWT
CI / Lint / Test / Vet (push) Successful in 17s
CI / Mirror to GitHub (push) Successful in 4s
A shell mangle that glued a key to a preceding word ('yes'+'sk-...') had
no word boundary, so the leading \b in the openai-sk rule failed to match
and a LiteLLM master key leaked past the scrubber into ingest (2026-06-11).

- openai-sk: drop leading \b, match sk- shape anywhere ({32,} floor keeps
  short task-/disk- words clean).
- add jwt rule for bare header.payload.sig tokens (no Bearer prefix).
- regression tests: the exact yessk- evasion, standalone sk-, bare JWT,
  plus clean-content guards for task-/disk-.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 08:55:33 +02:00
mathias 5dc247b994 fix(ci): quote "on" key so Gitea parses workflow triggers
CI / Lint / Test / Vet (push) Successful in 17s
CI / Mirror to GitHub (push) Successful in 4s
Bare on: parses as YAML boolean true (Norway problem); Gitea then ignores the triggers and silently skips jobs. Quoting forces the string key.
2026-06-03 08:38:46 +02:00
mathias 2125558196 docs: extend harness boundary decision to cover Crush as third harness
CI / Mirror to GitHub (push) Successful in 4s
CI / Lint / Test / Vet (push) Successful in 12s
2026-05-28 11:38:36 +00:00
mathias 2beaac2feb docs: add 2026-05-28 decisions — harness boundary + field benchmark definition
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 3s
2026-05-28 10:31:41 +00:00
mathias 525811bc1a docs: add hypothesis statement and harness boundary clarification
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 4s
2026-05-28 10:30:39 +00:00
mathias bad0581623 merge: client-name scrubber rule (refs hyperguild#27)
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 4s
2026-05-26 07:10:05 +02:00
mathias a94b860c2e feat(claudewatcher): client-name guard via RegisterRule + env
Pre-rollout guard. Source code stays clean — client identities come
from CLAUDE_INGEST_CLIENT_BLOCK env (sourced from a SOPS-encrypted k8s
secret in infra repo). Env value is a regex alternation; main wraps
it with `(?i)\b(...)\b` so word-boundary matching avoids false hits
inside longer identifiers (e.g. "Sebastian" doesn't trigger on "SEB").

DefaultRules (credential shapes) still take precedence so any leak
that's BOTH a client mention AND a credential shape logs as the
credential — strictly more dangerous, points triage at the right
thing. Tests cover precedence + case variations + word-boundary
respect + invalid-pattern rejection.

Refs: infra#73 Track E.1 pre-rollout grill (option B).

Bump-Type: minor
2026-05-26 07:10:05 +02:00
mathias f8cf27e5de merge: claudewatcher (closes hyperguild#27, refs infra#73)
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 4s
2026-05-25 19:59:13 +02:00
mathias 49b188e9c9 feat(server): wire claudewatcher behind CLAUDE_SESSIONS_DIR
Opt-in by setting CLAUDE_SESSIONS_DIR to the ~/.claude/projects path.
When set, the server starts claudewatcher.Watch in a goroutine that
ticks every CLAUDE_INGEST_INTERVAL seconds (default 60). Requires
BRAIN_PG_DSN for the cursor table — fail-fast if missing.

Each Batch becomes one wiki note at:
  brain/wiki/claude-sessions/facts/session-<host>-<session_id>.md

with frontmatter type=source + domain=<project basename>. Per-turn
content capped at 2000 chars (full transcripts stay in
~/.claude/projects already); the brain entry is a digest, not a
mirror.

CLAUDE_INGEST_HOST overrides the os.Hostname()-derived host label,
useful when multiple ingestion pods consume the same DSN from
different machines.

Closes hyperguild#27.

Bump-Type: minor
2026-05-25 19:59:07 +02:00
mathias bc011cc1f0 feat(claudewatcher): ingest Claude Code session transcripts into brain
New package internal/claudewatcher. The volume gate (24 turns/week of
agentsquad logs vs 500/week gate) exposed that the real signal lives
in daily Claude Code usage at ~/.claude/projects/*/<uuid>.jsonl, not
in agentsquad output. This package captures that signal. See infra#73
Track E + hyperguild#27 for the full reframe.

Components:
- parser: tolerant JSONL parser over the observed Claude Code session
  schema (user / assistant / attachment / system + bookkeeping types).
  Skip-flag fast-paths queue-operation, last-prompt, permission-mode,
  ai-title, bridge-session, file-history-snapshot.
- scrubber: 11-rule fail-closed regex set for credential shapes
  (bearer, postgres URIs, PEM, ssh-key, ghp_/sk-/sk-ant-/AKIA, homelab
  env tokens, SOPS markers). Drop turn + log on match.
- cursor: postgres-backed claude_session_cursors table, keyed by
  (host, file_path) with byte_offset. Resumable across pod restarts.
- watcher: poll loop. Walks SessionsDir, processes each .jsonl from
  its cursor offset, runs scrubber, emits a Batch per file to a
  Sink interface, advances cursor on successful Ingest.

No classifier integration in this commit — every kept turn is emitted
in a per-session batch. The cmd/server wiring (next commit) routes
batches to brain/wiki/claude-sessions/facts/. Classifier-driven hall
routing (decisions / failures / hypotheses) is a follow-up.

19 unit tests across parser + scrubber + watcher. task check green.

Refs: infra#73, hyperguild#27
2026-05-25 19:58:58 +02:00
mathiasandClaude Opus 4.7 2726896079 feat(mcp): wire brain_context tool
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 4s
Returns top-N relevant brain entries for a project context. Combines
BM25 hits on project name with 2-hop graph expansion via Track A's
graphstore (when BRAIN_GRAPH_ENABLED). Closes hyperguild#28.

Notes on implementation choices that deviate slightly from the spec:
- Excerpt length: 200 chars per spec (vs the 300 used by search.Result).
  truncateExcerpt clamps the already-stripped BM25 excerpt; graph-only
  neighbours load their excerpt from disk via a private readExcerpt
  helper (search.hydrate is unexported).
- Graph scoring: 0.6 / max(1, distance) per neighbour, so distance-1
  contributes 0.6 and distance-2 contributes 0.3. BM25 hits decay
  linearly from 3.0 (rank-0) to 1.0 (rank-2), giving BM25 hits a
  natural ceiling above pure-graph hits while still letting a doc
  surfaced via both edge types outrank a BM25-only one.
- Test placement: package mcp (internal) rather than mcp_test, because
  graphReader is unexported and WithGraph only accepts *PGStore; an
  internal test can install a dual-interface fake directly on s.graph
  without spinning up postgres.

Bump-Type: minor

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 18:53:14 +02:00
mathias 2b7bbe38c7 docs(eval): record M4 + M4b scorer runs — phase 2 gate cleared (infra#72)
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Successful in 4s
Tier-weighted retrieval against the qa-2026-05.md 20-question set:

| run                            | top-1 | top-3 |
|--------------------------------|-------|-------|
| baseline (pre-phase-1)         | 20%   | 65%   |
| post phase 1 (parser+content)  | 20%   | 70%   |
| post M4 (tier weighting)       | 30%   | 75%   |
| post M4b (entities → K tier)   | 35%   | 80%   |

Net Phase 2 lift: +15pt top-1, +15pt top-3 — comfortably above the
≥10pt close-gate set in infra#72.

Three remaining misses are content-keyword issues, not structure
issues (the questions don't share enough lexical surface with the
target entries to surface via BM25 alone). Vector search would
help here but the iguana embedder is off-mesh (see infra#64).
2026-05-25 18:51:29 +02:00
mathias 1b00cbc0ae fix(search,graph): M4b wiki/entities/ → tier=knowledge
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 3s
Initial M4 mapping put wiki/entities/* in tier=note. Post-M4 eval
regressed qwen35-9b-fast from rank 2 → off top-5: knowledge entries
that cite the entity in passing now outscore the entity page itself
(1.5× weight vs 1.0×).

Entity anchor pages are durable facts about concrete things — they
map cleanly to the knowledge/facts/ slot in the post-M3 layout
target. Promote them now so the path inference matches.

Eval re-run after deploy is in infra#72.
2026-05-25 18:49:37 +02:00
mathias 4f78fecd06 feat(search): M4 tier-weighted BM25 re-rank (infra#72)
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 3s
The eval set under brain/eval/qa-2026-05.md showed BM25 top-1 at 20%
with 5 of the missing slugs being short focused knowledge entries
that lost to long aggregate docs on raw term-frequency. Tier weighting
addresses that without touching the BM25 algorithm itself.

How

- Result struct gains a Tier field, populated during the file walk
  via extractTier (frontmatter wins, path prefix as fallback —
  mirrors the graph.inferTierFromPath logic so the two callers stay
  in lockstep).
- After the existing sort (and optional hybridMerge), do a final
  stable re-sort by float64(Score) * tierWeight(Tier). Knowledge
  ×1.5, note ×1.0, inbox ×0.3, unknown ×1.0.
- hydrate() (vector-only hits) also fills Tier so re-ranking covers
  the hybrid path.

Test covers the load-bearing case: a long note-tier doc with raw=10
loses to a short knowledge-tier doc with raw=8 after weighting
(8×1.5=12 vs 10×1.0=10).

Measurement gate is in infra#72: re-run brain/eval/score.py against
the live brain after this image lands; close the issue when top-1
hit rate lifts by ≥10 absolute points.
2026-05-25 18:45:20 +02:00
mathias d5f112b600 feat(graph,graphstore): M2 parse tier+topic from frontmatter, persist via Upsert (infra#72)
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 4s
extract.go now reads `tier:` and `topic:` from YAML frontmatter, with
a path-based fallback when frontmatter is absent (the pre-M3 state on
every existing entry):

  knowledge/* → tier=knowledge
  notes/*     → tier=note
  wiki/**     → tier=note   (sources + concepts + entities are I-level)
  inbox/**, raw/**, sessions/**, clips/** → tier=inbox

Frontmatter wins when present — covers the M3-migrated case where an
entry's path may not match the tier the author chose for it.

UpsertEntity persists both columns. M1's schema already has them.

Backfill on next pod start populates tier for the whole corpus
without any file moves; M3 will follow up with the actual layout
migration and explicit frontmatter writes.
2026-05-25 12:35:38 +02:00
mathias ea9518e712 feat(graphstore): M1 add tier + topic columns to brain_entities (infra#72)
CI / Lint / Test / Vet (push) Successful in 15s
CI / Mirror to GitHub (push) Successful in 3s
Schema-only change. DDL adds tier + topic on fresh tables and uses
ADD COLUMN IF NOT EXISTS on existing tables (idempotent across pod
restarts). New conditional indexes match the wing/hall pattern.

No behavior change in this commit — UpsertEntity still writes only
the original columns; tier + topic stay '' on every row. M2 plumbs
the parser through. The empty default means existing queries are
untouched until the rest of the chain lands.

Part of infra#72 — brain DIKW tier redesign.
2026-05-25 07:17:39 +02:00
mathias e34cd6c12b docs(eval): record post-fix scorer run — phase 1 lift insufficient
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 4s
Top-1 stayed at 20% (4/20), top-3 +5pt (65→70%) after:
- extract.go wing/topic parser fix (commit 3084c41)
- qwen35-9b-fast entity pad (was 239-byte stub → full entity)
- grafana entry: add "pod restart" synonym to lesson body
- dangling refs stripped from index.md + entities/k3s.md

The only retrieval move: qwen35-9b-fast climbed from rank 0 (off top-5)
to rank 2 — the entity pad worked. Other 5 misses are ranker behaviour
on already-keyword-overlapping entries; BM25 doesn't weight the right
slugs to the top.

Per the proposal's gate (≥10pt lift = stop, <10pt = Phase 2 justified),
the DIKW tier redesign earns its cost. Next session: tier column +
file moves + tier-weighted retrieval, then re-measure against this
same eval set.
2026-05-24 22:48:48 +02:00
mathias 3084c4173d fix(graph): route wiki/<flat>.md to Type=knowledge, not Type=hall with filename-as-wing
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 4s
classifyByPath had a hole: paths like wiki/index.md or wiki/<slug>.md
(direct children of wiki/, no subdirectory) hit the default branch and
wrote Wing=parts[1] — which IS the filename, not a wing. Symptom in
brain_entities: rows like (slug=index, wing=index.md) and
(slug=autobe-..., wing=autobe-evaluation-pattern-....md).

Fix: when len(parts) < 3 (no subdirectory at all), fall through to
Type=knowledge and let frontmatter set wing/hall if present.

Add brain/eval/ artifacts at the same time:
- qa-2026-05.md — 20 hand-authored Q→expected-slug pairs covering the
  homelab knowledge corpus across mcp, dex, gitops, postgres, go,
  models, methodology
- score.py — calls brain_query for each pair, scores top-1 + top-3,
  emits per-question detail. BRAIN_MCP_TOKEN via env.

Pre-fix baseline against the live brain: top-1 = 20% (4/20),
top-3 = 65% (13/20). Six hard misses where the expected slug doesn't
even land in the top-5.

Used to gate the phase 2 DIKW redesign (infra#62 follow-up): if
phase 1 fixes (this parser fix + 20 backlink authoring on top
orphans) lift top-1 by <10 absolute points, structure is the
bottleneck and the tier redesign is justified.
2026-05-24 22:33:04 +02:00
mathias 72be87b4e7 chore(routing): flip LITELLM_BASE_URL default to https://llm-api.d-ma.be
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 3s
Follow-up to infra#70. LiteLLM moved off piguard into k3s and the
public llm-api.d-ma.be hostname now upstreams to koala:30401. The
piguard:4000 default in the source bit-rots — works today because
piguard:4000 is still alive during the 7-day soak, breaks the moment
the compose comes down.

Pointing the default at the public hostname survives the cutover
without needing a follow-up. Production deploys via k3s already
override via env (in-cluster Service DNS) so this only affects local
dev shells without LITELLM_BASE_URL set.

- internal/config/routing.go: comment + envOr fallback
- internal/config/routing_test.go: expected value in defaults test
- scripts/smoke-routing.sh: shell default

task check: clean (tests + vet + govulncheck).
2026-05-24 15:06:23 +02:00
mathiasandClaude Opus 4.7 153ef6ccac feat(graph): GraphRAG augment brain_answer with top-hit subgraph
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 3s
Commit 4 of Track A — the no-shelfware close-out the grill demanded.
brain_answer now folds the 1-hop outgoing neighbourhood of its top
BM25/rerank hit into the LLM's context as a <related> block when
BRAIN_GRAPH_ENABLED is on. With the flag off the prompt is byte-for-
byte identical to the pre-Track-A behaviour, so existing tests still
pass without modification.

The hop list contains slug, edge_type, doc_path — no extra retrieval
pass, no second LLM call, no file reads. The model can ignore the
block when irrelevant; when it adds signal we get GraphRAG for free.

Refs: docs/superpowers/specs/2026-05-homelab-training-graph-next-step.md
in infra repo + grill addendum item "Track A: GraphRAG wiring into
brain_answer is mandatory in same commit chain (no shelfware risk)".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 15:24:45 +02:00
mathiasandClaude Opus 4.7 2148565ee6 feat(mcp): expose brain_graph tool — neighbors, subgraph, path
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 4s
Commit 3 of Track A. The MCP server now publishes a new tool that
opens the brain knowledge graph (entities + wikilink edges) for
external consumers (claude.ai connectors, gitea-mcp, agentsquad).

- tools_graph.go: brain_graph handler dispatches by op:
    neighbors  — 1-hop outgoing from slug, optional edge_type filter
    subgraph   — every reachable slug within depth hops (≤6)
    path       — shortest directed path src→dst within depth (≤8)
  Returns slug + entity metadata + edge_type + hop distance.

- server.go: handleCall routes "brain_graph" to brainGraph.

- handlers.go: tool descriptor with the op enum + per-op required
  fields documented in the description.

- server_test.go: TestServerToolsList expects brain_graph in the
  listing.

The tool returns an error when BRAIN_GRAPH_ENABLED is unset — same
shape as brain_answer when the answer LLM is unconfigured.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 15:23:18 +02:00
mathiasandClaude Opus 4.7 f43e0bccbf feat(graph): wire graphsync into MCP write/ingest/tunnel handlers
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 4s
Commit 2 of Track A. Service stays a no-op until BRAIN_GRAPH_ENABLED=
true; flipping it on creates the schema (idempotent), starts indexing
every successful write, and optionally backfills the existing brain
dir.

- internal/graphsync: best-effort wrapper around graph.Extract +
  graphstore. IndexDoc reads docPath under brainDir, parses, upserts
  entity + replaces edges. BackfillFromBrainDir walks wiki/ +
  knowledge/. Both are no-ops on nil store so callers wire
  unconditionally.

- mcp.Server gains WithGraph builder + graphsync.Store field.
  brain_write, brain_ingest, brain_ingest_raw, brain_tunnel call
  indexInGraph after success — failures slog.Warn but never
  propagate (graph is augmentation, not correctness).

- cmd/server gates the wiring on BRAIN_GRAPH_ENABLED=true (default
  off so first rollout doesn't surprise). BRAIN_GRAPH_BACKFILL=true
  triggers a one-shot walk of the brain dir on boot.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 15:21:33 +02:00
mathiasandClaude Opus 4.7 f53ee18cb6 feat(graph): add brain_entities + brain_edges store and wikilink parser
CI / Lint / Test / Vet (push) Successful in 12s
CI / Mirror to GitHub (push) Successful in 3s
Foundation for Track A (GraphRAG on top of existing wiki). Two new
packages, both unwired — service behaviour unchanged until commit 2
hooks the pipeline.

- internal/graph: pure parser. Extract() walks markdown + frontmatter
  and emits one Entity + N wikilink Edges per doc. Dedupes per (dst,
  line), ignores self-references, classifies hall/concept/entity/
  source/knowledge from path layout.

- internal/graphstore: pgx-backed PGStore mirroring vectorstore's
  shape. Idempotent Init() creates brain_entities + brain_edges with
  indexes on src_slug, dst_slug, src_doc, wing, type. Operations:
  UpsertEntity, ReplaceEdgesForDoc (tx), DeleteByDoc, Neighbors,
  Subgraph (recursive CTE, depth ≤6), Path (shortest path, depth ≤8).

Schema lives on the shared postgres18 instance alongside the
brain_embeddings table — no new datastore. See
docs/superpowers/specs/2026-05-homelab-training-graph-next-step.md
in infra repo + infra#62.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 15:18:08 +02:00
mathiasandClaude Opus 4.7 c153e9105c ci: retrigger build after chassis repo made public
CI / Lint / Test / Vet (push) Successful in 13s
CI / Mirror to GitHub (push) Successful in 3s
Same reason as gitea-mcp ci retrigger commit — mcp-chassis was created
private; the ingestion port (commit ca22df2) couldn't fetch it in CI.
Chassis is now public; this empty commit retriggers the Build and deploy
pipeline.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 12:17:55 +02:00
mathiasandClaude Opus 4.7 ce96a6a571 fix(ci): allow ingestion Dockerfile to fetch internal gitea modules
CI / Mirror to GitHub (push) Successful in 4s
CI / Lint / Test / Vet (push) Successful in 12s
Same fix as gitea-mcp commit for the same reason — mcp-chassis (added
in commit ca22df2) is hosted at gitea.d-ma.be and Gitea returns http://
in its go-import meta tag, breaking the default go module resolution
inside the Docker build.

GOPRIVATE+GOPROXY=direct+GOSUMDB=off plus a git config insteadOf rewrite
to flip http:// → https:// for gitea.d-ma.be clones.

Without this, hyperguild CI Build and deploy failed on the chassis
port (sha=ca22df2). Reapplying CI should now succeed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 12:12:09 +02:00
mathiasandClaude Opus 4.7 ca22df2d6a feat(ingestion): migrate to gitea.d-ma.be/mathias/mcp-chassis v0.1.0
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Successful in 3s
Second port of the MCP chassis (gitea-mcp was first, commit 658f4ba).
Closes the chassis-adoption loop on the two highest-LOC consumers.

Changes:
- Drop ingestion/internal/auth/ entirely (jwt.go + jwt_test.go +
  protected_resource.go + protected_resource_test.go) — chassis provides
  JWTValidator + ProtectedResourceHandler with identical semantics.
- Drop ingestion/internal/mcp/auth.go (BearerAuth function, ~65 LOC)
  and the integration test auth_test.go (~200 LOC) — chassis
  BearerMiddleware replaces it. Static-Bearer-or-Dex-JWT precedence and
  RFC 9728 resource_metadata challenge behavior preserved 1:1.
- cmd/server/main.go: import chassis as `chassisauth`, rewire the three
  call sites. Use realm="brain" in the BearerMiddleware call so a 401
  challenge identifies the resource as the brain MCP.

OAuth client_credentials handler (ingestion/internal/oauth) stays —
chassis v0.1.0 covers only the JWT path; OAuth flow is a candidate for
chassis v0.2.0 once a second MCP needs it (rule of three).

Net delta: -~330 LOC of duplicated auth code; +1 import; +1 GOPRIVATE
env requirement on dev machines (documented in the spike handoff
2026-05-22-mcp-chassis-spike.md).

task check green (lint + test + vet + govulncheck).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 10:43:11 +02:00
mathiasandClaude Opus 4.7 e49b36e463 feat(ingestion): expose Prometheus /metrics for brain query latency
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Successful in 3s
Closes infra#50.

Adds an internal/metrics package with a hand-rolled Prometheus
exposition layer (stdlib + sync/atomic only — no new dep) and wraps the
HTTP mux with a timing middleware. Every request emits one observation
on the `brain_query_duration_seconds` histogram labeled by
`path` (request Pattern, low cardinality) and `status` (2xx/3xx/4xx/5xx).

Dependency choice: hand-rolled rather than github.com/prometheus/client_golang
because the surface needed is small (one histogram + bucket constants)
and the repo CLAUDE.md keeps deps stdlib + jwx + testify only. ~150 LOC
of code + tests is cheaper than the chart of transitive prometheus deps.

Endpoints:
- GET /metrics  — OpenMetrics text exposition, no auth (cluster-internal)

Wire format pinned by tests in internal/metrics/metrics_test.go. The
ServiceMonitor that drives the kube-prometheus-stack scrape lives in
infra/k3s/apps/supervisor/ (separate commit on mathias/infra).

After this image deploys, the canary alert from
docs/superpowers/specs/2026-05-homelab-architecture-review.md becomes
wireable:

  histogram_quantile(0.95,
    sum(rate(brain_query_duration_seconds_bucket[5m])) by (le))
    > 1.5 * histogram_quantile(0.95,
        sum(rate(brain_query_duration_seconds_bucket[5m] offset 7d)) by (le))

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 07:13:05 +02:00
mathias 815739758e feat(vectorstore): re-embed on file mtime > store updated_at (#23)
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Has been skipped
Removes the TODO in Sync that left files static after their first embed.
Edits to brain/wiki/ and brain/knowledge/ now surface in subsequent
syncs without manual /backfill-embeddings calls.

Approach
- Store interface: KnownPaths → KnownPathsWithTime returning path →
  updated_at. Callers compare against file mtime to detect edits.
- PGStore: SELECT path, updated_at FROM brain_embeddings.
- Sync groups known chunks by parent path and tracks the EARLIEST
  updated_at per parent. A file is stale when its mtime is after that
  oldest chunk's timestamp — any chunk older than the file means at
  least one chunk hasn't been refreshed since the last edit.
- Stale-path rewrite: delete every old chunk for the parent (handles
  "file shrunk → fewer chunks → orphan rows at higher #NNNN" cleanly),
  then re-chunk + re-embed + re-upsert.

Tests
- New: TestSync_ReembedsFileWhenMtimeNewer — file mtime forced into the
  future vs store updated_at; Sync deletes old chunk + upserts fresh one.
- New: TestSync_SkipsFileWhenMtimeOlder — file mtime backdated; Sync is
  a no-op (no upserts, no deletes).
- Updated: stubStore.known is now map[string]time.Time. A zero value
  resolves to a far-future sentinel so existing "skip if already known"
  tests keep passing without per-test setup.
- pg_test renamed KnownPaths integration → KnownPathsWithTime; asserts
  updated_at is non-zero and within 5s of insert wall-clock.

Backward compat
- brain_embeddings rows pre-dating this change carry valid updated_at
  values (column was always populated via `DEFAULT now()` + ON CONFLICT
  `updated_at = now()`). No migration needed. Live pod will start
  re-embedding any file whose source has been edited since its chunks
  were originally written.

Closes gitea/mathias/hyperguild#23.
2026-05-20 09:50:45 +02:00
mathias 6f1cb53295 feat(project_create): mirror_to_github opt-in, default false (infra#34 ADR)
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Has been skipped
Per the Gitea-as-true-master ADR (infra#34), GitHub mirror is now an
explicit opt-in via mirror_to_github=true. Default (omit / false) provisions
a Gitea repo + staging namespace + experiment-brief issue only — no GitHub
repo, no push-mirror.

Rationale: US cloud providers (Microsoft/GitHub) are subject to CLOUD Act
and NSL. Client code, business logic, and infra-adjacent repos should
never live on US-owned infrastructure. Only open-source projects intended
for public community (hyperguild, gitea-mcp, template-*) should opt in.

Changes
- internal/skills/project/handlers.go
  - createArgs gains MirrorToGitHub bool (json:"mirror_to_github,omitempty").
  - res.GitHubURL is set only when MirrorToGitHub is true; empty string otherwise.
  - Steps 2 (create_github_repo) + 3 (mirror) are wrapped in `if args.MirrorToGitHub`.
  - experimentBrief renders "Gitea-only" line by default and the existing
    "Push-mirror configured" line only on opt-in.
- internal/skills/project/skill.go
  - Tool schema gains mirror_to_github (boolean, default false) with description
    spelling out when to opt in. Tool Description updated to reflect new default.
- internal/skills/project/handlers_test.go
  - Added mirroredArgs() helper (happyArgs + mirror_to_github:true).
  - Tests that exercise the GitHub flow (HappyPath, GitHubExists_Idempotent,
    GitHubFails, NoGitHubClient_DegradedMode, Idempotent_RepoExists,
    MirrorFails, InfraCommitFails) switched to mirroredArgs.
  - Added TestProjectCreate_DefaultSkipsGitHubMirror covering the Gitea-only
    path: 3 gitea-mcp calls, zero GitHub calls, empty github_url, reached=
    [create_repo, infra_commit, issue], body reflects Gitea-only.

Closes gitea/mathias/hyperguild#17. Moves infra#34 acceptance item
"project_create updated: mirror_to_github defaults to false".
2026-05-20 08:35:02 +02:00
mathias 37fdd33b2d feat(ingestion): chunk markdown before embedding (#38)
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Has been skipped
Long markdown files (>~8KB) silently failed to embed because nomic-embed-text
on iguana has a 2048-token context. embed sync logged errors=1 every cycle
with no useful body until #37 added per-item logging — three files exceed
the ceiling: finbert source (8 KB), koala-machine-state (7.1 KB),
litellm-absorption (8.8 KB). Curated knowledge entries should never be
vector-blind.

Approach: chunk-before-embed, no schema change.

vectorstore/chunk.go (new)
- ChunkMarkdown splits at H1/H2 boundaries; sections over maxBytes are
  further split at paragraph boundaries, packing greedily under budget.
- NumberChunks assigns "<parent>#NNNN" storage paths (1-based, zero-padded
  to 4 digits — handles files with up to ~10k sections in stable sort order).
- ParentPath strips the chunk suffix for retrieval-side dedup.

vectorstore/sync.go
- After ChunkMarkdown produces N pieces, each is embedded + upserted as a
  separate brain_embeddings row at "<parent>#NNNN". maxChunkBytes = 4000
  (≈1000 nomic tokens, well under the 2048 ceiling with headroom for
  unicode/code blocks).
- "Already embedded?" check now reduces known paths to parent set via
  ParentPath, so the first chunk hit short-circuits the file.
- Delete walk also reduces via ParentPath; when a parent file disappears,
  every chunk row (and any pre-existing bare-path row, for backward
  compatibility with rows written before this change) gets dropped.

search/search.go
- hybridMerge collapses chunk-path vector hits to parent via ParentPath
  before scope check, RRF accumulation, and hydration. A file with three
  chunk hits returns one result row, not three.

Backward compatibility: pre-existing bare-path rows in brain_embeddings
keep working — ParentPath returns them unchanged, knownParents handles
them as if they were "wiki/foo.md#NNNN" hits, sync skips re-embed, and
search dedup is a no-op for them. No migration required to ship.

Tests:
- chunk_test.go covers short / heading split / oversized section /
  content preservation / chunk numbering / parent-path stripping.
- sync_test.go adds long-file chunking, single-chunk-row short file,
  skip-if-any-chunk-known, delete-all-chunks-of-disappeared-file.
  Existing tests updated for #NNNN paths.
- search_test.go adds chunk-paths-dedupe-to-parent.

Closes gitea/mathias/infra#38.
2026-05-19 21:57:09 +02:00
mathias 078ec029da fix(ingestion): embed sync also scans brain/knowledge/ + logs per-item errors
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Has been skipped
The embed sync goroutine only walked brain/wiki/. brain/knowledge/ (112
curated entries, per CLAUDE.md the most-important brain content) had zero
coverage in brain_embeddings — vector retrieval was blind to it. Hybrid
BM25 + pgvector retrieval would never surface a curated knowledge entry
via the vector arm.

Extract the per-root walk into a loop over a small subdir list and add
"knowledge" alongside "wiki". scanDirs is package-level so it stays a
single source of truth for what gets embedded.

Also log each failing item's path + error string from StartSync.
Previously only the aggregate count was logged, so a persistent
`errors=1` per cycle was opaque. With per-item warnings, the actual
ollama "input length exceeds the context length" surface immediately.

Refs gitea/mathias/infra#37 (this commit covers the knowledge/ scan
bug; the long-file chunking bug is a separate change.)
2026-05-19 21:27:15 +02:00
mathias 4af1036423 fix(ingestion): redact password from BRAIN_PG_DSN log line
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Successful in 4s
The previous "crude redaction" — pgDSN[:strings.IndexByte(pgDSN+"@", '@')] —
sliced up to the `@` character, which sits *after* the password in a
postgres URL, so the log line included the password in plaintext (caught
on first activation, 2026-05-18 startup log).

Use url.Parse + URL.Redacted() instead. Falls back to "postgres://***"
if parsing fails — we never log a raw DSN.
2026-05-19 13:04:12 +02:00
mathias 7a13c75655 fix(scripts): brain-embeddings-init.sql psql-level conditionals
CI / Lint / Test / Vet (push) Successful in 24s
CI / Mirror to GitHub (push) Successful in 3s
CREATE DATABASE doesn't work inside a DO $$ ... $$ block (transactional
restriction). And psql `:'var'` substitutions resolve client-side, so
they can't reach inside a DO block either.

Replace both DO blocks with psql-native idioms:
- `\gexec` for the conditional CREATE DATABASE
- `\if` + `\gset` for the create-or-rotate-password branch on the
  brain_app role

Verified end-to-end on koala postgres18: brain DB created, vector
0.8.1 extension installed, brain_app role login works.
2026-05-18 23:28:56 +02:00
mathias 57462b52ff feat(brain): hybrid BM25 + pgvector retrieval (opt-in)
CI / Lint / Test / Vet (push) Successful in 15s
CI / Mirror to GitHub (push) Successful in 3s
Wires nomic-embed-text (iguana ollama) + pgvector on the shared
postgres18 into brain_query / brain_answer via Reciprocal Rank Fusion.
Pure BM25 stays the default; setting BRAIN_PG_DSN and BRAIN_EMBED_URL
together opts in. Setting one without the other is misconfiguration →
exit 1.

New packages:

- internal/embed
  Client.Embed(ctx, text) → []float32 via POST {URL}/api/embed.
  Defaults to nomic-embed-text:latest (768 dim). nil-on-empty-URL so
  callers gate on a single nil check.

- internal/vectorstore
  PGStore wraps a pgxpool against postgres18. Init creates
  brain_embeddings(path PK, vector(768), updated_at) + HNSW cosine
  index idempotently. Upsert / Delete / Search / KnownPaths.
  Sync(brainDir, store, embedder) diffs brain/wiki/ against the store
  and upserts new files / deletes removed ones; StartSync runs it on
  a ticker (default 300s). Integration tests gated by BRAIN_PG_TEST_DSN.

- scripts/brain-embeddings-init.sql
  One-time DBA setup: brain DB, brain_app role, vector extension,
  GRANTs. Idempotent.

Search layer:

- search.QueryOptions gains Vector + Embedder fields.
- QueryContext is the cancellable variant; Query stays for callers.
- When both are set, BM25 (top-N) and pgvector (top-4N) candidates
  merge via Reciprocal Rank Fusion (k=60, Cormack et al. 2009 — no
  tuning knob, robust to scale differences between rankers).
- Vector-only hits are hydrated from disk so callers see uniform
  Result records (path, title, excerpt, wing, hall, score).
- Wing/hall filters still apply to vector candidates via path-prefix.
- On embedder/vector errors the search falls back to BM25 — embedding
  outage degrades quality but doesn't take the brain offline.

MCP wiring:

- mcp.Server.WithHybridRetrieval(v, e) opt-in setter, same shape as
  WithReranker.
- brainQuery and brainAnswer pass the wired vector/embedder through
  to search.QueryContext.

REST:

- POST /backfill-embeddings drives Sync synchronously. Returns
  {added, deleted, errors[]}. 503 when feature is unconfigured.

cmd/server/main.go:

- BRAIN_PG_DSN + BRAIN_EMBED_URL together enable hybrid; one alone
  → exit 1.
- vectorAdapter bridges *PGStore (returns []Hit) to
  search.VectorSearcher (which takes []VectorHit) without either
  package importing the other.
- BRAIN_EMBED_SYNC_INTERVAL (default 300s) controls the background
  Sync ticker.

Backend pivot from Qdrant to pgvector recorded in DECISIONS.md
2026-05-18 (supersedes 2026-04-08): postgres18 already runs in
databases/ ns, Qdrant was never deployed, one engine beats two.

Dependency: github.com/jackc/pgx/v5 — modern, native pgvector via
parametric vector literals.

Tests:
- embed.Client: empty-URL nil, request shape, dimension, upstream
  error propagation, empty-text rejection.
- vectorstore.PGStore: dimension validation (unit); upsert/search/
  KnownPaths (integration, BRAIN_PG_TEST_DSN-gated).
- vectorstore.Sync: adds new files, skips known, deletes
  disappeared, skips _index.md, no-op when nil, collects embedder
  errors.
- search.Query: hybrid promotes vector-only hits via RRF; falls
  back to BM25 on embedder error.

Closes hyperguild#8.
2026-05-18 23:11:25 +02:00
mathias a56a4db963 feat(brain_answer): Qwen3-Reranker cross-encoder filter (opt-in)
CI / Lint / Test / Vet (push) Successful in 10s
CI / Mirror to GitHub (push) Successful in 3s
Adds an opt-in cross-encoder rerank step between BM25 retrieval and LLM
synthesis. With BRAIN_RERANKER_URL set, brain_answer retrieves BM25
top-20, scores each excerpt against the query via Qwen3-Reranker on
Ollama, drops the "no" answers, and forwards up to 5 surviving sources
to the LLM. Unset, behaviour is unchanged (BM25 top-10 → LLM).

The reranker is a *filter*, not a re-ranker: Qwen3-Reranker emits a
binary yes/no token under its native chat template, and ties within the
"yes" set are broken by BM25 rank — what got retrieved first stays
ahead.

New package ingestion/internal/reranker:
- Client with URL, Model, HTTP fields.
- New(url, model) returns nil on empty url so callers can treat
  "feature disabled" as a single nil check.
- Score(ctx, query, docs) issues one /api/generate call per doc using
  the Qwen3-Reranker yes/no chat template (verbatim, because the model
  was trained on this exact wording). Parses the first non-think token.

Wiring:
- mcp.Server gains a WithReranker fluent setter to keep NewServer
  signature stable.
- brain_answer's BM25 limit jumps to 20 only when a reranker is wired,
  to give the filter something to do.
- cmd/server/main.go reads BRAIN_RERANKER_URL (+ optional
  BRAIN_RERANKER_MODEL, default dengcao/Qwen3-Reranker-0.6B:F16).

Tests cover: nil-on-empty-url, ordered yes/no scoring, request shape
(model, prompt contents, yes/no template), ambiguous response → 0,
empty doc slice, upstream-error propagation, plus an end-to-end
brain_answer integration that proves only the relevant note reaches the
LLM when noise.md is rejected.

Closes hyperguild#7.
2026-05-18 22:55:46 +02:00
mathias 58c57412a9 feat(brain-mcp): OAuth 2.0 client_credentials flow for claude.ai
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Successful in 3s
Adds a minimal RFC 8414 + RFC 6749 client_credentials flow so claude.ai's
custom-MCP integration (no static-Bearer field in the UI) can exchange a
client_id + client_secret pair for the existing BRAIN_MCP_TOKEN and use
it as a Bearer on /mcp. No JWTs, no refresh, no expiry — the rest of
the auth middleware is unchanged.

New package ingestion/internal/oauth:
- MetadataHandler(issuer): serves /.well-known/oauth-authorization-server
  with grant_types=[client_credentials] and both
  token_endpoint_auth_methods (post + basic).
- TokenHandler(cfg): serves /oauth/token. Validates client_id and
  client_secret via constant-time compare; returns BRAIN_MCP_TOKEN as
  access_token. RFC 6749 §5.2 error JSON on bad grant / bad creds.

Wiring in cmd/server/main.go: opt-in by setting both OAUTH_CLIENT_ID and
OAUTH_CLIENT_SECRET. Setting only one is misconfiguration → exit 1.
Mounts both endpoints with no auth; MCP_RESOURCE_URL supplies the
issuer.

Also pivots issue #8's vector backend from Qdrant to pgvector (see
DECISIONS.md 2026-05-18) — Qdrant was never deployed and postgres18 with
pgvector already runs as the project default; supersedes 2026-04-08 for
this use case.

Tests cover post-auth, basic-auth, wrong secret, bad grant, GET
rejection, malformed Basic header, and Basic without colon.

Closes hyperguild#5.
2026-05-18 22:21:54 +02:00
mathias ddd07ae7eb feat(brain): cross-wing tunnels — bidirectional wikilinks + auto-detect
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Successful in 3s
Adds the `brain_tunnel` MCP tool and auto-tunnel behaviour for
`brain_write`, so concepts that appear in multiple wings become
navigable from any of them.

New surface in package brain:
- WriteTunnel(brainDir, src, tgt) — appends a `## See also` bidirectional
  wikilink between two notes in different wings. Idempotent (link not
  duplicated on re-call) and reuses an existing See also section.
- DetectTunnels(brainDir, content) — walks brain/wiki/, returns
  TunnelCandidates for notes whose title appears in content. Tags
  whole-word case-insensitive hits as Exact=true and substring-only hits
  as Exact=false.
- AutoTunnel(brainDir, src, content) — wraps DetectTunnels: writes
  cross-wing exact matches, stages fuzzy matches into
  brain/raw/tunnel-candidates-<YYYY-MM-DD>.md for human review.

MCP wiring:
- `brain_tunnel` tool: explicit manual link (source, target).
- `brain_write` with wing+hall now triggers AutoTunnel on the new
  content. Failures are logged and never abort the primary write.

readTitleAndCreated also humanises the slug fallback (hyphens → spaces)
so titleless notes participate in content matching.

Closes hyperguild#16.

Tests: idempotency, same-wing rejection, missing-note rejection,
See-also reuse, exact/fuzzy detection, slug fallback, MCP tool happy
path, auto-tunnel hook (cross-wing exact → linked; same-wing → skipped;
fuzzy → candidates file).
2026-05-18 21:32:49 +02:00
mathias 61b6247df9 fix(brain-mcp): static Bearer short-circuits before OAuth challenge
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Successful in 3s
Reorders BearerAuth so a valid BRAIN_MCP_TOKEN match wins instantly and
never emits WWW-Authenticate. Adds RFC 9728 resource_metadata challenge
header on 401 (only when MCP_RESOURCE_URL is configured) so claude.ai's
OAuth-discovery path still works.

Why: claude CLI on koala/flamingo with `.mcp.json` `Authorization: Bearer
$BRAIN_MCP_TOKEN` was being kicked into RFC 7591 dynamic client
registration against Dex (static-only) and dying. Cause was the auth
middleware running JWT validation first and emitting an OAuth challenge
on the fall-through 401 even when the caller had a valid static token.
Inverting the precedence and gating the challenge on resourceMetadataURL
keeps the LAN/Tailscale CLI path silent and only invites OAuth discovery
on actually-unauthenticated requests.

Regression guards in the test file:
- valid static Bearer 200 has no WWW-Authenticate
- 401 with resourceMetadataURL set carries the challenge
- 401 with empty resourceMetadataURL emits no challenge

Closes hyperguild#9 in code. Live verification (claude CLI on koala
listing brain tools) blocked on ingestion image rebuild + redeploy.
2026-05-18 21:00:05 +02:00
mathias 75685e7b67 feat(brain): structured wing/hall taxonomy + obsidian-compatible layout
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Successful in 4s
Adds a two-dimensional address (wing, hall) to brain notes. A wing is a
topic domain (e.g. jepa-fx, hyperguild); a hall is one of a closed
vocabulary of memory types (facts, decisions, failures, hypotheses,
sources). Notes route to brain/wiki/<wing>/<hall>/<slug>.md with
wing/hall/created_at YAML frontmatter, making the directory a valid
Obsidian vault.

Changes:
- new package ingestion/internal/brain (NotePath, ValidHalls, Sanitise,
  BuildWingIndex, BuildAllWingIndexes)
- api.WriteNote refactored to WriteNoteOptions; wing+hall routes to
  brain/wiki/, otherwise falls back to brain/knowledge/ (legacy)
- search.Query → QueryOptions with optional Wing/Hall filtering; Result
  carries wing/hall extracted from frontmatter or path segments
- MCP tools brain_write and brain_query gain optional wing/hall params
  (hall enum-validated); new brain_index tool regenerates _index.md MOC
- POST /index REST endpoint mirrors brain_index
- brain_write auto-rebuilds the wing's _index.md after a wing+hall write
- scripts/migrate-brain-halls.sh migrates flat brain/wiki/{concepts,entities}/
  into the new layout (dry-run by default, --commit applies)

All existing tests pass; new tests cover wing/hall write routing, scope
filtering, invalid hall rejection, _index.md generation, and migration
script paths.

Closes hyperguild#1.
2026-05-18 20:47:08 +02:00
mathias fe18e4ee77 test(routing): de-flake TestRoutingPodEndToEnd
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Successful in 4s
- Random port via net.Listen(":0") replaces hardcoded 33310 (was the
  primary failure mode under parallel test load).
- Bump waitForPort deadline 5s → 30s — `go build` under -race can exceed
  5s on a loaded machine.
- Replace osPath() (always returned empty PATH because exec.Command("env").Env
  is the *child's* env, not the parent's) with explicit PATH+HOME via
  os.Getenv. Don't inherit full env: would leak ROUTING_MCP_TOKEN from the
  parent shell and flip the routing pod into auth-required mode, breaking
  the test.

Closes #15. Verified: 10 cold-cache test runs pass, 3 consecutive task check
runs pass.
2026-05-18 20:00:18 +02:00
mathias 937355cabe fix(project_create): commit staging namespace directly to infra main
CI / Lint / Test / Vet (push) Successful in 11s
CI / Mirror to GitHub (push) Successful in 3s
Drops the intermediate `staging/<name>` branch so Flux begins reconciling the
namespace within ~60s of `project_create` instead of waiting on a human PR
merge. Consistent with project-wide trunk-based development.

Rationale: ADR 2026-05-18 in DECISIONS.md.

Closes hyperguild#14 (item 1). Item 2 (GITEA_MCP_TOKEN in SOPS) verified
already-present in infra@408a527 secrets.enc.yaml.

Note: TestRoutingPodEndToEnd is failing on main pre-existing this commit
(context deadline waiting for port 33310 in <5s). Not caused by this change;
project skill tests pass. To track in a separate issue.
2026-05-18 17:20:53 +02:00
mathiasandClaude Opus 4.7 5950ef5f0f feat(mcpclient): fail-fast on empty bearer token
CI / Lint / Test / Vet (push) Successful in 10s
CI / Mirror to GitHub (push) Successful in 4s
mcpclient.New previously accepted an empty token and silently omitted
the Authorization header at request time. When the env var sourcing
the token was missing from a Kubernetes Secret (envFrom doesn't warn
on missing keys), this surfaced as an opaque 401 from the upstream
MCP server with no log trail — see hyperguild #13 and brain entry
"mcpclient-empty-token-silent-401-envfrom-missing-key".

mcpclient.New now returns ErrTokenRequired when token is empty.
The routing pod's project_create init checks the error and exits
with a clear message pointing at routing-secrets, turning a runtime
401 storm into a startup crashloop the operator can fix immediately.

Tests pass a dummy "test" token (httptest servers don't enforce
bearer auth, so any non-empty value works). Added a regression
test asserting empty-token construction returns ErrTokenRequired.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 16:28:09 +02:00
90 changed files with 9960 additions and 733 deletions
+54 -8
View File
@@ -27,6 +27,14 @@ and climate/sustainability tech.
These rules apply to every task across every project, regardless of harness. These rules apply to every task across every project, regardless of harness.
0. **Pre-task ritual — before ANY implementation (non-negotiable).** Run this before writing a single line:
- **Query the brain** (`brain_query`) for the domain + symptom. If the result changes your approach, surface it before acting. 5 seconds beats 5 hours.
- **Load the relevant skill** — see trigger table in *Engineering Skills* below.
- **Write the failing test first.** Name the test before the function. If the target is untestable (e.g. `main()` wiring), extract the logic into a testable function first. No implementation without a red test.
- **State the observable success criterion** — what specific behavior, output, or passing test proves this is done?
**TDD is non-negotiable.** "Tests pass" is not proof of correctness — only proof the tests ran. Write tests that would catch the bug before writing code that fixes it.
1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly. 1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly.
Think before coding; if the problem is unclear, ask or state assumptions before acting. Think before coding; if the problem is unclear, ask or state assumptions before acting.
2. **Minimum viable code.** Solve with the smallest change that works. Nothing 2. **Minimum viable code.** Solve with the smallest change that works. Nothing
@@ -49,6 +57,22 @@ These rules apply to every task across every project, regardless of harness.
PR flow only when a human reviewer outside the project is required. Document PR flow only when a human reviewer outside the project is required. Document
the reason in PROJECT.md. the reason in PROJECT.md.
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
the code is not the end of the task; capturing it is. Run this unprompted:
- **Tag + bump SemVer** on the change (annotated tag; minor for a feature or
new/changed ADR, patch for a fix; docs in the same commit). Check the repo's
actual last tag — stated versions in docs drift stale.
- **Push** main and the tag (CI is the gate).
- **Persist generalizable learnings to the brain** (`brain_write`, wing/hall) —
the reusable patterns and the footguns that would bite anyone again, never
project status. See *Knowledge base — when to write* below.
- **File discovered-but-deferred work as tracker issues** on the project's own
repo — token-budget gaps, recorded ADR limitations, v2 follow-ups. Don't let
"out of scope, recorded" rot in a commit message; make it a ticket with a
source pointer.
- Surface the brain entries and issue numbers in the closing summary so the
trail is auditable.
## Default stack ## Default stack
| Layer | Default | Fallback | Last resort | | Layer | Default | Fallback | Last resort |
@@ -78,6 +102,26 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
- **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config - **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config
- **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message - **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message
## Secret handling (every harness, every command)
Tool output is persisted: terminal → `~/.claude/projects` transcripts →
claudewatcher → brain/wiki → gitea history. A secret printed once is
searchable forever, and clearing it means rotating the key. So:
1. **Never print, echo, log, or transform a secret to inspect it.** No
`base64`/`xxd`/`cat` of a key, and never pipe a secret through a transform
to defeat `op run`'s output masking (it masks raw values; base64 hides them
from the mask — that exact trick leaked a key on 2026-06-11).
2. **Secrets stay in the subprocess.** Reference them only as env vars consumed
*inside* `op run --env-file ~/.op-env -- <cmd>`. Never place a literal secret
in a command's argv (it lands in the tool call and the transcript).
3. **Existence check without revealing the value:** `[ -n "$X" ] && echo set`
never `${X:-...}` (returns the value when set) and never echo a substring of it.
4. **Cross-host secrets:** run the secret-consuming command on the host that has
the secret; do not forward a raw key over ssh argv/stdout.
5. If a secret does leak into output, say so immediately and flag it for rotation —
don't bury it.
## Infrastructure ## Infrastructure
Three machines on Tailscale: Three machines on Tailscale:
@@ -157,7 +201,7 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool | | **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same | | **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` | | **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` |
| **Browser / human inspection** | `https://gitea.d-ma.be/mathias/hyperguild``knowledge/` and `wiki/` markdown files | | **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild``knowledge/` and `wiki/` markdown files |
- **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`. - **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`.
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as - **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
@@ -219,15 +263,17 @@ unconditionally on every host, every harness.
## Engineering Skills ## Engineering Skills
Shared engineering skills are available in `~/dev/.skills/`. Load on demand via the index. Shared engineering skills are available in `~/dev/.skills/`. Load at task start — not "on demand" but on schedule, before writing code. See `~/dev/.skills/SKILLS_INDEX.md` for the full list.
See `~/dev/.skills/SKILLS_INDEX.md` for the full list with descriptions and "use when" triggers. **Skill trigger table — load before starting, not after getting stuck:**
Key skills: | Task type | Load |
- **TDD**: always write tests first — load `tdd` skill |-----------|------|
- **Code Review**: load `code-review` skill before any review | Any feature or bug fix | `tdd` |
- **SOLID/Clean Code**: load `solid` or `clean-code` skill for design work | Refactor or design | `clean-code` or `solid` |
- **Problem first**: load `problem-analysis` skill before coding non-trivial features | Debug | `problem-analysis` |
| Review code or PRs | `code-review` |
| Frame a problem before coding | `problem-analysis` |
--- ---
+54 -8
View File
@@ -32,6 +32,14 @@ and climate/sustainability tech.
These rules apply to every task across every project, regardless of harness. These rules apply to every task across every project, regardless of harness.
0. **Pre-task ritual — before ANY implementation (non-negotiable).** Run this before writing a single line:
- **Query the brain** (`brain_query`) for the domain + symptom. If the result changes your approach, surface it before acting. 5 seconds beats 5 hours.
- **Load the relevant skill** — see trigger table in *Engineering Skills* below.
- **Write the failing test first.** Name the test before the function. If the target is untestable (e.g. `main()` wiring), extract the logic into a testable function first. No implementation without a red test.
- **State the observable success criterion** — what specific behavior, output, or passing test proves this is done?
**TDD is non-negotiable.** "Tests pass" is not proof of correctness — only proof the tests ran. Write tests that would catch the bug before writing code that fixes it.
1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly. 1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly.
Think before coding; if the problem is unclear, ask or state assumptions before acting. Think before coding; if the problem is unclear, ask or state assumptions before acting.
2. **Minimum viable code.** Solve with the smallest change that works. Nothing 2. **Minimum viable code.** Solve with the smallest change that works. Nothing
@@ -54,6 +62,22 @@ These rules apply to every task across every project, regardless of harness.
PR flow only when a human reviewer outside the project is required. Document PR flow only when a human reviewer outside the project is required. Document
the reason in PROJECT.md. the reason in PROJECT.md.
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
the code is not the end of the task; capturing it is. Run this unprompted:
- **Tag + bump SemVer** on the change (annotated tag; minor for a feature or
new/changed ADR, patch for a fix; docs in the same commit). Check the repo's
actual last tag — stated versions in docs drift stale.
- **Push** main and the tag (CI is the gate).
- **Persist generalizable learnings to the brain** (`brain_write`, wing/hall) —
the reusable patterns and the footguns that would bite anyone again, never
project status. See *Knowledge base — when to write* below.
- **File discovered-but-deferred work as tracker issues** on the project's own
repo — token-budget gaps, recorded ADR limitations, v2 follow-ups. Don't let
"out of scope, recorded" rot in a commit message; make it a ticket with a
source pointer.
- Surface the brain entries and issue numbers in the closing summary so the
trail is auditable.
## Default stack ## Default stack
| Layer | Default | Fallback | Last resort | | Layer | Default | Fallback | Last resort |
@@ -83,6 +107,26 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
- **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config - **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config
- **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message - **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message
## Secret handling (every harness, every command)
Tool output is persisted: terminal → `~/.claude/projects` transcripts →
claudewatcher → brain/wiki → gitea history. A secret printed once is
searchable forever, and clearing it means rotating the key. So:
1. **Never print, echo, log, or transform a secret to inspect it.** No
`base64`/`xxd`/`cat` of a key, and never pipe a secret through a transform
to defeat `op run`'s output masking (it masks raw values; base64 hides them
from the mask — that exact trick leaked a key on 2026-06-11).
2. **Secrets stay in the subprocess.** Reference them only as env vars consumed
*inside* `op run --env-file ~/.op-env -- <cmd>`. Never place a literal secret
in a command's argv (it lands in the tool call and the transcript).
3. **Existence check without revealing the value:** `[ -n "$X" ] && echo set` —
never `${X:-...}` (returns the value when set) and never echo a substring of it.
4. **Cross-host secrets:** run the secret-consuming command on the host that has
the secret; do not forward a raw key over ssh argv/stdout.
5. If a secret does leak into output, say so immediately and flag it for rotation —
don't bury it.
## Infrastructure ## Infrastructure
Three machines on Tailscale: Three machines on Tailscale:
@@ -162,7 +206,7 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool | | **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same | | **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` | | **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` |
| **Browser / human inspection** | `https://gitea.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files | | **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files |
- **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`. - **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`.
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as - **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
@@ -224,15 +268,17 @@ unconditionally on every host, every harness.
## Engineering Skills ## Engineering Skills
Shared engineering skills are available in `~/dev/.skills/`. Load on demand via the index. Shared engineering skills are available in `~/dev/.skills/`. Load at task start — not "on demand" but on schedule, before writing code. See `~/dev/.skills/SKILLS_INDEX.md` for the full list.
See `~/dev/.skills/SKILLS_INDEX.md` for the full list with descriptions and "use when" triggers. **Skill trigger table — load before starting, not after getting stuck:**
Key skills: | Task type | Load |
- **TDD**: always write tests first — load `tdd` skill |-----------|------|
- **Code Review**: load `code-review` skill before any review | Any feature or bug fix | `tdd` |
- **SOLID/Clean Code**: load `solid` or `clean-code` skill for design work | Refactor or design | `clean-code` or `solid` |
- **Problem first**: load `problem-analysis` skill before coding non-trivial features | Debug | `problem-analysis` |
| Review code or PRs | `code-review` |
| Frame a problem before coding | `problem-analysis` |
--- ---
+54 -8
View File
@@ -30,6 +30,14 @@ and climate/sustainability tech.
These rules apply to every task across every project, regardless of harness. These rules apply to every task across every project, regardless of harness.
0. **Pre-task ritual — before ANY implementation (non-negotiable).** Run this before writing a single line:
- **Query the brain** (`brain_query`) for the domain + symptom. If the result changes your approach, surface it before acting. 5 seconds beats 5 hours.
- **Load the relevant skill** — see trigger table in *Engineering Skills* below.
- **Write the failing test first.** Name the test before the function. If the target is untestable (e.g. `main()` wiring), extract the logic into a testable function first. No implementation without a red test.
- **State the observable success criterion** — what specific behavior, output, or passing test proves this is done?
**TDD is non-negotiable.** "Tests pass" is not proof of correctness — only proof the tests ran. Write tests that would catch the bug before writing code that fixes it.
1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly. 1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly.
Think before coding; if the problem is unclear, ask or state assumptions before acting. Think before coding; if the problem is unclear, ask or state assumptions before acting.
2. **Minimum viable code.** Solve with the smallest change that works. Nothing 2. **Minimum viable code.** Solve with the smallest change that works. Nothing
@@ -52,6 +60,22 @@ These rules apply to every task across every project, regardless of harness.
PR flow only when a human reviewer outside the project is required. Document PR flow only when a human reviewer outside the project is required. Document
the reason in PROJECT.md. the reason in PROJECT.md.
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
the code is not the end of the task; capturing it is. Run this unprompted:
- **Tag + bump SemVer** on the change (annotated tag; minor for a feature or
new/changed ADR, patch for a fix; docs in the same commit). Check the repo's
actual last tag — stated versions in docs drift stale.
- **Push** main and the tag (CI is the gate).
- **Persist generalizable learnings to the brain** (`brain_write`, wing/hall) —
the reusable patterns and the footguns that would bite anyone again, never
project status. See *Knowledge base — when to write* below.
- **File discovered-but-deferred work as tracker issues** on the project's own
repo — token-budget gaps, recorded ADR limitations, v2 follow-ups. Don't let
"out of scope, recorded" rot in a commit message; make it a ticket with a
source pointer.
- Surface the brain entries and issue numbers in the closing summary so the
trail is auditable.
## Default stack ## Default stack
| Layer | Default | Fallback | Last resort | | Layer | Default | Fallback | Last resort |
@@ -81,6 +105,26 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
- **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config - **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config
- **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message - **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message
## Secret handling (every harness, every command)
Tool output is persisted: terminal → `~/.claude/projects` transcripts →
claudewatcher → brain/wiki → gitea history. A secret printed once is
searchable forever, and clearing it means rotating the key. So:
1. **Never print, echo, log, or transform a secret to inspect it.** No
`base64`/`xxd`/`cat` of a key, and never pipe a secret through a transform
to defeat `op run`'s output masking (it masks raw values; base64 hides them
from the mask — that exact trick leaked a key on 2026-06-11).
2. **Secrets stay in the subprocess.** Reference them only as env vars consumed
*inside* `op run --env-file ~/.op-env -- <cmd>`. Never place a literal secret
in a command's argv (it lands in the tool call and the transcript).
3. **Existence check without revealing the value:** `[ -n "$X" ] && echo set` —
never `${X:-...}` (returns the value when set) and never echo a substring of it.
4. **Cross-host secrets:** run the secret-consuming command on the host that has
the secret; do not forward a raw key over ssh argv/stdout.
5. If a secret does leak into output, say so immediately and flag it for rotation —
don't bury it.
## Infrastructure ## Infrastructure
Three machines on Tailscale: Three machines on Tailscale:
@@ -160,7 +204,7 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool | | **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same | | **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` | | **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` |
| **Browser / human inspection** | `https://gitea.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files | | **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files |
- **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`. - **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`.
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as - **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
@@ -222,15 +266,17 @@ unconditionally on every host, every harness.
## Engineering Skills ## Engineering Skills
Shared engineering skills are available in `~/dev/.skills/`. Load on demand via the index. Shared engineering skills are available in `~/dev/.skills/`. Load at task start — not "on demand" but on schedule, before writing code. See `~/dev/.skills/SKILLS_INDEX.md` for the full list.
See `~/dev/.skills/SKILLS_INDEX.md` for the full list with descriptions and "use when" triggers. **Skill trigger table — load before starting, not after getting stuck:**
Key skills: | Task type | Load |
- **TDD**: always write tests first — load `tdd` skill |-----------|------|
- **Code Review**: load `code-review` skill before any review | Any feature or bug fix | `tdd` |
- **SOLID/Clean Code**: load `solid` or `clean-code` skill for design work | Refactor or design | `clean-code` or `solid` |
- **Problem first**: load `problem-analysis` skill before coding non-trivial features | Debug | `problem-analysis` |
| Review code or PRs | `code-review` |
| Frame a problem before coding | `problem-analysis` |
--- ---
+9 -9
View File
@@ -1,6 +1,6 @@
name: cd name: cd
on: "on":
workflow_run: workflow_run:
workflows: ["CI"] workflows: ["CI"]
types: [completed] types: [completed]
@@ -13,9 +13,9 @@ jobs:
if: ${{ github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.event == 'push' }} if: ${{ github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.event == 'push' }}
environment: staging environment: staging
env: env:
INGESTION_IMAGE: gitea.d-ma.be/mathias/ingestion INGESTION_IMAGE: git.d-ma.be/mathias/ingestion
ROUTING_IMAGE: gitea.d-ma.be/mathias/routing ROUTING_IMAGE: git.d-ma.be/mathias/routing
INFRA_REPO: git@gitea.d-ma.be:mathias/infra.git INFRA_REPO: git@git.d-ma.be:mathias/infra.git
BUILDKIT_HOST: unix:///run/buildkit/buildkitd.sock BUILDKIT_HOST: unix:///run/buildkit/buildkitd.sock
steps: steps:
- name: Checkout - name: Checkout
@@ -71,17 +71,17 @@ jobs:
mkdir -p ~/.ssh mkdir -p ~/.ssh
echo "${{ secrets.INFRA_DEPLOY_KEY }}" > ~/.ssh/infra_deploy_key echo "${{ secrets.INFRA_DEPLOY_KEY }}" > ~/.ssh/infra_deploy_key
chmod 600 ~/.ssh/infra_deploy_key chmod 600 ~/.ssh/infra_deploy_key
printf 'Host gitea.d-ma.be\n HostName 127.0.0.1\n Port 30022\n StrictHostKeyChecking no\n' >> ~/.ssh/config printf 'Host git.d-ma.be\n HostName 127.0.0.1\n Port 30022\n StrictHostKeyChecking no\n' >> ~/.ssh/config
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \ GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \
git clone "${INFRA_REPO}" /tmp/infra-update git clone "${INFRA_REPO}" /tmp/infra-update
cd /tmp/infra-update cd /tmp/infra-update
sed -i "s|gitea.d-ma.be/mathias/ingestion:.*|gitea.d-ma.be/mathias/ingestion:${IMAGE_TAG}|" \ sed -i "s|git.d-ma.be/mathias/ingestion:.*|git.d-ma.be/mathias/ingestion:${IMAGE_TAG}|" \
"k3s/apps/supervisor/ingestion-deployment.yaml" "k3s/apps/supervisor/ingestion-deployment.yaml"
sed -i "s|gitea.d-ma.be/mathias/routing:.*|gitea.d-ma.be/mathias/routing:${IMAGE_TAG}|" \ sed -i "s|git.d-ma.be/mathias/routing:.*|git.d-ma.be/mathias/routing:${IMAGE_TAG}|" \
"k3s/apps/routing/deployment.yaml" "k3s/apps/routing/deployment.yaml"
git config user.email "cd-bot@d-ma.be" git config user.email "cd-bot@d-ma.be"
@@ -103,7 +103,7 @@ jobs:
- name: Wait for Flux to apply new ingestion image - name: Wait for Flux to apply new ingestion image
run: | run: |
EXPECTED="gitea.d-ma.be/mathias/ingestion:${{ github.sha }}" EXPECTED="git.d-ma.be/mathias/ingestion:${{ github.sha }}"
for i in $(seq 1 60); do for i in $(seq 1 60); do
CURRENT=$(kubectl get deploy ingestion -n supervisor \ CURRENT=$(kubectl get deploy ingestion -n supervisor \
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || echo "") -o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || echo "")
@@ -135,7 +135,7 @@ jobs:
- name: Wait for Flux to apply new routing image - name: Wait for Flux to apply new routing image
run: | run: |
EXPECTED="gitea.d-ma.be/mathias/routing:${{ github.sha }}" EXPECTED="git.d-ma.be/mathias/routing:${{ github.sha }}"
for i in $(seq 1 60); do for i in $(seq 1 60); do
CURRENT=$(kubectl get deploy routing -n routing \ CURRENT=$(kubectl get deploy routing -n routing \
-o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || echo "") -o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null || echo "")
+1 -1
View File
@@ -1,6 +1,6 @@
name: CI name: CI
on: "on":
push: push:
branches: [main] branches: [main]
tags: ["v*"] tags: ["v*"]
+54 -8
View File
@@ -27,6 +27,14 @@ and climate/sustainability tech.
These rules apply to every task across every project, regardless of harness. These rules apply to every task across every project, regardless of harness.
0. **Pre-task ritual — before ANY implementation (non-negotiable).** Run this before writing a single line:
- **Query the brain** (`brain_query`) for the domain + symptom. If the result changes your approach, surface it before acting. 5 seconds beats 5 hours.
- **Load the relevant skill** — see trigger table in *Engineering Skills* below.
- **Write the failing test first.** Name the test before the function. If the target is untestable (e.g. `main()` wiring), extract the logic into a testable function first. No implementation without a red test.
- **State the observable success criterion** — what specific behavior, output, or passing test proves this is done?
**TDD is non-negotiable.** "Tests pass" is not proof of correctness — only proof the tests ran. Write tests that would catch the bug before writing code that fixes it.
1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly. 1. **No assumptions.** Don't hide confusion — surface it. Surface tradeoffs explicitly.
Think before coding; if the problem is unclear, ask or state assumptions before acting. Think before coding; if the problem is unclear, ask or state assumptions before acting.
2. **Minimum viable code.** Solve with the smallest change that works. Nothing 2. **Minimum viable code.** Solve with the smallest change that works. Nothing
@@ -49,6 +57,22 @@ These rules apply to every task across every project, regardless of harness.
PR flow only when a human reviewer outside the project is required. Document PR flow only when a human reviewer outside the project is required. Document
the reason in PROJECT.md. the reason in PROJECT.md.
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
the code is not the end of the task; capturing it is. Run this unprompted:
- **Tag + bump SemVer** on the change (annotated tag; minor for a feature or
new/changed ADR, patch for a fix; docs in the same commit). Check the repo's
actual last tag — stated versions in docs drift stale.
- **Push** main and the tag (CI is the gate).
- **Persist generalizable learnings to the brain** (`brain_write`, wing/hall) —
the reusable patterns and the footguns that would bite anyone again, never
project status. See *Knowledge base — when to write* below.
- **File discovered-but-deferred work as tracker issues** on the project's own
repo — token-budget gaps, recorded ADR limitations, v2 follow-ups. Don't let
"out of scope, recorded" rot in a commit message; make it a ticket with a
source pointer.
- Surface the brain entries and issue numbers in the closing summary so the
trail is auditable.
## Default stack ## Default stack
| Layer | Default | Fallback | Last resort | | Layer | Default | Fallback | Last resort |
@@ -78,6 +102,26 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
- **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config - **Security**: no secrets in code, govulncheck before adding deps, SOPS for encrypted config
- **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message - **Dependencies**: prefer stdlib. testify, slog, templ, sqlc, google.golang.org/adk (agent projects only) are pre-approved; anything else needs justification in the commit message
## Secret handling (every harness, every command)
Tool output is persisted: terminal → `~/.claude/projects` transcripts →
claudewatcher → brain/wiki → gitea history. A secret printed once is
searchable forever, and clearing it means rotating the key. So:
1. **Never print, echo, log, or transform a secret to inspect it.** No
`base64`/`xxd`/`cat` of a key, and never pipe a secret through a transform
to defeat `op run`'s output masking (it masks raw values; base64 hides them
from the mask — that exact trick leaked a key on 2026-06-11).
2. **Secrets stay in the subprocess.** Reference them only as env vars consumed
*inside* `op run --env-file ~/.op-env -- <cmd>`. Never place a literal secret
in a command's argv (it lands in the tool call and the transcript).
3. **Existence check without revealing the value:** `[ -n "$X" ] && echo set`
never `${X:-...}` (returns the value when set) and never echo a substring of it.
4. **Cross-host secrets:** run the secret-consuming command on the host that has
the secret; do not forward a raw key over ssh argv/stdout.
5. If a secret does leak into output, say so immediately and flag it for rotation —
don't bury it.
## Infrastructure ## Infrastructure
Three machines on Tailscale: Three machines on Tailscale:
@@ -157,7 +201,7 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool | | **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same | | **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` | | **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` |
| **Browser / human inspection** | `https://gitea.d-ma.be/mathias/hyperguild``knowledge/` and `wiki/` markdown files | | **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild``knowledge/` and `wiki/` markdown files |
- **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`. - **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`.
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as - **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
@@ -219,15 +263,17 @@ unconditionally on every host, every harness.
## Engineering Skills ## Engineering Skills
Shared engineering skills are available in `~/dev/.skills/`. Load on demand via the index. Shared engineering skills are available in `~/dev/.skills/`. Load at task start — not "on demand" but on schedule, before writing code. See `~/dev/.skills/SKILLS_INDEX.md` for the full list.
See `~/dev/.skills/SKILLS_INDEX.md` for the full list with descriptions and "use when" triggers. **Skill trigger table — load before starting, not after getting stuck:**
Key skills: | Task type | Load |
- **TDD**: always write tests first — load `tdd` skill |-----------|------|
- **Code Review**: load `code-review` skill before any review | Any feature or bug fix | `tdd` |
- **SOLID/Clean Code**: load `solid` or `clean-code` skill for design work | Refactor or design | `clean-code` or `solid` |
- **Problem first**: load `problem-analysis` skill before coding non-trivial features | Debug | `problem-analysis` |
| Review code or PRs | `code-review` |
| Frame a problem before coding | `problem-analysis` |
--- ---
+117
View File
@@ -4,6 +4,74 @@ Record *why* things are the way they are. Future-you will thank present-you.
--- ---
## 2026-05-28 — three active harnesses: hyperguild, agentsquad, Crush (extends earlier boundary decision)
**Context:** After wiring Crush to LiteLLM in May 2026, there are now three active harnesses.
The earlier boundary decision only covered hyperguild vs agentsquad. Crush's role was undefined.
**Decision:** Three harnesses, three distinct roles, shared skills layer.
| Harness | Engine | Primary use | Brain MCP? | Routing pod? | Skills? |
|---------|--------|-------------|------------|--------------|---------|
| **hyperguild** | Claude Code + MCP | Disciplined solo coding sessions, TDD/review/debug workflows | Yes | Yes | Yes (SKILL.md) |
| **agentsquad** | OpenCode + LiteLLM | Multi-agent task execution, executor/reviewer pipelines | No | No (own routing) | Yes (SKILL.md) |
| **Crush** | Charmbracelet TUI + LiteLLM | Interactive local coding, quick iterations on flamingo | No (not yet) | No (direct LiteLLM) | Yes (SKILL.md) |
**Crush specifics (as of 2026-05-28):**
- Config: `~/.config/crush/crush.json` on flamingo (see brain: `homelab/facts/crush-litellm-wiring-2026-05`)
- Connects directly to LiteLLM at `http://koala:4000/v1/` using `sk-local-123`
- Auth type: `openai-compat` (not `openai`)
- Does NOT go through the routing pod — model selection is manual in the Crush UI
- Brain MCP not wired — Crush has no MCP client capability today; revisit if Crush adds MCP support
**Shared across all three:**
- `mathias/skills` — any SKILL.md file works in all three harnesses
- LiteLLM proxy on koala (`http://koala:4000/v1/`) — Crush and agentsquad both route through it; hyperguild does too for local model calls
**Consequences:** No consolidation needed. crush.json must be kept in sync when litellm_config.yaml model names change. The `crush.json` canonical location is `~/.config/crush/crush.json` on flamingo — not yet tracked in a dotfiles repo (track as tech debt).
---
## 2026-05-28 — "field benchmark" for local models = pass-rate at scale (supersedes GOTTH eval suite)
**Context:** The GOTTH eval suite (45 offline prompts across 5 categories) was replaced by
a "field benchmark" in May 2026, but the replacement was never defined concretely.
**Decision:** The field benchmark is per-skill pass rate over real routing pod usage,
collected automatically by `internal/routing/passrate.go` and exposed at:
```
GET /pass-rate?skill=<name>&window=<duration>
```
No separate eval suite. No synthetic prompts. The benchmark runs itself once the routing
pod receives real traffic. Target: 30-day rolling window per skill, reviewed monthly.
**Bootstrap note:** With no session history, `passrate.go` returns `nil` and the router
defaults to the thinking model for every call. The fast-model path activates only after
real pass-rate data accumulates. Seed with real usage — do not pre-populate.
**Consequences:** Zero maintenance overhead for the benchmark. The tradeoff is that results
are only meaningful after ~2 weeks of real usage, and skills that are rarely invoked will
have statistically thin pass-rate data. Revisit if a skill has fewer than 20 calls in 30 days.
---
## 2026-05-28 — brain injection in skill handlers: review is done, others unverified
**Context:** The April 2026 scope reset listed "brain_query injection into skill handlers"
as the top priority. As of 2026-05-28, `internal/skills/review/handlers.go` calls
`brain.Query(ctx, ...)` before dispatching to the LLM — confirmed in code review.
Status of debug, retrospective, and trainer handlers is unverified.
**Decision:** Treat review as the reference implementation. Verify debug, retrospective,
trainer against the same pattern before shipping new skill work. Tracked in issue #32.
**Consequences:** The April concern may be stale for review. A one-pass audit of the other
three skill handlers closes this fully.
---
## 2026-04-08 — AGENTS.md as cross-tool standard, not CLAUDE.md ## 2026-04-08 — AGENTS.md as cross-tool standard, not CLAUDE.md
**Context**: Multiple tools (Crush, Pi, Antigravity) read `AGENTS.md` natively. Claude Code reads `CLAUDE.md`. Building on `CLAUDE.md` as the primary format locks into one vendor. **Context**: Multiple tools (Crush, Pi, Antigravity) read `AGENTS.md` natively. Claude Code reads `CLAUDE.md`. Building on `CLAUDE.md` as the primary format locks into one vendor.
@@ -118,3 +186,52 @@ When berget.ai tokens run out, flip `BRAIN_LLM_PRIMARY_URL` to iguana.
**Decision**: The root context-sync generates a `mathias.md` prompt and `mathias.toml` agent config in `~/.vibe/`. This is the one tool that needs a custom adapter path. **Decision**: The root context-sync generates a `mathias.md` prompt and `mathias.toml` agent config in `~/.vibe/`. This is the one tool that needs a custom adapter path.
**Consequences**: Run `vibe --agent mathias` to use your conventions. Other Vibe users on the machine aren't affected. **Consequences**: Run `vibe --agent mathias` to use your conventions. Other Vibe users on the machine aren't affected.
---
## 2026-05-18 — project_create commits staging namespace directly to infra main
**Context:** `project_create` writes a k8s namespace manifest into the infra
repo so Flux brings up a staging environment for the new project. Initial
implementation pushed to a `staging/<name>` branch, which required manual PR
merge before Flux saw the namespace — defeating the "one tool call, project
exists, staging reconciling within 60s" goal.
**Decision:** Option A — commit directly to `main`. `callInfraCommit` passes
`branch: "main"` to gitea-mcp's `file_write_branch`; no PR, no merge step.
**Consequences:** Staging namespace appears in cluster within ~60s of the
`project_create` call. Consistent with project-wide TBD policy (CLAUDE.md):
commit directly to main, every commit deployable. Acceptable because the
manifest is a fresh namespace under `k3s/staging/<name>/` — isolated, low
blast-radius, and Flux will simply recreate it if the file is bad. Manual
review gating was friction for no compensating safety gain on experiment
namespaces.
---
## 2026-05-18 — pgvector over Qdrant for brain hybrid retrieval (supersedes 2026-04-08)
**Context:** The 2026-04-08 ADR chose Qdrant for vector store. Since then,
postgres18 with pgvector has been deployed in the `databases` namespace on
koala and is already the shared default for the rest of the project
(CLAUDE.md lists `pgvector (vector), BM25` as the primary search layer and
Qdrant only as a fallback "when >1M vectors or hybrid retrieval"). Qdrant
itself has never been deployed — `kubectl get` finds no pod, service, or
manifest. Standing up a new vector engine for a single consumer is friction
that the original ADR did not weigh.
**Decision:** Use pgvector for brain hybrid retrieval. Issue #8 — and any
follow-on embedding work — targets the existing `postgres18` instance:
- one table `brain_embeddings(path TEXT PRIMARY KEY, embedding VECTOR(768), updated_at TIMESTAMPTZ)`,
IVFFlat or HNSW index by feel once volume warrants
- BM25 stays as today (file walk + token frequency); cosine via pgvector
- hybrid scoring done in SQL or Go; pick once we measure
- nomic-embed-text on iguana ollama provides 768-dim vectors
**Consequences:** One database engine instead of two. Backups, monitoring,
and connection pooling already solved. Trade-off: pgvector at >1M vectors
or under hybrid-search load may underperform Qdrant — revisit only when
benchmarks hurt. The 2026-04-08 ADR is superseded for the brain use case;
Qdrant remains the noted fallback path in CLAUDE.md if scale demands it.
+72 -49
View File
@@ -5,14 +5,31 @@ Instead of letting Claude Code do whatever it wants, hyperguild enforces structu
workflows (TDD red/green/refactor), logs every session, and accumulates learnings workflows (TDD red/green/refactor), logs every session, and accumulates learnings
into a searchable brain. into a searchable brain.
## Hypothesis
> We believe routing skill tasks through local models, backed by brain context,
> produces measurably better outcomes than raw Claude Code alone —
> measurable by per-skill pass rate over rolling 30-day windows
> (available at `GET /pass-rate?skill=<name>&window=30d` on the brain pod).
This is the falsifiable claim the routing pod and pass-rate infrastructure exist to test.
If per-skill pass rates don't improve over baseline (all-cloud) after 30 days of real
usage, the fast-model routing path should be reconsidered.
## Harness
**hyperguild = Claude Code + MCP.** This is a supervisor for Claude Code sessions specifically.
For multi-agent orchestration (OpenCode + LiteLLM, executor/reviewer pipelines), see
[agentsquad](http://gitea.d-ma.be/mathias/agentsquad) — a separate harness for a different
orchestration model. Skills (mathias/skills) are shared between both.
## How it works ## How it works
``` ```
Your Claude Code session (in any project) Your Claude Code session (in any project)
│ MCP over HTTP (Tailscale) │ MCP over HTTP (Tailscale)
├──▶ supervisor :3200 (NodePort 30320 on koala) — skill workers: tdd, debug, spec, … ├──▶ routing :3210 (NodePort 30310 on koala) — review, debug, retrospective, trainer
├──▶ routing :3210 (NodePort 30310 on koala) — Mode 2 only: review, debug, retrospective, trainer
└──▶ brain :3300 (NodePort 30330 on koala) — brain_query, brain_write, brain_ingest, session_log └──▶ brain :3300 (NodePort 30330 on koala) — brain_query, brain_write, brain_ingest, session_log
└─ also serves the legacy REST endpoints (/query, /write, /ingest, …) └─ also serves the legacy REST endpoints (/query, /write, /ingest, …)
@@ -20,34 +37,28 @@ Your Claude Code session (in any project)
brain/ brain/
├── sessions/ — JSONL log, one file per session_id ├── sessions/ — JSONL log, one file per session_id
├── wiki/ — searchable knowledge (full-text) ├── wiki/ — searchable knowledge (wing/hall layout)
│ ├── concepts/ │ ├── homelab/
│ ├── entities/ │ ├── claude-sessions/
│ └── sources/ │ └── ...
── raw/ — retrospective output, staged for review ── knowledge/ — legacy flat notes (migration pending: hyperguild#22)
└── training-data/ — SFT/DPO/RL data (Phase 2)
``` ```
## Phase 1 tools (available now) ## Phase 1 tools (available now)
| Tool | What it does | | Tool | What it does |
|------|-------------| |------|-------------|
| `tdd_red` | Writes a failing test for a spec, verifies it fails |
| `tdd_green` | Writes the minimal implementation to make tests pass |
| `tdd_refactor` | Cleans up implementation while keeping tests green |
| `session_log` | Appends a structured entry to the session JSONL log | | `session_log` | Appends a structured entry to the session JSONL log |
| `retrospective` | Reads the session log, identifies novel learnings, writes to brain/raw/ | | `retrospective` | Reads the session log, identifies novel learnings, writes to brain |
| `review` | Structured code review via local model, brain-context injected |
| `debug` | Hypothesis-driven debugging via local model |
| `brain_query` | Full-text search over brain/wiki/ | | `brain_query` | Full-text search over brain/wiki/ |
| `brain_write` | Writes a note to brain/raw/ (with optional YAML frontmatter) | | `brain_write` | Writes a note to brain (with wing/hall routing) |
| `brain_answer` | BM25 + LLM synthesis — Q&A over brain corpus |
| `tier` | Returns the current connectivity tier (1=cloud, 2=LAN, 3=offline) | | `tier` | Returns the current connectivity tier (1=cloud, 2=LAN, 3=offline) |
## Start the servers > **Note:** `tdd_red/green/refactor` and `spec` were retired in Plan 7 (2026-05-12).
> They are now SKILL.md files in [mathias/skills](http://gitea.d-ma.be/mathias/skills).
```bash
# Requires goreman: go install github.com/mattn/goreman@latest
task start # starts ingestion (:3300) + supervisor (:3200) via goreman
task stop # kills both by port
```
## Connect a project ## Connect a project
@@ -56,9 +67,9 @@ Create `.mcp.json` in your project root:
```json ```json
{ {
"mcpServers": { "mcpServers": {
"supervisor": { "routing": {
"type": "http", "type": "http",
"url": "http://koala:30320/mcp" "url": "http://koala:30310/mcp"
}, },
"brain": { "brain": {
"type": "http", "type": "http",
@@ -68,33 +79,29 @@ Create `.mcp.json` in your project root:
} }
``` ```
Two MCP servers are exposed today, both reachable over Tailscale: Two MCP servers are exposed, both reachable over Tailscale:
- **`supervisor`** at `koala:30320` — skill workers (`tdd_red/green/refactor`, - **`routing`** at `koala:30310` — skill workers (`review`, `debug`, `retrospective`, `trainer`).
`review`, `debug`, `spec`, `retrospective`, `trainer`, `tier`). Routes each call to fast local model or thinking model based on per-skill pass rate.
- **`brain`** at `koala:30330` — knowledge access (`brain_query`, `brain_write`, - **`brain`** at `koala:30330` — knowledge access (`brain_query`, `brain_write`,
`brain_ingest`, `brain_ingest_raw`) and `session_log`. Hosted by the ingestion `brain_ingest`, `brain_ingest_raw`, `brain_answer`, `brain_classify`) and `session_log`.
service directly, no separate pod.
No local binary or stdio shim is required — Claude Code talks to both via HTTP. No local binary or stdio shim is required — Claude Code talks to both via HTTP.
Open Claude Code in your project — run `/mcp` to confirm both servers are listed. Open Claude Code in your project — run `/mcp` to confirm both servers are listed.
## A typical TDD session ## A typical session
``` ```
1. Call tdd_red → spec in, failing test file out 1. Call review → brain context injected + local model review → findings
2. Call tdd_green → test path in, implementation out 2. Call session_log → log each phase result
3. Call tdd_refactor → impl + test in, cleaned code out 3. Call retrospective → extracts learnings → brain
4. Call session_log → log each phase result 4. Future sessions: call brain_query / brain_answer to retrieve relevant context
5. Call retrospective → extracts learnings → brain/raw/
6. Review brain/raw/, move worthy notes to brain/wiki/concepts/
7. Future sessions: call brain_query to retrieve relevant context
``` ```
## Tier detection ## Tier detection
The supervisor probes connectivity at call time: The routing pod probes connectivity at call time:
| Tier | Label | Condition | | Tier | Label | Condition |
|------|-------|-----------| |------|-------|-----------|
@@ -102,31 +109,47 @@ The supervisor probes connectivity at call time:
| 2 | lan-only | Can reach LiteLLM but not Anthropic | | 2 | lan-only | Can reach LiteLLM but not Anthropic |
| 3 | airplane | No external connectivity | | 3 | airplane | No external connectivity |
## Model routing
The routing pod selects models per skill call based on historical pass rate:
| Pass rate | Decision |
|-----------|----------|
| ≥ 0.90 (FLOOR) | Fast model (`HYPERGUILD_FAST_MODEL`) |
| ≤ 0.70 (CEIL) | Thinking model (`HYPERGUILD_THINKING_MODEL`) |
| between CEIL and FLOOR | Sample band — probabilistic routing |
| nil (no history yet) | Defaults to thinking model |
> **Bootstrap note:** With no session history, all calls route to the thinking model.
> The fast-model path activates only after real pass-rate data accumulates at `/pass-rate`.
> Seed with real usage — don't try to pre-populate.
## Key env vars ## Key env vars
| Variable | Default | Purpose | | Variable | Default | Purpose |
|----------|---------|---------| |----------|---------|---------|
| `INGEST_BRAIN_DIR` | `../brain` | Brain directory for ingestion server | | `INGEST_BRAIN_DIR` | `../brain` | Brain directory for ingestion server |
| `INGEST_PORT` | `3300` | Ingestion server port | | `INGEST_PORT` | `3300` | Ingestion server port |
| `SUPERVISOR_CONFIG_DIR` | `./config/supervisor` | Skill discipline files | | `INGEST_BASE_URL` | `http://localhost:3300` | Routing pod → brain |
| `SUPERVISOR_SESSIONS_DIR` | `./brain/sessions` | JSONL session logs |
| `INGEST_BASE_URL` | `http://localhost:3300` | Supervisor → ingestion |
| `LITELLM_BASE_URL` | — | LiteLLM proxy for Tier 2 model routing | | `LITELLM_BASE_URL` | — | LiteLLM proxy for Tier 2 model routing |
| `SUPERVISOR_MCP_TOKEN` | — | Optional bearer token for the supervisor MCP HTTP endpoint; when empty, no auth is enforced |
| `ROUTING_PORT` | `3210` | Routing pod's listen port | | `ROUTING_PORT` | `3210` | Routing pod's listen port |
| `ROUTING_MCP_TOKEN` | — | Optional bearer token for the routing MCP HTTP endpoint | | `ROUTING_MCP_TOKEN` | — | Optional bearer token; when empty, no auth enforced |
| `BRAIN_URL` | `http://ingestion.supervisor:3300` | Routing pod → brain (in-cluster) | | `BRAIN_URL` | `http://ingestion.supervisor:3300` | Routing pod → brain (in-cluster) |
| `HYPERGUILD_FAST_MODEL` | `koala/qwen35-9b-fast` | Fast model for high-pass-rate skill calls | | `HYPERGUILD_FAST_MODEL` | `koala/qwen35-9b-fast` | Fast model for high-pass-rate skill calls |
| `HYPERGUILD_THINKING_MODEL` | `iguana/gemma4-26b` | Thinking model for low-pass-rate skill calls | | `HYPERGUILD_THINKING_MODEL` | `iguana/gemma4-26b` | Thinking model for low-pass-rate skill calls |
| `HYPERGUILD_ROUTE_LOCAL_FLOOR` | `0.90` | At/above pass rate, route to fast model | | `HYPERGUILD_ROUTE_LOCAL_FLOOR` | `0.90` | Fast model threshold |
| `HYPERGUILD_ROUTE_LOCAL_CEIL` | `0.70` | Below pass rate, route to thinking model. Between CEIL and FLOOR is the sample band. | | `HYPERGUILD_ROUTE_LOCAL_CEIL` | `0.70` | Thinking model threshold |
| `HYPERGUILD_PASS_RATE_TTL_SECONDS` | `60` | Per-skill pass-rate cache TTL | | `HYPERGUILD_PASS_RATE_TTL_SECONDS` | `60` | Per-skill pass-rate cache TTL |
> **Operator note:** LiteLLM at `LITELLM_BASE_URL` must register both `HYPERGUILD_FAST_MODEL` and `HYPERGUILD_THINKING_MODEL` for routing to do useful work. If a model is missing, LiteLLM returns 4xx, the routing pod's fast route fails, the fail-open retry on the thinking model likely also fails (since both are missing), and the only signal is `final_status: "fail"` on `_routing` entries in the brain. > **Operator note:** LiteLLM at `LITELLM_BASE_URL` must register both `HYPERGUILD_FAST_MODEL`
> and `HYPERGUILD_THINKING_MODEL`. If a model is missing, the fail-open retry also fails and
> the only signal is `final_status: "fail"` on `_routing` entries in the brain.
## Phase 2 (planned) ## Open issues
- `review` skill — structured code review with iron law enforcement See [issues](http://gitea.d-ma.be/mathias/hyperguild/issues) — key open items:
- `debug` skill — hypothesis-driven debugging sessions
- `spec` skill — generates specs from conversations - **#25** — skills platform overhaul (audit first, then lazy loading + brain feedback loop)
- `trainer` — extracts SFT/DPO pairs from session logs for fine-tuning - **#24** — reduce context burn from skill listing
- **#22** — migrate legacy brain notes to wing/hall layout (one-shot script, low risk)
- **#31** — connect routing-mcp to claude.ai as custom connector
+167
View File
@@ -0,0 +1,167 @@
# baseline-pre-fix — 20 questions, k=5
top-1 hit rate: 4/20 = 20%
top-3 hit rate: 13/20 = 65%
## per-question detail
· rank=3 expected=dex-in-memory-storage-wipes-oauth-tokens-on-every-pod-restart
q: how do I stop dex from logging users out on every pod restart?
1. homelab-network-perimeter-model
2. 2026-05-12-koala-machine-state
3. dex-in-memory-storage-wipes-oauth-tokens-on-every-pod-restart <-- expected
4. infra-litellm-absorption-2026-05-16
5. Financial Sentiment Analysis on Stock Market Headlines With FinBERT & HuggingFace
★ rank=1 expected=postgres-least-privilege-migration-tenant-grant-bypass-2026-05
q: my postgres-exporter broke after revoking PUBLIC CONNECT — why?
1. postgres-least-privilege-migration-tenant-grant-bypass-2026-05 <-- expected
2. infra-litellm-absorption-2026-05-16
3. brain-mcp-activation-runbook
4. extension-version-lags-platform-major-upgrade
5. ntfy-deny-all-rollout-ordering-keep-alert-pipeline-live-during-auth-flip
★ rank=1 expected=homelab-network-perimeter-model
q: when is a NodePort acceptable vs needing a public ingress with bearer gate?
1. homelab-network-perimeter-model <-- expected
2. qwen3-thinking-model-empty-content-trap
3. mcpclient-empty-token-silent-401-envfrom-missing-key
4. 2026-05-12-koala-machine-state
5. koala-llama-swap-native-tool-calls-survey-2026-05
· rank=3 expected=exit-255-unknown-reason-not-oom
q: what does container exit code 255 with reason Unknown mean?
1. qwen3-thinking-model-empty-content-trap
2. infra-litellm-absorption-2026-05-16
3. exit-255-unknown-reason-not-oom <-- expected
4. mcpclient-empty-token-silent-401-envfrom-missing-key
5. koala-llama-swap-native-tool-calls-survey-2026-05
· rank=3 expected=gitea-push-mirror-cannot-create-remote-repo-needs-pre-existing-github-repo
q: can gitea push-mirror create the github repo automatically?
1. infra-litellm-absorption-2026-05-16
2. Autoresearch
3. gitea-push-mirror-cannot-create-remote-repo-needs-pre-existing-github-repo <-- expected
4. adr-new-project-gitea-first-github-mirror
5. adr-github-as-primary-remote
✗ rank=0 expected=flux-healthcheck-stale-on-resource-removal
q: a flux kustomization is stuck after I removed a resource — why?
1. qwen3-thinking-model-empty-content-trap
2. 2026-05-12-koala-machine-state
3. homelab-architecture-principles-2026-05
4. gitea-mcp: full stack shipped end-to-end (2026-05-05)
5. k8s-configmap-mount-no-reload-needs-pod-restart
· rank=2 expected=go-bytes-buffer-bytes-reset-aliasing-trap
q: the bytes buffer aliasing trap with Reset in a loop — what's the bug?
1. Financial Sentiment Analysis on Stock Market Headlines With FinBERT & HuggingFace
2. go-bytes-buffer-bytes-reset-aliasing-trap <-- expected
3. homelab-security-chains-not-bugs
4. training-on-rtx-5070-pretraining-vs-finetuning
5. Hash Encoding
★ rank=1 expected=homelab-architecture-principles-2026-05
q: what are the homelab architecture principles from may 2026?
1. homelab-architecture-principles-2026-05 <-- expected
2. homelab-network-perimeter-model
3. Claude Managed Agents — architecture notes relevant to homelab agent platform
4. homelab-core-glossary
5. 2026-05-12-koala-machine-state
✗ rank=0 expected=2026-05-04-sops-age-key-from-flux-cluster
q: where does the sops age private key live in the cluster?
1. 2026-05-12-koala-machine-state
2. homelab-network-perimeter-model
3. postgres-least-privilege-migration-tenant-grant-bypass-2026-05
4. brain-mcp-activation-runbook
5. dex-in-memory-storage-wipes-oauth-tokens-on-every-pod-restart
✗ rank=0 expected=grafana-dashboards-as-code-not-ui-state
q: why do my grafana dashboards disappear after a pod restart?
1. infra-litellm-absorption-2026-05-16
2. 2026-05-12-koala-machine-state
3. Financial Sentiment Analysis on Stock Market Headlines With FinBERT & HuggingFace
4. brain-mcp-activation-runbook
5. dex-in-memory-storage-wipes-oauth-tokens-on-every-pod-restart
· rank=2 expected=double-diamond-methodology
q: what is the double diamond methodology?
1. Harnessing the Power of Hash Encoding for Categorical Data in Data Science
2. double-diamond-methodology <-- expected
3. unified-methodology-diamond-futures-autoresearch
4. futures-thinking-extended-double-diamond
5. insight-exploration-as-diamond-1
· rank=3 expected=2026-05-04-mcp-transport-version-claude-ai-strict
q: my MCP server works from claude code but fails on claude.ai — what's different?
1. qwen3-thinking-model-empty-content-trap
2. mcp-resource-url-empty-breaks-claude-ai-discovery-silently
3. 2026-05-04-mcp-transport-version-claude-ai-strict <-- expected
4. 2026-05-04-claude-ai-custom-mcp-connectors
5. finding-github-mcp-claudeai-vs-claudecode
· rank=2 expected=homelab-security-chains-not-bugs
q: how should I rate security findings — isolated bugs or exploit chains?
1. homelab-network-perimeter-model
2. homelab-security-chains-not-bugs <-- expected
3. Financial Sentiment Analysis on Stock Market Headlines With FinBERT & HuggingFace
4. policy-audit-mode-blocks-nothing
5. homelab-document-accepted-risk-to-break-audit-cycle
· rank=2 expected=2026-05-03-canonical-vs-derived-context-flow
q: how should canonical context files relate to derived adapter files?
1. qwen3-thinking-model-empty-content-trap
2. 2026-05-03-canonical-vs-derived-context-flow <-- expected
3. 2026-05-12-koala-machine-state
4. 2026-05-04-claude-ai-custom-mcp-connectors
5. koala-llama-swap-native-tool-calls-survey-2026-05
· rank=2 expected=homelab-core-glossary
q: what is the homelab core vocabulary glossary?
1. homelab-architecture-principles-2026-05
2. homelab-core-glossary <-- expected
3. Claude Managed Agents — architecture notes relevant to homelab agent platform
4. 2026-05-12-koala-machine-state
5. Autoresearch
★ rank=1 expected=koala-llama-swap-native-tool-calls-survey-2026-05
q: which models on koala llama-swap actually emit native tool_calls correctly?
1. koala-llama-swap-native-tool-calls-survey-2026-05 <-- expected
2. 2026-05-12-koala-machine-state
3. infra-litellm-absorption-2026-05-16
4. training-on-rtx-5070-pretraining-vs-finetuning
5. qwen3-thinking-model-empty-content-trap
✗ rank=0 expected=qwen35-9b-fast
q: what is qwen35-9b-fast and what's it used for?
1. koala-llama-swap-native-tool-calls-survey-2026-05
2. qwen3-thinking-model-empty-content-trap
3. Qwen35-9b-fast
4. infra-litellm-absorption-2026-05-16
5. 2026-05-12-koala-machine-state
✗ rank=0 expected=go-defer-errcheck-body-close
q: in go, how do I prevent defer body close from silently dropping errors?
1. infra-litellm-absorption-2026-05-16
2. homelab-network-perimeter-model
3. go-bytes-buffer-bytes-reset-aliasing-trap
4. mcpclient-empty-token-silent-401-envfrom-missing-key
5. brain-mcp-activation-runbook
✗ rank=0 expected=hyperguild-level3-pipeline-rewrite
q: what was the level 3 rewrite of hyperguild's ingestion pipeline?
1. 2026-05-12-koala-machine-state
2. homelab-core-glossary
3. brain-mcp-activation-runbook
4. koala-llama-swap-native-tool-calls-survey-2026-05
5. infra-litellm-absorption-2026-05-16
? rank=4 expected=adr-new-project-gitea-first-github-mirror
q: what's the new-project ADR — is it gitea-first or github-first?
1. gitea-push-mirror-cannot-create-remote-repo-needs-pre-existing-github-repo
2. gitea-mcp: full stack shipped end-to-end (2026-05-05)
3. mcp-tool-design-get-needs-list-partner
4. adr-new-project-gitea-first-github-mirror <-- expected
5. 2026-05-04-gitea-mcp-build-session
+167
View File
@@ -0,0 +1,167 @@
# post-fix — 20 questions, k=5
top-1 hit rate: 4/20 = 20%
top-3 hit rate: 14/20 = 70%
## per-question detail
· rank=3 expected=dex-in-memory-storage-wipes-oauth-tokens-on-every-pod-restart
q: how do I stop dex from logging users out on every pod restart?
1. homelab-network-perimeter-model
2. 2026-05-12-koala-machine-state
3. dex-in-memory-storage-wipes-oauth-tokens-on-every-pod-restart <-- expected
4. infra-litellm-absorption-2026-05-16
5. Financial Sentiment Analysis on Stock Market Headlines With FinBERT & HuggingFace
★ rank=1 expected=postgres-least-privilege-migration-tenant-grant-bypass-2026-05
q: my postgres-exporter broke after revoking PUBLIC CONNECT — why?
1. postgres-least-privilege-migration-tenant-grant-bypass-2026-05 <-- expected
2. infra-litellm-absorption-2026-05-16
3. brain-mcp-activation-runbook
4. extension-version-lags-platform-major-upgrade
5. ntfy-deny-all-rollout-ordering-keep-alert-pipeline-live-during-auth-flip
★ rank=1 expected=homelab-network-perimeter-model
q: when is a NodePort acceptable vs needing a public ingress with bearer gate?
1. homelab-network-perimeter-model <-- expected
2. qwen3-thinking-model-empty-content-trap
3. mcpclient-empty-token-silent-401-envfrom-missing-key
4. 2026-05-12-koala-machine-state
5. koala-llama-swap-native-tool-calls-survey-2026-05
· rank=3 expected=exit-255-unknown-reason-not-oom
q: what does container exit code 255 with reason Unknown mean?
1. qwen3-thinking-model-empty-content-trap
2. infra-litellm-absorption-2026-05-16
3. exit-255-unknown-reason-not-oom <-- expected
4. mcpclient-empty-token-silent-401-envfrom-missing-key
5. koala-llama-swap-native-tool-calls-survey-2026-05
· rank=3 expected=gitea-push-mirror-cannot-create-remote-repo-needs-pre-existing-github-repo
q: can gitea push-mirror create the github repo automatically?
1. infra-litellm-absorption-2026-05-16
2. Autoresearch
3. gitea-push-mirror-cannot-create-remote-repo-needs-pre-existing-github-repo <-- expected
4. adr-new-project-gitea-first-github-mirror
5. adr-github-as-primary-remote
✗ rank=0 expected=flux-healthcheck-stale-on-resource-removal
q: a flux kustomization is stuck after I removed a resource — why?
1. qwen3-thinking-model-empty-content-trap
2. 2026-05-12-koala-machine-state
3. homelab-architecture-principles-2026-05
4. gitea-mcp: full stack shipped end-to-end (2026-05-05)
5. k8s-configmap-mount-no-reload-needs-pod-restart
· rank=2 expected=go-bytes-buffer-bytes-reset-aliasing-trap
q: the bytes buffer aliasing trap with Reset in a loop — what's the bug?
1. Financial Sentiment Analysis on Stock Market Headlines With FinBERT & HuggingFace
2. go-bytes-buffer-bytes-reset-aliasing-trap <-- expected
3. homelab-security-chains-not-bugs
4. training-on-rtx-5070-pretraining-vs-finetuning
5. Hash Encoding
★ rank=1 expected=homelab-architecture-principles-2026-05
q: what are the homelab architecture principles from may 2026?
1. homelab-architecture-principles-2026-05 <-- expected
2. homelab-network-perimeter-model
3. Claude Managed Agents — architecture notes relevant to homelab agent platform
4. homelab-core-glossary
5. 2026-05-12-koala-machine-state
✗ rank=0 expected=2026-05-04-sops-age-key-from-flux-cluster
q: where does the sops age private key live in the cluster?
1. 2026-05-12-koala-machine-state
2. homelab-network-perimeter-model
3. postgres-least-privilege-migration-tenant-grant-bypass-2026-05
4. brain-mcp-activation-runbook
5. dex-in-memory-storage-wipes-oauth-tokens-on-every-pod-restart
✗ rank=0 expected=grafana-dashboards-as-code-not-ui-state
q: why do my grafana dashboards disappear after a pod restart?
1. infra-litellm-absorption-2026-05-16
2. 2026-05-12-koala-machine-state
3. Financial Sentiment Analysis on Stock Market Headlines With FinBERT & HuggingFace
4. brain-mcp-activation-runbook
5. dex-in-memory-storage-wipes-oauth-tokens-on-every-pod-restart
· rank=2 expected=double-diamond-methodology
q: what is the double diamond methodology?
1. Harnessing the Power of Hash Encoding for Categorical Data in Data Science
2. double-diamond-methodology <-- expected
3. unified-methodology-diamond-futures-autoresearch
4. futures-thinking-extended-double-diamond
5. insight-exploration-as-diamond-1
· rank=3 expected=2026-05-04-mcp-transport-version-claude-ai-strict
q: my MCP server works from claude code but fails on claude.ai — what's different?
1. qwen3-thinking-model-empty-content-trap
2. mcp-resource-url-empty-breaks-claude-ai-discovery-silently
3. 2026-05-04-mcp-transport-version-claude-ai-strict <-- expected
4. 2026-05-04-claude-ai-custom-mcp-connectors
5. finding-github-mcp-claudeai-vs-claudecode
· rank=2 expected=homelab-security-chains-not-bugs
q: how should I rate security findings — isolated bugs or exploit chains?
1. homelab-network-perimeter-model
2. homelab-security-chains-not-bugs <-- expected
3. Financial Sentiment Analysis on Stock Market Headlines With FinBERT & HuggingFace
4. policy-audit-mode-blocks-nothing
5. homelab-document-accepted-risk-to-break-audit-cycle
· rank=2 expected=2026-05-03-canonical-vs-derived-context-flow
q: how should canonical context files relate to derived adapter files?
1. qwen3-thinking-model-empty-content-trap
2. 2026-05-03-canonical-vs-derived-context-flow <-- expected
3. 2026-05-12-koala-machine-state
4. 2026-05-04-claude-ai-custom-mcp-connectors
5. koala-llama-swap-native-tool-calls-survey-2026-05
· rank=2 expected=homelab-core-glossary
q: what is the homelab core vocabulary glossary?
1. homelab-architecture-principles-2026-05
2. homelab-core-glossary <-- expected
3. Claude Managed Agents — architecture notes relevant to homelab agent platform
4. 2026-05-12-koala-machine-state
5. Autoresearch
★ rank=1 expected=koala-llama-swap-native-tool-calls-survey-2026-05
q: which models on koala llama-swap actually emit native tool_calls correctly?
1. koala-llama-swap-native-tool-calls-survey-2026-05 <-- expected
2. 2026-05-12-koala-machine-state
3. infra-litellm-absorption-2026-05-16
4. training-on-rtx-5070-pretraining-vs-finetuning
5. qwen3-thinking-model-empty-content-trap
· rank=2 expected=qwen35-9b-fast
q: what is qwen35-9b-fast and what's it used for?
1. koala-llama-swap-native-tool-calls-survey-2026-05
2. qwen35-9b-fast <-- expected
3. qwen3-thinking-model-empty-content-trap
4. infra-litellm-absorption-2026-05-16
5. 2026-05-12-koala-machine-state
✗ rank=0 expected=go-defer-errcheck-body-close
q: in go, how do I prevent defer body close from silently dropping errors?
1. infra-litellm-absorption-2026-05-16
2. homelab-network-perimeter-model
3. go-bytes-buffer-bytes-reset-aliasing-trap
4. mcpclient-empty-token-silent-401-envfrom-missing-key
5. brain-mcp-activation-runbook
✗ rank=0 expected=hyperguild-level3-pipeline-rewrite
q: what was the level 3 rewrite of hyperguild's ingestion pipeline?
1. 2026-05-12-koala-machine-state
2. homelab-core-glossary
3. brain-mcp-activation-runbook
4. koala-llama-swap-native-tool-calls-survey-2026-05
5. infra-litellm-absorption-2026-05-16
? rank=4 expected=adr-new-project-gitea-first-github-mirror
q: what's the new-project ADR — is it gitea-first or github-first?
1. gitea-push-mirror-cannot-create-remote-repo-needs-pre-existing-github-repo
2. gitea-mcp: full stack shipped end-to-end (2026-05-05)
3. mcp-tool-design-get-needs-list-partner
4. adr-new-project-gitea-first-github-mirror <-- expected
5. 2026-05-04-gitea-mcp-build-session
+167
View File
@@ -0,0 +1,167 @@
# post-m4-tier-weighting — 20 questions, k=5
top-1 hit rate: 6/20 = 30%
top-3 hit rate: 15/20 = 75%
## per-question detail
· rank=3 expected=dex-in-memory-storage-wipes-oauth-tokens-on-every-pod-restart
q: how do I stop dex from logging users out on every pod restart?
1. homelab-network-perimeter-model
2. 2026-05-12-koala-machine-state
3. dex-in-memory-storage-wipes-oauth-tokens-on-every-pod-restart <-- expected
4. infra-litellm-absorption-2026-05-16
5. k8s-configmap-mount-no-reload-needs-pod-restart
· rank=2 expected=postgres-least-privilege-migration-tenant-grant-bypass-2026-05
q: my postgres-exporter broke after revoking PUBLIC CONNECT — why?
1. infra-litellm-absorption-2026-05-16
2. postgres-least-privilege-migration-tenant-grant-bypass-2026-05 <-- expected
3. extension-version-lags-platform-major-upgrade
4. ntfy-deny-all-rollout-ordering-keep-alert-pipeline-live-during-auth-flip
5. gitea-push-mirror-cannot-create-remote-repo-needs-pre-existing-github-repo
★ rank=1 expected=homelab-network-perimeter-model
q: when is a NodePort acceptable vs needing a public ingress with bearer gate?
1. homelab-network-perimeter-model <-- expected
2. qwen3-thinking-model-empty-content-trap
3. mcpclient-empty-token-silent-401-envfrom-missing-key
4. 2026-05-12-koala-machine-state
5. koala-llama-swap-native-tool-calls-survey-2026-05
· rank=3 expected=exit-255-unknown-reason-not-oom
q: what does container exit code 255 with reason Unknown mean?
1. qwen3-thinking-model-empty-content-trap
2. infra-litellm-absorption-2026-05-16
3. exit-255-unknown-reason-not-oom <-- expected
4. mcpclient-empty-token-silent-401-envfrom-missing-key
5. koala-llama-swap-native-tool-calls-survey-2026-05
· rank=2 expected=gitea-push-mirror-cannot-create-remote-repo-needs-pre-existing-github-repo
q: can gitea push-mirror create the github repo automatically?
1. infra-litellm-absorption-2026-05-16
2. gitea-push-mirror-cannot-create-remote-repo-needs-pre-existing-github-repo <-- expected
3. adr-new-project-gitea-first-github-mirror
4. adr-github-as-primary-remote
5. 2026-05-12-koala-machine-state
✗ rank=0 expected=flux-healthcheck-stale-on-resource-removal
q: a flux kustomization is stuck after I removed a resource — why?
1. qwen3-thinking-model-empty-content-trap
2. 2026-05-12-koala-machine-state
3. homelab-architecture-principles-2026-05
4. k8s-configmap-mount-no-reload-needs-pod-restart
5. training-on-rtx-5070-pretraining-vs-finetuning
★ rank=1 expected=go-bytes-buffer-bytes-reset-aliasing-trap
q: the bytes buffer aliasing trap with Reset in a loop — what's the bug?
1. go-bytes-buffer-bytes-reset-aliasing-trap <-- expected
2. homelab-security-chains-not-bugs
3. Financial Sentiment Analysis on Stock Market Headlines With FinBERT & HuggingFace
4. training-on-rtx-5070-pretraining-vs-finetuning
5. flux-healthcheck-stale-on-resource-removal
★ rank=1 expected=homelab-architecture-principles-2026-05
q: what are the homelab architecture principles from may 2026?
1. homelab-architecture-principles-2026-05 <-- expected
2. homelab-network-perimeter-model
3. homelab-core-glossary
4. 2026-05-12-koala-machine-state
5. pattern-reddit-tmux-multiagent-conductor
? rank=4 expected=2026-05-04-sops-age-key-from-flux-cluster
q: where does the sops age private key live in the cluster?
1. 2026-05-12-koala-machine-state
2. homelab-network-perimeter-model
3. dex-in-memory-storage-wipes-oauth-tokens-on-every-pod-restart
4. 2026-05-04-sops-age-key-from-flux-cluster <-- expected
5. homelab-security-chains-not-bugs
★ rank=1 expected=grafana-dashboards-as-code-not-ui-state
q: why do my grafana dashboards disappear after a pod restart?
1. grafana-dashboards-as-code-not-ui-state <-- expected
2. infra-litellm-absorption-2026-05-16
3. 2026-05-12-koala-machine-state
4. dex-in-memory-storage-wipes-oauth-tokens-on-every-pod-restart
5. k8s-configmap-mount-no-reload-needs-pod-restart
★ rank=1 expected=double-diamond-methodology
q: what is the double diamond methodology?
1. double-diamond-methodology <-- expected
2. unified-methodology-diamond-futures-autoresearch
3. futures-thinking-extended-double-diamond
4. insight-exploration-as-diamond-1
5. workflow-idea-to-running-service
· rank=3 expected=2026-05-04-mcp-transport-version-claude-ai-strict
q: my MCP server works from claude code but fails on claude.ai — what's different?
1. qwen3-thinking-model-empty-content-trap
2. mcp-resource-url-empty-breaks-claude-ai-discovery-silently
3. 2026-05-04-mcp-transport-version-claude-ai-strict <-- expected
4. 2026-05-04-claude-ai-custom-mcp-connectors
5. finding-github-mcp-claudeai-vs-claudecode
· rank=2 expected=homelab-security-chains-not-bugs
q: how should I rate security findings — isolated bugs or exploit chains?
1. homelab-network-perimeter-model
2. homelab-security-chains-not-bugs <-- expected
3. policy-audit-mode-blocks-nothing
4. homelab-document-accepted-risk-to-break-audit-cycle
5. audit-shortcut-tls-blocks-zero-equals-edge-only
· rank=2 expected=2026-05-03-canonical-vs-derived-context-flow
q: how should canonical context files relate to derived adapter files?
1. qwen3-thinking-model-empty-content-trap
2. 2026-05-03-canonical-vs-derived-context-flow <-- expected
3. 2026-05-12-koala-machine-state
4. 2026-05-04-claude-ai-custom-mcp-connectors
5. koala-llama-swap-native-tool-calls-survey-2026-05
· rank=2 expected=homelab-core-glossary
q: what is the homelab core vocabulary glossary?
1. homelab-architecture-principles-2026-05
2. homelab-core-glossary <-- expected
3. 2026-05-12-koala-machine-state
4. flux-kustomization-depends-on-bootstrap-ordering
5. brain-ingest-ntfy-service
★ rank=1 expected=koala-llama-swap-native-tool-calls-survey-2026-05
q: which models on koala llama-swap actually emit native tool_calls correctly?
1. koala-llama-swap-native-tool-calls-survey-2026-05 <-- expected
2. 2026-05-12-koala-machine-state
3. infra-litellm-absorption-2026-05-16
4. training-on-rtx-5070-pretraining-vs-finetuning
5. qwen3-thinking-model-empty-content-trap
✗ rank=0 expected=qwen35-9b-fast
q: what is qwen35-9b-fast and what's it used for?
1. koala-llama-swap-native-tool-calls-survey-2026-05
2. qwen3-thinking-model-empty-content-trap
3. infra-litellm-absorption-2026-05-16
4. 2026-05-12-koala-machine-state
5. index
✗ rank=0 expected=go-defer-errcheck-body-close
q: in go, how do I prevent defer body close from silently dropping errors?
1. homelab-network-perimeter-model
2. infra-litellm-absorption-2026-05-16
3. go-bytes-buffer-bytes-reset-aliasing-trap
4. mcpclient-empty-token-silent-401-envfrom-missing-key
5. koala-llama-swap-native-tool-calls-survey-2026-05
✗ rank=0 expected=hyperguild-level3-pipeline-rewrite
q: what was the level 3 rewrite of hyperguild's ingestion pipeline?
1. 2026-05-12-koala-machine-state
2. homelab-core-glossary
3. koala-llama-swap-native-tool-calls-survey-2026-05
4. infra-litellm-absorption-2026-05-16
5. homelab-architecture-principles-2026-05
· rank=3 expected=adr-new-project-gitea-first-github-mirror
q: what's the new-project ADR — is it gitea-first or github-first?
1. gitea-push-mirror-cannot-create-remote-repo-needs-pre-existing-github-repo
2. mcp-tool-design-get-needs-list-partner
3. adr-new-project-gitea-first-github-mirror <-- expected
4. 2026-05-04-gitea-mcp-build-session
5. adr-local-dev-vs-hyperguild-new-project
+167
View File
@@ -0,0 +1,167 @@
# post-m4b-entities-promoted — 20 questions, k=5
top-1 hit rate: 7/20 = 35%
top-3 hit rate: 16/20 = 80%
## per-question detail
· rank=3 expected=dex-in-memory-storage-wipes-oauth-tokens-on-every-pod-restart
q: how do I stop dex from logging users out on every pod restart?
1. homelab-network-perimeter-model
2. 2026-05-12-koala-machine-state
3. dex-in-memory-storage-wipes-oauth-tokens-on-every-pod-restart <-- expected
4. infra-litellm-absorption-2026-05-16
5. k8s-configmap-mount-no-reload-needs-pod-restart
· rank=2 expected=postgres-least-privilege-migration-tenant-grant-bypass-2026-05
q: my postgres-exporter broke after revoking PUBLIC CONNECT — why?
1. infra-litellm-absorption-2026-05-16
2. postgres-least-privilege-migration-tenant-grant-bypass-2026-05 <-- expected
3. extension-version-lags-platform-major-upgrade
4. ntfy-deny-all-rollout-ordering-keep-alert-pipeline-live-during-auth-flip
5. gitea-push-mirror-cannot-create-remote-repo-needs-pre-existing-github-repo
★ rank=1 expected=homelab-network-perimeter-model
q: when is a NodePort acceptable vs needing a public ingress with bearer gate?
1. homelab-network-perimeter-model <-- expected
2. qwen3-thinking-model-empty-content-trap
3. mcpclient-empty-token-silent-401-envfrom-missing-key
4. 2026-05-12-koala-machine-state
5. koala-llama-swap-native-tool-calls-survey-2026-05
· rank=3 expected=exit-255-unknown-reason-not-oom
q: what does container exit code 255 with reason Unknown mean?
1. qwen3-thinking-model-empty-content-trap
2. infra-litellm-absorption-2026-05-16
3. exit-255-unknown-reason-not-oom <-- expected
4. mcpclient-empty-token-silent-401-envfrom-missing-key
5. koala-llama-swap-native-tool-calls-survey-2026-05
· rank=2 expected=gitea-push-mirror-cannot-create-remote-repo-needs-pre-existing-github-repo
q: can gitea push-mirror create the github repo automatically?
1. infra-litellm-absorption-2026-05-16
2. gitea-push-mirror-cannot-create-remote-repo-needs-pre-existing-github-repo <-- expected
3. adr-new-project-gitea-first-github-mirror
4. adr-github-as-primary-remote
5. 2026-05-12-koala-machine-state
✗ rank=0 expected=flux-healthcheck-stale-on-resource-removal
q: a flux kustomization is stuck after I removed a resource — why?
1. qwen3-thinking-model-empty-content-trap
2. 2026-05-12-koala-machine-state
3. homelab-architecture-principles-2026-05
4. k8s-configmap-mount-no-reload-needs-pod-restart
5. training-on-rtx-5070-pretraining-vs-finetuning
★ rank=1 expected=go-bytes-buffer-bytes-reset-aliasing-trap
q: the bytes buffer aliasing trap with Reset in a loop — what's the bug?
1. go-bytes-buffer-bytes-reset-aliasing-trap <-- expected
2. homelab-security-chains-not-bugs
3. Financial Sentiment Analysis on Stock Market Headlines With FinBERT & HuggingFace
4. training-on-rtx-5070-pretraining-vs-finetuning
5. flux-healthcheck-stale-on-resource-removal
★ rank=1 expected=homelab-architecture-principles-2026-05
q: what are the homelab architecture principles from may 2026?
1. homelab-architecture-principles-2026-05 <-- expected
2. homelab-network-perimeter-model
3. homelab-core-glossary
4. 2026-05-12-koala-machine-state
5. pattern-reddit-tmux-multiagent-conductor
? rank=4 expected=2026-05-04-sops-age-key-from-flux-cluster
q: where does the sops age private key live in the cluster?
1. 2026-05-12-koala-machine-state
2. homelab-network-perimeter-model
3. dex-in-memory-storage-wipes-oauth-tokens-on-every-pod-restart
4. 2026-05-04-sops-age-key-from-flux-cluster <-- expected
5. homelab-security-chains-not-bugs
★ rank=1 expected=grafana-dashboards-as-code-not-ui-state
q: why do my grafana dashboards disappear after a pod restart?
1. grafana-dashboards-as-code-not-ui-state <-- expected
2. infra-litellm-absorption-2026-05-16
3. 2026-05-12-koala-machine-state
4. dex-in-memory-storage-wipes-oauth-tokens-on-every-pod-restart
5. k8s-configmap-mount-no-reload-needs-pod-restart
★ rank=1 expected=double-diamond-methodology
q: what is the double diamond methodology?
1. double-diamond-methodology <-- expected
2. unified-methodology-diamond-futures-autoresearch
3. futures-thinking-extended-double-diamond
4. insight-exploration-as-diamond-1
5. workflow-idea-to-running-service
· rank=3 expected=2026-05-04-mcp-transport-version-claude-ai-strict
q: my MCP server works from claude code but fails on claude.ai — what's different?
1. qwen3-thinking-model-empty-content-trap
2. mcp-resource-url-empty-breaks-claude-ai-discovery-silently
3. 2026-05-04-mcp-transport-version-claude-ai-strict <-- expected
4. 2026-05-04-claude-ai-custom-mcp-connectors
5. finding-github-mcp-claudeai-vs-claudecode
· rank=2 expected=homelab-security-chains-not-bugs
q: how should I rate security findings — isolated bugs or exploit chains?
1. homelab-network-perimeter-model
2. homelab-security-chains-not-bugs <-- expected
3. policy-audit-mode-blocks-nothing
4. homelab-document-accepted-risk-to-break-audit-cycle
5. audit-shortcut-tls-blocks-zero-equals-edge-only
· rank=2 expected=2026-05-03-canonical-vs-derived-context-flow
q: how should canonical context files relate to derived adapter files?
1. qwen3-thinking-model-empty-content-trap
2. 2026-05-03-canonical-vs-derived-context-flow <-- expected
3. 2026-05-12-koala-machine-state
4. 2026-05-04-claude-ai-custom-mcp-connectors
5. koala-llama-swap-native-tool-calls-survey-2026-05
· rank=2 expected=homelab-core-glossary
q: what is the homelab core vocabulary glossary?
1. homelab-architecture-principles-2026-05
2. homelab-core-glossary <-- expected
3. 2026-05-12-koala-machine-state
4. qwen35-9b-fast
5. flux-kustomization-depends-on-bootstrap-ordering
★ rank=1 expected=koala-llama-swap-native-tool-calls-survey-2026-05
q: which models on koala llama-swap actually emit native tool_calls correctly?
1. koala-llama-swap-native-tool-calls-survey-2026-05 <-- expected
2. 2026-05-12-koala-machine-state
3. infra-litellm-absorption-2026-05-16
4. training-on-rtx-5070-pretraining-vs-finetuning
5. qwen3-thinking-model-empty-content-trap
★ rank=1 expected=qwen35-9b-fast
q: what is qwen35-9b-fast and what's it used for?
1. qwen35-9b-fast <-- expected
2. koala-llama-swap-native-tool-calls-survey-2026-05
3. qwen3-thinking-model-empty-content-trap
4. infra-litellm-absorption-2026-05-16
5. 2026-05-12-koala-machine-state
✗ rank=0 expected=go-defer-errcheck-body-close
q: in go, how do I prevent defer body close from silently dropping errors?
1. homelab-network-perimeter-model
2. infra-litellm-absorption-2026-05-16
3. go-bytes-buffer-bytes-reset-aliasing-trap
4. mcpclient-empty-token-silent-401-envfrom-missing-key
5. koala-llama-swap-native-tool-calls-survey-2026-05
✗ rank=0 expected=hyperguild-level3-pipeline-rewrite
q: what was the level 3 rewrite of hyperguild's ingestion pipeline?
1. 2026-05-12-koala-machine-state
2. homelab-core-glossary
3. koala-llama-swap-native-tool-calls-survey-2026-05
4. infra-litellm-absorption-2026-05-16
5. homelab-architecture-principles-2026-05
· rank=3 expected=adr-new-project-gitea-first-github-mirror
q: what's the new-project ADR — is it gitea-first or github-first?
1. gitea-push-mirror-cannot-create-remote-repo-needs-pre-existing-github-repo
2. mcp-tool-design-get-needs-list-partner
3. adr-new-project-gitea-first-github-mirror <-- expected
4. 2026-05-04-gitea-mcp-build-session
5. adr-local-dev-vs-hyperguild-new-project
+76
View File
@@ -0,0 +1,76 @@
# Brain retrieval eval set — 2026-05-24
20 hand-authored Q→expected-top-1-slug pairs. Used by `score.sh` to
measure brain_query top-1 + top-3 hit rate against the live brain.
Authoring rules:
- Each question maps to **one** clear-best entry. Avoid ambiguous
questions where multiple slugs could be the right answer.
- Questions are phrased the way a future-me would actually ask, not
the way the entry's title reads. Some lexical distance is the point.
- `expected` is the slug as stored in `brain_entities.slug`. Update
if the slug renames.
## Pairs
```
q: how do I stop dex from logging users out on every pod restart?
expected: dex-in-memory-storage-wipes-oauth-tokens-on-every-pod-restart
q: my postgres-exporter broke after revoking PUBLIC CONNECT — why?
expected: postgres-least-privilege-migration-tenant-grant-bypass-2026-05
q: when is a NodePort acceptable vs needing a public ingress with bearer gate?
expected: homelab-network-perimeter-model
q: what does container exit code 255 with reason Unknown mean?
expected: exit-255-unknown-reason-not-oom
q: can gitea push-mirror create the github repo automatically?
expected: gitea-push-mirror-cannot-create-remote-repo-needs-pre-existing-github-repo
q: a flux kustomization is stuck after I removed a resource — why?
expected: flux-healthcheck-stale-on-resource-removal
q: the bytes buffer aliasing trap with Reset in a loop — what's the bug?
expected: go-bytes-buffer-bytes-reset-aliasing-trap
q: what are the homelab architecture principles from may 2026?
expected: homelab-architecture-principles-2026-05
q: where does the sops age private key live in the cluster?
expected: 2026-05-04-sops-age-key-from-flux-cluster
q: why do my grafana dashboards disappear after a pod restart?
expected: grafana-dashboards-as-code-not-ui-state
q: what is the double diamond methodology?
expected: double-diamond-methodology
q: my MCP server works from claude code but fails on claude.ai — what's different?
expected: 2026-05-04-mcp-transport-version-claude-ai-strict
q: how should I rate security findings — isolated bugs or exploit chains?
expected: homelab-security-chains-not-bugs
q: how should canonical context files relate to derived adapter files?
expected: 2026-05-03-canonical-vs-derived-context-flow
q: what is the homelab core vocabulary glossary?
expected: homelab-core-glossary
q: which models on koala llama-swap actually emit native tool_calls correctly?
expected: koala-llama-swap-native-tool-calls-survey-2026-05
q: what is qwen35-9b-fast and what's it used for?
expected: qwen35-9b-fast
q: in go, how do I prevent defer body close from silently dropping errors?
expected: go-defer-errcheck-body-close
q: what was the level 3 rewrite of hyperguild's ingestion pipeline?
expected: hyperguild-level3-pipeline-rewrite
q: what's the new-project ADR — is it gitea-first or github-first?
expected: adr-new-project-gitea-first-github-mirror
```
+131
View File
@@ -0,0 +1,131 @@
#!/usr/bin/env python3
"""Score brain_query against the qa-2026-05.md eval set.
Reads `q:` / `expected:` pairs, calls brain_query MCP for each, records
top-1 + top-3 hit rate. Run:
BRAIN_MCP_TOKEN=$(grep '^export BRAIN_MCP_TOKEN=' ~/.llmkeys | cut -d= -f2-) \\
python3 score.py qa-2026-05.md
Optionally pass --baseline <name> to save the result as a labeled run.
"""
import argparse
import json
import os
import re
import sys
import time
import urllib.request
ENDPOINT = "https://brain-mcp.d-ma.be/mcp"
def load_pairs(path):
pairs = []
q = None
with open(path) as f:
for line in f:
line = line.rstrip()
if line.startswith("q:"):
q = line[2:].strip()
elif line.startswith("expected:") and q is not None:
expected = line[len("expected:"):].strip()
pairs.append((q, expected))
q = None
return pairs
def brain_query(token, query, k=5):
body = json.dumps({
"jsonrpc": "2.0",
"id": 1,
"method": "tools/call",
"params": {"name": "brain_query", "arguments": {"query": query, "k": k}},
}).encode()
req = urllib.request.Request(
ENDPOINT,
data=body,
headers={
"Authorization": f"Bearer {token}",
"Content-Type": "application/json",
"Accept": "application/json, text/event-stream",
},
method="POST",
)
with urllib.request.urlopen(req, timeout=30) as r:
raw = r.read().decode()
for line in raw.splitlines():
if line.startswith("data:"):
raw = line[5:].strip()
break
d = json.loads(raw)
if "error" in d:
raise RuntimeError(d["error"])
text = d["result"]["content"][0]["text"]
return json.loads(text).get("results", [])
def slug_of(result):
# `title` mirrors the slug in brain_entities for normal entries.
# Fall back to basename(path) if title is missing.
t = result.get("title", "")
if t:
return t
p = result.get("path", "")
return re.sub(r"\.md$", "", os.path.basename(p))
def main():
ap = argparse.ArgumentParser()
ap.add_argument("evalset")
ap.add_argument("--baseline", default="run")
ap.add_argument("--k", type=int, default=5)
args = ap.parse_args()
token = os.environ.get("BRAIN_MCP_TOKEN")
if not token:
sys.exit("BRAIN_MCP_TOKEN not set")
pairs = load_pairs(args.evalset)
if not pairs:
sys.exit(f"no pairs in {args.evalset}")
print(f"# {args.baseline}{len(pairs)} questions, k={args.k}")
print()
hits1 = 0
hits3 = 0
detail = []
for q, expected in pairs:
try:
results = brain_query(token, q, k=args.k)
except Exception as e:
detail.append((q, expected, [], f"ERR {e}"))
continue
slugs = [slug_of(r) for r in results]
rank = slugs.index(expected) + 1 if expected in slugs else 0
h1 = 1 if rank == 1 else 0
h3 = 1 if 0 < rank <= 3 else 0
hits1 += h1
hits3 += h3
detail.append((q, expected, slugs, rank))
total = len(pairs)
print(f"top-1 hit rate: {hits1}/{total} = {100*hits1/total:.0f}%")
print(f"top-3 hit rate: {hits3}/{total} = {100*hits3/total:.0f}%")
print()
print("## per-question detail")
print()
for q, expected, slugs, rank in detail:
marker = {0: "", 1: "", 2: "·", 3: "·"}.get(rank, "?")
if isinstance(rank, str):
marker = "!"
print(f"{marker} rank={rank} expected={expected}")
print(f" q: {q}")
for i, s in enumerate(slugs[:args.k], 1):
mark = " <-- expected" if s == expected else ""
print(f" {i}. {s}{mark}")
print()
if __name__ == "__main__":
main()
@@ -0,0 +1,48 @@
{"_meta":true,"note":"Agent-consumer column of brain-MCP intent analysis. consumer_type fixed=autonomous_agent. CAVEAT: canonical schema file brain-intent-extraction.md is NOT present on this host (koala) — only this session's own task prompt references it. The closed intent vocabulary below was RECONSTRUCTED from the task prompt's framing + brain/schema.md. Re-map intent labels if the canonical vocab differs. schema_source=reconstructed on every row.","closed_intent_vocab":["semantic_retrieval","lexical_lookup","check_prior_art","synthesized_answer","store_new_knowledge","update_or_supersede","ingest_raw_source","verify_write_landed","discover_capability","intent_unclear"],"intent_tool_match_values":["match","mismatch","partial"],"corpus":"~/.claude/projects/*/*.jsonl (Claude Code agent transcripts on koala). brain/sessions/*.jsonl empty. agentsquad docs/eval/*.jsonl are code-review eval results, NOT brain calls. No separate Crush logs found. Zero brain calls appear under any mcp__ name with a human typing the call — all brain acts are agent-initiated (CLAUDE.md reflex), so all qualify as autonomous_agent."}
{"id":"a01","session":"tapir-c","ts":"2026-06-?T15:01:51","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"single pre-task query 'YouTube Data API captions download ownership limitation timedtext adapter Go' — named-entity lexical lookup, fit BM25 well, no reformulation."}
{"id":"a02","session":"tapir","ts":"2026-06-05T21:44:10","tool":"brain_ingest","intent":"ingest_raw_source","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_ingest 14s earlier — tool not ambient, had to be discovered/loaded first.","evidence":"source=tapir-scheduled-discovery-session-2026-06-05, a session learnings dump."}
{"id":"a03","session":"tapir","ts":"2026-06-?T14:00:59","tool":"brain_ingest","intent":"ingest_raw_source","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"source=tapir-rls-identity-bootstrapping, first write of RLS lesson."}
{"id":"a04","session":"tapir","ts":"2026-06-?T14:02:08","tool":"brain_ingest","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-INGESTED same source name 'tapir-rls-identity-bootstrapping' 69s later with edited/condensed body. No update/patch/supersede verb exists, so the agent overwrote-by-re-ingest. Whether this dedups or creates a v2 duplicate is opaque to the agent.","observed_friction":"agent revised content within 70s of first write — classic edit-after-write with no edit primitive.","evidence":"two brain_ingest, identical source string, divergent content."}
{"id":"a05","session":"tapir","ts":"2026-06-?T14:56:27","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"postgres-cascade-skips-tables-without-fk.md — distinct new lesson."}
{"id":"a06","session":"tapir","ts":"2026-06-?T21:08:57","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_query — discovery tax again.","evidence":"'Dex passwords.dex.coreos.com CRD ...' keyword-rich, single shot."}
{"id":"a07","session":"AI-infra","ts":"2026-05-?T15:38:03","tool":"brain_query(HTTP-curl)","intent":"discover_capability","intent_tool_match":"mismatch","workaround":"raw `curl -X POST` to brain-mcp endpoint instead of MCP tool. Preceded by two ToolSearch ('brain knowledge memory' then 'brain') that did not yield a usable loaded tool, so agent fell back to HTTP.","observed_friction":"3-step ladder: ToolSearch 'brain knowledge memory' -> ToolSearch 'brain' -> curl. Agent did not know which act maps to which tool name.","evidence":"curl -s -o /tmp/brain-init.txt -w code:%{http_code} -X POST ..."}
{"id":"a08","session":"AI-infra","ts":"2026-05-?T04:46:13","tool":"brain_query(HTTP-curl)","intent":"discover_capability","intent_tool_match":"mismatch","workaround":"hand-set TOKEN=... then curl brain-test endpoint — probing whether the HTTP brain path is reachable/authed at all. MCP path not used.","observed_friction":"agent testing connectivity by hand; MCP auth/availability not trusted.","evidence":"TOKEN=...; curl -s -o /tmp/brain-test ..."}
{"id":"a09","session":"AI-infra","ts":"2026-05-?T05:23:35","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_query (after an earlier 05:03 ToolSearch 'brain ingestion knowledge wiki' that explored layers).","evidence":"'koala machine state RTX 5070 llama-swap' — named-entity recall, fits lexical."}
{"id":"a10","session":"AI-infra","ts":"2026-05-?T05:27:36","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"batch of 3 in ~1s (k3s/flux gitops; llama-swap ai-stack GPU; MCP Dex OAuth claude.ai) — parallel prior-art sweep, all named-entity."}
{"id":"a11","session":"AI-infra","ts":"2026-05-?T07:13:50","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"batch of 4 in ~2s before a debugging session (flux healthCheck; exit 255 restart loop; NVML mismatch; mirror rebase). Named symptoms, lexical fit OK on first pass."}
{"id":"a12","session":"AI-infra","ts":"2026-05-?T07:14:26","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"4 writes in ~35s (flux-healthcheck-stale; exit-255-unknown-reason-not-oom; nvidia-nvml-mismatch; mcp-static-bearer) — answers to the 4 queries just run, captured as lessons. Healthy query->fix->write loop."}
{"id":"a13","session":"AI-infra","ts":"2026-05-?T07:15:25","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"25s after writing exit-255-unknown-reason-not-oom.md, re-queried 'exit 255 unknown SIGKILL containerd' — reformulated terms (SIGKILL/containerd not in original query 'exit 255 unknown reason restart loop diagnosis'). Either confirming the fresh write is retrievable or re-searching because first lexical query missed. No read-after-write / get-by-id act exists.","observed_friction":"reformulation chain: 'exit 255 unknown reason restart loop diagnosis' -> 'exit 255 unknown SIGKILL containerd'. Same need, different keywords.","evidence":"query at 07:13:51 vs 07:15:25 bracketing the 07:14:37 write."}
{"id":"a14","session":"AI-infra","ts":"2026-05-?T07:22:55","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"batch of 5 in ~20s before a homelab security audit (piguard/iguana tailscale; unifi UCG firewall; SOPS age; ingress TLS cert-manager; koala UFW iptables). Named-entity sweep."}
{"id":"a15","session":"AI-infra","ts":"2026-05-?T09:27:53","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"audit-shortcut-tls-blocks-zero; policy-audit-mode-blocks-nothing — distinct new audit lessons."}
{"id":"a16","session":"AI-infra","ts":"2026-05-?T09:28:22","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"homelab-security-chains-not-bugs.md FIRST write (worked example: koala 2026-05-13)."}
{"id":"a17","session":"AI-infra","ts":"2026-05-?T10:32:49","tool":"brain_write","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-WROTE homelab-security-chains-not-bugs.md ~64min later with a different/expanded worked example (host-user dotfile, over-broad ClusterRole). Same filename, additive revision, no patch/append/supersede verb — agent overwrites and hopes the index replaces rather than duplicates.","observed_friction":"the in-between hour of audit work produced a better example; only way to fold it in was a full re-write of the same slug.","evidence":"two brain_write same filename at 09:28:22 and 10:32:49, divergent worked examples."}
{"id":"a18","session":"AI-infra","ts":"2026-05-?T10:18:25","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"homelab-document-accepted-risk-to-break-audit-cycle.md — distinct."}
{"id":"a19","session":"AI-infra","ts":"2026-05-?T10:32:55","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"6s after the homelab-chains re-write, queried 'RBAC MCP cluster pods log chain' — checking the chain reasoning is retrievable / finding the related entry. Read-after-write done via lexical search.","observed_friction":null,"evidence":"query immediately follows the 10:32:49 write."}
{"id":"a20","session":"AI-infra","ts":"2026-05-?T18:57:34","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_write,brain_query — re-discovered tools this session.","evidence":"'extension build pinned version major version upgrade postgres pgvector'."}
{"id":"a21","session":"AI-infra","ts":"2026-05-?T18:57:58","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"extension-version-lags-platform-major-upgrade.md."}
{"id":"a22","session":"AI-infra","ts":"2026-05-?T18:58:06","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"8s after writing extension-version-lags, re-queried 'pgvector postgres extension version compile error bump' — reformulated from the 18:57:34 query ('extension build pinned version...'). Lexical re-search to confirm the just-written lesson is findable, with different keyword guess.","observed_friction":"reformulation: 'extension build pinned version major version upgrade postgres pgvector' -> 'pgvector postgres extension version compile error bump'.","evidence":"write at 18:57:58 bracketed by queries 18:57:34 and 18:58:06."}
{"id":"a23","session":"AI-infra","ts":"2026-05-?T18:31:23","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"webfetch-readme-when-image-or-flag-uncertain.md FIRST write."}
{"id":"a24","session":"AI-infra","ts":"2026-05-?T18:34:23","tool":"brain_write","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-WROTE webfetch-readme-when-image-or-flag-uncertain.md 3min later, near-identical body. Looks like a retry/overwrite (uncertain the first landed, or minor edit). No idempotent upsert with confirmation, so agent re-fires the write.","observed_friction":"followed 7s later by a brain_query on the same topic ('OSS tool image registry CLI flag webhook path schema drift README pre-flight') — write-write-query, i.e. overwrite then verify-by-search.","evidence":"two brain_write same filename 18:31:23 / 18:34:23, then query 18:34:31."}
{"id":"a25","session":"AI-infra","ts":"2026-05-?T18:34:31","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"keyword-stuffed lexical query 'OSS tool image registry CLI flag webhook path schema drift README pre-flight' fired right after the webfetch-readme write — agent dumps every concept token hoping BM25 surfaces its own fresh note. This is semantic intent (find that conceptual lesson) coerced into a bag-of-keywords.","observed_friction":"query is a concatenation of the note's section headings — a tell that the agent is groping lexically for content it knows by meaning.","evidence":"query text mirrors the just-written note's bullet topics."}
{"id":"a26","session":"dev","ts":"2026-06-?T21:18:26","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_answer.","evidence":"'tapir transcript persistence shared cross-user dedup table RLS isolation ADR-021 ...' -> 22min later a brain_write (acted on the answer). Answer consumed, not re-queried. Healthy."}
{"id":"a27","session":"dev","ts":"2026-06-?T21:40:20","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"tapir-migration-and-rls-test-infra-gotchas, with wing/hall absent here (flat) — see schema-confusion note a40."}
{"id":"a28","session":"dev","ts":"2026-06-?T05:35:04","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"partial","workaround":"asked 'gitea MCP not working workaround file issue via API which token ... how to authenticate gitea API' — a how-do-I question. Next brain act (05:39 query) is a different topic (tapir transcript), so the answer was apparently sufficient OR abandoned; ambiguous.","observed_friction":null,"evidence":"brain_answer then unrelated brain_query 4min later."}
{"id":"a29","session":"dev","ts":"2026-06-?T05:39:54","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"'tapir transcript persistence ADR-021 shared non-RLS'."}
{"id":"a30","session":"dev","ts":"2026-06-?T05:57:37","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"gitea-mcp-per-repo-tools-404-and-rest-fallback FIRST write."}
{"id":"a31","session":"dev","ts":"2026-06-?T05:57:55","tool":"brain_write","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-WROTE gitea-mcp-per-repo-tools-404-and-rest-fallback 18s later — overwrite/retry of same slug, no upsert confirmation.","observed_friction":"sub-20s gap = almost certainly a content tweak the agent could not express as an edit.","evidence":"two brain_write same filename 05:57:37 / 05:57:55."}
{"id":"a32","session":"dev","ts":"2026-05-?T11:51:47","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"infra-litellm-absorption-2026-05-16.md."}
{"id":"a33","session":"dev","ts":"2026-05-?T12:07:04","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"batch of 3 (litellm rebuild time piguard; docker compose orphaned volumes; prometheus_client ModuleNotFoundError) — lexical, error-string driven."}
{"id":"a34","session":"dev","ts":"2026-05-?T15:08:15","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"mismatch","workaround":"THREE brain_answer at 15:08 (moved compose volumes? / pi rebuild time? / litellm ModuleNotFound prometheus) — the SAME three topics queried lexically an hour earlier (12:07) — were IMMEDIATELY followed at 15:09 by THREE brain_query on the same three topics. The agent asked the synthesizer, was unsatisfied, and fell straight back to raw lexical search. Strongest answer->query fallback in the corpus.","observed_friction":"answer/query duplication across one intent: agent hedges by firing both interfaces, trusting neither.","evidence":"15:08 answers vs 15:09 queries, topic-for-topic aligned."}
{"id":"a35","session":"dev","ts":"2026-05-?T15:09:16","tool":"brain_query","intent":"semantic_retrieval","intent_tool_match":"mismatch","workaround":"after the 3 brain_answer calls failed to satisfy, re-issued as lexical brain_query ('moved compose stack to new directory volumes disappeared empty'; 'raspberry pi docker build time arm slow'; 'how to enable prometheus metrics on litellm proxy callback'). The want is meaning-based ('did my volumes move?') but the only retrieval that 'worked' was keyword search — and these are full natural-language sentences crammed into a BM25 box.","observed_friction":"natural-language questions ('how to enable...', 'moved ... disappeared') passed to a lexical query tool — semantic intent, lexical interface.","evidence":"3 queries at 15:09 mirror the 3 answers at 15:08."}
{"id":"a36","session":"dev","ts":"2026-05-?T20:31:50","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"'What happened with the litellm migration on 2026-05-16?' — episodic recall question, answer fit; no re-query followed."}
{"id":"a37","session":"dev","ts":"2026-05-?T21:07:17","tool":"brain_query","intent":"semantic_retrieval","intent_tool_match":"mismatch","workaround":"FOUR-step reformulation chain over one Go bug: 'bytes.Buffer Bytes Reset aliasing slice sharing' -> 'go buffer reuse map backing array bug' -> [write go-bytes-buffer-bytes-reset-aliasing-trap.md] -> 'go map values all show same content after loop' -> 'bytes.Buffer Bytes returns same data every iteration'. The agent knows the SYMPTOM (all map values identical) and the CAUSE (Bytes() aliasing) but cannot phrase a single lexical query that bridges them — it wants concept retrieval and is forced to brute-force keyword variants.","observed_friction":"4 distinct phrasings of the same bug, two before and two after writing the lesson — also doubles as verify_write_landed on the trailing queries.","evidence":"21:07:17, 21:07:17, (write 21:08:01), 21:08:10, 21:08:19."}
{"id":"a38","session":"dev","ts":"2026-05-?T21:08:01","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"go-bytes-buffer-bytes-reset-aliasing-trap.md."}
{"id":"a39","session":"dev","ts":"2026-05-?T07:54:26","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"mcp-tool-design-get-needs-list-partner.md — a design principle."}
{"id":"a40","session":"dev","ts":"2026-06-?T06:25:00","tool":"brain_query","intent":"check_prior_art","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"'Dex to Authentik migration auth.d-ma.be issuer cutover OIDC subject ...'."}
{"id":"a41","session":"dev","ts":"2026-06-?T06:25:08","tool":"brain_answer","intent":"synthesized_answer","intent_tool_match":"mismatch","workaround":"brain_query (a40) and brain_answer (a41) fired ~8s apart on the SAME intent (Dex->Authentik subject-keyed token orphan). Agent runs lexical search AND synthesized answer in parallel for one question rather than choosing — it cannot predict which interface will return usable knowledge, so it pays both.","observed_friction":"query+answer doublet on one need.","evidence":"06:25:00 query then 06:25:08 answer, same topic."}
{"id":"a42","session":"dev","ts":"2026-06-?T13:48:49","tool":"brain_query","intent":"verify_write_landed","intent_tool_match":"mismatch","workaround":"'authentik cutover validation probe' then 6min later 'authentik cutover post-flip validation' — reformulated pair, likely searching for the agent's own earlier cutover notes / confirming validation steps are recorded. Lexical re-search standing in for recall-my-recent-context.","observed_friction":"reformulation: 'validation probe' -> 'post-flip validation'.","evidence":"13:48:49 and 13:54:56."}
{"id":"a43","session":"dev","ts":"2026-06-?T13:59:17","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":"preceded by ToolSearch select:brain_write.","evidence":"oidc-issuer-host-change-vs-idp-swap-subject."}
{"id":"a44","session":"dev","ts":"2026-06-?T13:59:50","tool":"brain_write","intent":"store_new_knowledge","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"cannot-move-ingress-host-across-namespaces-flux-dryrun FIRST write."}
{"id":"a45","session":"dev","ts":"2026-06-?T14:00:13","tool":"brain_write","intent":"update_or_supersede","intent_tool_match":"mismatch","workaround":"RE-WROTE cannot-move-ingress-host-across-namespaces-flux-dryrun 23s later — overwrite of same slug, no edit/upsert primitive.","observed_friction":"sub-30s gap = content correction expressed as a full re-write.","evidence":"two brain_write same filename 13:59:50 / 14:00:13."}
{"id":"a46","session":"dev","ts":"2026-06-?T13:53:54","tool":"brain_write(HTTP-staged)","intent":"store_new_knowledge","intent_tool_match":"mismatch","workaround":"after `ToolSearch select:mcp__claude_ai_brain__authenticate` (MCP auth flow), the agent staged the entry as `cat > /tmp/brain_entry.json` ({filename:'postgres-force-rls-cross-u...', content}) for a curl write rather than calling brain_write directly — MCP write path was not usable (auth/loading), so it dropped to the HTTP bodge.","observed_friction":"reached for an 'authenticate' tool, then abandoned MCP for hand-built JSON + curl. Matches known pattern: brain/op MCP auth lapses often.","evidence":"ToolSearch authenticate 13:53:27 -> cat /tmp/brain_entry.json 13:53:54."}
{"id":"a47","session":"template-go-agent","ts":"2026-05-?T18:46:26","tool":"BASH(not-a-brain-act)","intent":"intent_unclear","intent_tool_match":"match","workaround":null,"observed_friction":null,"evidence":"'brain_' substring was inside a git commit message body ('agent boundaries, network policy, agent s...'), NOT a brain call. Excluded from knowledge-act analysis; logged for audit completeness."}
@@ -0,0 +1,148 @@
# Agent-Consumer Brain Intent Analysis — koala column
**Consumer:** `autonomous_agent` (all rows). **Host:** koala. **Date:** 2026-06-15.
**Raw rows:** `agent-intent-column.jsonl` (46 real knowledge-acts + 1 excluded false-positive).
## Caveat — canonical schema not on this host
The shared closed-vocabulary file `brain-intent-extraction.md` **does not exist on
koala** — the only reference to it is inside *this task's own prompt*. The intent
vocabulary below was **reconstructed** from the prompt's framing + `brain/schema.md`.
Every row carries `schema_source: reconstructed`. If the canonical vocab differs,
re-map the `intent` field; the `intent_tool_match` / `workaround` / `observed_friction`
evidence stands regardless of label names.
**Reconstructed closed vocab:** `semantic_retrieval`, `lexical_lookup`,
`check_prior_art`, `synthesized_answer`, `store_new_knowledge`, `update_or_supersede`,
`ingest_raw_source`, `verify_write_landed`, `discover_capability`, `intent_unclear`.
## Corpus
- `~/.claude/projects/*/*.jsonl` — Claude Code agent transcripts (98 files). **The only
source with brain calls.**
- `brain/sessions/*.jsonl` — empty (only `.gitkeep`).
- `agentsquad docs/eval/*.jsonl` — code-review eval results, **not** brain calls.
- No separate Crush session logs on this host.
- **Zero** brain calls were human-typed. Every brain act is agent-initiated (the
CLAUDE.md "query as reflex / close-the-loop write" behaviour), so all qualify as
`autonomous_agent`. The human gave the top-level task; the agent chose every brain act.
## 1. Intent histogram, split by `intent_tool_match`
| intent | match | mismatch | partial | total |
|---|---|---|---|---|
| check_prior_art | 10 | 0 | 0 | 10 |
| store_new_knowledge | 14 | 1 | 0 | 15 |
| update_or_supersede | 0 | 5 | 0 | 5 |
| synthesized_answer | 2 | 2 | 1 | 5 |
| verify_write_landed | 0 | 4 | 0 | 4 |
| semantic_retrieval | 0 | 3 | 0 | 3 |
| ingest_raw_source | 2 | 0 | 0 | 2 |
| discover_capability | 0 | 2 | 0 | 2 |
| **total** | **28** | **17** | **1** | **46** |
> Batch note: several rows collapse a same-second fan-out of identical-intent calls
> (a10=3, a11=4, a14=5, a33=3, a34=3, a35=3). Call-level the corpus is ~62 brain calls;
> the table counts the 46 distinct knowledge-acts. Frequency is deliberately *not* the
> point — the mismatch column is.
**37% of agent knowledge-acts (17/46) are interface mismatches.** Every mismatch falls
into one of four intents: `update_or_supersede`, `verify_write_landed`,
`semantic_retrieval`, `discover_capability` — plus one `store` that had to use HTTP.
## 2. Mismatch list, grouped by intent (primary deliverable)
### update_or_supersede → re-write same slug (5/5 mismatch) — HIGHEST VALUE
There is **no update / patch / append / supersede verb**. When an agent improves a note
it already wrote, the only move is to call `brain_write`/`brain_ingest` **again with the
same filename/source** and hope the index replaces rather than duplicates. Observed:
| slug | 1st write | 2nd write | gap | what changed |
|---|---|---|---|---|
| `tapir-rls-identity-bootstrapping` (ingest) | 14:00:59 | 14:02:08 | 69s | condensed body |
| `homelab-security-chains-not-bugs.md` | 09:28:22 | 10:32:49 | 64m | new worked example |
| `webfetch-readme-when-image-or-flag-uncertain.md` | 18:31:23 | 18:34:23 | 3m | near-identical (retry) |
| `gitea-mcp-per-repo-tools-404-and-rest-fallback` | 05:57:37 | 05:57:55 | 18s | content tweak |
| `cannot-move-ingress-host-across-namespaces-flux-dryrun` | 13:59:50 | 14:00:13 | 23s | content tweak |
Sub-30s gaps (3 of 5) read as "I wanted to edit but can only overwrite." The agent has
no way to know whether the second write deduped or created a contradictory v2 — opacity
the brain's own design principle (`mcp-tool-design-get-needs-list-partner.md`, written
*by one of these very agents*) would flag: every `_write` needs a `_get`/`_update` partner.
### verify_write_landed → lexical re-query (4/4 mismatch)
No read-after-write / get-by-id confirmation. After every substantive write, agents
re-query lexically to check the note is retrievable — and *reformulate the keywords*
because they can't predict what BM25 indexed:
- `exit-255` lesson: query `exit 255 unknown reason restart loop diagnosis` → write →
query `exit 255 unknown SIGKILL containerd`.
- `extension-version-lags`: query `extension build pinned version...pgvector` → write →
query `pgvector postgres extension version compile error bump`.
- `webfetch-readme`: write → write → query stuffed with the note's own section headings.
### semantic_retrieval → BM25 keyword-stuffing (3/3 mismatch)
Agent knows the *meaning* but not the *indexed words*, so it brute-forces phrasings of
one need against a lexical tool:
- **4-step chain on one Go bug:** `bytes.Buffer Bytes Reset aliasing slice sharing`
`go buffer reuse map backing array bug` → (write) → `go map values all show same
content after loop``bytes.Buffer Bytes returns same data every iteration`. Symptom
and cause both known; no single lexical query bridges them.
- Natural-language questions (`how to enable prometheus metrics on litellm proxy
callback`, `moved compose stack to new directory volumes disappeared empty`) shoved
into `brain_query`.
### synthesized_answer → fall back to / hedge with brain_query (2 mismatch + 1 partial)
`brain_answer` is frequently **not trusted as terminal**:
- **Strongest signal:** 3× `brain_answer` at 15:08 (compose volumes / pi rebuild time /
litellm ModuleNotFound) → 3× `brain_query` at 15:09 on the *same three topics*. The
agent asked the synthesizer, was unsatisfied, and immediately re-ran raw search.
- Dex→Authentik: `brain_query` and `brain_answer` fired **8s apart on one question** —
the agent pays both interfaces because it can't predict which returns usable knowledge.
- (Counter-examples exist: `brain_answer` for episodic recall — "what happened with the
litellm migration on 2026-05-16?" — was consumed and not re-queried. So `answer`
works for *episodic/temporal* recall, fails for *how-do-I / does-X-hold* reasoning.)
### discover_capability + store-via-HTTP (3 mismatch)
brain tools are **not ambient** — they are deferred and must be `ToolSearch`-loaded each
session. Agents fumble the discovery (`ToolSearch 'brain knowledge memory'` →
`'brain'` → `'brain ingestion knowledge wiki'`) and, when MCP load/auth fails, drop to
**raw `curl` against `brain-mcp` / hand-built `/tmp/brain_entry.json`**. One agent even
`ToolSearch`-ed an `authenticate` tool, then abandoned MCP for the HTTP bodge — matching
the known "brain/op MCP auth lapses too often" footgun.
### Write-interface / layer schema confusion (cross-cutting)
`brain_write` was called with **three different param shapes** in the same corpus:
`{filename, type:"lesson", content}`, `{filename, content}` (no type), and
`{wing:"tapir", hall:"failures", filename, content}` — plus `brain_ingest {source,
content}`. Agents are unsure which verb and which layer (flat slug vs `wing`/`hall`
knowledge routing vs raw ingest) a given knowledge-act maps to. This is the
`knowledge/ vs wiki/` confusion expressed at the parameter level.
## 3. `intent_unclear` rate
**0 / 46 genuine brain acts (0%).** Agent intent is unusually legible because these are
Claude Code transcripts: the surrounding task, the query/filename strings, and the
write content all disambiguate. One row (`a47`) was tagged `intent_unclear` and
**excluded** — its `brain_` substring was inside a git commit message, not a brain call.
Example of the only ambiguity that arose: a `brain_answer` on "gitea MCP not working...
how to authenticate" followed by an unrelated query — can't tell if the answer satisfied
or was abandoned (`partial`, row a28).
## 4. The single biggest intent↔interface gap
**The brain offers one write verb and one lexical read verb, but autonomous agents
perform four distinct knowledge-acts against them — and three of the four have no fitting
interface.** The deepest gap is the **missing update/supersede path**: agents close every
task by writing a lesson (the CLAUDE.md ritual), routinely improve it minutes-to-an-hour
later, and — having no edit primitive — re-write the same slug blind, unable to tell
whether they corrected the entry or forked a contradiction into the index. This compounds
with the lexical-only read side: because there is no `get-by-id` or semantic retrieval,
agents can't even reliably *find their own just-written note* to check it, so they
keyword-stuff reformulated queries and hedge `brain_answer` with parallel `brain_query`.
The interface is built for *append-and-keyword-search*; the agents are trying to
*curate a living, deduplicated knowledge base*, and the seam between those two shows up
as the 5 blind re-writes, 4 read-after-write re-queries, and 3 semantic-as-lexical chains
that dominate the mismatch column.
---
*Evidence-only per task scope — no redesign proposed.*
@@ -0,0 +1,140 @@
# Brain-MCP Intent↔Interface Findings — Unified (two-column merge)
**Status — 2026-06-16**
-**Agent column** filled from `agent-intent-column.jsonl` (46 acts, koala).
-**Human column** = `PENDING`. Drop the Claude.ai-history analysis into
`human-intent-column.jsonl` (same dir, schema below), then fill the `PENDING`
cells and the synthesis blocks marked `<<SYNTH>>`.
- ⚠️ Canonical `brain-intent-extraction.md` still absent on koala. Vocab below is
the **reconstructed** lock both columns must share. If the real file surfaces,
re-map `intent` labels in *both* columns identically before merging.
---
## Shared schema (LOCKED — both columns conform)
Per-call row, JSONL:
| field | values / form | notes |
|---|---|---|
| `id` | `a01..` (agent) / `h01..` (human) | column prefix kept distinct |
| `session` | string | source session/conversation id |
| `ts` | ISO-8601 | best-effort |
| `tool` | brain tool name (+ `(HTTP-curl)` / `(HTTP-staged)` suffix for bodges) | |
| `intent` | closed vocab ↓ | the knowledge-act WANTED |
| `intent_tool_match` | `match` \| `mismatch` \| `partial` | does the called tool fit the want |
| `consumer_type` | `autonomous_agent` \| `human_interactive` | fixed per column |
| `workaround` | string \| null | the bodge when mismatch — **primary signal** |
| `observed_friction` | string \| null | reformulation chains, discovery tax, hedging |
| `evidence` | string | excerpt anchoring the classification |
| `schema_source` | `reconstructed` | flip to `canonical` if real vocab lands |
### Closed intent vocab (LOCKED)
`semantic_retrieval`, `lexical_lookup`, `check_prior_art`, `synthesized_answer`,
`store_new_knowledge`, `update_or_supersede`, `ingest_raw_source`,
`verify_write_landed`, `discover_capability`, `intent_unclear`.
---
## Master comparison — by intent
| intent | agent acts | agent mismatch | human acts | human mismatch | shared gap |
|---|---|---|---|---|---|
| check_prior_art | 10 | 0% | `PENDING` | `PENDING` | — |
| store_new_knowledge | 15 | 7% (1/15) | `PENDING` | `PENDING` | `<<SYNTH>>` |
| update_or_supersede | 5 | **100%** (5/5) | `PENDING` | `PENDING` | `<<SYNTH>>` no edit verb |
| synthesized_answer | 5 | 40% (2/5)+1 partial | `PENDING` | `PENDING` | `<<SYNTH>>` |
| verify_write_landed | 4 | **100%** (4/4) | `PENDING` | `PENDING` | `<<SYNTH>>` no read-after-write |
| semantic_retrieval | 3 | **100%** (3/3) | `PENDING` | `PENDING` | `<<SYNTH>>` lexical-only read |
| ingest_raw_source | 2 | 0% | `PENDING` | `PENDING` | — |
| discover_capability | 2 | **100%** (2/2) | `PENDING` | `PENDING` | agent-specific (ToolSearch/auth)? |
| intent_unclear | 0 | — | `PENDING` | `PENDING` | divergence expected ↓ |
| **TOTAL** | **46** | **37% (17)** | `PENDING` | `PENDING` | |
---
## Per-intent merged findings
### update_or_supersede — agent: 5/5 mismatch (highest value)
**Agent:** no edit/patch/append verb. Agents re-write same slug blind:
`homelab-security-chains-not-bugs.md` (+64m), `tapir-rls-identity-bootstrapping`,
`webfetch-readme...`, `gitea-mcp-per-repo-tools-404...`,
`cannot-move-ingress-host...` — 3 of 5 sub-30s ("wanted edit, got overwrite").
Cannot tell if write deduped or forked a contradiction.
**Human:** `PENDING` — *look for: user editing a prior note, asking "update what I
saved about X", or expressing frustration that an old fact is stale/duplicated.*
**<<SYNTH>>** shared verdict once both filled.
### verify_write_landed — agent: 4/4 mismatch
**Agent:** no `get-by-id`/read-after-write. Agents lexically re-query their own
fresh note with reformulated keywords (`exit 255 unknown reason``...SIGKILL
containerd`; `extension build pinned...``pgvector ...compile error bump`).
**Human:** `PENDING` — *humans may not exhibit this (they trust the write UI
confirmation). If absent in human column, it's an agent-specific gap → flag.*
**<<SYNTH>>**.
### semantic_retrieval — agent: 3/3 mismatch
**Agent:** meaning known, indexed words unknown → BM25 keyword-stuffing. 4-step
chain on one Go `bytes.Buffer` bug; NL questions shoved into `brain_query`.
**Human:** `PENDING` — *humans likely hit this HARDER (they phrase conversationally).
Compare reformulation-chain length agent vs human.*
**<<SYNTH>>** — likely the strongest cross-consumer overlap.
### synthesized_answer — agent: 2 mismatch + 1 partial
**Agent:** `brain_answer` not trusted terminal — 3 answers → 3 same-topic queries
1min later; query+answer fired 8s apart hedging one need. Works for *episodic*
recall, fails for *how-do-I / does-X-hold*.
**Human:** `PENDING` — *humans may prefer `brain_answer` as primary (chat-native).
If human match-rate >> agent, the tool fits humans not agents → key divergence.*
**<<SYNTH>>**.
### store_new_knowledge — agent: 14/15 match
**Agent:** healthy, except 1 HTTP-staged bodge when MCP auth lapsed. Also surfaced
write-schema confusion: 3 param shapes (`{filename,type}` / `{filename}` /
`{wing,hall,filename}`) + `ingest{source}`.
**Human:** `PENDING`*humans rarely write directly; expect low volume.*
**<<SYNTH>>**.
### check_prior_art / ingest_raw_source — agent: 0% mismatch
Lexical fits named-entity recall and raw-source capture. **Human:** `PENDING`.
### discover_capability — agent: 2/2 mismatch (agent-specific)
Brain tools deferred → `ToolSearch`-load each session; auth lapse → `curl` bodge.
**Likely has NO human analog** (humans get ambient connectors). Candidate for
"agent-only gap" bucket. **Human:** `PENDING` to confirm absent.
---
## Cross-consumer divergence — questions to resolve at merge
1. **intent_unclear rate.** Agent = 0% (transcripts self-document). Human expected
higher (conversational, implicit). Big delta = the columns measure legibility
differently, not just intent.
2. **Where does each consumer's mismatch concentrate?** Agent mismatch is
write-side-heavy (supersede + verify-landed = 9/17). Hypothesis: human mismatch
is read-side-heavy (semantic + answer). If true → **the interface fails the two
consumers at opposite ends.**
3. **Agent-only gaps** (`discover_capability`, `verify_write_landed`) vs
**shared gaps** (`semantic_retrieval`, `update_or_supersede`). Shared gaps =
highest-priority evidence; agent-only = harness/auth issues.
---
## Combined headline — `<<SYNTH>>` (fill when human column lands)
> Agent-side draft (to be reconciled with human-side):
> Brain = append + keyword-search; agents want a curated, dedup'd, self-verifying KB.
> Missing update/supersede path + lexical-only reads are the seam. **Open question
> for the merge: do humans hit the same read-side wall, making semantic-retrieval the
> universal gap — or do agents uniquely suffer the write-side (supersede / verify)
> wall that humans sidestep via the chat UI?**
---
## Drop-in checklist (when human column arrives)
1. Place `human-intent-column.jsonl` in this dir; conform to LOCKED schema.
2. Fill every `PENDING` cell in master table + per-intent blocks.
3. Resolve the 3 divergence questions with evidence.
4. Replace each `<<SYNTH>>` with the reconciled verdict; write the combined headline.
5. If canonical vocab surfaced: re-map both columns' `intent`, flip `schema_source`.
6. Commit as `docs(brain): merge human+agent intent columns`.
+6 -1
View File
@@ -103,12 +103,17 @@ func main() {
})) }))
if cfg.GiteaMCPURL != "" { if cfg.GiteaMCPURL != "" {
mcpC, err := mcpclient.New(cfg.GiteaMCPURL, cfg.GiteaMCPToken)
if err != nil {
logger.Error("mcpclient init for project_create — GITEA_MCP_URL is set but GITEA_MCP_TOKEN is empty (check routing-secrets)", "err", err)
os.Exit(1)
}
var ghClient *githubclient.Client var ghClient *githubclient.Client
if cfg.GitHubPAT != "" { if cfg.GitHubPAT != "" {
ghClient = githubclient.New(cfg.GitHubPAT) ghClient = githubclient.New(cfg.GitHubPAT)
} }
reg.Register(project.New(project.Config{ reg.Register(project.New(project.Config{
Client: mcpclient.New(cfg.GiteaMCPURL, cfg.GiteaMCPToken), Client: mcpC,
GitHub: ghClient, GitHub: ghClient,
GiteaOwner: cfg.GiteaOwner, GiteaOwner: cfg.GiteaOwner,
GitHubOwner: cfg.GitHubOwner, GitHubOwner: cfg.GitHubOwner,
+28 -16
View File
@@ -4,9 +4,12 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"io" "io"
"net"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"os"
"os/exec" "os/exec"
"strconv"
"strings" "strings"
"testing" "testing"
"time" "time"
@@ -42,28 +45,33 @@ func TestRoutingPodEndToEnd(t *testing.T) {
})) }))
defer brain.Close() defer brain.Close()
port := freePort(t)
addr := "127.0.0.1:" + port
baseURL := "http://" + addr
bin := buildRouting(t) bin := buildRouting(t)
cmd := exec.Command(bin) cmd := exec.Command(bin)
cmd.Env = append(cmd.Env, cmd.Env = []string{
"ROUTING_PORT=33310", "ROUTING_PORT=" + port,
"LITELLM_BASE_URL="+llm.URL, "LITELLM_BASE_URL=" + llm.URL,
"LITELLM_API_KEY=stub", "LITELLM_API_KEY=stub",
"BRAIN_URL="+brain.URL, "BRAIN_URL=" + brain.URL,
"SUPERVISOR_CONFIG_DIR=../../config/supervisor", "SUPERVISOR_CONFIG_DIR=../../config/supervisor",
"PATH="+osPath(), "PATH=" + os.Getenv("PATH"),
) "HOME=" + os.Getenv("HOME"),
}
require.NoError(t, cmd.Start()) require.NoError(t, cmd.Start())
t.Cleanup(func() { _ = cmd.Process.Kill() }) t.Cleanup(func() { _ = cmd.Process.Kill() })
require.NoError(t, waitForPort(t, "127.0.0.1:33310", 5*time.Second)) require.NoError(t, waitForPort(t, addr, 30*time.Second))
resp := mcpCall(t, "http://127.0.0.1:33310/mcp", `{"jsonrpc":"2.0","id":1,"method":"tools/list"}`) resp := mcpCall(t, baseURL+"/mcp", `{"jsonrpc":"2.0","id":1,"method":"tools/list"}`)
assert.Contains(t, resp, `"review"`) assert.Contains(t, resp, `"review"`)
assert.Contains(t, resp, `"debug"`) assert.Contains(t, resp, `"debug"`)
assert.Contains(t, resp, `"retrospective"`) assert.Contains(t, resp, `"retrospective"`)
assert.Contains(t, resp, `"trainer"`) assert.Contains(t, resp, `"trainer"`)
resp = mcpCall(t, "http://127.0.0.1:33310/mcp", `{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"review","arguments":{"project_root":"/tmp","files":["README.md"]}}}`) resp = mcpCall(t, baseURL+"/mcp", `{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"review","arguments":{"project_root":"/tmp","files":["README.md"]}}}`)
_ = resp // shape varies by skill; we only need a 200 _ = resp // shape varies by skill; we only need a 200
// Wait briefly for the async session_log to land. // Wait briefly for the async session_log to land.
@@ -113,11 +121,15 @@ func mcpCall(t *testing.T, url, body string) string {
return string(raw) return string(raw)
} }
func osPath() string { // freePort grabs an OS-assigned TCP port and releases it. There is a small
for _, e := range append([]string{}, exec.Command("env").Env...) { // race window before the subprocess re-binds it, but it is acceptable for
if strings.HasPrefix(e, "PATH=") { // test isolation against a hardcoded port colliding with another test or
return strings.TrimPrefix(e, "PATH=") // stray process.
} func freePort(t *testing.T) string {
} t.Helper()
return "/usr/bin:/bin" l, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
port := l.Addr().(*net.TCPAddr).Port
require.NoError(t, l.Close())
return strconv.Itoa(port)
} }
+9
View File
@@ -5,6 +5,15 @@ FROM golang:1.26-bookworm AS builder
ARG VERSION=dev ARG VERSION=dev
WORKDIR /src WORKDIR /src
# Fetch internal gitea-hosted Go modules (mcp-chassis) without going through
# proxy.golang.org and without HTTP→HTTPS surprises. The Gitea server returns
# http:// in its go-import meta tag (config-level limitation), so rewrite to
# https here and bypass the module proxy + sumdb.
RUN git config --global url."https://gitea.d-ma.be/".insteadOf "http://gitea.d-ma.be/"
ENV GOPRIVATE=gitea.d-ma.be
ENV GOPROXY=direct
ENV GOSUMDB=off
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
+272 -15
View File
@@ -6,18 +6,102 @@ import (
"fmt" "fmt"
"log/slog" "log/slog"
"net/http" "net/http"
"net/url"
"os" "os"
"strconv" "strconv"
"strings"
"time" "time"
chassisauth "gitea.d-ma.be/mathias/mcp-chassis/auth"
"github.com/mathiasbq/hyperguild/ingestion/internal/api" "github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/auth" "github.com/mathiasbq/hyperguild/ingestion/internal/claudewatcher"
"github.com/mathiasbq/hyperguild/ingestion/internal/embed"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphsync"
"github.com/mathiasbq/hyperguild/ingestion/internal/llm" "github.com/mathiasbq/hyperguild/ingestion/internal/llm"
"github.com/mathiasbq/hyperguild/ingestion/internal/mcp" "github.com/mathiasbq/hyperguild/ingestion/internal/mcp"
"github.com/mathiasbq/hyperguild/ingestion/internal/metrics"
"github.com/mathiasbq/hyperguild/ingestion/internal/oauth"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline" "github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
"github.com/mathiasbq/hyperguild/ingestion/internal/reranker"
"github.com/mathiasbq/hyperguild/ingestion/internal/search"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/watcher" "github.com/mathiasbq/hyperguild/ingestion/internal/watcher"
) )
// claudeSink converts each claudewatcher.Batch into one wiki note under
// brain/wiki/claude-sessions/facts/. v1 emits one note per session
// keyed by host + session id; classifier-driven hall routing is a
// follow-up (hyperguild#27 v2).
type claudeSink struct {
brainDir string
logger *slog.Logger
}
func (s *claudeSink) Ingest(ctx context.Context, b claudewatcher.Batch) error {
if len(b.Turns) == 0 {
return nil
}
var sb strings.Builder
fmt.Fprintf(&sb, "# Claude session %s (%s)\n\n", b.SessionID, b.Host)
fmt.Fprintf(&sb, "_Project: `%s`. File: `%s`. Turns: %d._\n\n", b.ProjectID, b.FilePath, len(b.Turns))
for _, t := range b.Turns {
fmt.Fprintf(&sb, "## %s — %s\n\n", t.Type, t.Timestamp.UTC().Format(time.RFC3339))
if t.ToolName != "" {
fmt.Fprintf(&sb, "_tool: `%s`_\n\n", t.ToolName)
}
// Cap per-turn excerpt to keep page size bounded; the full
// transcript lives on disk under ~/.claude/projects/ already.
content := t.Content
if len(content) > 2000 {
content = content[:2000] + "…"
}
sb.WriteString(content)
sb.WriteString("\n\n")
}
slug := "session-" + b.Host + "-" + b.SessionID
if _, err := api.WriteNote(s.brainDir, api.WriteNoteOptions{
Filename: slug,
Wing: "claude-sessions",
Hall: "facts",
Type: "source",
Domain: b.ProjectID,
Content: sb.String(),
}); err != nil {
return fmt.Errorf("write claude session note: %w", err)
}
return nil
}
// redactDSN parses a Postgres URL and replaces its password with `***`
// for safe inclusion in logs. Falls back to a non-leaking placeholder
// if parsing fails — we never log a raw DSN.
func redactDSN(dsn string) string {
u, err := url.Parse(dsn)
if err != nil || u.User == nil {
return "postgres://***"
}
return u.Redacted()
}
// vectorAdapter bridges *vectorstore.PGStore (returns []vectorstore.Hit)
// to the search.VectorSearcher interface (which uses []search.VectorHit).
// Kept here, not in either package, so neither has to import the other.
type vectorAdapter struct{ s *vectorstore.PGStore }
func (a vectorAdapter) Search(ctx context.Context, q []float32, limit int) ([]search.VectorHit, error) {
hits, err := a.s.Search(ctx, q, limit)
if err != nil {
return nil, err
}
out := make([]search.VectorHit, len(hits))
for i, h := range hits {
out[i] = search.VectorHit{Path: h.Path, Distance: h.Distance}
}
return out, nil
}
func envOr(key, fallback string) string { func envOr(key, fallback string) string {
if v := os.Getenv(key); v != "" { if v := os.Getenv(key); v != "" {
return v return v
@@ -34,6 +118,16 @@ func envInt(key string, fallback int) int {
return fallback return fallback
} }
// systemHostname returns os.Hostname() with a "unknown" fallback so the
// caller never has to handle the rare error path.
func systemHostname() string {
h, err := os.Hostname()
if err != nil || h == "" {
return "unknown"
}
return h
}
func main() { func main() {
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil)) logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
@@ -75,6 +169,68 @@ func main() {
} }
mcpSrv := mcp.NewServer(brainDir, &pipelineCfg, llmClient.Complete, answerComplete) mcpSrv := mcp.NewServer(brainDir, &pipelineCfg, llmClient.Complete, answerComplete)
if rerankURL := os.Getenv("BRAIN_RERANKER_URL"); rerankURL != "" {
rerankModel := envOr("BRAIN_RERANKER_MODEL", "dengcao/Qwen3-Reranker-0.6B:F16")
mcpSrv = mcpSrv.WithReranker(reranker.New(rerankURL, rerankModel))
logger.Info("brain reranker configured", "url", rerankURL, "model", rerankModel)
}
// Hybrid retrieval (pgvector + nomic-embed-text). Both env vars must
// be set together for the path to wire on; otherwise BM25-only.
var vectorStore *vectorstore.PGStore
pgDSN := os.Getenv("BRAIN_PG_DSN")
embedURL := os.Getenv("BRAIN_EMBED_URL")
switch {
case pgDSN != "" && embedURL != "":
embedModel := envOr("BRAIN_EMBED_MODEL", "nomic-embed-text:latest")
store, err := vectorstore.New(context.Background(), pgDSN)
if err != nil {
logger.Error("vector store init", "err", err)
os.Exit(1)
}
if err := store.Init(context.Background()); err != nil {
logger.Error("vector store migrate", "err", err)
os.Exit(1)
}
vectorStore = store
embedder := embed.New(embedURL, embedModel)
mcpSrv = mcpSrv.WithHybridRetrieval(vectorAdapter{s: store}, embedder)
h.WithEmbedSync(store, embedder)
logger.Info("brain hybrid retrieval enabled",
"pg", redactDSN(pgDSN),
"embed_url", embedURL, "embed_model", embedModel)
// Graph store shares the same postgres18 DSN as the vector
// store and is opt-in via BRAIN_GRAPH_ENABLED=true. Defaults
// to off so first rollout doesn't surprise — flip on after
// the migration completes and the backfill finishes.
if envOr("BRAIN_GRAPH_ENABLED", "false") == "true" {
gstore, gerr := graphstore.New(context.Background(), pgDSN)
if gerr != nil {
logger.Error("graph store init", "err", gerr)
os.Exit(1)
}
if gerr := gstore.Init(context.Background()); gerr != nil {
logger.Error("graph store migrate", "err", gerr)
os.Exit(1)
}
mcpSrv = mcpSrv.WithGraph(gstore)
if envOr("BRAIN_GRAPH_BACKFILL", "false") == "true" {
n, berr := graphsync.BackfillFromBrainDir(context.Background(), gstore, brainDir)
if berr != nil {
logger.Warn("graph backfill incomplete", "indexed", n, "err", berr)
} else {
logger.Info("graph backfill complete", "indexed", n)
}
}
logger.Info("brain graph enabled", "pg", redactDSN(pgDSN))
}
case pgDSN == "" && embedURL == "":
// disabled — fine
default:
logger.Error("BRAIN_PG_DSN and BRAIN_EMBED_URL must be set together")
os.Exit(1)
}
mcpToken := os.Getenv("BRAIN_MCP_TOKEN") mcpToken := os.Getenv("BRAIN_MCP_TOKEN")
if mcpToken == "" { if mcpToken == "" {
@@ -91,34 +247,135 @@ func main() {
}) })
} }
// Claude Code session ingestion (hyperguild#27 / infra#73 Track E.1).
// Off by default — explicitly opt in by setting CLAUDE_SESSIONS_DIR
// to the ~/.claude/projects path. Requires BRAIN_PG_DSN for the
// cursor table (resumable offsets across restarts).
if claudeDir := os.Getenv("CLAUDE_SESSIONS_DIR"); claudeDir != "" {
if pgDSN == "" {
logger.Error("CLAUDE_SESSIONS_DIR set but BRAIN_PG_DSN missing — claudewatcher needs the cursor table")
os.Exit(1)
}
// Client-name guard. The env value is a regex alternation
// (e.g. "SEB|Mastercard"); we wrap it with word boundaries
// and case-insensitive flag so substrings inside longer
// identifiers don't false-match. Sourced from a SOPS secret
// so client identities never live in source.
if clientBlock := os.Getenv("CLAUDE_INGEST_CLIENT_BLOCK"); clientBlock != "" {
pattern := `(?i)\b(` + clientBlock + `)\b`
if err := claudewatcher.RegisterRule("client-name", pattern); err != nil {
logger.Error("claudewatcher client-block rule invalid", "err", err)
os.Exit(1)
}
logger.Info("claudewatcher client-block guard registered")
}
cursorStore, cerr := claudewatcher.NewCursorStore(ctx, pgDSN)
if cerr != nil {
logger.Error("claudewatcher cursor init", "err", cerr)
os.Exit(1)
}
if cerr := cursorStore.Init(ctx); cerr != nil {
logger.Error("claudewatcher cursor migrate", "err", cerr)
os.Exit(1)
}
host := envOr("CLAUDE_INGEST_HOST", systemHostname())
interval := time.Duration(envInt("CLAUDE_INGEST_INTERVAL", 60)) * time.Second
sink := &claudeSink{brainDir: brainDir, logger: logger}
go func() {
if err := claudewatcher.Watch(ctx, claudewatcher.Config{
SessionsDir: claudeDir,
Host: host,
Interval: interval,
Sink: sink,
Cursors: cursorStore,
Logger: logger,
}); err != nil && err != context.Canceled {
logger.Error("claudewatcher exited", "err", err)
}
}()
logger.Info("claudewatcher started",
"sessions_dir", claudeDir, "host", host, "interval", interval)
}
if vectorStore != nil {
embedSyncInterval := envInt("BRAIN_EMBED_SYNC_INTERVAL", 300)
vectorstore.StartSync(ctx, brainDir, vectorStore,
embed.New(os.Getenv("BRAIN_EMBED_URL"),
envOr("BRAIN_EMBED_MODEL", "nomic-embed-text:latest")),
time.Duration(embedSyncInterval)*time.Second)
logger.Info("embed sync started", "interval_s", embedSyncInterval)
}
mux := http.NewServeMux() mux := http.NewServeMux()
mux.HandleFunc("POST /query", h.Query) mux.HandleFunc("POST /query", h.Query)
mux.HandleFunc("POST /write", h.Write) mux.HandleFunc("POST /write", h.Write)
mux.HandleFunc("POST /index", h.Index)
mux.HandleFunc("POST /ingest", h.Ingest) mux.HandleFunc("POST /ingest", h.Ingest)
mux.HandleFunc("POST /ingest-path", h.IngestPath) mux.HandleFunc("POST /ingest-path", h.IngestPath)
mux.HandleFunc("POST /ingest-raw", h.IngestRaw) mux.HandleFunc("POST /ingest-raw", h.IngestRaw)
mux.HandleFunc("POST /backfill-refs", h.BackfillRefs) mux.HandleFunc("POST /backfill-refs", h.BackfillRefs)
mux.HandleFunc("POST /backfill-embeddings", h.BackfillEmbeddings)
mux.HandleFunc("GET /pass-rate", h.PassRate) mux.HandleFunc("GET /pass-rate", h.PassRate)
var jwtValidator *auth.Validator jwtValidator, err := chassisauth.NewJWTValidator(ctx, os.Getenv("DEX_ISSUER_URL"), os.Getenv("MCP_AUDIENCE"))
if dexURL := os.Getenv("DEX_ISSUER_URL"); dexURL != "" { if err != nil {
audience := os.Getenv("MCP_AUDIENCE") logger.Error("build jwt validator", "err", err)
v, err := auth.NewValidator(dexURL, audience) os.Exit(1)
if err != nil { }
logger.Error("build jwt validator", "err", err) if jwtValidator != nil {
os.Exit(1) logger.Info("jwt auth enabled", "issuer", os.Getenv("DEX_ISSUER_URL"))
}
jwtValidator = v
logger.Info("jwt auth enabled", "issuer", dexURL)
} }
mux.Handle("/mcp", mcp.BearerAuth(mcpToken, jwtValidator, mcpSrv)) // Resource-metadata URL is only emitted on 401 when Dex OAuth is
// configured. Static-Bearer-only deployments leave this empty so
// clients never see an OAuth challenge.
var resourceMetadataURL string
if dexURL := os.Getenv("DEX_ISSUER_URL"); dexURL != "" { if dexURL := os.Getenv("DEX_ISSUER_URL"); dexURL != "" {
resourceURL := os.Getenv("MCP_RESOURCE_URL") resourceURL := os.Getenv("MCP_RESOURCE_URL")
mux.HandleFunc("GET /.well-known/oauth-protected-resource", mux.HandleFunc("GET /.well-known/oauth-protected-resource",
auth.ProtectedResourceHandler(resourceURL, os.Getenv("DEX_ISSUER_URL"))) chassisauth.ProtectedResourceHandler(resourceURL, dexURL))
if resourceURL != "" {
resourceMetadataURL = strings.TrimRight(resourceURL, "/") + "/.well-known/oauth-protected-resource"
}
} }
mux.Handle("/mcp", chassisauth.BearerMiddleware(mcpToken, jwtValidator, "brain", resourceMetadataURL, mcpSrv))
// Opt-in OAuth 2.0 client_credentials flow for claude.ai's custom-MCP
// integration UI, which has no static-Bearer field. Setting both
// OAUTH_CLIENT_ID and OAUTH_CLIENT_SECRET enables the token exchange;
// setting only one is misconfiguration → fail fast.
oauthID := os.Getenv("OAUTH_CLIENT_ID")
oauthSecret := os.Getenv("OAUTH_CLIENT_SECRET")
switch {
case oauthID != "" && oauthSecret != "":
issuer := os.Getenv("MCP_RESOURCE_URL")
if issuer == "" {
logger.Error("OAUTH_CLIENT_ID/SECRET set but MCP_RESOURCE_URL is empty; cannot derive issuer")
os.Exit(1)
}
mux.HandleFunc("GET /.well-known/oauth-authorization-server",
oauth.MetadataHandler(issuer))
mux.HandleFunc("POST /oauth/token", oauth.TokenHandler(oauth.TokenConfig{
ClientID: oauthID,
ClientSecret: oauthSecret,
AccessToken: mcpToken,
}))
logger.Info("oauth client_credentials enabled", "issuer", strings.TrimRight(issuer, "/"))
case oauthID == "" && oauthSecret == "":
// disabled — that's fine
default:
logger.Error("OAUTH_CLIENT_ID and OAUTH_CLIENT_SECRET must be set together")
os.Exit(1)
}
// /metrics — unauthenticated Prometheus endpoint. kube-prometheus-stack
// scrapes it via the ServiceMonitor in k3s/apps/supervisor/. The metrics
// middleware below wraps every other registered handler so it observes
// real request latency. /metrics itself is excluded from its own
// observation by registering it on the outer mux (post-wrap).
reg := metrics.New()
mux.HandleFunc("GET /metrics", reg.Handler())
logger.Info("metrics endpoint registered", "path", "/metrics")
addr := ":" + port addr := ":" + port
watchIntervalLog := "disabled" watchIntervalLog := "disabled"
if watchInterval > 0 { if watchInterval > 0 {
@@ -133,7 +390,7 @@ func main() {
"watch_interval", watchIntervalLog, "watch_interval", watchIntervalLog,
"mcp_enabled", true, "mcp_enabled", true,
) )
if err := http.ListenAndServe(addr, mux); err != nil { if err := http.ListenAndServe(addr, reg.Middleware(mux)); err != nil {
logger.Error("server stopped", "err", err) logger.Error("server stopped", "err", err)
os.Exit(1) os.Exit(1)
} }
+7
View File
@@ -8,9 +8,14 @@ require (
) )
require ( require (
gitea.d-ma.be/mathias/mcp-chassis v0.1.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect github.com/davecgh/go-spew v1.1.1 // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 // indirect github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 // indirect
github.com/goccy/go-json v0.10.3 // indirect github.com/goccy/go-json v0.10.3 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/pgx/v5 v5.9.2 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/lestrrat-go/blackmagic v1.0.3 // indirect github.com/lestrrat-go/blackmagic v1.0.3 // indirect
github.com/lestrrat-go/httpcc v1.0.1 // indirect github.com/lestrrat-go/httpcc v1.0.1 // indirect
github.com/lestrrat-go/httprc v1.0.6 // indirect github.com/lestrrat-go/httprc v1.0.6 // indirect
@@ -19,6 +24,8 @@ require (
github.com/pmezard/go-difflib v1.0.0 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/segmentio/asm v1.2.0 // indirect github.com/segmentio/asm v1.2.0 // indirect
golang.org/x/crypto v0.32.0 // indirect golang.org/x/crypto v0.32.0 // indirect
golang.org/x/sync v0.17.0 // indirect
golang.org/x/sys v0.31.0 // indirect golang.org/x/sys v0.31.0 // indirect
golang.org/x/text v0.29.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect
) )
+17
View File
@@ -1,3 +1,5 @@
gitea.d-ma.be/mathias/mcp-chassis v0.1.0 h1:8RXO34+n7Vu8HnUMagars6fc4oemqRpMu7MVtjaj4qY=
gitea.d-ma.be/mathias/mcp-chassis v0.1.0/go.mod h1:ajbLlwr2L7FAN3TBU39KucZkKJM02wTbKbDKDEW2YvE=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
@@ -5,6 +7,14 @@ github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 h1:NMZiJj8QnKe1LgsbDayM4UoHwbvw
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0/go.mod h1:ZXNYxsqcloTdSy/rNShjYzMhyjf0LaoftYK0p+A3h40= github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0/go.mod h1:ZXNYxsqcloTdSy/rNShjYzMhyjf0LaoftYK0p+A3h40=
github.com/goccy/go-json v0.10.3 h1:KZ5WoDbxAIgm2HNbYckL0se1fHD6rz5j4ywS6ebzDqA= github.com/goccy/go-json v0.10.3 h1:KZ5WoDbxAIgm2HNbYckL0se1fHD6rz5j4ywS6ebzDqA=
github.com/goccy/go-json v0.10.3/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M= github.com/goccy/go-json v0.10.3/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.9.2 h1:3ZhOzMWnR4yJ+RW1XImIPsD1aNSz4T4fyP7zlQb56hw=
github.com/jackc/pgx/v5 v5.9.2/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/lestrrat-go/blackmagic v1.0.3 h1:94HXkVLxkZO9vJI/w2u1T0DAoprShFd13xtnSINtDWs= github.com/lestrrat-go/blackmagic v1.0.3 h1:94HXkVLxkZO9vJI/w2u1T0DAoprShFd13xtnSINtDWs=
github.com/lestrrat-go/blackmagic v1.0.3/go.mod h1:6AWFyKNNj0zEXQYfTMPfZrAXUWUfTIZ5ECEUEJaijtw= github.com/lestrrat-go/blackmagic v1.0.3/go.mod h1:6AWFyKNNj0zEXQYfTMPfZrAXUWUfTIZ5ECEUEJaijtw=
github.com/lestrrat-go/httpcc v1.0.1 h1:ydWCStUeJLkpYyjLDHihupbn2tYmZ7m22BGkcvZZrIE= github.com/lestrrat-go/httpcc v1.0.1 h1:ydWCStUeJLkpYyjLDHihupbn2tYmZ7m22BGkcvZZrIE=
@@ -22,16 +32,23 @@ github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZN
github.com/segmentio/asm v1.2.0 h1:9BQrFxC+YOHJlTlHGkTrFWf59nbL3XnCoFLTwDCI7ys= github.com/segmentio/asm v1.2.0 h1:9BQrFxC+YOHJlTlHGkTrFWf59nbL3XnCoFLTwDCI7ys=
github.com/segmentio/asm v1.2.0/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs= github.com/segmentio/asm v1.2.0/go.mod h1:BqMnlJP91P8d+4ibuonYZw9mfnzI9HfxselHZr5aAcs=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
golang.org/x/crypto v0.32.0 h1:euUpcYgM8WcP71gNpTqQCn6rC2t6ULUPiOzfWaXVVfc= golang.org/x/crypto v0.32.0 h1:euUpcYgM8WcP71gNpTqQCn6rC2t6ULUPiOzfWaXVVfc=
golang.org/x/crypto v0.32.0/go.mod h1:ZnnJkOaASj8g0AjIduWNlq2NRxL0PlBrbKVyZ6V/Ugc= golang.org/x/crypto v0.32.0/go.mod h1:ZnnJkOaASj8g0AjIduWNlq2NRxL0PlBrbKVyZ6V/Ugc=
golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug=
golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik= golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk=
golang.org/x/text v0.29.0/go.mod h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+97
View File
@@ -0,0 +1,97 @@
package api
import "strings"
// frontmatter is an ordered, line-preserving view of a note's YAML
// frontmatter block. It deliberately avoids a full YAML round-trip: the
// brain writes flat `key: value` frontmatter by hand, and a yaml.v3
// re-marshal would reorder keys and strip comments. Preserving the
// original lines verbatim keeps brain_update a surgical edit — only the
// keys it manages (updated_at, supersedes, supersede_reason) change.
type frontmatter struct {
lines []fmLine
}
// fmLine is one frontmatter line. For `key: value` lines, key and value
// are populated; for blank lines, comments, or anything that isn't a
// simple scalar pair, key is empty and raw holds the line verbatim.
type fmLine struct {
key string
value string
raw string
}
// parseFrontmatter splits src into its frontmatter block and body. A
// frontmatter block is recognised only when the file opens with a `---`
// fence and a closing `---` fence follows. Otherwise the whole input is
// the body and the returned frontmatter is empty.
func parseFrontmatter(src string) (frontmatter, string) {
var fm frontmatter
if !strings.HasPrefix(src, "---\n") {
return fm, src
}
rest := src[len("---\n"):]
end := strings.Index(rest, "\n---\n")
if end < 0 {
// Opening fence with no closing fence — treat as bodyless content.
return fm, src
}
block := rest[:end]
body := rest[end+len("\n---\n"):]
for _, line := range strings.Split(block, "\n") {
key, val, ok := strings.Cut(line, ":")
key = strings.TrimSpace(key)
if !ok || key == "" || strings.HasPrefix(strings.TrimSpace(line), "#") {
fm.lines = append(fm.lines, fmLine{raw: line})
continue
}
fm.lines = append(fm.lines, fmLine{key: key, value: strings.TrimSpace(val)})
}
return fm, body
}
// get returns the value for key, or "" if absent.
func (f *frontmatter) get(key string) string {
for _, l := range f.lines {
if l.key == key {
return l.value
}
}
return ""
}
// set overrides the value for an existing key in place, or appends a new
// `key: value` line when the key is absent.
func (f *frontmatter) set(key, value string) {
for i := range f.lines {
if f.lines[i].key == key {
f.lines[i].value = value
return
}
}
f.lines = append(f.lines, fmLine{key: key, value: value})
}
// render serialises the frontmatter back into a `---`-fenced block. An
// empty frontmatter renders to the empty string so bodies without a
// header stay header-less.
func (f *frontmatter) render() string {
if len(f.lines) == 0 {
return ""
}
var b strings.Builder
b.WriteString("---\n")
for _, l := range f.lines {
if l.key == "" {
b.WriteString(l.raw)
} else {
b.WriteString(l.key)
b.WriteString(": ")
b.WriteString(l.value)
}
b.WriteByte('\n')
}
b.WriteString("---\n")
return b.String()
}
@@ -0,0 +1,61 @@
package api
import (
"strings"
"testing"
"github.com/stretchr/testify/assert"
)
func TestParseFrontmatterSplitsHeaderAndBody(t *testing.T) {
src := "---\nwing: jepa-fx\nhall: facts\ncreated_at: 2026-01-01T00:00:00Z\n---\n# Title\n\nbody text\n"
fm, body := parseFrontmatter(src)
assert.Equal(t, "jepa-fx", fm.get("wing"))
assert.Equal(t, "facts", fm.get("hall"))
assert.Equal(t, "2026-01-01T00:00:00Z", fm.get("created_at"))
assert.Equal(t, "# Title\n\nbody text\n", body)
}
func TestParseFrontmatterNoHeader(t *testing.T) {
src := "# Just a body\n\nno frontmatter here\n"
fm, body := parseFrontmatter(src)
assert.Empty(t, fm.lines)
assert.Equal(t, src, body)
}
func TestFrontmatterSetOverridesExistingKey(t *testing.T) {
fm, _ := parseFrontmatter("---\nwing: a\nupdated_at: old\n---\nbody\n")
fm.set("updated_at", "new")
assert.Equal(t, "new", fm.get("updated_at"))
// No duplicate key.
assert.Equal(t, 1, strings.Count(fm.render(), "updated_at:"))
}
func TestFrontmatterSetAppendsNewKey(t *testing.T) {
fm, _ := parseFrontmatter("---\nwing: a\n---\nbody\n")
fm.set("supersedes", "abc123")
out := fm.render()
assert.Contains(t, out, "wing: a")
assert.Contains(t, out, "supersedes: abc123")
}
func TestFrontmatterRenderPreservesCustomFields(t *testing.T) {
src := "---\nwing: a\nhall: facts\ncustom_field: keep-me\ntags: [x, y]\n---\nbody\n"
fm, _ := parseFrontmatter(src)
fm.set("updated_at", "2026-06-22T00:00:00Z")
out := fm.render()
assert.Contains(t, out, "custom_field: keep-me")
assert.Contains(t, out, "tags: [x, y]")
assert.Contains(t, out, "updated_at: 2026-06-22T00:00:00Z")
}
func TestFrontmatterRenderRoundTrips(t *testing.T) {
src := "---\nwing: a\nhall: facts\n---\n"
fm, _ := parseFrontmatter(src)
assert.Equal(t, src, fm.render())
}
+164 -18
View File
@@ -11,16 +11,20 @@ import (
"strings" "strings"
"time" "time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/extract" "github.com/mathiasbq/hyperguild/ingestion/internal/extract"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline" "github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
"github.com/mathiasbq/hyperguild/ingestion/internal/search" "github.com/mathiasbq/hyperguild/ingestion/internal/search"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
) )
// Handler serves the ingestion HTTP API. // Handler serves the ingestion HTTP API.
type Handler struct { type Handler struct {
brainDir string brainDir string
logger *slog.Logger logger *slog.Logger
pipeline pipeline.Config pipeline pipeline.Config
embedStore vectorstore.Store
embedClient vectorstore.Embedder
} }
// NewHandler constructs a Handler. brainDir is the absolute path to brain/. // NewHandler constructs a Handler. brainDir is the absolute path to brain/.
@@ -31,9 +35,19 @@ func NewHandler(brainDir string, logger *slog.Logger, pipelineCfg pipeline.Confi
return &Handler{brainDir: brainDir, logger: logger, pipeline: pipelineCfg} return &Handler{brainDir: brainDir, logger: logger, pipeline: pipelineCfg}
} }
// WithEmbedSync wires the optional vector store + embedder used by the
// POST /backfill-embeddings endpoint. Calling with either nil is a no-op.
func (h *Handler) WithEmbedSync(store vectorstore.Store, embedder vectorstore.Embedder) *Handler {
h.embedStore = store
h.embedClient = embedder
return h
}
type queryRequest struct { type queryRequest struct {
Query string `json:"query"` Query string `json:"query"`
Limit int `json:"limit,omitempty"` Limit int `json:"limit,omitempty"`
Wing string `json:"wing,omitempty"`
Hall string `json:"hall,omitempty"`
} }
type writeRequest struct { type writeRequest struct {
@@ -41,6 +55,8 @@ type writeRequest struct {
Filename string `json:"filename,omitempty"` Filename string `json:"filename,omitempty"`
Type string `json:"type,omitempty"` Type string `json:"type,omitempty"`
Domain string `json:"domain,omitempty"` Domain string `json:"domain,omitempty"`
Wing string `json:"wing,omitempty"`
Hall string `json:"hall,omitempty"`
} }
type ingestRequest struct { type ingestRequest struct {
@@ -75,7 +91,12 @@ func (h *Handler) Query(w http.ResponseWriter, r *http.Request) {
req.Limit = 5 req.Limit = 5
} }
results, err := search.Query(h.brainDir, req.Query, req.Limit) results, err := search.Query(h.brainDir, search.QueryOptions{
Query: req.Query,
Limit: req.Limit,
Wing: req.Wing,
Hall: req.Hall,
})
if err != nil { if err != nil {
h.logger.Error("query failed", "err", err) h.logger.Error("query failed", "err", err)
writeError(w, http.StatusInternalServerError, "search error") writeError(w, http.StatusInternalServerError, "search error")
@@ -85,13 +106,78 @@ func (h *Handler) Query(w http.ResponseWriter, r *http.Request) {
writeJSON(w, map[string]any{"results": results}) writeJSON(w, map[string]any{"results": results})
} }
// WriteNote writes a markdown file to brainDir/knowledge/<filename>, optionally // WriteNoteOptions configures how a brain note is written.
// prefixed with YAML frontmatter built from typ and domain. Returns the path //
// When both Wing and Hall are non-empty, the note routes into the
// structured wiki at brain/wiki/<wing>/<hall>/<slug>.md and gets
// wing/hall/created_at injected into its YAML frontmatter.
//
// When either is empty, the note falls back to brain/knowledge/<filename>
// with optional type/domain frontmatter (legacy behaviour).
type WriteNoteOptions struct {
Content string
Filename string
Type string
Domain string
Wing string
Hall string
}
// WriteNote writes a markdown note into the brain. Returns the path
// relative to brainDir (forward-slashed). Filename traversal is rejected. // relative to brainDir (forward-slashed). Filename traversal is rejected.
func WriteNote(brainDir, content, filename, typ, domain string) (string, error) { func WriteNote(brainDir string, opts WriteNoteOptions) (string, error) {
if content == "" { if opts.Content == "" {
return "", fmt.Errorf("content is required") return "", fmt.Errorf("content is required")
} }
if opts.Wing != "" && opts.Hall != "" {
return writeHallNote(brainDir, opts)
}
if opts.Wing != "" || opts.Hall != "" {
return "", fmt.Errorf("wing and hall must be set together")
}
return writeLegacyNote(brainDir, opts)
}
// writeHallNote routes a note into brain/wiki/<wing>/<hall>/ and injects
// wing/hall/created_at frontmatter.
func writeHallNote(brainDir string, opts WriteNoteOptions) (string, error) {
slug := opts.Filename
if slug == "" {
slug = time.Now().UTC().Format("2006-01-02-150405") + "-auto"
}
dest, err := brain.NotePath(brainDir, opts.Wing, opts.Hall, slug)
if err != nil {
return "", err
}
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
return "", fmt.Errorf("create hall dir: %w", err)
}
var fm strings.Builder
fm.WriteString("---\n")
fmt.Fprintf(&fm, "wing: %s\n", brain.Sanitise(opts.Wing))
fmt.Fprintf(&fm, "hall: %s\n", opts.Hall)
fmt.Fprintf(&fm, "created_at: %s\n", time.Now().UTC().Format(time.RFC3339))
if opts.Type != "" {
fmt.Fprintf(&fm, "type: %s\n", opts.Type)
}
if opts.Domain != "" {
fmt.Fprintf(&fm, "domain: %s\n", opts.Domain)
}
fm.WriteString("---\n")
if err := os.WriteFile(dest, []byte(fm.String()+opts.Content), 0o644); err != nil {
return "", fmt.Errorf("write: %w", err)
}
rel, _ := filepath.Rel(brainDir, dest)
return filepath.ToSlash(rel), nil
}
// writeLegacyNote preserves the original brain/knowledge/ behaviour for
// callers that have not adopted the wing/hall taxonomy.
func writeLegacyNote(brainDir string, opts WriteNoteOptions) (string, error) {
filename := opts.Filename
if filename == "" { if filename == "" {
filename = fmt.Sprintf("%s-auto.md", time.Now().UTC().Format("2006-01-02-150405")) filename = fmt.Sprintf("%s-auto.md", time.Now().UTC().Format("2006-01-02-150405"))
} }
@@ -101,26 +187,24 @@ func WriteNote(brainDir, content, filename, typ, domain string) (string, error)
return "", fmt.Errorf("create raw dir: %w", err) return "", fmt.Errorf("create raw dir: %w", err)
} }
finalContent := content finalContent := opts.Content
if typ != "" || domain != "" { if opts.Type != "" || opts.Domain != "" {
var fm strings.Builder var fm strings.Builder
fm.WriteString("---\n") fm.WriteString("---\n")
if typ != "" { if opts.Type != "" {
fmt.Fprintf(&fm, "type: %s\n", typ) fmt.Fprintf(&fm, "type: %s\n", opts.Type)
} }
if domain != "" { if opts.Domain != "" {
fmt.Fprintf(&fm, "domain: %s\n", domain) fmt.Fprintf(&fm, "domain: %s\n", opts.Domain)
} }
fm.WriteString("---\n") fm.WriteString("---\n")
finalContent = fm.String() + content finalContent = fm.String() + opts.Content
} }
// Reject path separators outright; any non-flat filename is misuse.
if strings.ContainsAny(filename, `/\`) { if strings.ContainsAny(filename, `/\`) {
return "", fmt.Errorf("invalid filename") return "", fmt.Errorf("invalid filename")
} }
base := filepath.Base(filename) base := filepath.Base(filename)
// After Base, "." and ".." remain. Reject those before adding .md.
if base == "." || base == ".." || base == "" { if base == "." || base == ".." || base == "" {
return "", fmt.Errorf("invalid filename") return "", fmt.Errorf("invalid filename")
} }
@@ -143,15 +227,77 @@ func (h *Handler) Write(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "invalid JSON") writeError(w, http.StatusBadRequest, "invalid JSON")
return return
} }
relPath, err := WriteNote(h.brainDir, req.Content, req.Filename, req.Type, req.Domain) relPath, err := WriteNote(h.brainDir, WriteNoteOptions(req))
if err != nil { if err != nil {
h.logger.Error("write failed", "err", err) h.logger.Error("write failed", "err", err)
writeError(w, http.StatusBadRequest, err.Error()) writeError(w, http.StatusBadRequest, err.Error())
return return
} }
if req.Wing != "" && req.Hall != "" {
if err := brain.BuildWingIndex(h.brainDir, req.Wing); err != nil {
h.logger.Warn("auto-index failed", "wing", req.Wing, "err", err)
}
}
writeJSON(w, map[string]string{"path": relPath}) writeJSON(w, map[string]string{"path": relPath})
} }
// BackfillEmbeddings handles POST /backfill-embeddings — synchronously
// embeds every note under brain/wiki/ that's not yet in the vector
// store, and deletes rows for files no longer on disk.
func (h *Handler) BackfillEmbeddings(w http.ResponseWriter, r *http.Request) {
if h.embedStore == nil || h.embedClient == nil {
writeError(w, http.StatusServiceUnavailable,
"embeddings not configured (set BRAIN_PG_DSN and BRAIN_EMBED_URL)")
return
}
res, err := vectorstore.Sync(r.Context(), h.brainDir, h.embedStore, h.embedClient)
if err != nil {
h.logger.Error("backfill failed", "err", err)
writeError(w, http.StatusInternalServerError, "backfill error")
return
}
errStrs := make([]string, 0, len(res.Errors))
for _, e := range res.Errors {
errStrs = append(errStrs, e.Error())
}
writeJSON(w, map[string]any{
"added": res.Added,
"deleted": res.Deleted,
"errors": errStrs,
})
}
type indexRequest struct {
Wing string `json:"wing,omitempty"`
}
// Index handles POST /index — regenerate the _index.md MOC for one wing
// (when "wing" is set) or for every wing (when omitted).
func (h *Handler) Index(w http.ResponseWriter, r *http.Request) {
var req indexRequest
if r.ContentLength > 0 {
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid JSON")
return
}
}
if req.Wing == "" {
if err := brain.BuildAllWingIndexes(h.brainDir); err != nil {
h.logger.Error("index all failed", "err", err)
writeError(w, http.StatusInternalServerError, "index error")
return
}
writeJSON(w, map[string]any{"status": "ok", "scope": "all"})
return
}
if err := brain.BuildWingIndex(h.brainDir, req.Wing); err != nil {
h.logger.Error("index failed", "wing", req.Wing, "err", err)
writeError(w, http.StatusBadRequest, err.Error())
return
}
writeJSON(w, map[string]any{"status": "ok", "scope": req.Wing})
}
// Ingest handles POST /ingest — run the pipeline on provided content. // Ingest handles POST /ingest — run the pipeline on provided content.
func (h *Handler) Ingest(w http.ResponseWriter, r *http.Request) { func (h *Handler) Ingest(w http.ResponseWriter, r *http.Request) {
var req ingestRequest var req ingestRequest
+131
View File
@@ -0,0 +1,131 @@
package api
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"os"
"path/filepath"
"strings"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
)
// ContentHash returns the lowercase hex sha256 of b. It is the note's
// content_hash handle: brain_write / brain_update return it, brain_get
// recomputes it from the file on disk, and brain_update stamps the prior
// note's hash into the new note's `supersedes` frontmatter.
func ContentHash(b []byte) string {
sum := sha256.Sum256(b)
return hex.EncodeToString(sum[:])
}
// resolveWithin maps a brainDir-relative path to an absolute path and
// guarantees it does not escape brainDir. Returns the cleaned relPath
// (forward-slashed) and the absolute path.
func resolveWithin(brainDir, relPath string) (rel, abs string, err error) {
clean := filepath.Clean("/" + filepath.ToSlash(relPath))
rel = strings.TrimPrefix(clean, "/")
abs = filepath.Join(brainDir, filepath.FromSlash(rel))
check, err := filepath.Rel(brainDir, abs)
if err != nil || check == ".." || strings.HasPrefix(check, ".."+string(filepath.Separator)) {
return "", "", fmt.Errorf("path %q escapes brain dir", relPath)
}
return rel, abs, nil
}
// UpdateNoteOptions identifies the note to supersede and supplies its new
// body. Path takes precedence; otherwise the target is resolved from
// Wing/Hall/Slug via brain.NotePath.
type UpdateNoteOptions struct {
Path string // brainDir-relative path; takes precedence over wing/hall/slug
Wing string
Hall string
Slug string
Content string // new full body (whole-note replace)
Reason string // optional; stamped as supersede_reason
}
// UpdateNote supersedes an existing note in place. It replaces the body
// with opts.Content, preserves the existing frontmatter (created_at,
// wing, hall, and any custom fields), and stamps updated_at, supersedes
// (the prior content hash), and supersede_reason (when given).
//
// It never creates: if the target does not exist, it returns an error so
// the caller can fall back to brain_write. Returns the note's relPath,
// the new content hash, and the prior content hash.
//
// Embeddings are NOT refreshed here. The rewritten file's mtime advances,
// which the mtime-driven vectorstore.Sync ticker uses to re-embed it on
// its next pass — the same out-of-band mechanism brain_write relies on.
func UpdateNote(brainDir string, opts UpdateNoteOptions) (relPath, contentHash, priorHash string, err error) {
if opts.Content == "" {
return "", "", "", fmt.Errorf("content is required")
}
var rel string
if opts.Path != "" {
rel = opts.Path
} else {
full, perr := brain.NotePath(brainDir, opts.Wing, opts.Hall, opts.Slug)
if perr != nil {
return "", "", "", perr
}
rel, _ = filepath.Rel(brainDir, full)
rel = filepath.ToSlash(rel)
}
rel, abs, err := resolveWithin(brainDir, rel)
if err != nil {
return "", "", "", err
}
prior, err := os.ReadFile(abs)
if err != nil {
if os.IsNotExist(err) {
return "", "", "", fmt.Errorf("note %q does not exist: use brain_write to create", rel)
}
return "", "", "", fmt.Errorf("read target: %w", err)
}
priorHash = ContentHash(prior)
fm, _ := parseFrontmatter(string(prior))
fm.set("updated_at", time.Now().UTC().Format(time.RFC3339))
fm.set("supersedes", priorHash)
if opts.Reason != "" {
fm.set("supersede_reason", opts.Reason)
}
out := []byte(fm.render() + opts.Content)
if err := os.WriteFile(abs, out, 0o644); err != nil {
return "", "", "", fmt.Errorf("write: %w", err)
}
return rel, ContentHash(out), priorHash, nil
}
// ReadNote reads the note at the brainDir-relative relPath and returns
// its parsed frontmatter, body, and content hash. It is the read-after-
// write primitive behind brain_get: the hash it returns equals the hash
// brain_write / brain_update returned for the same bytes.
func ReadNote(brainDir, relPath string) (fm map[string]string, body, contentHash string, err error) {
_, abs, err := resolveWithin(brainDir, relPath)
if err != nil {
return nil, "", "", err
}
raw, err := os.ReadFile(abs)
if err != nil {
if os.IsNotExist(err) {
return nil, "", "", fmt.Errorf("note %q does not exist", relPath)
}
return nil, "", "", fmt.Errorf("read note: %w", err)
}
parsed, body := parseFrontmatter(string(raw))
fm = make(map[string]string, len(parsed.lines))
for _, l := range parsed.lines {
if l.key != "" {
fm[l.key] = l.value
}
}
return fm, body, ContentHash(raw), nil
}
+130
View File
@@ -0,0 +1,130 @@
package api
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// seedNote writes a note directly to disk and returns its relPath.
func seedNote(t *testing.T, brainDir, rel, content string) string {
t.Helper()
full := filepath.Join(brainDir, filepath.FromSlash(rel))
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
require.NoError(t, os.WriteFile(full, []byte(content), 0o644))
return rel
}
func TestUpdateNoteSupersedesAndStamps(t *testing.T) {
brainDir := t.TempDir()
rel := seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md",
"---\nwing: jepa-fx\nhall: facts\ncreated_at: 2026-01-01T00:00:00Z\ncustom: keep-me\n---\n# Old\n\nold body\n")
relPath, hash, priorHash, err := UpdateNote(brainDir, UpdateNoteOptions{
Path: rel,
Content: "# New\n\nnew body\n",
Reason: "facts changed",
})
require.NoError(t, err)
assert.Equal(t, rel, relPath)
assert.NotEmpty(t, hash)
assert.NotEmpty(t, priorHash)
assert.NotEqual(t, hash, priorHash)
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
require.NoError(t, err)
s := string(got)
// Body replaced.
assert.Contains(t, s, "# New")
assert.NotContains(t, s, "old body")
// Prior fields preserved.
assert.Contains(t, s, "wing: jepa-fx")
assert.Contains(t, s, "hall: facts")
assert.Contains(t, s, "created_at: 2026-01-01T00:00:00Z")
assert.Contains(t, s, "custom: keep-me")
// Supersession stamped.
assert.Contains(t, s, "updated_at:")
assert.Contains(t, s, "supersedes: "+priorHash)
assert.Contains(t, s, "supersede_reason: facts changed")
}
func TestUpdateNoteResolvesByWingHallSlug(t *testing.T) {
brainDir := t.TempDir()
seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md",
"---\nwing: jepa-fx\nhall: facts\n---\nold\n")
relPath, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
Wing: "jepa-fx", Hall: "facts", Slug: "val-vol",
Content: "new\n",
})
require.NoError(t, err)
assert.Equal(t, "wiki/jepa-fx/facts/val-vol.md", relPath)
}
func TestUpdateNoteErrorsOnMissingAndDoesNotCreate(t *testing.T) {
brainDir := t.TempDir()
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
Wing: "jepa-fx", Hall: "facts", Slug: "ghost",
Content: "x\n",
})
require.Error(t, err)
assert.Contains(t, err.Error(), "does not exist")
// No file created.
_, statErr := os.Stat(filepath.Join(brainDir, "wiki/jepa-fx/facts/ghost.md"))
assert.True(t, os.IsNotExist(statErr), "missing-target update must not create a note")
}
func TestUpdateNoteRejectsTraversal(t *testing.T) {
brainDir := t.TempDir()
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
Path: "../escape.md",
Content: "x\n",
})
require.Error(t, err)
}
func TestReadNoteReturnsFrontmatterBodyHash(t *testing.T) {
brainDir := t.TempDir()
rel := seedNote(t, brainDir, "wiki/jepa-fx/facts/n.md",
"---\nwing: jepa-fx\nhall: facts\n---\n# Body\n\ntext\n")
fm, body, hash, err := ReadNote(brainDir, rel)
require.NoError(t, err)
assert.Equal(t, "jepa-fx", fm["wing"])
assert.Equal(t, "facts", fm["hall"])
assert.Equal(t, "# Body\n\ntext\n", body)
// Hash matches ContentHash of the raw bytes on disk (round-trip).
raw, _ := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
assert.Equal(t, ContentHash(raw), hash)
}
func TestReadNoteRejectsTraversal(t *testing.T) {
brainDir := t.TempDir()
_, _, _, err := ReadNote(brainDir, "../../etc/passwd")
require.Error(t, err)
}
func TestUpdateThenReadRoundTripsHash(t *testing.T) {
brainDir := t.TempDir()
rel := seedNote(t, brainDir, "wiki/a/facts/n.md", "---\nwing: a\nhall: facts\n---\nold\n")
_, hash, _, err := UpdateNote(brainDir, UpdateNoteOptions{Path: rel, Content: "new\n"})
require.NoError(t, err)
_, _, readHash, err := ReadNote(brainDir, rel)
require.NoError(t, err)
assert.Equal(t, hash, readHash, "update content_hash must round-trip through ReadNote")
}
func TestContentHashStable(t *testing.T) {
assert.Equal(t, ContentHash([]byte("abc")), ContentHash([]byte("abc")))
assert.NotEqual(t, ContentHash([]byte("abc")), ContentHash([]byte("abd")))
assert.True(t, strings.HasPrefix(ContentHash([]byte("")), "")) // hex, non-panicking
}
-84
View File
@@ -1,84 +0,0 @@
package auth
import (
"context"
"encoding/json"
"fmt"
"net/http"
"time"
"github.com/lestrrat-go/jwx/v2/jwk"
"github.com/lestrrat-go/jwx/v2/jwt"
)
// Validator validates Bearer JWTs issued by a Dex (OIDC) authorization server.
// Audience is optional; leave empty to skip audience validation.
type Validator struct {
issuer string
audience string
jwksURI string
cache *jwk.Cache
}
// NewValidator fetches the OIDC discovery document from issuerURL, extracts
// jwks_uri, seeds the JWKS cache, and returns a ready Validator.
// If DEX_ISSUER_URL is not set the caller should pass "" and skip construction.
func NewValidator(issuerURL, audience string) (*Validator, error) {
resp, err := http.Get(issuerURL + "/.well-known/openid-configuration") //nolint:noctx
if err != nil {
return nil, fmt.Errorf("fetch oidc discovery: %w", err)
}
defer resp.Body.Close() //nolint:errcheck
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("oidc discovery: status %d", resp.StatusCode)
}
var doc struct {
JWKSURI string `json:"jwks_uri"`
}
if err := json.NewDecoder(resp.Body).Decode(&doc); err != nil {
return nil, fmt.Errorf("decode oidc discovery: %w", err)
}
if doc.JWKSURI == "" {
return nil, fmt.Errorf("oidc discovery: empty jwks_uri")
}
ctx := context.Background()
cache := jwk.NewCache(ctx)
if err := cache.Register(doc.JWKSURI, jwk.WithMinRefreshInterval(time.Hour)); err != nil {
return nil, fmt.Errorf("register jwks cache: %w", err)
}
if _, err := cache.Refresh(ctx, doc.JWKSURI); err != nil {
return nil, fmt.Errorf("initial jwks fetch: %w", err)
}
return &Validator{
issuer: issuerURL,
audience: audience,
jwksURI: doc.JWKSURI,
cache: cache,
}, nil
}
// Validate parses and validates rawToken. Returns the subject claim on success.
func (v *Validator) Validate(ctx context.Context, rawToken string) (string, error) {
keySet, err := v.cache.Get(ctx, v.jwksURI)
if err != nil {
return "", fmt.Errorf("get jwks: %w", err)
}
opts := []jwt.ParseOption{
jwt.WithKeySet(keySet),
jwt.WithValidate(true),
jwt.WithIssuer(v.issuer),
}
if v.audience != "" {
opts = append(opts, jwt.WithAudience(v.audience))
}
tok, err := jwt.ParseString(rawToken, opts...)
if err != nil {
return "", fmt.Errorf("validate jwt: %w", err)
}
return tok.Subject(), nil
}
-169
View File
@@ -1,169 +0,0 @@
package auth_test
import (
"context"
"crypto/rand"
"crypto/rsa"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/lestrrat-go/jwx/v2/jwa"
"github.com/lestrrat-go/jwx/v2/jwk"
"github.com/lestrrat-go/jwx/v2/jwt"
"github.com/mathiasbq/hyperguild/ingestion/internal/auth"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
type testKeys struct {
priv jwk.Key
pub jwk.Key
}
func generateRSAKeys(t *testing.T) testKeys {
t.Helper()
raw, err := rsa.GenerateKey(rand.Reader, 2048)
require.NoError(t, err)
priv, err := jwk.FromRaw(raw)
require.NoError(t, err)
require.NoError(t, priv.Set(jwk.KeyIDKey, "test-kid"))
require.NoError(t, priv.Set(jwk.AlgorithmKey, jwa.RS256))
pub, err := jwk.PublicKeyOf(priv)
require.NoError(t, err)
return testKeys{priv: priv, pub: pub}
}
func mockOIDCServer(t *testing.T, keys testKeys) *httptest.Server {
t.Helper()
set := jwk.NewSet()
require.NoError(t, set.AddKey(keys.pub))
jwksBytes, err := json.Marshal(set)
require.NoError(t, err)
mux := http.NewServeMux()
var srv *httptest.Server
mux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]string{
"issuer": srv.URL,
"jwks_uri": srv.URL + "/jwks",
})
})
mux.HandleFunc("/jwks", func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write(jwksBytes)
})
srv = httptest.NewServer(mux)
t.Cleanup(srv.Close)
return srv
}
func signToken(t *testing.T, keys testKeys, issuer, audience, subject string, exp time.Time) string {
t.Helper()
b := jwt.NewBuilder().
Issuer(issuer).
Subject(subject).
Expiration(exp)
if audience != "" {
b = b.Audience([]string{audience})
}
tok, err := b.Build()
require.NoError(t, err)
signed, err := jwt.Sign(tok, jwt.WithKey(jwa.RS256, keys.priv))
require.NoError(t, err)
return string(signed)
}
func TestValidator(t *testing.T) {
keys := generateRSAKeys(t)
srv := mockOIDCServer(t, keys)
ctx := context.Background()
v, err := auth.NewValidator(srv.URL, "brain")
require.NoError(t, err)
tests := []struct {
name string
token string
wantSub string
wantErr bool
}{
{
name: "valid jwt",
token: signToken(t, keys, srv.URL, "brain", "test-user", time.Now().Add(time.Hour)),
wantSub: "test-user",
},
{
name: "expired jwt",
token: signToken(t, keys, srv.URL, "brain", "test-user", time.Now().Add(-time.Hour)),
wantErr: true,
},
{
name: "wrong issuer",
token: signToken(t, keys, "https://evil.example.com", "brain", "test-user", time.Now().Add(time.Hour)),
wantErr: true,
},
{
name: "wrong audience",
token: signToken(t, keys, srv.URL, "other-service", "test-user", time.Now().Add(time.Hour)),
wantErr: true,
},
{
name: "tampered token",
token: signToken(t, keys, srv.URL, "brain", "test-user", time.Now().Add(time.Hour)) + "tampered",
wantErr: true,
},
{
name: "not a jwt",
token: "not-a-jwt",
wantErr: true,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
sub, err := v.Validate(ctx, tc.token)
if tc.wantErr {
assert.Error(t, err)
assert.Empty(t, sub)
} else {
require.NoError(t, err)
assert.Equal(t, tc.wantSub, sub)
}
})
}
}
func TestNewValidator_NoAudience(t *testing.T) {
keys := generateRSAKeys(t)
srv := mockOIDCServer(t, keys)
ctx := context.Background()
v, err := auth.NewValidator(srv.URL, "")
require.NoError(t, err)
// Token without audience passes when audience validation is disabled.
tok, err := jwt.NewBuilder().
Issuer(srv.URL).
Subject("sub").
Expiration(time.Now().Add(time.Hour)).
Build()
require.NoError(t, err)
signed, err := jwt.Sign(tok, jwt.WithKey(jwa.RS256, keys.priv))
require.NoError(t, err)
sub, err := v.Validate(ctx, string(signed))
require.NoError(t, err)
assert.Equal(t, "sub", sub)
}
func TestNewValidator_BadDiscoveryURL(t *testing.T) {
_, err := auth.NewValidator("http://127.0.0.1:1", "brain")
assert.Error(t, err)
}
@@ -1,23 +0,0 @@
package auth
import (
"encoding/json"
"net/http"
)
// ProtectedResourceHandler returns an RFC 9728 oauth-protected-resource metadata
// handler. Mount at GET /.well-known/oauth-protected-resource (no auth required).
func ProtectedResourceHandler(resourceURL, issuerURL string) http.HandlerFunc {
type metadata struct {
Resource string `json:"resource"`
AuthorizationServers []string `json:"authorization_servers"`
}
body, _ := json.Marshal(metadata{
Resource: resourceURL,
AuthorizationServers: []string{issuerURL},
})
return func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write(body)
}
}
@@ -1,28 +0,0 @@
package auth_test
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/auth"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestProtectedResourceHandler(t *testing.T) {
h := auth.ProtectedResourceHandler("https://brain-mcp.d-ma.be", "https://auth.d-ma.be")
req := httptest.NewRequest(http.MethodGet, "/.well-known/oauth-protected-resource", nil)
rr := httptest.NewRecorder()
h(rr, req)
assert.Equal(t, http.StatusOK, rr.Code)
assert.Equal(t, "application/json", rr.Header().Get("Content-Type"))
var body map[string]any
require.NoError(t, json.Unmarshal(rr.Body.Bytes(), &body))
assert.Equal(t, "https://brain-mcp.d-ma.be", body["resource"])
servers := body["authorization_servers"].([]any)
assert.Equal(t, "https://auth.d-ma.be", servers[0])
}
+161
View File
@@ -0,0 +1,161 @@
package brain
import (
"bufio"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"time"
)
// noteEntry is one row in a Wing _index.md.
type noteEntry struct {
Hall string
Slug string
Title string
Created string
}
// BuildWingIndex regenerates brain/wiki/<wing>/_index.md as a Map of
// Content listing every note in that wing with its Hall and creation
// date. Returns nil if the wing directory does not exist.
func BuildWingIndex(brainDir, wing string) error {
w := Sanitise(wing)
if w == "" {
return fmt.Errorf("invalid wing %q", wing)
}
wingDir := filepath.Join(brainDir, "wiki", w)
if _, err := os.Stat(wingDir); os.IsNotExist(err) {
return nil
} else if err != nil {
return fmt.Errorf("stat wing: %w", err)
}
entries, err := collectWingEntries(wingDir)
if err != nil {
return err
}
sort.Slice(entries, func(i, j int) bool {
if entries[i].Hall != entries[j].Hall {
return entries[i].Hall < entries[j].Hall
}
return entries[i].Slug < entries[j].Slug
})
var b strings.Builder
fmt.Fprintf(&b, "# %s\n\n", w)
b.WriteString("| Hall | Note | Created |\n")
b.WriteString("|------|------|---------|\n")
for _, e := range entries {
fmt.Fprintf(&b, "| %s | [%s](%s/%s.md) | %s |\n", e.Hall, e.Title, e.Hall, e.Slug, e.Created)
}
dest := filepath.Join(wingDir, "_index.md")
return os.WriteFile(dest, []byte(b.String()), 0o644)
}
// BuildAllWingIndexes regenerates _index.md for every wing under brain/wiki/.
func BuildAllWingIndexes(brainDir string) error {
wikiDir := filepath.Join(brainDir, "wiki")
ents, err := os.ReadDir(wikiDir)
if os.IsNotExist(err) {
return nil
}
if err != nil {
return fmt.Errorf("read wiki: %w", err)
}
for _, e := range ents {
if !e.IsDir() {
continue
}
if err := BuildWingIndex(brainDir, e.Name()); err != nil {
return fmt.Errorf("index %s: %w", e.Name(), err)
}
}
return nil
}
func collectWingEntries(wingDir string) ([]noteEntry, error) {
var out []noteEntry
ents, err := os.ReadDir(wingDir)
if err != nil {
return nil, fmt.Errorf("read wing: %w", err)
}
for _, hallEnt := range ents {
if !hallEnt.IsDir() {
continue
}
hall := hallEnt.Name()
if !IsValidHall(hall) {
continue
}
hallDir := filepath.Join(wingDir, hall)
notes, err := os.ReadDir(hallDir)
if err != nil {
return nil, fmt.Errorf("read hall %s: %w", hall, err)
}
for _, n := range notes {
if n.IsDir() || !strings.HasSuffix(n.Name(), ".md") || n.Name() == "_index.md" {
continue
}
slug := strings.TrimSuffix(n.Name(), ".md")
full := filepath.Join(hallDir, n.Name())
title, created := readTitleAndCreated(full, slug)
out = append(out, noteEntry{Hall: hall, Slug: slug, Title: title, Created: created})
}
}
return out, nil
}
// readTitleAndCreated reads YAML frontmatter for title + created_at; falls
// back to slug and file mtime when absent.
func readTitleAndCreated(path, slug string) (string, string) {
f, err := os.Open(path)
if err != nil {
return slug, ""
}
defer func() { _ = f.Close() }()
title, created := "", ""
scanner := bufio.NewScanner(f)
inFrontmatter := false
for scanner.Scan() {
line := scanner.Text()
if strings.TrimSpace(line) == "---" {
if !inFrontmatter {
inFrontmatter = true
continue
}
break
}
if !inFrontmatter {
continue
}
key, val, ok := strings.Cut(line, ":")
if !ok {
continue
}
v := strings.Trim(strings.TrimSpace(val), `"'`)
switch strings.TrimSpace(key) {
case "title":
title = v
case "created_at":
if t, err := time.Parse(time.RFC3339, v); err == nil {
created = t.UTC().Format("2006-01-02")
} else {
created = v
}
}
}
if title == "" {
title = strings.ReplaceAll(slug, "-", " ")
}
if created == "" {
if info, err := os.Stat(path); err == nil {
created = info.ModTime().UTC().Format("2006-01-02")
}
}
return title, created
}
+85
View File
@@ -0,0 +1,85 @@
package brain_test
import (
"os"
"path/filepath"
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestBuildWingIndex(t *testing.T) {
dir := t.TempDir()
for _, p := range []struct{ rel, body string }{
{"wiki/jepa-fx/decisions/val-vol.md", "---\ntitle: Val Vol R2\ncreated_at: 2026-05-06T10:00:00Z\n---\nbody\n"},
{"wiki/jepa-fx/facts/architecture.md", "---\ntitle: Architecture\ncreated_at: 2026-05-04T10:00:00Z\n---\nbody\n"},
{"wiki/jepa-fx/sources/paper.md", "---\n---\nbody\n"},
} {
full := filepath.Join(dir, p.rel)
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
require.NoError(t, os.WriteFile(full, []byte(p.body), 0o644))
}
require.NoError(t, brain.BuildWingIndex(dir, "jepa-fx"))
got, err := os.ReadFile(filepath.Join(dir, "wiki", "jepa-fx", "_index.md"))
require.NoError(t, err)
s := string(got)
assert.Contains(t, s, "# jepa-fx")
assert.Contains(t, s, "| Hall | Note | Created |")
assert.Contains(t, s, "| decisions | [Val Vol R2](decisions/val-vol.md) | 2026-05-06 |")
assert.Contains(t, s, "| facts | [Architecture](facts/architecture.md) | 2026-05-04 |")
assert.Contains(t, s, "| sources | [paper](sources/paper.md) |")
// Halls sorted alphabetically.
assert.Less(t, indexOf(s, "decisions"), indexOf(s, "facts"))
assert.Less(t, indexOf(s, "facts"), indexOf(s, "sources"))
}
func TestBuildWingIndex_SkipsInvalidHalls(t *testing.T) {
dir := t.TempDir()
wingDir := filepath.Join(dir, "wiki", "jepa-fx")
require.NoError(t, os.MkdirAll(filepath.Join(wingDir, "garbage"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(wingDir, "garbage", "x.md"), []byte("x"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(wingDir, "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(wingDir, "facts", "y.md"), []byte("y"), 0o644))
require.NoError(t, brain.BuildWingIndex(dir, "jepa-fx"))
got, err := os.ReadFile(filepath.Join(wingDir, "_index.md"))
require.NoError(t, err)
s := string(got)
assert.Contains(t, s, "facts")
assert.NotContains(t, s, "garbage")
}
func TestBuildAllWingIndexes(t *testing.T) {
dir := t.TempDir()
for _, p := range []struct{ rel, body string }{
{"wiki/a/facts/x.md", "x"},
{"wiki/b/facts/y.md", "y"},
} {
full := filepath.Join(dir, p.rel)
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
require.NoError(t, os.WriteFile(full, []byte(p.body), 0o644))
}
require.NoError(t, brain.BuildAllWingIndexes(dir))
_, err := os.Stat(filepath.Join(dir, "wiki", "a", "_index.md"))
require.NoError(t, err)
_, err = os.Stat(filepath.Join(dir, "wiki", "b", "_index.md"))
require.NoError(t, err)
}
func TestBuildWingIndex_NoWingDir(t *testing.T) {
dir := t.TempDir()
require.NoError(t, brain.BuildWingIndex(dir, "ghost"))
}
func indexOf(s, sub string) int {
for i := 0; i+len(sub) <= len(s); i++ {
if s[i:i+len(sub)] == sub {
return i
}
}
return -1
}
+70
View File
@@ -0,0 +1,70 @@
// Package brain provides the wing/hall path taxonomy used by the brain
// wiki layout. A note's canonical location is
// brain/wiki/<wing>/<hall>/<slug>.md, where Wing is a free-form topic
// domain and Hall is one of a closed vocabulary of memory types.
package brain
import (
"fmt"
"path/filepath"
"strings"
)
// ValidHalls is the closed vocabulary of hall names. A hall captures the
// memory type of a note within any wing.
var ValidHalls = map[string]bool{
"facts": true,
"decisions": true,
"failures": true,
"hypotheses": true,
"sources": true,
}
// IsValidHall reports whether h is in the closed Hall vocabulary.
func IsValidHall(h string) bool {
return ValidHalls[h]
}
// NotePath resolves the canonical filesystem path for a note given a
// wing, hall, and slug. Returns an error if hall is not in ValidHalls
// or if wing/slug sanitise to empty strings.
//
// The returned path is brain/wiki/<wing>/<hall>/<slug>.md with all
// segments sanitised: lowercased, alphanumerics and hyphens only.
func NotePath(brainDir, wing, hall, slug string) (string, error) {
if !IsValidHall(hall) {
return "", fmt.Errorf("invalid hall %q: must be one of facts/decisions/failures/hypotheses/sources", hall)
}
w := Sanitise(wing)
if w == "" {
return "", fmt.Errorf("invalid wing %q: must contain at least one alphanumeric character", wing)
}
s := Sanitise(strings.TrimSuffix(slug, ".md"))
if s == "" {
return "", fmt.Errorf("invalid slug %q: must contain at least one alphanumeric character", slug)
}
return filepath.Join(brainDir, "wiki", w, hall, s+".md"), nil
}
// Sanitise lowercases s and keeps only [a-z0-9-], collapsing any other
// character (including path separators) to a hyphen. Leading/trailing
// hyphens and runs of hyphens are collapsed.
func Sanitise(s string) string {
s = strings.ToLower(strings.TrimSpace(s))
var b strings.Builder
prevHyphen := true
for _, r := range s {
switch {
case r >= 'a' && r <= 'z', r >= '0' && r <= '9':
b.WriteRune(r)
prevHyphen = false
case r == '-' || r == '_' || r == ' ' || r == '/' || r == '\\' || r == '.':
if !prevHyphen {
b.WriteByte('-')
prevHyphen = true
}
}
}
out := b.String()
return strings.Trim(out, "-")
}
+73
View File
@@ -0,0 +1,73 @@
package brain_test
import (
"path/filepath"
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestNotePath_Valid(t *testing.T) {
got, err := brain.NotePath("/b", "jepa-fx", "decisions", "val-vol-r2")
require.NoError(t, err)
assert.Equal(t, filepath.Join("/b", "wiki", "jepa-fx", "decisions", "val-vol-r2.md"), got)
}
func TestNotePath_StripsMdSuffix(t *testing.T) {
got, err := brain.NotePath("/b", "x", "facts", "note.md")
require.NoError(t, err)
assert.Equal(t, filepath.Join("/b", "wiki", "x", "facts", "note.md"), got)
}
func TestNotePath_SanitisesWingAndSlug(t *testing.T) {
got, err := brain.NotePath("/b", "Jepa FX!", "facts", "Val Vol R2")
require.NoError(t, err)
assert.Equal(t, filepath.Join("/b", "wiki", "jepa-fx", "facts", "val-vol-r2.md"), got)
}
func TestNotePath_RejectsInvalidHall(t *testing.T) {
_, err := brain.NotePath("/b", "x", "garbage", "y")
require.Error(t, err)
assert.Contains(t, err.Error(), "invalid hall")
}
func TestNotePath_RejectsEmptyWing(t *testing.T) {
_, err := brain.NotePath("/b", "!!!", "facts", "y")
require.Error(t, err)
assert.Contains(t, err.Error(), "invalid wing")
}
func TestNotePath_RejectsEmptySlug(t *testing.T) {
_, err := brain.NotePath("/b", "x", "facts", "!!!")
require.Error(t, err)
assert.Contains(t, err.Error(), "invalid slug")
}
func TestSanitise(t *testing.T) {
cases := map[string]string{
"Jepa-FX": "jepa-fx",
" foo bar ": "foo-bar",
"Val/Vol\\R2.md": "val-vol-r2-md",
"!!!": "",
"___leading": "leading",
"trailing___": "trailing",
"multi---hyphen": "multi-hyphen",
"UPPER 123 mixed": "upper-123-mixed",
}
for in, want := range cases {
t.Run(in, func(t *testing.T) {
assert.Equal(t, want, brain.Sanitise(in))
})
}
}
func TestIsValidHall(t *testing.T) {
for _, h := range []string{"facts", "decisions", "failures", "hypotheses", "sources"} {
assert.True(t, brain.IsValidHall(h), h)
}
for _, h := range []string{"", "Facts", "facts ", "rooms", "concepts", "entities"} {
assert.False(t, brain.IsValidHall(h), h)
}
}
+286
View File
@@ -0,0 +1,286 @@
package brain
import (
"bufio"
"fmt"
"os"
"path/filepath"
"strings"
"time"
)
// seeAlsoHeader is the markdown heading used to group cross-wing links.
const seeAlsoHeader = "## See also"
// TunnelCandidate is a cross-wing match surfaced by DetectTunnels. It is
// not yet a written link — the caller decides whether confidence is high
// enough to commit it via WriteTunnel.
type TunnelCandidate struct {
// TargetPath is the candidate note's path relative to brainDir
// (forward-slashed), e.g. "wiki/hyperguild/decisions/routing.md".
TargetPath string
// MatchedTerm is the title that matched in the source content.
MatchedTerm string
// Exact is true when the match was a case-insensitive whole-token
// hit on the target's frontmatter title. Fuzzy matches (substring
// only) are flagged Exact=false and should not be auto-written.
Exact bool
}
// DetectTunnels scans brain/wiki/ for notes whose title appears in
// content. Returns one TunnelCandidate per matching note. Exact is true
// when content contains the title as a whole-word case-insensitive
// token; false when only a substring matched (caller treats these as
// fuzzy and should not auto-write them).
//
// A note's title is read from YAML frontmatter `title:`; failing that,
// the filename slug (sans `.md`, hyphens → spaces) is used.
func DetectTunnels(brainDir, content string) ([]TunnelCandidate, error) {
wikiDir := filepath.Join(brainDir, "wiki")
if _, err := os.Stat(wikiDir); os.IsNotExist(err) {
return nil, nil
} else if err != nil {
return nil, fmt.Errorf("stat wiki: %w", err)
}
lowerContent := strings.ToLower(content)
var out []TunnelCandidate
err := filepath.WalkDir(wikiDir, func(path string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() || !strings.HasSuffix(path, ".md") || d.Name() == "_index.md" {
return nil
}
title, _ := readTitleAndCreated(path, strings.TrimSuffix(d.Name(), ".md"))
needle := strings.ToLower(strings.TrimSpace(title))
if needle == "" {
return nil
}
idx := strings.Index(lowerContent, needle)
if idx == -1 {
return nil
}
rel, err := filepath.Rel(brainDir, path)
if err != nil {
return err
}
out = append(out, TunnelCandidate{
TargetPath: filepath.ToSlash(rel),
MatchedTerm: title,
Exact: isWholeWord(lowerContent, idx, len(needle)),
})
return nil
})
if err != nil {
return nil, err
}
return out, nil
}
// isWholeWord reports whether the substring at [idx, idx+n) in s is
// bounded by non-alphanumeric characters (or string edges).
func isWholeWord(s string, idx, n int) bool {
left := idx == 0 || !isWordByte(s[idx-1])
right := idx+n == len(s) || !isWordByte(s[idx+n])
return left && right
}
func isWordByte(b byte) bool {
return (b >= 'a' && b <= 'z') ||
(b >= 'A' && b <= 'Z') ||
(b >= '0' && b <= '9')
}
// AutoTunnel runs DetectTunnels against content and, for each
// candidate, either writes a bidirectional tunnel (when the match is
// exact and in a different wing) or stages it for human review in
// brain/raw/tunnel-candidates-<YYYY-MM-DD>.md.
//
// sourcePath is the note that originated the content — used to skip
// self-matches and same-wing tunnels. Errors writing individual
// tunnels are recorded into the candidates file but never abort the
// rest of the scan; the caller's primary write has already succeeded
// and auto-linking is best-effort.
func AutoTunnel(brainDir, sourcePath, content string) error {
srcWing, err := wingOf(sourcePath)
if err != nil {
return err
}
candidates, err := DetectTunnels(brainDir, content)
if err != nil {
return err
}
var fuzzy []TunnelCandidate
for _, c := range candidates {
if c.TargetPath == sourcePath {
continue
}
tgtWing, err := wingOf(c.TargetPath)
if err != nil || tgtWing == srcWing {
continue
}
if !c.Exact {
fuzzy = append(fuzzy, c)
continue
}
if err := WriteTunnel(brainDir, sourcePath, c.TargetPath); err != nil {
fuzzy = append(fuzzy, c)
}
}
return logFuzzyCandidates(brainDir, sourcePath, fuzzy)
}
// logFuzzyCandidates appends one row per candidate to
// brain/raw/tunnel-candidates-<YYYY-MM-DD>.md, creating the file with a
// header on first write of the day. No-op when the candidate list is empty.
func logFuzzyCandidates(brainDir, sourcePath string, cs []TunnelCandidate) error {
if len(cs) == 0 {
return nil
}
rawDir := filepath.Join(brainDir, "raw")
if err := os.MkdirAll(rawDir, 0o755); err != nil {
return err
}
stamp := time.Now().UTC().Format("2006-01-02")
path := filepath.Join(rawDir, "tunnel-candidates-"+stamp+".md")
existed := fileExists(path)
f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
if err != nil {
return err
}
defer func() { _ = f.Close() }()
if !existed {
if _, err := f.WriteString("# Tunnel candidates " + stamp + "\n\nFuzzy cross-wing matches surfaced by AutoTunnel. Review and promote to a tunnel via `brain_tunnel` if relevant.\n\n"); err != nil {
return err
}
}
for _, c := range cs {
line := fmt.Sprintf("- `%s` ↔ `%s` (term: %q)\n", sourcePath, c.TargetPath, c.MatchedTerm)
if _, err := f.WriteString(line); err != nil {
return err
}
}
return nil
}
func fileExists(p string) bool {
_, err := os.Stat(p)
return err == nil
}
// WriteTunnel appends a bidirectional wikilink between sourcePath and
// targetPath under a `## See also` section in each note. Paths are
// relative to brainDir (forward-slashed), e.g. wiki/<wing>/<hall>/<slug>.md.
//
// Idempotent: re-calling with the same pair does not duplicate links or
// section headers. Rejects same-wing pairs (a tunnel is by definition
// cross-wing) and missing notes.
func WriteTunnel(brainDir, sourcePath, targetPath string) error {
srcWing, err := wingOf(sourcePath)
if err != nil {
return fmt.Errorf("source: %w", err)
}
tgtWing, err := wingOf(targetPath)
if err != nil {
return fmt.Errorf("target: %w", err)
}
if srcWing == tgtWing {
return fmt.Errorf("tunnel must cross wings; got both in %q", srcWing)
}
srcFull := filepath.Join(brainDir, filepath.FromSlash(sourcePath))
tgtFull := filepath.Join(brainDir, filepath.FromSlash(targetPath))
if _, err := os.Stat(srcFull); err != nil {
return fmt.Errorf("source note: %w", err)
}
if _, err := os.Stat(tgtFull); err != nil {
return fmt.Errorf("target note: %w", err)
}
if err := appendSeeAlso(srcFull, wikilinkOf(targetPath)); err != nil {
return fmt.Errorf("update source: %w", err)
}
if err := appendSeeAlso(tgtFull, wikilinkOf(sourcePath)); err != nil {
return fmt.Errorf("update target: %w", err)
}
return nil
}
// wikilinkOf turns "wiki/<wing>/<hall>/<slug>.md" into "<wing>/<hall>/<slug>"
// for use inside `[[...]]`.
func wikilinkOf(relPath string) string {
p := strings.TrimSuffix(relPath, ".md")
p = strings.TrimPrefix(p, "wiki/")
return p
}
// wingOf extracts the wing segment from a relative wiki path
// "wiki/<wing>/<hall>/<slug>.md".
func wingOf(relPath string) (string, error) {
parts := strings.Split(relPath, "/")
if len(parts) < 4 || parts[0] != "wiki" {
return "", fmt.Errorf("not a wiki path: %q", relPath)
}
if parts[1] == "" {
return "", fmt.Errorf("empty wing in path: %q", relPath)
}
return parts[1], nil
}
// appendSeeAlso inserts `- [[link]]` under the file's See also section,
// creating the section if absent. No-op when the link is already present.
func appendSeeAlso(filePath, link string) error {
content, err := os.ReadFile(filePath)
if err != nil {
return err
}
wikilink := "[[" + link + "]]"
if strings.Contains(string(content), wikilink) {
return nil
}
bullet := "- " + wikilink
if !strings.Contains(string(content), seeAlsoHeader) {
// No section yet — append a fresh one. Always emit a trailing
// newline so subsequent appends don't merge into the previous line.
trimmed := strings.TrimRight(string(content), "\n")
out := trimmed + "\n\n" + seeAlsoHeader + "\n\n" + bullet + "\n"
return os.WriteFile(filePath, []byte(out), 0o644)
}
// Section exists — splice the bullet in just before the next `## `
// heading (or EOF). Reading the file line-by-line keeps this robust
// against arbitrary section ordering.
var b strings.Builder
scanner := bufio.NewScanner(strings.NewReader(string(content)))
scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024)
inSeeAlso, inserted := false, false
for scanner.Scan() {
line := scanner.Text()
if !inserted && inSeeAlso && strings.HasPrefix(line, "## ") &&
strings.TrimSpace(line) != seeAlsoHeader {
b.WriteString(bullet)
b.WriteByte('\n')
b.WriteByte('\n')
inserted = true
}
if strings.TrimSpace(line) == seeAlsoHeader {
inSeeAlso = true
}
b.WriteString(line)
b.WriteByte('\n')
}
if err := scanner.Err(); err != nil {
return err
}
if !inserted {
// section was the last thing in the file — just append bullet
out := strings.TrimRight(b.String(), "\n") + "\n" + bullet + "\n"
return os.WriteFile(filePath, []byte(out), 0o644)
}
return os.WriteFile(filePath, []byte(b.String()), 0o644)
}
+177
View File
@@ -0,0 +1,177 @@
package brain_test
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// seedNote writes a minimal markdown note at brainDir/relPath with the given body.
func seedNote(t *testing.T, brainDir, relPath, body string) {
t.Helper()
full := filepath.Join(brainDir, relPath)
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
require.NoError(t, os.WriteFile(full, []byte(body), 0o644))
}
func TestWriteTunnel_AppendsBidirectionalLinks(t *testing.T) {
dir := t.TempDir()
seedNote(t, dir, "wiki/jepa-fx/decisions/val-vol.md",
"---\nwing: jepa-fx\nhall: decisions\n---\n# Val Vol R2\n\nbody.\n")
seedNote(t, dir, "wiki/hyperguild/decisions/routing.md",
"---\nwing: hyperguild\nhall: decisions\n---\n# Routing\n\nbody.\n")
err := brain.WriteTunnel(dir,
"wiki/jepa-fx/decisions/val-vol.md",
"wiki/hyperguild/decisions/routing.md",
)
require.NoError(t, err)
src, err := os.ReadFile(filepath.Join(dir, "wiki/jepa-fx/decisions/val-vol.md"))
require.NoError(t, err)
assert.Contains(t, string(src), "## See also")
assert.Contains(t, string(src), "[[hyperguild/decisions/routing]]")
tgt, err := os.ReadFile(filepath.Join(dir, "wiki/hyperguild/decisions/routing.md"))
require.NoError(t, err)
assert.Contains(t, string(tgt), "## See also")
assert.Contains(t, string(tgt), "[[jepa-fx/decisions/val-vol]]")
}
func TestWriteTunnel_Idempotent(t *testing.T) {
dir := t.TempDir()
seedNote(t, dir, "wiki/a/facts/x.md", "# X\n\nbody.\n")
seedNote(t, dir, "wiki/b/facts/y.md", "# Y\n\nbody.\n")
for i := 0; i < 3; i++ {
require.NoError(t, brain.WriteTunnel(dir,
"wiki/a/facts/x.md", "wiki/b/facts/y.md"))
}
src, err := os.ReadFile(filepath.Join(dir, "wiki/a/facts/x.md"))
require.NoError(t, err)
assert.Equal(t, 1, strings.Count(string(src), "[[b/facts/y]]"),
"link should appear exactly once after 3 calls")
assert.Equal(t, 1, strings.Count(string(src), "## See also"))
tgt, err := os.ReadFile(filepath.Join(dir, "wiki/b/facts/y.md"))
require.NoError(t, err)
assert.Equal(t, 1, strings.Count(string(tgt), "[[a/facts/x]]"))
assert.Equal(t, 1, strings.Count(string(tgt), "## See also"))
}
func TestWriteTunnel_RejectsSameWing(t *testing.T) {
dir := t.TempDir()
seedNote(t, dir, "wiki/jepa-fx/facts/x.md", "x")
seedNote(t, dir, "wiki/jepa-fx/facts/y.md", "y")
err := brain.WriteTunnel(dir,
"wiki/jepa-fx/facts/x.md", "wiki/jepa-fx/facts/y.md")
require.Error(t, err)
assert.Contains(t, err.Error(), "cross wings")
}
func TestWriteTunnel_RejectsMissingNote(t *testing.T) {
dir := t.TempDir()
seedNote(t, dir, "wiki/a/facts/x.md", "x")
err := brain.WriteTunnel(dir,
"wiki/a/facts/x.md", "wiki/b/facts/ghost.md")
require.Error(t, err)
}
func TestDetectTunnels_ExactTitleMatch(t *testing.T) {
dir := t.TempDir()
seedNote(t, dir, "wiki/jepa-fx/decisions/val-vol.md",
"---\nwing: jepa-fx\nhall: decisions\ntitle: Val Vol R2\n---\nbody.\n")
seedNote(t, dir, "wiki/jepa-fx/facts/lejpa.md",
"---\nwing: jepa-fx\nhall: facts\ntitle: LeJPA Architecture\n---\nbody.\n")
candidates, err := brain.DetectTunnels(dir,
"We need to revisit Val Vol R2 in light of new tier data.")
require.NoError(t, err)
require.Len(t, candidates, 1)
assert.Equal(t, "wiki/jepa-fx/decisions/val-vol.md", candidates[0].TargetPath)
assert.Equal(t, "Val Vol R2", candidates[0].MatchedTerm)
assert.True(t, candidates[0].Exact)
}
func TestDetectTunnels_FuzzyMatch(t *testing.T) {
dir := t.TempDir()
seedNote(t, dir, "wiki/x/facts/routing.md",
"---\ntitle: Routing\n---\nbody.\n")
// Substring of title appears in content, but not as a whole word.
candidates, err := brain.DetectTunnels(dir, "rerouting handles failover")
require.NoError(t, err)
require.Len(t, candidates, 1)
assert.False(t, candidates[0].Exact, "substring-only match should be fuzzy")
}
func TestDetectTunnels_NoFrontmatterFallsBackToSlug(t *testing.T) {
dir := t.TempDir()
seedNote(t, dir, "wiki/x/facts/widget-flags.md", "# widget flags\n\nbody.\n")
candidates, err := brain.DetectTunnels(dir,
"Documented Widget Flags after the deploy issue.")
require.NoError(t, err)
require.Len(t, candidates, 1)
assert.True(t, candidates[0].Exact)
assert.Equal(t, "widget flags", candidates[0].MatchedTerm)
}
func TestAutoTunnel_FuzzyGoesToCandidatesFile(t *testing.T) {
dir := t.TempDir()
// Existing note in a different wing whose title is "Routing".
seedNote(t, dir, "wiki/other/facts/routing.md",
"---\nwing: other\nhall: facts\ntitle: Routing\n---\nbody.\n")
// Source note in another wing whose body mentions "rerouting" (substring match only).
seedNote(t, dir, "wiki/jepa-fx/facts/new.md",
"---\nwing: jepa-fx\nhall: facts\n---\nrerouting traffic\n")
require.NoError(t, brain.AutoTunnel(dir,
"wiki/jepa-fx/facts/new.md", "rerouting traffic"))
// Source must not get auto-linked (fuzzy).
got, err := os.ReadFile(filepath.Join(dir, "wiki/jepa-fx/facts/new.md"))
require.NoError(t, err)
assert.NotContains(t, string(got), "[[other/facts/routing]]")
// Candidates file must list the pair.
matches, err := filepath.Glob(filepath.Join(dir, "raw", "tunnel-candidates-*.md"))
require.NoError(t, err)
require.Len(t, matches, 1)
body, err := os.ReadFile(matches[0])
require.NoError(t, err)
assert.Contains(t, string(body), "wiki/jepa-fx/facts/new.md")
assert.Contains(t, string(body), "wiki/other/facts/routing.md")
assert.Contains(t, string(body), "Routing")
}
func TestDetectTunnels_EmptyWiki(t *testing.T) {
dir := t.TempDir()
cs, err := brain.DetectTunnels(dir, "anything")
require.NoError(t, err)
assert.Empty(t, cs)
}
func TestWriteTunnel_AppendsToExistingSeeAlso(t *testing.T) {
dir := t.TempDir()
seedNote(t, dir, "wiki/a/facts/x.md",
"# X\n\nbody.\n\n## See also\n\n- [[a/facts/old]]\n")
seedNote(t, dir, "wiki/b/facts/y.md", "# Y\n\nbody.\n")
require.NoError(t, brain.WriteTunnel(dir,
"wiki/a/facts/x.md", "wiki/b/facts/y.md"))
src, err := os.ReadFile(filepath.Join(dir, "wiki/a/facts/x.md"))
require.NoError(t, err)
s := string(src)
assert.Equal(t, 1, strings.Count(s, "## See also"), "should reuse existing section")
assert.Contains(t, s, "[[a/facts/old]]")
assert.Contains(t, s, "[[b/facts/y]]")
}
+110
View File
@@ -0,0 +1,110 @@
package claudewatcher
import (
"context"
"errors"
"fmt"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
)
// CursorStore tracks how far the watcher has ingested into each
// session JSONL file. Keyed by (host, file_path) so the same `~/.claude`
// path on different hosts doesn't collide and resumability survives
// pod restarts. Idempotent Init lives alongside the rest of the
// claudewatcher schema; no separate migration framework.
type CursorStore struct {
pool *pgxpool.Pool
}
// NewCursorStore opens a pool against dsn. Caller closes the store.
func NewCursorStore(ctx context.Context, dsn string) (*CursorStore, error) {
pool, err := pgxpool.New(ctx, dsn)
if err != nil {
return nil, fmt.Errorf("pgxpool: %w", err)
}
if err := pool.Ping(ctx); err != nil {
pool.Close()
return nil, fmt.Errorf("ping: %w", err)
}
return &CursorStore{pool: pool}, nil
}
// NewCursorStoreFromPool wraps an existing pool (so the watcher can
// share the brain DSN pool with vectorstore/graphstore without a
// second connection set). Caller must NOT close the wrapped pool via
// the store — close the pool directly.
func NewCursorStoreFromPool(pool *pgxpool.Pool) *CursorStore {
return &CursorStore{pool: pool}
}
// Close releases the underlying connection pool when this store owns
// it. No-op when the pool was injected via NewCursorStoreFromPool —
// pgxpool.Close is idempotent so we lean on that.
func (s *CursorStore) Close() {
if s.pool != nil {
s.pool.Close()
}
}
// Init creates the claude_session_cursors table when missing.
func (s *CursorStore) Init(ctx context.Context) error {
const ddl = `
CREATE TABLE IF NOT EXISTS claude_session_cursors (
host TEXT NOT NULL,
file_path TEXT NOT NULL,
byte_offset BIGINT NOT NULL DEFAULT 0,
last_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (host, file_path)
);
CREATE INDEX IF NOT EXISTS claude_session_cursors_host_idx
ON claude_session_cursors (host);
`
_, err := s.pool.Exec(ctx, ddl)
return err
}
// GetOffset returns the last recorded byte offset for (host, filePath).
// Missing rows are reported as offset=0, ok=false so the caller can
// distinguish "never ingested" from "ingested at the start of the
// file" (both produce identical behaviour but the metric is useful).
func (s *CursorStore) GetOffset(ctx context.Context, host, filePath string) (int64, bool, error) {
if host == "" || filePath == "" {
return 0, false, errors.New("host and file_path are required")
}
var offset int64
err := s.pool.QueryRow(ctx, `
SELECT byte_offset FROM claude_session_cursors WHERE host = $1 AND file_path = $2
`, host, filePath).Scan(&offset)
if errors.Is(err, pgx.ErrNoRows) {
return 0, false, nil
}
if err != nil {
return 0, false, fmt.Errorf("query: %w", err)
}
return offset, true, nil
}
// SetOffset writes the new offset for (host, filePath). Used after
// every successful parse + ingest batch so a crash mid-file rewinds
// only to the last committed checkpoint.
func (s *CursorStore) SetOffset(ctx context.Context, host, filePath string, offset int64) error {
if host == "" || filePath == "" {
return errors.New("host and file_path are required")
}
if offset < 0 {
return errors.New("offset must be >= 0")
}
_, err := s.pool.Exec(ctx, `
INSERT INTO claude_session_cursors (host, file_path, byte_offset, last_seen_at)
VALUES ($1, $2, $3, now())
ON CONFLICT (host, file_path) DO UPDATE
SET byte_offset = EXCLUDED.byte_offset,
last_seen_at = now()
`, host, filePath, offset)
if err != nil {
return fmt.Errorf("upsert offset: %w", err)
}
return nil
}
+305
View File
@@ -0,0 +1,305 @@
// Package claudewatcher ingests Claude Code session transcripts
// (`~/.claude/projects/*/<uuid>.jsonl`) into the brain corpus.
//
// Schema (observed 2026-05-25 across ~30 session files on koala):
//
// type=user — user prompts + tool results
// type=assistant — model turns; tool_use blocks live in message.content
// type=attachment — hook outputs, ingested files
// type=system — turn-boundary metadata
// type=file-history-snapshot — git-style snapshot of edited files
// type=queue-operation, last-prompt, permission-mode, ai-title,
// bridge-session — internal bookkeeping, ignored
//
// The parser is intentionally tolerant: malformed lines are skipped
// (caller logs and advances), missing optional fields default to "",
// and unknown `type` values are returned as Turn entries with
// `Skip=true` so callers can filter cheaply.
package claudewatcher
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
"strings"
"time"
)
// Turn is one parsed JSONL entry from a Claude Code session log.
//
// Skip is true for entry types we never want to ingest (queue
// bookkeeping, snapshots, etc.). Callers fast-path these without
// running the scrubber or classifier.
type Turn struct {
SessionID string
Type string
ParentUUID string
Timestamp time.Time
Cwd string
GitBranch string
Content string // plain-text projection of the entry, ready for the scrubber/classifier
ToolName string // populated when an assistant turn invokes a tool
OffsetAfter int64 // byte offset in the file just past this entry
Skip bool
ParseWarning string // non-empty when the entry parsed but had a sub-field we couldn't normalise
}
// ParseStream reads JSONL lines from r starting at startOffset and
// invokes emit for each parsed entry. emit may return ErrStop to
// terminate the scan cleanly. Other emit errors propagate.
//
// startOffset is informational — the caller is expected to have already
// seeked the underlying reader to that offset. ParseStream adds the
// number of bytes consumed per line to it to compute Turn.OffsetAfter.
//
// Lines that fail to unmarshal are logged via warnf and skipped; they
// do NOT advance OffsetAfter past the malformed line by themselves,
// but the next valid line resumes correctly because bufio.Scanner
// preserves stream position.
func ParseStream(
r io.Reader,
startOffset int64,
warnf func(format string, args ...any),
emit func(Turn) error,
) (int64, error) {
scanner := bufio.NewScanner(r)
scanner.Buffer(make([]byte, 0, 64*1024), 8*1024*1024) // some lines are big (tool outputs)
offset := startOffset
for scanner.Scan() {
raw := scanner.Bytes()
lineLen := int64(len(raw)) + 1 // +1 for the newline
t, err := parseTurn(raw)
if err != nil {
if warnf != nil {
warnf("parse: %v (%d bytes)", err, len(raw))
}
offset += lineLen
continue
}
t.OffsetAfter = offset + lineLen
if err := emit(t); err != nil {
if errors.Is(err, ErrStop) {
return t.OffsetAfter, nil
}
return offset, fmt.Errorf("emit: %w", err)
}
offset = t.OffsetAfter
}
if err := scanner.Err(); err != nil {
return offset, fmt.Errorf("scan: %w", err)
}
return offset, nil
}
// ErrStop terminates a ParseStream loop without surfacing an error.
var ErrStop = errors.New("claudewatcher: stop")
// rawEntry is a permissive shape that covers every type observed in
// the JSONL files. Fields we don't care about are intentionally
// omitted to keep the unmarshal cheap.
type rawEntry struct {
Type string `json:"type"`
SessionID string `json:"sessionId"`
ParentUUID string `json:"parentUuid"`
Timestamp string `json:"timestamp"`
Cwd string `json:"cwd"`
GitBranch string `json:"gitBranch"`
Message json.RawMessage `json:"message"`
Attachment json.RawMessage `json:"attachment"`
Content string `json:"content"` // queue-operation
LastPrompt string `json:"lastPrompt"` // last-prompt
Subtype string `json:"subtype"` // system
}
// skipTypes lists every entry type we want to never ingest. Marked Skip
// at parse time so the caller's filter is a single boolean check.
var skipTypes = map[string]struct{}{
"queue-operation": {},
"last-prompt": {},
"permission-mode": {},
"ai-title": {},
"bridge-session": {},
"file-history-snapshot": {},
}
func parseTurn(raw []byte) (Turn, error) {
var e rawEntry
if err := json.Unmarshal(raw, &e); err != nil {
return Turn{}, fmt.Errorf("unmarshal: %w", err)
}
t := Turn{
Type: e.Type,
SessionID: e.SessionID,
ParentUUID: e.ParentUUID,
Cwd: e.Cwd,
GitBranch: e.GitBranch,
}
if _, skip := skipTypes[e.Type]; skip {
t.Skip = true
return t, nil
}
if e.Timestamp != "" {
if ts, err := time.Parse(time.RFC3339Nano, e.Timestamp); err == nil {
t.Timestamp = ts
} else {
t.ParseWarning = "timestamp"
}
}
switch e.Type {
case "user":
t.Content = extractMessageText(e.Message)
case "assistant":
t.Content, t.ToolName = extractAssistantTurn(e.Message)
case "attachment":
t.Content = extractAttachmentText(e.Attachment)
case "system":
t.Content = "[system " + e.Subtype + "]"
default:
// Unknown type — keep the row but mark Skip so callers ignore.
t.Skip = true
}
return t, nil
}
// extractMessageText pulls the textual projection out of a user/assistant
// message field. The shape is the Anthropic Messages API content-block
// array (an array of {type, text|tool_use|tool_result, ...}). We
// concatenate every text-bearing block and ignore the rest.
func extractMessageText(raw json.RawMessage) string {
if len(raw) == 0 {
return ""
}
var msg struct {
Role string `json:"role"`
Content json.RawMessage `json:"content"`
Stop string `json:"stop_reason"`
Model string `json:"model"`
Usage map[string]any `json:"usage"`
Meta map[string]string `json:"meta"`
}
if err := json.Unmarshal(raw, &msg); err != nil {
// Some user turns have message as plain string.
var s string
if err2 := json.Unmarshal(raw, &s); err2 == nil {
return s
}
return ""
}
// Content can be a string OR an array.
var asString string
if err := json.Unmarshal(msg.Content, &asString); err == nil {
return asString
}
var blocks []struct {
Type string `json:"type"`
Text string `json:"text"`
Content json.RawMessage `json:"content"`
}
if err := json.Unmarshal(msg.Content, &blocks); err != nil {
return ""
}
var sb strings.Builder
for _, b := range blocks {
switch b.Type {
case "text":
sb.WriteString(b.Text)
sb.WriteByte('\n')
case "tool_result":
// Tool result content may itself be a string or array of blocks.
var s string
if err := json.Unmarshal(b.Content, &s); err == nil {
sb.WriteString("[tool_result] ")
sb.WriteString(s)
sb.WriteByte('\n')
continue
}
var sub []struct {
Type string `json:"type"`
Text string `json:"text"`
}
if err := json.Unmarshal(b.Content, &sub); err == nil {
for _, s := range sub {
if s.Type == "text" {
sb.WriteString("[tool_result] ")
sb.WriteString(s.Text)
sb.WriteByte('\n')
}
}
}
}
}
return strings.TrimRight(sb.String(), "\n")
}
// extractAssistantTurn pulls text + the first tool name (if any) from
// an assistant content-block array. Multi-tool turns lose the second
// name; the goal is signal for classification, not perfect fidelity.
func extractAssistantTurn(raw json.RawMessage) (string, string) {
if len(raw) == 0 {
return "", ""
}
var msg struct {
Content json.RawMessage `json:"content"`
}
if err := json.Unmarshal(raw, &msg); err != nil {
return "", ""
}
var blocks []struct {
Type string `json:"type"`
Text string `json:"text"`
Name string `json:"name"`
Tool json.RawMessage `json:"input"`
}
if err := json.Unmarshal(msg.Content, &blocks); err != nil {
return "", ""
}
var sb strings.Builder
var firstTool string
for _, b := range blocks {
switch b.Type {
case "text":
sb.WriteString(b.Text)
sb.WriteByte('\n')
case "tool_use":
if firstTool == "" {
firstTool = b.Name
}
sb.WriteString("[tool_use:")
sb.WriteString(b.Name)
sb.WriteString("]\n")
}
}
return strings.TrimRight(sb.String(), "\n"), firstTool
}
// extractAttachmentText pulls text content from an attachment payload,
// or returns a short tag when the attachment is a hook event.
func extractAttachmentText(raw json.RawMessage) string {
if len(raw) == 0 {
return ""
}
var a struct {
Type string `json:"type"`
HookName string `json:"hookName"`
HookEvent string `json:"hookEvent"`
Content string `json:"content"`
Text string `json:"text"`
}
if err := json.Unmarshal(raw, &a); err != nil {
return ""
}
if a.Content != "" {
return a.Content
}
if a.Text != "" {
return a.Text
}
if a.HookName != "" {
return "[hook " + a.HookEvent + ":" + a.HookName + "]"
}
return ""
}
@@ -0,0 +1,157 @@
package claudewatcher
import (
"errors"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func collect(t *testing.T, body string) ([]Turn, int64, error) {
t.Helper()
var out []Turn
end, err := ParseStream(strings.NewReader(body), 0, nil, func(tr Turn) error {
out = append(out, tr)
return nil
})
return out, end, err
}
func TestParseStream_UserTurnStringContent(t *testing.T) {
body := `{"type":"user","sessionId":"S","timestamp":"2026-05-25T07:00:00Z","message":"hello world"}
`
turns, end, err := collect(t, body)
require.NoError(t, err)
require.Len(t, turns, 1)
assert.Equal(t, "user", turns[0].Type)
assert.Equal(t, "S", turns[0].SessionID)
assert.Equal(t, "hello world", turns[0].Content)
assert.False(t, turns[0].Skip)
assert.Equal(t, int64(len(body)), end)
}
func TestParseStream_UserTurnContentBlocks(t *testing.T) {
body := `{"type":"user","sessionId":"S","timestamp":"2026-05-25T07:00:00Z","message":{"role":"user","content":[{"type":"text","text":"line 1"},{"type":"text","text":"line 2"}]}}
`
turns, _, err := collect(t, body)
require.NoError(t, err)
require.Len(t, turns, 1)
assert.Equal(t, "line 1\nline 2", turns[0].Content)
}
func TestParseStream_AssistantToolUse(t *testing.T) {
body := `{"type":"assistant","sessionId":"S","timestamp":"2026-05-25T07:00:00Z","message":{"content":[{"type":"text","text":"calling now"},{"type":"tool_use","name":"Edit","input":{}}]}}
`
turns, _, err := collect(t, body)
require.NoError(t, err)
require.Len(t, turns, 1)
assert.Equal(t, "Edit", turns[0].ToolName)
assert.Contains(t, turns[0].Content, "calling now")
assert.Contains(t, turns[0].Content, "[tool_use:Edit]")
}
func TestParseStream_AssistantToolResult(t *testing.T) {
body := `{"type":"user","sessionId":"S","timestamp":"2026-05-25T07:00:00Z","message":{"content":[{"type":"tool_result","content":"output of cmd"}]}}
`
turns, _, err := collect(t, body)
require.NoError(t, err)
require.Len(t, turns, 1)
assert.Contains(t, turns[0].Content, "[tool_result] output of cmd")
}
func TestParseStream_SkipsBookkeepingTypes(t *testing.T) {
body := strings.Join([]string{
`{"type":"queue-operation","sessionId":"S","content":"x"}`,
`{"type":"last-prompt","sessionId":"S","lastPrompt":"y"}`,
`{"type":"permission-mode","sessionId":"S","permissionMode":"auto"}`,
`{"type":"ai-title","sessionId":"S","aiTitle":"My session"}`,
`{"type":"file-history-snapshot","messageId":"abc"}`,
}, "\n") + "\n"
turns, _, err := collect(t, body)
require.NoError(t, err)
require.Len(t, turns, 5)
for _, tr := range turns {
assert.True(t, tr.Skip, "expected Skip=true for %q", tr.Type)
}
}
func TestParseStream_UnknownTypeIsSkip(t *testing.T) {
body := `{"type":"future-thing","sessionId":"S"}` + "\n"
turns, _, err := collect(t, body)
require.NoError(t, err)
require.Len(t, turns, 1)
assert.True(t, turns[0].Skip)
}
func TestParseStream_MalformedLineIsSkippedNotFatal(t *testing.T) {
body := strings.Join([]string{
`{"type":"user","sessionId":"S","message":"first"}`,
`{not valid json`,
`{"type":"user","sessionId":"S","message":"third"}`,
}, "\n") + "\n"
var warnings int
var turns []Turn
_, err := ParseStream(strings.NewReader(body), 0, func(format string, args ...any) {
warnings++
}, func(tr Turn) error {
turns = append(turns, tr)
return nil
})
require.NoError(t, err)
require.Len(t, turns, 2, "first + third should make it through")
assert.Equal(t, 1, warnings)
}
func TestParseStream_EmitErrStopHaltsCleanly(t *testing.T) {
body := strings.Join([]string{
`{"type":"user","sessionId":"S","message":"a"}`,
`{"type":"user","sessionId":"S","message":"b"}`,
`{"type":"user","sessionId":"S","message":"c"}`,
}, "\n") + "\n"
count := 0
end, err := ParseStream(strings.NewReader(body), 0, nil, func(tr Turn) error {
count++
if count == 2 {
return ErrStop
}
return nil
})
require.NoError(t, err)
assert.Equal(t, 2, count)
assert.Greater(t, end, int64(0))
}
func TestParseStream_EmitOtherErrorPropagates(t *testing.T) {
body := `{"type":"user","sessionId":"S","message":"a"}` + "\n"
want := errors.New("boom")
_, err := ParseStream(strings.NewReader(body), 0, nil, func(tr Turn) error {
return want
})
require.Error(t, err)
assert.Contains(t, err.Error(), "boom")
}
func TestParseStream_AttachmentHookEvent(t *testing.T) {
body := `{"type":"attachment","sessionId":"S","timestamp":"2026-05-25T07:00:00Z","attachment":{"type":"hook_success","hookName":"SessionStart:startup","hookEvent":"SessionStart","content":"hook body"}}
`
turns, _, err := collect(t, body)
require.NoError(t, err)
require.Len(t, turns, 1)
assert.Equal(t, "hook body", turns[0].Content)
}
func TestParseStream_OffsetAdvances(t *testing.T) {
body := `{"type":"user","sessionId":"S","message":"a"}` + "\n" +
`{"type":"user","sessionId":"S","message":"b"}` + "\n"
var offsets []int64
_, err := ParseStream(strings.NewReader(body), 100, nil, func(tr Turn) error {
offsets = append(offsets, tr.OffsetAfter)
return nil
})
require.NoError(t, err)
require.Len(t, offsets, 2)
assert.Greater(t, offsets[0], int64(100))
assert.Greater(t, offsets[1], offsets[0])
}
@@ -0,0 +1,126 @@
package claudewatcher
import (
"fmt"
"regexp"
"sync"
)
// Scrubber drops any turn whose content matches a known-bad pattern.
// Fail-closed by design: we'd rather lose signal than ingest credentials
// into a public-readable brain. The caller logs the drop reason.
//
// Rules cover the credential shapes most common to leak through Claude
// Code sessions: bearer tokens, postgres URIs with embedded auth, OAuth
// secret values, SOPS-encrypted secret blobs (we don't want the
// ciphertext either — it's a marker that the original message contained
// secret state), PEM-encoded private keys, and the explicit env-var
// naming conventions used in the homelab.
//
// Pattern philosophy: match by shape, not by content. A 40-char hex
// string in isolation is fine; the same string after `Authorization:
// Bearer ` is not. Tuned to catch known leak vectors from prior
// secret-hygiene incidents (POSTGRES_PASSWORD via kubectl exec env,
// INFRA_MCP_TOKEN via sops -d output) without dropping every Edit on a
// config file.
// Rule is a single named regex with a redact hint shown in the warn log.
type Rule struct {
Name string
RE *regexp.Regexp
}
// DefaultRules is the regex set applied by Scrub. Mutable for tests but
// callers should treat it as read-only at runtime.
var DefaultRules = []Rule{
// authorization-header is checked before the bare bearer rule so
// contextual hits ("Authorization: Bearer X") report the more
// specific match name in logs.
{Name: "authorization-header", RE: regexp.MustCompile(`(?i)Authorization\s*:\s*[A-Za-z]+\s+\S{8,}`)},
{Name: "bearer-token", RE: regexp.MustCompile(`(?i)Bearer\s+[A-Za-z0-9._\-]{16,}`)},
// JWT (header.payload.sig), e.g. a Dex/OAuth token dumped to stdout
// without a "Bearer " prefix. Both header and payload base64url-encode
// JSON, so both segments begin with "eyJ".
{Name: "jwt", RE: regexp.MustCompile(`eyJ[A-Za-z0-9_\-]{8,}\.eyJ[A-Za-z0-9_\-]{8,}\.[A-Za-z0-9_\-]{8,}`)},
{Name: "postgres-uri-with-password", RE: regexp.MustCompile(`postgres(?:ql)?://[^:\s/]+:[^@\s/]+@`)},
{Name: "private-key", RE: regexp.MustCompile(`-----BEGIN[^-]*PRIVATE KEY-----`)},
{Name: "ssh-key", RE: regexp.MustCompile(`ssh-(?:rsa|ed25519|ecdsa)\s+[A-Za-z0-9+/=]{40,}`)},
{Name: "github-pat", RE: regexp.MustCompile(`\b(?:ghp|gho|ghu|ghr|gha)_[A-Za-z0-9]{30,}\b`)},
// 1Password service-account token (ops_<base64url>). Long, high-value root
// credential; guard the bare value (the _TOKEN= form also hits homelab-env-token).
{Name: "op-service-account", RE: regexp.MustCompile(`\bops_[A-Za-z0-9_\-]{40,}`)},
// No leading \b: a shell mangle can glue the key to a preceding word
// ("yes"+"sk-...") which has no word boundary, and that exact case
// leaked a LiteLLM master key past this rule (2026-06-11). Match the
// sk- shape wherever it appears; the {32,} length floor keeps short
// "task-"/"disk-" words from tripping it.
{Name: "openai-sk", RE: regexp.MustCompile(`sk-(?:proj-)?[A-Za-z0-9]{32,}`)},
{Name: "anthropic-sk", RE: regexp.MustCompile(`\bsk-ant-[A-Za-z0-9_\-]{32,}\b`)},
{Name: "aws-access-key", RE: regexp.MustCompile(`\bAKIA[0-9A-Z]{16}\b`)},
{Name: "homelab-env-token", RE: regexp.MustCompile(`(?i)(?:_TOKEN|_PASSWORD|_API_KEY|_SECRET)\s*[:=]\s*['"]?[A-Za-z0-9._/+\-]{12,}`)},
{Name: "sops-encrypted-marker", RE: regexp.MustCompile(`ENC\[AES256_GCM,data:[A-Za-z0-9+/=]{8,}`)},
}
// extraRules is appended to DefaultRules at process startup via
// RegisterRule. The mutex guards concurrent RegisterRule calls (rare)
// against concurrent Scrub reads (hot path). Scrub takes a read lock
// only when extraRules is non-empty, so steady-state cost is zero
// when no client-name guard is configured.
var (
extraRulesMu sync.RWMutex
extraRules []Rule
)
// RegisterRule appends a runtime-configured regex to the scrubber's
// rule set. Used by main to inject client-name guards from
// CLAUDE_INGEST_CLIENT_BLOCK env var (or equivalent SOPS-encrypted
// secret) without baking client identities into source code.
//
// pattern is compiled as-is — callers wrap with `\b...\b` and case
// flags as needed. Duplicate names are accepted (rules are positional);
// the second registration just fires after the first.
func RegisterRule(name, pattern string) error {
re, err := regexp.Compile(pattern)
if err != nil {
return fmt.Errorf("compile rule %q: %w", name, err)
}
extraRulesMu.Lock()
extraRules = append(extraRules, Rule{Name: name, RE: re})
extraRulesMu.Unlock()
return nil
}
// ResetExtraRules clears every RegisterRule-added rule. Test-only.
func ResetExtraRules() {
extraRulesMu.Lock()
extraRules = nil
extraRulesMu.Unlock()
}
// Scrub reports the first matching rule, or empty when content is clean.
// Empty string is treated as clean. Caller decides what to do on a hit;
// the convention in claudewatcher is to drop the turn entirely and emit
// a slog.Warn naming the rule.
//
// Rule order: DefaultRules first (credential shapes), then runtime
// RegisterRule additions (client-name guards). Credential leaks
// outrank client-name hits in the log because they're strictly more
// dangerous.
func Scrub(content string) string {
if content == "" {
return ""
}
for _, r := range DefaultRules {
if r.RE.MatchString(content) {
return r.Name
}
}
extraRulesMu.RLock()
defer extraRulesMu.RUnlock()
for _, r := range extraRules {
if r.RE.MatchString(content) {
return r.Name
}
}
return ""
}
@@ -0,0 +1,132 @@
package claudewatcher
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestScrub_PoisonedFixtures(t *testing.T) {
// One representative bad-string per rule. If a rule fires for the
// wrong content shape later, this table localises the regression.
cases := []struct {
name string
content string
want string
}{
{"bearer-token", "curl -H 'Authorization: Bearer abcdef1234567890ghijklmnop'", "authorization-header"},
{"bearer-no-header", "header = Bearer eyJhbGciOiJIUzI1NiJ9.payload.sig", "bearer-token"},
{"postgres-uri", "DATABASE_URL=postgres://user:s3cret@10.0.1.20:5432/brain", "postgres-uri-with-password"},
{"private-key", "-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNzaC1rZXktdjEAAAAA", "private-key"},
{"ssh-public", "deploy: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK1234567890abcdefghij user@host", "ssh-key"},
{"github-pat-classic", "GH_TOKEN=ghp_aBcD1234EfGh5678IjKl9012MnOp3456QrSt", "github-pat"},
{"openai-key", "OPENAI_API_KEY=sk-proj-AAAABBBBCCCCDDDDEEEEFFFFGGGGHHHHIIII", "openai-sk"},
{"anthropic-key", "ANTHROPIC_API_KEY=sk-ant-api03-aaaaBBBBccccDDDDeeeeFFFFggggHHHHiiiiJJJJkkkk", "anthropic-sk"},
{"aws-access-key", "AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE", "aws-access-key"},
{"homelab-env", "POSTGRES_PASSWORD=hunter2supersecretvalue", "homelab-env-token"},
{"sops-marker", "value: ENC[AES256_GCM,data:abc123def456,iv:zzz]", "sops-encrypted-marker"},
// Regression: a shell mangle glued the key to a preceding word
// ("yes"+"sk-..."), defeating the leading \b in the sk- rule and
// leaking a LiteLLM master key past the scrubber (2026-06-11).
{"sk-glued-to-word", "master key resolved: yessk-7181ca984603239d8c4819361bf33b94b9c3c07018791868", "openai-sk"},
{"sk-standalone-hex", "sk-7181ca984603239d8c4819361bf33b94b9c3c07018791868", "openai-sk"},
// Bare JWT not preceded by "Bearer" (e.g. a Dex token dumped to stdout).
{"jwt-bare", "token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0.dQw4w9WgXcQabcdef", "jwt"},
// 1Password service-account token (ops_<base64url>), env-assigned and bare.
// Both hit the dedicated op-service-account rule (ordered before the
// generic homelab-env-token). Guards ~/.zshrc reads etc. (2026-06-14).
{"op-sa-env", "export OP_SERVICE_ACCOUNT_TOKEN=ops_eyJzaWduSW5BZGRyZXNzIjoibXkuMXBhc3N3b3JkLmNvbSJ9", "op-service-account"},
{"op-sa-bare", "ops_eyJzaWduSW5BZGRyZXNzIjoibXkuMXBhc3N3b3JkLmNvbSIsInVzZXJBdXRoIjp7fX0aGVsbG8", "op-service-account"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := Scrub(tc.content)
assert.Equal(t, tc.want, got)
})
}
}
func TestScrub_CleanContentPassesThrough(t *testing.T) {
cases := []string{
"",
"plain text with no credentials",
"a 40 char hex string aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa is fine in isolation",
"`Bearer` token mentioned in docs without an actual value",
"file at ~/.ssh/id_ed25519",
"the function Authorization() takes no args",
"comment: see API key in 1Password",
// loosened sk- rule must not trip on short "task-"/"disk-" words
"run task-build then task-test in the pipeline",
"mounted /dev/disk-by-id/wwn-0x5000",
}
for _, c := range cases {
assert.Empty(t, Scrub(c), "expected clean for %q", c)
}
}
func TestScrub_FirstMatchWins(t *testing.T) {
// Content matching multiple rules: report the first rule order in
// DefaultRules. Stability matters for log triage.
content := "Authorization: Bearer ghp_aBcD1234EfGh5678IjKl9012MnOp3456QrSt"
assert.Equal(t, "authorization-header", Scrub(content))
}
func TestRegisterRule_ClientNameGuard(t *testing.T) {
t.Cleanup(ResetExtraRules)
require := func(err error) {
if err != nil {
t.Fatalf("unexpected err: %v", err)
}
}
require(RegisterRule("client-name", `(?i)\b(SEB|Mastercard)\b`))
// Hits — case variations + word-boundary respect.
for _, hit := range []string{
"mentioned SEB in this commit",
"the Mastercard project deadline",
"working on mastercard scope",
"SEB internal review",
} {
assert.Equal(t, "client-name", Scrub(hit), "should match %q", hit)
}
// Misses — substring within a longer word should NOT match
// thanks to \b. "Sebastian" contains "seb" but \b prevents hit.
for _, miss := range []string{
"Sebastian wrote the docs",
"unrelated text",
"researcher",
"https://example.com/search?seb=1", // 'seb' bounded by ?=, still matches \b
} {
got := Scrub(miss)
if miss == "https://example.com/search?seb=1" {
// `seb=` has word-boundary at '='; this DOES match \bseb\b.
// Accept either outcome; document the tradeoff.
assert.Contains(t, []string{"", "client-name"}, got)
continue
}
assert.Empty(t, got, "should NOT match %q", miss)
}
}
func TestRegisterRule_CredentialsTakePrecedence(t *testing.T) {
t.Cleanup(ResetExtraRules)
require := func(err error) {
if err != nil {
t.Fatalf("unexpected err: %v", err)
}
}
require(RegisterRule("client-name", `\b(SEB)\b`))
// Content matches both a credential rule AND a client rule —
// credential rule wins by ordering, so log triage points at the
// strictly more dangerous leak.
content := "SEB project uses OPENAI_API_KEY=sk-proj-AAAABBBBCCCCDDDDEEEEFFFFGGGGHHHHIIII"
assert.Equal(t, "openai-sk", Scrub(content))
}
func TestRegisterRule_RejectsInvalidPattern(t *testing.T) {
t.Cleanup(ResetExtraRules)
err := RegisterRule("bad", "[unclosed")
assert.Error(t, err)
}
+234
View File
@@ -0,0 +1,234 @@
package claudewatcher
import (
"context"
"fmt"
"log/slog"
"os"
"path/filepath"
"strings"
"time"
)
// Sink consumes batches of ingest-ready turns from the watcher. The
// production implementation builds wiki pages and calls pipeline.RunRaw
// against the brain. Tests substitute a counter.
//
// A Batch represents the turns ingested from one session file between
// two cursor checkpoints. Implementations must be idempotent — the
// watcher only advances the cursor on a nil return.
type Sink interface {
Ingest(ctx context.Context, b Batch) error
}
// Batch is a per-file slice of turns plus identifying metadata.
type Batch struct {
Host string // origin host, e.g. "koala"
FilePath string // absolute path to the source .jsonl file
SessionID string // first session_id seen in the batch
ProjectID string // basename of the parent dir, e.g. "-home-mathias-dev"
Turns []Turn // never empty; caller filters Skip + scrubber matches
}
// Config drives one Watch loop. SessionsDir is the absolute path to the
// Claude Code projects directory (~/.claude/projects). Host is the
// label written into cursors and ingested page frontmatter. Interval
// is the poll cadence; a zero or negative value disables the loop.
//
// Sink is required. Cursors is optional — when nil the watcher
// re-reads from byte 0 on every tick (useful for first-run testing
// without a postgres dependency).
type Config struct {
SessionsDir string
Host string
Interval time.Duration
Sink Sink
Cursors *CursorStore
Logger *slog.Logger
}
// Watch runs the polling loop until ctx is cancelled. Returns ctx.Err()
// on shutdown. Each tick walks SessionsDir for *.jsonl files, advances
// each file's cursor, and emits one Batch per file with new turns.
// Errors during a single file's parse or ingest are logged but do not
// abort the loop — a single bad file shouldn't block the others.
func Watch(ctx context.Context, cfg Config) error {
if cfg.SessionsDir == "" {
return fmt.Errorf("sessions dir is required")
}
if cfg.Sink == nil {
return fmt.Errorf("sink is required")
}
if cfg.Interval <= 0 {
return fmt.Errorf("interval must be positive")
}
if cfg.Host == "" {
cfg.Host = "unknown"
}
if cfg.Logger == nil {
cfg.Logger = slog.Default()
}
cfg.Logger.Info("claudewatcher: started",
"sessions_dir", cfg.SessionsDir,
"host", cfg.Host,
"interval", cfg.Interval)
ticker := time.NewTicker(cfg.Interval)
defer ticker.Stop()
// Run an immediate first sweep so first-launch users don't wait one
// tick before anything happens.
runTick(ctx, cfg)
for {
select {
case <-ctx.Done():
return ctx.Err()
case <-ticker.C:
runTick(ctx, cfg)
}
}
}
// runTick is one polling pass. Exposed (lowercase) for tests via
// TickOnce.
func runTick(ctx context.Context, cfg Config) {
files, err := listSessionFiles(cfg.SessionsDir)
if err != nil {
cfg.Logger.Warn("claudewatcher: list session files", "err", err)
return
}
for _, f := range files {
if ctx.Err() != nil {
return
}
if err := processFile(ctx, cfg, f); err != nil {
cfg.Logger.Warn("claudewatcher: file failed",
"path", f, "err", err)
}
}
}
// TickOnce runs one sweep synchronously and returns. Used by tests +
// by ad-hoc CLI invocations.
func TickOnce(ctx context.Context, cfg Config) error {
if cfg.SessionsDir == "" || cfg.Sink == nil {
return fmt.Errorf("config invalid")
}
if cfg.Host == "" {
cfg.Host = "unknown"
}
if cfg.Logger == nil {
cfg.Logger = slog.Default()
}
runTick(ctx, cfg)
return nil
}
func listSessionFiles(root string) ([]string, error) {
var out []string
err := filepath.WalkDir(root, func(path string, d os.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if d.IsDir() {
return nil
}
if !strings.HasSuffix(path, ".jsonl") {
return nil
}
out = append(out, path)
return nil
})
if err != nil {
return nil, fmt.Errorf("walk %s: %w", root, err)
}
return out, nil
}
func processFile(ctx context.Context, cfg Config, path string) error {
startOffset := int64(0)
if cfg.Cursors != nil {
off, _, err := cfg.Cursors.GetOffset(ctx, cfg.Host, path)
if err != nil {
return fmt.Errorf("get cursor: %w", err)
}
startOffset = off
}
stat, err := os.Stat(path)
if err != nil {
return fmt.Errorf("stat: %w", err)
}
if stat.Size() <= startOffset {
return nil // nothing new
}
f, err := os.Open(path)
if err != nil {
return fmt.Errorf("open: %w", err)
}
defer func() { _ = f.Close() }()
if _, err := f.Seek(startOffset, 0); err != nil {
return fmt.Errorf("seek: %w", err)
}
var keep []Turn
var sessionID string
var droppedScrub int
endOffset, err := ParseStream(f, startOffset,
func(format string, args ...any) {
cfg.Logger.Warn(fmt.Sprintf("claudewatcher: parse: "+format, args...))
},
func(t Turn) error {
if t.Skip || t.Content == "" {
return nil
}
if rule := Scrub(t.Content); rule != "" {
droppedScrub++
cfg.Logger.Warn("claudewatcher: turn dropped by scrubber",
"rule", rule, "path", path, "session_id", t.SessionID)
return nil
}
if sessionID == "" {
sessionID = t.SessionID
}
keep = append(keep, t)
return nil
})
if err != nil {
return fmt.Errorf("parse stream: %w", err)
}
if len(keep) == 0 {
if cfg.Cursors != nil {
if err := cfg.Cursors.SetOffset(ctx, cfg.Host, path, endOffset); err != nil {
return fmt.Errorf("advance cursor (no-turns): %w", err)
}
}
if droppedScrub > 0 {
cfg.Logger.Info("claudewatcher: only scrubbed turns this tick",
"path", path, "dropped", droppedScrub)
}
return nil
}
batch := Batch{
Host: cfg.Host,
FilePath: path,
SessionID: sessionID,
ProjectID: filepath.Base(filepath.Dir(path)),
Turns: keep,
}
if err := cfg.Sink.Ingest(ctx, batch); err != nil {
return fmt.Errorf("sink ingest: %w", err)
}
if cfg.Cursors != nil {
if err := cfg.Cursors.SetOffset(ctx, cfg.Host, path, endOffset); err != nil {
return fmt.Errorf("advance cursor: %w", err)
}
}
cfg.Logger.Info("claudewatcher: ingested batch",
"path", path, "session_id", sessionID,
"turns_kept", len(keep), "dropped_scrub", droppedScrub,
"new_offset", endOffset)
return nil
}
@@ -0,0 +1,174 @@
package claudewatcher
import (
"context"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// memSink captures batches without touching postgres. Thread-safe so
// TickOnce can run from any goroutine in concurrent tests.
type memSink struct {
mu sync.Mutex
batches []Batch
failOn string // file basename to error on
}
func (m *memSink) Ingest(_ context.Context, b Batch) error {
m.mu.Lock()
defer m.mu.Unlock()
if m.failOn != "" && strings.Contains(b.FilePath, m.failOn) {
return assert.AnError
}
m.batches = append(m.batches, b)
return nil
}
func writeSession(t *testing.T, dir, sessionID string, lines []string) string {
t.Helper()
path := filepath.Join(dir, sessionID+".jsonl")
body := strings.Join(lines, "\n") + "\n"
require.NoError(t, os.WriteFile(path, []byte(body), 0o644))
return path
}
func TestTickOnce_NoCursorReingestsEverythingEveryTick(t *testing.T) {
tmp := t.TempDir()
projectDir := filepath.Join(tmp, "-home-mathias-dev")
require.NoError(t, os.MkdirAll(projectDir, 0o755))
writeSession(t, projectDir, "sess1", []string{
`{"type":"user","sessionId":"sess1","message":"first prompt"}`,
`{"type":"assistant","sessionId":"sess1","message":{"content":[{"type":"text","text":"first answer"}]}}`,
})
sink := &memSink{}
cfg := Config{
SessionsDir: tmp,
Host: "koala",
Sink: sink,
}
require.NoError(t, TickOnce(context.Background(), cfg))
require.NoError(t, TickOnce(context.Background(), cfg))
require.Len(t, sink.batches, 2, "no cursor => re-emits same batch every tick")
assert.Equal(t, "sess1", sink.batches[0].SessionID)
assert.Equal(t, "koala", sink.batches[0].Host)
assert.Equal(t, "-home-mathias-dev", sink.batches[0].ProjectID)
assert.Len(t, sink.batches[0].Turns, 2)
}
func TestTickOnce_FiltersSkipTurnsAndScrubberMatches(t *testing.T) {
tmp := t.TempDir()
proj := filepath.Join(tmp, "-home-mathias-dev")
require.NoError(t, os.MkdirAll(proj, 0o755))
writeSession(t, proj, "sess-scrub", []string{
`{"type":"queue-operation","sessionId":"sess-scrub","content":"x"}`, // Skip
`{"type":"user","sessionId":"sess-scrub","message":"normal prompt"}`,
`{"type":"assistant","sessionId":"sess-scrub","message":{"content":[{"type":"text","text":"value POSTGRES_PASSWORD=hunter2supersecretvalue"}]}}`, // scrubbed
})
sink := &memSink{}
require.NoError(t, TickOnce(context.Background(), Config{
SessionsDir: tmp, Host: "koala", Sink: sink,
}))
require.Len(t, sink.batches, 1)
turns := sink.batches[0].Turns
require.Len(t, turns, 1, "skip + scrubbed turns must not reach the sink")
assert.Equal(t, "user", turns[0].Type)
}
func TestTickOnce_AllScrubbedNoBatchEmitted(t *testing.T) {
tmp := t.TempDir()
proj := filepath.Join(tmp, "-home-mathias-dev")
require.NoError(t, os.MkdirAll(proj, 0o755))
writeSession(t, proj, "all-bad", []string{
`{"type":"user","sessionId":"all-bad","message":"Authorization: Bearer abcdef1234567890ghijklmnop"}`,
})
sink := &memSink{}
require.NoError(t, TickOnce(context.Background(), Config{
SessionsDir: tmp, Host: "koala", Sink: sink,
}))
assert.Empty(t, sink.batches, "no usable turns => no batch")
}
func TestTickOnce_IgnoresNonJsonlFiles(t *testing.T) {
tmp := t.TempDir()
proj := filepath.Join(tmp, "-home-mathias-dev")
require.NoError(t, os.MkdirAll(proj, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(proj, "README.md"), []byte("ignore me"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(proj, "config.json"), []byte("{}"), 0o644))
sink := &memSink{}
require.NoError(t, TickOnce(context.Background(), Config{
SessionsDir: tmp, Host: "koala", Sink: sink,
}))
assert.Empty(t, sink.batches)
}
func TestTickOnce_HandlesMultipleProjectsAndSessions(t *testing.T) {
tmp := t.TempDir()
projA := filepath.Join(tmp, "-home-mathias-dev")
projB := filepath.Join(tmp, "-home-mathias-AI-infra")
require.NoError(t, os.MkdirAll(projA, 0o755))
require.NoError(t, os.MkdirAll(projB, 0o755))
writeSession(t, projA, "a1", []string{`{"type":"user","sessionId":"a1","message":"q1"}`})
writeSession(t, projA, "a2", []string{`{"type":"user","sessionId":"a2","message":"q2"}`})
writeSession(t, projB, "b1", []string{`{"type":"user","sessionId":"b1","message":"q3"}`})
sink := &memSink{}
require.NoError(t, TickOnce(context.Background(), Config{
SessionsDir: tmp, Host: "koala", Sink: sink,
}))
require.Len(t, sink.batches, 3)
projects := map[string]int{}
for _, b := range sink.batches {
projects[b.ProjectID]++
}
assert.Equal(t, 2, projects["-home-mathias-dev"])
assert.Equal(t, 1, projects["-home-mathias-AI-infra"])
}
func TestTickOnce_SinkErrorDoesNotKillOtherFiles(t *testing.T) {
tmp := t.TempDir()
proj := filepath.Join(tmp, "-home-mathias-dev")
require.NoError(t, os.MkdirAll(proj, 0o755))
writeSession(t, proj, "good", []string{`{"type":"user","sessionId":"good","message":"q"}`})
writeSession(t, proj, "bad-session", []string{`{"type":"user","sessionId":"bad-session","message":"q"}`})
sink := &memSink{failOn: "bad-session"}
require.NoError(t, TickOnce(context.Background(), Config{
SessionsDir: tmp, Host: "koala", Sink: sink,
}))
require.Len(t, sink.batches, 1, "good session still ingested")
assert.Equal(t, "good", sink.batches[0].SessionID)
}
func TestWatch_RespectsContextCancel(t *testing.T) {
tmp := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(tmp, "-home-mathias-dev"), 0o755))
sink := &memSink{}
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() {
done <- Watch(ctx, Config{
SessionsDir: tmp,
Host: "koala",
Interval: 10 * time.Millisecond,
Sink: sink,
})
}()
time.Sleep(50 * time.Millisecond)
cancel()
select {
case err := <-done:
assert.ErrorIs(t, err, context.Canceled)
case <-time.After(2 * time.Second):
t.Fatal("Watch did not return after cancel")
}
}
+76
View File
@@ -0,0 +1,76 @@
// Package embed produces dense vector embeddings for brain content.
//
// Wire format is Ollama's `/api/embed`, with the canonical request shape
// `{"model": "...", "input": "..."}` and a 2-D `embeddings` response.
// Default deployment runs `nomic-embed-text` on iguana, which returns
// 768-dim vectors compatible with the brain_embeddings table schema.
package embed
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"time"
)
// Client posts embedding requests to an Ollama-compatible endpoint.
type Client struct {
URL string
Model string
HTTP *http.Client
}
// New constructs a Client. Returns nil when url is empty so callers can
// treat a missing BRAIN_EMBED_URL as "feature disabled" via a single nil
// check.
func New(url, model string) *Client {
if url == "" {
return nil
}
return &Client{
URL: strings.TrimRight(url, "/"),
Model: model,
HTTP: &http.Client{Timeout: 30 * time.Second},
}
}
// Embed returns the embedding vector for text. Empty text is rejected
// up-front to keep upstream errors from masking caller mistakes.
func (c *Client) Embed(ctx context.Context, text string) ([]float32, error) {
if strings.TrimSpace(text) == "" {
return nil, fmt.Errorf("embed: empty text")
}
reqBody, _ := json.Marshal(map[string]any{
"model": c.Model,
"input": text,
})
req, err := http.NewRequestWithContext(ctx, http.MethodPost,
c.URL+"/api/embed", bytes.NewReader(reqBody))
if err != nil {
return nil, err
}
req.Header.Set("Content-Type", "application/json")
resp, err := c.HTTP.Do(req)
if err != nil {
return nil, err
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode/100 != 2 {
body, _ := io.ReadAll(resp.Body)
return nil, fmt.Errorf("embed: status %d: %s", resp.StatusCode, string(body))
}
var out struct {
Embeddings [][]float32 `json:"embeddings"`
}
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
return nil, fmt.Errorf("embed: decode: %w", err)
}
if len(out.Embeddings) == 0 || len(out.Embeddings[0]) == 0 {
return nil, fmt.Errorf("embed: empty embeddings in response")
}
return out.Embeddings[0], nil
}
+74
View File
@@ -0,0 +1,74 @@
package embed_test
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/embed"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestNew_EmptyURLReturnsNil(t *testing.T) {
assert.Nil(t, embed.New("", "model"))
}
func TestEmbed_ReturnsVector(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, "/api/embed", r.URL.Path)
var req map[string]any
require.NoError(t, json.NewDecoder(r.Body).Decode(&req))
assert.Equal(t, "nomic", req["model"])
assert.Equal(t, "hello", req["input"])
_ = json.NewEncoder(w).Encode(map[string]any{
"embeddings": [][]float32{{0.1, 0.2, 0.3}},
})
}))
defer srv.Close()
c := embed.New(srv.URL, "nomic")
require.NotNil(t, c)
v, err := c.Embed(context.Background(), "hello")
require.NoError(t, err)
assert.Equal(t, []float32{0.1, 0.2, 0.3}, v)
}
func TestEmbed_StripsTrailingSlashFromURL(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, "/api/embed", r.URL.Path)
_ = json.NewEncoder(w).Encode(map[string]any{"embeddings": [][]float32{{1.0}}})
}))
defer srv.Close()
c := embed.New(srv.URL+"/", "nomic")
_, err := c.Embed(context.Background(), "x")
require.NoError(t, err)
}
func TestEmbed_PropagatesUpstreamError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusBadGateway)
}))
defer srv.Close()
c := embed.New(srv.URL, "m")
_, err := c.Embed(context.Background(), "x")
require.Error(t, err)
}
func TestEmbed_RejectsEmptyEmbeddingsArray(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{"embeddings": [][]float32{}})
}))
defer srv.Close()
c := embed.New(srv.URL, "m")
_, err := c.Embed(context.Background(), "x")
require.Error(t, err)
}
func TestEmbed_RejectsEmptyText(t *testing.T) {
c := embed.New("http://127.0.0.1:1", "m")
_, err := c.Embed(context.Background(), "")
require.Error(t, err)
}
+263
View File
@@ -0,0 +1,263 @@
// Package graph extracts entity + edge records from brain markdown
// documents for the brain_entities / brain_edges relational graph.
//
// The extractor is pure: it takes markdown bytes and a document path and
// returns the entity (one per doc) and the wikilink edges (zero or more)
// it found, with source line numbers so the graph store can record
// provenance.
//
// Edge types in v1: only "wikilink" — derived from [[slug]] and
// [[slug|Display]] occurrences in the body. Section-header edges are
// deferred (see infra#62 grill addendum).
package graph
import (
"bufio"
"bytes"
"path/filepath"
"regexp"
"strings"
)
// Entity represents one brain document for graph indexing.
//
// Slug is the basename without ".md" — the same identity used by
// wiki canonicalization and the wikilink target syntax.
//
// Type categorises the doc into a coarse bucket so callers can filter
// graph traversals (e.g. "only entity nodes"). When the doc lives
// under brain/wiki/<wing>/<hall>/, Wing and Hall capture the
// taxonomy; otherwise they're empty (legacy brain/knowledge/ docs).
type Entity struct {
DocPath string // forward-slash, relative to brainDir
Slug string
Type string // "concept" | "entity" | "source" | "hall" | "knowledge"
Wing string // optional; from frontmatter or path
Hall string // optional; from frontmatter or path
Title string // optional; from frontmatter
// DIKW tier — infra#72. Empty until M3 migration writes `tier:`
// frontmatter to every entry. Path-inferred tier kicks in as a
// fallback so the column populates immediately on backfill even
// for entries that haven't had their frontmatter rewritten yet.
Tier string // "inbox" | "note" | "knowledge"
Topic string // kebab-slug; the thing the entry is about
}
// Edge represents a directed relationship between two slugs.
//
// SrcLine is the 1-indexed line in the source document where the link
// was found, so callers can re-find the linking text after an edit.
type Edge struct {
SrcDoc string // forward-slash, relative to brainDir
SrcSlug string // == Entity.Slug for SrcDoc
DstSlug string
EdgeType string // "wikilink" in v1
SrcLine int // 1-indexed
}
// linkRE matches both [[slug]] and [[slug|Display Name]] wikilinks.
// Group 1 is the slug; group 2 (if present) is the display.
var linkRE = regexp.MustCompile(`\[\[([^\]|]+)(?:\|([^\]]+))?\]\]`)
// Extract parses one markdown document and returns its Entity plus the
// outgoing wikilink Edges. docPath is forward-slash, relative to
// brainDir; content is the raw markdown bytes.
//
// Returns ok=false when docPath does not yield a usable slug (e.g.
// non-markdown file slipped through).
func Extract(docPath string, content []byte) (Entity, []Edge, bool) {
slug := slugFromPath(docPath)
if slug == "" {
return Entity{}, nil, false
}
ent := Entity{DocPath: docPath, Slug: slug}
classifyByPath(&ent, docPath)
readFrontmatter(&ent, content)
inferTierFromPath(&ent, docPath)
edges := extractEdges(docPath, slug, content)
return ent, edges, true
}
// inferTierFromPath fills Tier when frontmatter didn't already set it.
// The new layout has dedicated subtrees per tier; pre-migration paths
// (knowledge/, wiki/, raw/, sessions/) get their best-guess mapping so
// the column populates on backfill before the M3 file moves run.
func inferTierFromPath(e *Entity, docPath string) {
if e.Tier != "" {
return
}
parts := strings.Split(docPath, "/")
if len(parts) == 0 {
return
}
switch parts[0] {
case "inbox":
e.Tier = "inbox"
case "notes":
e.Tier = "note"
case "knowledge":
e.Tier = "knowledge"
case "wiki":
// Pre-M3 wiki layout. Most subdirs are I-level:
// wiki/sources/ — synth summaries of raw inbox material
// wiki/concepts/ — definitions, not lessons
// One exception: wiki/entities/ holds anchor facts about
// concrete things (models, services, people) that the eval
// expects to surface when queried directly. Those map to K
// to match the post-M3 layout target (knowledge/facts/).
if len(parts) >= 2 && parts[1] == "entities" {
e.Tier = "knowledge"
} else {
e.Tier = "note"
}
case "raw", "sessions", "clips":
e.Tier = "inbox"
}
}
func slugFromPath(docPath string) string {
base := filepath.Base(docPath)
if !strings.HasSuffix(base, ".md") {
return ""
}
return strings.TrimSuffix(base, ".md")
}
// classifyByPath fills Type / Wing / Hall from the path layout when the
// doc lives under brain/wiki/. Layout: wiki/<wing>/<hall>/<slug>.md
// or wiki/<bucket>/<slug>.md for the legacy concept/entity/source dirs.
//
// Files directly under wiki/ (no subdirectory — e.g. wiki/index.md) used
// to incorrectly land Type="hall" Wing="index.md" because the path's
// second segment was the file itself. Now they fall through to Type
// "knowledge" and leave wing/hall to frontmatter.
func classifyByPath(e *Entity, docPath string) {
parts := strings.Split(docPath, "/")
if len(parts) < 2 || parts[0] != "wiki" {
e.Type = "knowledge"
return
}
if len(parts) < 3 {
// wiki/<slug>.md — no subdirectory. Treat as plain knowledge
// and let frontmatter set wing/hall if they're present.
e.Type = "knowledge"
return
}
switch parts[1] {
case "concepts":
e.Type = "concept"
case "entities":
e.Type = "entity"
case "sources":
e.Type = "source"
default:
// wiki/<wing>/<hall>/<slug>.md
e.Type = "hall"
e.Wing = parts[1]
if len(parts) >= 4 {
e.Hall = parts[2]
}
}
}
// readFrontmatter pulls title/wing/hall from a YAML frontmatter block.
// Frontmatter is optional; missing fields leave the entity unchanged.
func readFrontmatter(e *Entity, content []byte) {
scanner := bufio.NewScanner(bytes.NewReader(content))
inFM := false
for scanner.Scan() {
line := scanner.Text()
if strings.TrimSpace(line) == "---" {
if !inFM {
inFM = true
continue
}
return
}
if !inFM {
return
}
key, val, ok := strings.Cut(line, ":")
if !ok {
continue
}
v := strings.Trim(strings.TrimSpace(val), `"'`)
switch strings.TrimSpace(key) {
case "title":
if e.Title == "" {
e.Title = v
}
case "wing":
if e.Wing == "" {
e.Wing = v
}
case "hall":
if e.Hall == "" {
e.Hall = v
}
case "tier":
if e.Tier == "" {
e.Tier = v
}
case "topic":
if e.Topic == "" {
e.Topic = v
}
}
}
}
func extractEdges(docPath, srcSlug string, content []byte) []Edge {
var edges []Edge
seen := make(map[string]struct{}) // dedupe (dst, line)
scanner := bufio.NewScanner(bytes.NewReader(content))
line := 0
for scanner.Scan() {
line++
matches := linkRE.FindAllStringSubmatch(scanner.Text(), -1)
for _, m := range matches {
dst := strings.TrimSpace(m[1])
if dst == "" || dst == srcSlug {
continue
}
key := dst + "|" + itoa(line)
if _, dup := seen[key]; dup {
continue
}
seen[key] = struct{}{}
edges = append(edges, Edge{
SrcDoc: docPath,
SrcSlug: srcSlug,
DstSlug: dst,
EdgeType: "wikilink",
SrcLine: line,
})
}
}
return edges
}
// itoa avoids the fmt dependency on a hot path. Single-digit fast path
// keeps overhead negligible for typical line counts.
func itoa(n int) string {
if n == 0 {
return "0"
}
var buf [20]byte
i := len(buf)
neg := n < 0
if neg {
n = -n
}
for n > 0 {
i--
buf[i] = byte('0' + n%10)
n /= 10
}
if neg {
i--
buf[i] = '-'
}
return string(buf[i:])
}
+179
View File
@@ -0,0 +1,179 @@
package graph
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestExtract_HallDoc(t *testing.T) {
content := []byte(`---
wing: jepa-fx
hall: decisions
title: Val Vol Decision
---
# Val Vol
See also [[other-decision]] and [[parent-concept|Parent Concept]].
Linking to [[unrelated]].
`)
ent, edges, ok := Extract("wiki/jepa-fx/decisions/val-vol.md", content)
require.True(t, ok)
assert.Equal(t, "val-vol", ent.Slug)
assert.Equal(t, "hall", ent.Type)
assert.Equal(t, "jepa-fx", ent.Wing)
assert.Equal(t, "decisions", ent.Hall)
assert.Equal(t, "Val Vol Decision", ent.Title)
require.Len(t, edges, 3)
assert.Equal(t, "other-decision", edges[0].DstSlug)
assert.Equal(t, "parent-concept", edges[1].DstSlug)
assert.Equal(t, "unrelated", edges[2].DstSlug)
for _, e := range edges {
assert.Equal(t, "wikilink", e.EdgeType)
assert.Equal(t, "val-vol", e.SrcSlug)
assert.Equal(t, "wiki/jepa-fx/decisions/val-vol.md", e.SrcDoc)
assert.Greater(t, e.SrcLine, 0)
}
}
func TestExtract_LegacyConceptDoc(t *testing.T) {
content := []byte(`---
title: Hash Encoding
---
# Hash Encoding
Linked to [[financial-sentiment-analysis|FSA]].
`)
ent, edges, ok := Extract("wiki/concepts/hash-encoding.md", content)
require.True(t, ok)
assert.Equal(t, "hash-encoding", ent.Slug)
assert.Equal(t, "concept", ent.Type)
assert.Empty(t, ent.Wing)
assert.Empty(t, ent.Hall)
assert.Equal(t, "Hash Encoding", ent.Title)
require.Len(t, edges, 1)
assert.Equal(t, "financial-sentiment-analysis", edges[0].DstSlug)
}
func TestExtract_KnowledgeDoc(t *testing.T) {
content := []byte("# No frontmatter, no links here.\n")
ent, edges, ok := Extract("knowledge/some-note.md", content)
require.True(t, ok)
assert.Equal(t, "some-note", ent.Slug)
assert.Equal(t, "knowledge", ent.Type)
assert.Empty(t, edges)
}
func TestExtract_DedupesRepeatedLinkOnSameLine(t *testing.T) {
content := []byte("See [[foo]] and [[foo]] again on the same line.\n")
_, edges, ok := Extract("knowledge/dup.md", content)
require.True(t, ok)
require.Len(t, edges, 1)
assert.Equal(t, "foo", edges[0].DstSlug)
}
func TestExtract_KeepsMultipleEdgesOnDifferentLines(t *testing.T) {
content := []byte("First mention [[foo]].\n\nSecond mention [[foo]].\n")
_, edges, ok := Extract("knowledge/multi.md", content)
require.True(t, ok)
require.Len(t, edges, 2)
assert.NotEqual(t, edges[0].SrcLine, edges[1].SrcLine)
}
func TestExtract_IgnoresSelfLinks(t *testing.T) {
content := []byte("Self-reference [[self]] should be ignored.\n")
_, edges, ok := Extract("knowledge/self.md", content)
require.True(t, ok)
assert.Empty(t, edges)
}
func TestExtract_RejectsNonMarkdown(t *testing.T) {
_, _, ok := Extract("wiki/concepts/not-markdown.txt", []byte("anything"))
assert.False(t, ok)
}
func TestExtract_LineNumbersAre1Indexed(t *testing.T) {
content := []byte("line 1\nline 2 [[bar]]\n")
_, edges, ok := Extract("knowledge/lines.md", content)
require.True(t, ok)
require.Len(t, edges, 1)
assert.Equal(t, 2, edges[0].SrcLine)
}
// Files directly under wiki/ (no subdirectory) used to land
// Type="hall" Wing="<filename>.md" because the path's second segment
// was the file itself. The fix routes them to Type="knowledge" with
// empty Wing/Hall and lets frontmatter set them if present.
func TestExtract_WikiRootFileIsKnowledgeNotHall(t *testing.T) {
content := []byte("# Index\n\n- [[foo]]\n")
ent, _, ok := Extract("wiki/index.md", content)
require.True(t, ok)
assert.Equal(t, "index", ent.Slug)
assert.Equal(t, "knowledge", ent.Type)
assert.Empty(t, ent.Wing)
assert.Empty(t, ent.Hall)
}
func TestExtract_TierFromFrontmatter(t *testing.T) {
content := []byte(`---
tier: knowledge
topic: postgres-roles
title: Least-privilege migration trap
---
# body
`)
ent, _, ok := Extract("knowledge/some-lesson.md", content)
require.True(t, ok)
assert.Equal(t, "knowledge", ent.Tier)
assert.Equal(t, "postgres-roles", ent.Topic)
}
func TestExtract_TierInferredFromPath(t *testing.T) {
cases := []struct {
path string
want string
}{
{"knowledge/foo.md", "knowledge"},
{"wiki/sources/x.md", "note"},
{"wiki/concepts/x.md", "note"},
{"wiki/x.md", "note"},
{"inbox/clips/x.md", "inbox"},
{"notes/x.md", "note"},
{"raw/x.md", "inbox"},
{"sessions/x.md", "inbox"},
}
for _, tc := range cases {
ent, _, ok := Extract(tc.path, []byte("# x\n"))
require.True(t, ok, tc.path)
assert.Equal(t, tc.want, ent.Tier, tc.path)
}
}
func TestExtract_FrontmatterTierBeatsPathInference(t *testing.T) {
// A clip explicitly promoted via frontmatter wins over the path's
// inbox inference. Catches the case where a file has been moved
// to a new location but frontmatter hasn't been updated.
content := []byte("---\ntier: knowledge\n---\n# x\n")
ent, _, ok := Extract("inbox/clips/x.md", content)
require.True(t, ok)
assert.Equal(t, "knowledge", ent.Tier)
}
func TestExtract_WikiRootFileWithFrontmatterWingHall(t *testing.T) {
content := []byte(`---
wing: homelab
hall: facts
---
# Some root note
`)
ent, _, ok := Extract("wiki/some-note.md", content)
require.True(t, ok)
assert.Equal(t, "knowledge", ent.Type)
assert.Equal(t, "homelab", ent.Wing)
assert.Equal(t, "facts", ent.Hall)
}
+365
View File
@@ -0,0 +1,365 @@
// Package graphstore stores the brain knowledge graph (entities +
// directed edges) in PostgreSQL on the shared postgres18 instance,
// alongside the pgvector embeddings in [vectorstore].
//
// Schema (created idempotently by Init):
//
// brain_entities(slug PK, type, wing, hall, doc_path, title, updated_at)
// brain_edges(id PK, src_slug FK, dst_slug, edge_type, src_doc, src_line,
// weight, updated_at)
//
// Edges fan-out from a source document; calling [PGStore.ReplaceEdgesForDoc]
// replaces every edge previously emitted from that document so re-ingest is
// idempotent without bookkeeping.
//
// All slug strings are stored verbatim — callers are expected to canonicalise
// before persisting. Dst slugs may reference entities that don't yet exist
// (dangling edges); resolution is deferred to query time so ingestion order
// doesn't matter.
package graphstore
import (
"context"
"errors"
"fmt"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/mathiasbq/hyperguild/ingestion/internal/graph"
)
// PGStore is the postgres-backed brain knowledge-graph store. Construct
// with New + call Init once to create tables and indexes. Use Close to
// release the pool.
type PGStore struct {
pool *pgxpool.Pool
}
// New opens a pgxpool against dsn and pings to verify connectivity. The
// caller owns the resulting PGStore and must invoke Close.
func New(ctx context.Context, dsn string) (*PGStore, error) {
pool, err := pgxpool.New(ctx, dsn)
if err != nil {
return nil, fmt.Errorf("pgxpool: %w", err)
}
if err := pool.Ping(ctx); err != nil {
pool.Close()
return nil, fmt.Errorf("ping: %w", err)
}
return &PGStore{pool: pool}, nil
}
// Close releases the underlying connection pool.
func (s *PGStore) Close() {
if s.pool != nil {
s.pool.Close()
}
}
// Init creates brain_entities + brain_edges tables and their indexes if
// they don't yet exist. Safe to call on every startup. No-op when the
// schema already matches.
func (s *PGStore) Init(ctx context.Context) error {
const ddl = `
CREATE TABLE IF NOT EXISTS brain_entities (
slug TEXT PRIMARY KEY,
type TEXT NOT NULL DEFAULT 'knowledge',
wing TEXT NOT NULL DEFAULT '',
hall TEXT NOT NULL DEFAULT '',
doc_path TEXT NOT NULL,
title TEXT NOT NULL DEFAULT '',
tier TEXT NOT NULL DEFAULT '',
topic TEXT NOT NULL DEFAULT '',
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- Idempotent migration for clusters created before the DIKW tier
-- redesign (infra#72). ADD COLUMN IF NOT EXISTS is safe across
-- repeated startups.
ALTER TABLE brain_entities
ADD COLUMN IF NOT EXISTS tier TEXT NOT NULL DEFAULT '',
ADD COLUMN IF NOT EXISTS topic TEXT NOT NULL DEFAULT '';
CREATE INDEX IF NOT EXISTS brain_entities_wing_idx
ON brain_entities (wing) WHERE wing <> '';
CREATE INDEX IF NOT EXISTS brain_entities_type_idx
ON brain_entities (type);
CREATE INDEX IF NOT EXISTS brain_entities_tier_idx
ON brain_entities (tier) WHERE tier <> '';
CREATE INDEX IF NOT EXISTS brain_entities_topic_idx
ON brain_entities (topic) WHERE topic <> '';
CREATE TABLE IF NOT EXISTS brain_edges (
id BIGSERIAL PRIMARY KEY,
src_slug TEXT NOT NULL,
dst_slug TEXT NOT NULL,
edge_type TEXT NOT NULL DEFAULT 'wikilink',
src_doc TEXT NOT NULL,
src_line INTEGER NOT NULL DEFAULT 0,
weight REAL NOT NULL DEFAULT 1.0,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS brain_edges_src_idx
ON brain_edges (src_slug, edge_type);
CREATE INDEX IF NOT EXISTS brain_edges_dst_idx
ON brain_edges (dst_slug, edge_type);
CREATE INDEX IF NOT EXISTS brain_edges_src_doc_idx
ON brain_edges (src_doc);
`
_, err := s.pool.Exec(ctx, ddl)
return err
}
// UpsertEntity inserts or updates one entity by slug.
func (s *PGStore) UpsertEntity(ctx context.Context, e graph.Entity) error {
if e.Slug == "" {
return errors.New("entity slug is required")
}
if e.Type == "" {
e.Type = "knowledge"
}
_, err := s.pool.Exec(ctx, `
INSERT INTO brain_entities (slug, type, wing, hall, doc_path, title, tier, topic, updated_at)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, now())
ON CONFLICT (slug) DO UPDATE
SET type = EXCLUDED.type,
wing = EXCLUDED.wing,
hall = EXCLUDED.hall,
doc_path = EXCLUDED.doc_path,
title = EXCLUDED.title,
tier = EXCLUDED.tier,
topic = EXCLUDED.topic,
updated_at = now()
`, e.Slug, e.Type, e.Wing, e.Hall, e.DocPath, e.Title, e.Tier, e.Topic)
if err != nil {
return fmt.Errorf("upsert entity %q: %w", e.Slug, err)
}
return nil
}
// ReplaceEdgesForDoc deletes every edge previously emitted from docPath
// and inserts the new set in one transaction. Caller should pass the
// complete edge set for the doc — partial updates are not supported.
func (s *PGStore) ReplaceEdgesForDoc(ctx context.Context, docPath string, edges []graph.Edge) error {
if docPath == "" {
return errors.New("doc path is required")
}
tx, err := s.pool.BeginTx(ctx, pgx.TxOptions{})
if err != nil {
return fmt.Errorf("begin: %w", err)
}
defer func() { _ = tx.Rollback(ctx) }()
if _, err := tx.Exec(ctx, `DELETE FROM brain_edges WHERE src_doc = $1`, docPath); err != nil {
return fmt.Errorf("delete prior edges for %q: %w", docPath, err)
}
for _, e := range edges {
if e.SrcSlug == "" || e.DstSlug == "" {
continue
}
if _, err := tx.Exec(ctx, `
INSERT INTO brain_edges (src_slug, dst_slug, edge_type, src_doc, src_line, weight)
VALUES ($1, $2, $3, $4, $5, 1.0)
`, e.SrcSlug, e.DstSlug, e.EdgeType, e.SrcDoc, e.SrcLine); err != nil {
return fmt.Errorf("insert edge %s->%s: %w", e.SrcSlug, e.DstSlug, err)
}
}
if err := tx.Commit(ctx); err != nil {
return fmt.Errorf("commit: %w", err)
}
return nil
}
// DeleteByDoc removes the entity at docPath and every edge it sourced.
// Use when a wiki page is deleted on disk.
func (s *PGStore) DeleteByDoc(ctx context.Context, docPath string) error {
if docPath == "" {
return errors.New("doc path is required")
}
tx, err := s.pool.BeginTx(ctx, pgx.TxOptions{})
if err != nil {
return fmt.Errorf("begin: %w", err)
}
defer func() { _ = tx.Rollback(ctx) }()
if _, err := tx.Exec(ctx, `DELETE FROM brain_edges WHERE src_doc = $1`, docPath); err != nil {
return fmt.Errorf("delete edges: %w", err)
}
if _, err := tx.Exec(ctx, `DELETE FROM brain_entities WHERE doc_path = $1`, docPath); err != nil {
return fmt.Errorf("delete entity: %w", err)
}
return tx.Commit(ctx)
}
// Neighbor is one row in a Neighbors / Subgraph response.
type Neighbor struct {
Slug string
Type string
Wing string
Hall string
DocPath string
Title string
EdgeType string
Distance int // hop count from origin; 1 for direct neighbors
}
// Neighbors returns the direct (1-hop) outgoing neighbours of slug.
// edgeType filters by relationship kind; "" returns all kinds.
// limit defaults to 25 when <= 0.
func (s *PGStore) Neighbors(ctx context.Context, slug, edgeType string, limit int) ([]Neighbor, error) {
if slug == "" {
return nil, errors.New("slug is required")
}
if limit <= 0 {
limit = 25
}
q := `
SELECT e.dst_slug, COALESCE(t.type,''), COALESCE(t.wing,''), COALESCE(t.hall,''),
COALESCE(t.doc_path,''), COALESCE(t.title,''), e.edge_type, 1
FROM brain_edges e
LEFT JOIN brain_entities t ON t.slug = e.dst_slug
WHERE e.src_slug = $1
AND ($2 = '' OR e.edge_type = $2)
ORDER BY e.updated_at DESC
LIMIT $3
`
rows, err := s.pool.Query(ctx, q, slug, edgeType, limit)
if err != nil {
return nil, fmt.Errorf("query neighbors: %w", err)
}
defer rows.Close()
return scanNeighbors(rows)
}
// Subgraph returns every distinct slug reachable from origin within
// depth outgoing hops, annotated with the shortest hop distance. The
// origin itself is omitted. depth defaults to 2 when <= 0; values
// above 6 are clamped to 6 to bound traversal cost.
func (s *PGStore) Subgraph(ctx context.Context, origin string, depth int) ([]Neighbor, error) {
if origin == "" {
return nil, errors.New("origin slug is required")
}
if depth <= 0 {
depth = 2
}
if depth > 6 {
depth = 6
}
q := `
WITH RECURSIVE walk(slug, edge_type, distance) AS (
SELECT e.dst_slug, e.edge_type, 1
FROM brain_edges e
WHERE e.src_slug = $1
UNION
SELECT e.dst_slug, e.edge_type, w.distance + 1
FROM walk w
JOIN brain_edges e ON e.src_slug = w.slug
WHERE w.distance < $2
)
SELECT w.slug, COALESCE(t.type,''), COALESCE(t.wing,''), COALESCE(t.hall,''),
COALESCE(t.doc_path,''), COALESCE(t.title,''), w.edge_type, MIN(w.distance)
FROM walk w
LEFT JOIN brain_entities t ON t.slug = w.slug
WHERE w.slug <> $1
GROUP BY w.slug, t.type, t.wing, t.hall, t.doc_path, t.title, w.edge_type
ORDER BY MIN(w.distance), w.slug
`
rows, err := s.pool.Query(ctx, q, origin, depth)
if err != nil {
return nil, fmt.Errorf("query subgraph: %w", err)
}
defer rows.Close()
return scanNeighbors(rows)
}
// PathStep is one hop in a Path response.
type PathStep struct {
FromSlug string
ToSlug string
EdgeType string
}
// Path returns the shortest directed path from src to dst within
// maxDepth hops, as an ordered list of edges. Empty slice means no
// path exists. maxDepth defaults to 4 when <= 0; values above 8 are
// clamped to 8.
func (s *PGStore) Path(ctx context.Context, src, dst string, maxDepth int) ([]PathStep, error) {
if src == "" || dst == "" {
return nil, errors.New("src and dst are required")
}
if maxDepth <= 0 {
maxDepth = 4
}
if maxDepth > 8 {
maxDepth = 8
}
q := `
WITH RECURSIVE walk(cur, path_slugs, path_edges, distance) AS (
SELECT e.dst_slug,
ARRAY[e.src_slug, e.dst_slug]::TEXT[],
ARRAY[e.edge_type]::TEXT[],
1
FROM brain_edges e
WHERE e.src_slug = $1
UNION ALL
SELECT e.dst_slug,
w.path_slugs || e.dst_slug,
w.path_edges || e.edge_type,
w.distance + 1
FROM walk w
JOIN brain_edges e ON e.src_slug = w.cur
WHERE w.distance < $3
AND NOT (e.dst_slug = ANY(w.path_slugs))
)
SELECT path_slugs, path_edges
FROM walk
WHERE cur = $2
ORDER BY distance ASC
LIMIT 1
`
row := s.pool.QueryRow(ctx, q, src, dst, maxDepth)
var (
slugs []string
kinds []string
)
if err := row.Scan(&slugs, &kinds); err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return nil, nil
}
return nil, fmt.Errorf("scan path: %w", err)
}
if len(slugs) < 2 || len(kinds) == 0 {
return nil, nil
}
steps := make([]PathStep, 0, len(kinds))
for i := 0; i < len(kinds) && i+1 < len(slugs); i++ {
steps = append(steps, PathStep{
FromSlug: slugs[i],
ToSlug: slugs[i+1],
EdgeType: kinds[i],
})
}
return steps, nil
}
// CountEdges is a debug helper — returns the total edges currently stored.
// Used by tests and by the volume-gate diagnostic.
func (s *PGStore) CountEdges(ctx context.Context) (int64, error) {
var n int64
err := s.pool.QueryRow(ctx, `SELECT count(*) FROM brain_edges`).Scan(&n)
return n, err
}
func scanNeighbors(rows pgx.Rows) ([]Neighbor, error) {
var out []Neighbor
for rows.Next() {
var n Neighbor
if err := rows.Scan(
&n.Slug, &n.Type, &n.Wing, &n.Hall,
&n.DocPath, &n.Title, &n.EdgeType, &n.Distance,
); err != nil {
return nil, fmt.Errorf("scan: %w", err)
}
out = append(out, n)
}
return out, rows.Err()
}
+112
View File
@@ -0,0 +1,112 @@
// Package graphsync glues the disk-resident brain markdown documents to
// the relational graph in [graphstore]. It is a tiny seam so that the
// MCP handlers can call one function after every successful write or
// ingest without having to know either the parser or the postgres
// schema.
//
// Every operation is best-effort from the caller's perspective: if the
// graph store is unconfigured or the doc parses to nothing usable, the
// helpers return nil. Real database errors are surfaced so the caller
// can log them.
package graphsync
import (
"context"
"fmt"
"os"
"path/filepath"
"github.com/mathiasbq/hyperguild/ingestion/internal/graph"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphstore"
)
// Store is the subset of graphstore.PGStore that graphsync requires.
// Tests can substitute a fake by satisfying this interface.
type Store interface {
UpsertEntity(ctx context.Context, e graph.Entity) error
ReplaceEdgesForDoc(ctx context.Context, docPath string, edges []graph.Edge) error
DeleteByDoc(ctx context.Context, docPath string) error
}
// Compile-time assertion that *graphstore.PGStore satisfies Store.
var _ Store = (*graphstore.PGStore)(nil)
// IndexDoc reads docPath under brainDir and pushes one Entity + its
// outgoing wikilink Edges into store. relPath must be the
// forward-slash path relative to brainDir (the same shape returned by
// api.WriteNote).
//
// nil store is a valid no-op so callers can wire the helper
// unconditionally and let configuration decide whether the graph is
// populated.
func IndexDoc(ctx context.Context, store Store, brainDir, relPath string) error {
if store == nil {
return nil
}
if relPath == "" {
return nil
}
abs := filepath.Join(brainDir, filepath.FromSlash(relPath))
content, err := os.ReadFile(abs)
if err != nil {
return fmt.Errorf("read %q: %w", relPath, err)
}
ent, edges, ok := graph.Extract(relPath, content)
if !ok {
return nil
}
if err := store.UpsertEntity(ctx, ent); err != nil {
return fmt.Errorf("upsert entity: %w", err)
}
if err := store.ReplaceEdgesForDoc(ctx, relPath, edges); err != nil {
return fmt.Errorf("replace edges: %w", err)
}
return nil
}
// BackfillFromBrainDir walks every markdown file under brainDir/wiki/
// and brainDir/knowledge/, parses each, and upserts the resulting
// Entity + Edges. Existing rows are overwritten; orphan rows for
// already-deleted files are NOT cleaned up — call this only on a
// fresh store, or follow with a separate prune pass.
//
// Intended for one-shot startup runs against a populated brain dir.
// Cost scales linearly with corpus size; ~30 wiki pages plus the
// knowledge corpus is a few hundred ms.
func BackfillFromBrainDir(ctx context.Context, store Store, brainDir string) (indexed int, _ error) {
if store == nil {
return 0, nil
}
roots := []string{"wiki", "knowledge"}
for _, root := range roots {
base := filepath.Join(brainDir, root)
if _, err := os.Stat(base); os.IsNotExist(err) {
continue
}
err := filepath.WalkDir(base, func(path string, d os.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if d.IsDir() {
return nil
}
if filepath.Ext(path) != ".md" {
return nil
}
rel, relErr := filepath.Rel(brainDir, path)
if relErr != nil {
return fmt.Errorf("rel %q: %w", path, relErr)
}
rel = filepath.ToSlash(rel)
if err := IndexDoc(ctx, store, brainDir, rel); err != nil {
return fmt.Errorf("index %q: %w", rel, err)
}
indexed++
return nil
})
if err != nil {
return indexed, fmt.Errorf("walk %s: %w", root, err)
}
}
return indexed, nil
}
@@ -0,0 +1,134 @@
package graphsync
import (
"context"
"errors"
"os"
"path/filepath"
"sync"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/graph"
)
// fakeStore captures the calls IndexDoc / BackfillFromBrainDir made.
type fakeStore struct {
mu sync.Mutex
upserts []graph.Entity
replaces map[string][]graph.Edge
deletes []string
failOn string // upsert fails when entity slug == failOn
}
func newFakeStore() *fakeStore {
return &fakeStore{replaces: make(map[string][]graph.Edge)}
}
func (f *fakeStore) UpsertEntity(_ context.Context, e graph.Entity) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.failOn != "" && e.Slug == f.failOn {
return errors.New("synthetic failure")
}
f.upserts = append(f.upserts, e)
return nil
}
func (f *fakeStore) ReplaceEdgesForDoc(_ context.Context, docPath string, edges []graph.Edge) error {
f.mu.Lock()
defer f.mu.Unlock()
f.replaces[docPath] = append([]graph.Edge(nil), edges...)
return nil
}
func (f *fakeStore) DeleteByDoc(_ context.Context, docPath string) error {
f.mu.Lock()
defer f.mu.Unlock()
f.deletes = append(f.deletes, docPath)
return nil
}
func writeBrain(t *testing.T, brainDir, relPath, body string) {
t.Helper()
full := filepath.Join(brainDir, filepath.FromSlash(relPath))
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
require.NoError(t, os.WriteFile(full, []byte(body), 0o644))
}
func TestIndexDoc_UpsertsEntityAndEdges(t *testing.T) {
tmp := t.TempDir()
writeBrain(t, tmp, "wiki/concepts/foo.md", `---
title: Foo
---
# Foo
Linking to [[bar]] and [[baz|Baz]].
`)
fs := newFakeStore()
require.NoError(t, IndexDoc(context.Background(), fs, tmp, "wiki/concepts/foo.md"))
require.Len(t, fs.upserts, 1)
assert.Equal(t, "foo", fs.upserts[0].Slug)
assert.Equal(t, "concept", fs.upserts[0].Type)
edges := fs.replaces["wiki/concepts/foo.md"]
require.Len(t, edges, 2)
assert.Equal(t, "bar", edges[0].DstSlug)
assert.Equal(t, "baz", edges[1].DstSlug)
}
func TestIndexDoc_NoopOnNilStore(t *testing.T) {
require.NoError(t, IndexDoc(context.Background(), nil, "anywhere", "foo.md"))
}
func TestIndexDoc_NoopOnEmptyRelPath(t *testing.T) {
fs := newFakeStore()
require.NoError(t, IndexDoc(context.Background(), fs, "anywhere", ""))
assert.Empty(t, fs.upserts)
}
func TestIndexDoc_ErrorsOnMissingFile(t *testing.T) {
fs := newFakeStore()
err := IndexDoc(context.Background(), fs, t.TempDir(), "wiki/nope.md")
require.Error(t, err)
}
func TestIndexDoc_SurfacesStoreFailure(t *testing.T) {
tmp := t.TempDir()
writeBrain(t, tmp, "wiki/concepts/boom.md", "# Boom\n")
fs := newFakeStore()
fs.failOn = "boom"
err := IndexDoc(context.Background(), fs, tmp, "wiki/concepts/boom.md")
require.Error(t, err)
}
func TestBackfillFromBrainDir_WalksWikiAndKnowledge(t *testing.T) {
tmp := t.TempDir()
writeBrain(t, tmp, "wiki/concepts/foo.md", "# Foo\n[[bar]]\n")
writeBrain(t, tmp, "wiki/entities/bar.md", "# Bar\n")
writeBrain(t, tmp, "knowledge/legacy.md", "# Legacy [[foo]]\n")
// non-markdown file should be skipped
writeBrain(t, tmp, "wiki/concepts/skip.txt", "ignore me")
fs := newFakeStore()
n, err := BackfillFromBrainDir(context.Background(), fs, tmp)
require.NoError(t, err)
assert.Equal(t, 3, n)
assert.Len(t, fs.upserts, 3)
}
func TestBackfillFromBrainDir_TolerantOfMissingDirs(t *testing.T) {
tmp := t.TempDir()
fs := newFakeStore()
n, err := BackfillFromBrainDir(context.Background(), fs, tmp)
require.NoError(t, err)
assert.Equal(t, 0, n)
}
func TestBackfillFromBrainDir_NilStoreNoop(t *testing.T) {
n, err := BackfillFromBrainDir(context.Background(), nil, t.TempDir())
require.NoError(t, err)
assert.Equal(t, 0, n)
}
-36
View File
@@ -1,36 +0,0 @@
package mcp
import (
"crypto/subtle"
"net/http"
"strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/auth"
)
// BearerAuth returns a middleware that enforces authentication on every request.
// It tries a valid Dex JWT first (when v is non-nil), then falls back to the
// static token. Rejects if token is empty and no valid JWT is presented.
func BearerAuth(token string, v *auth.Validator, next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
rawToken, ok := strings.CutPrefix(r.Header.Get("Authorization"), "Bearer ")
if !ok {
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
if v != nil {
if _, err := v.Validate(r.Context(), rawToken); err == nil {
next.ServeHTTP(w, r)
return
}
}
if token != "" && subtle.ConstantTimeCompare([]byte(rawToken), []byte(token)) == 1 {
next.ServeHTTP(w, r)
return
}
http.Error(w, "unauthorized", http.StatusUnauthorized)
})
}
-161
View File
@@ -1,161 +0,0 @@
package mcp_test
import (
"context"
"crypto/rand"
"crypto/rsa"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/lestrrat-go/jwx/v2/jwa"
"github.com/lestrrat-go/jwx/v2/jwk"
"github.com/lestrrat-go/jwx/v2/jwt"
"github.com/mathiasbq/hyperguild/ingestion/internal/auth"
"github.com/mathiasbq/hyperguild/ingestion/internal/mcp"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func okHandler() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
})
}
func TestBearerAuth_MissingHeader(t *testing.T) {
handler := mcp.BearerAuth("secret", nil, okHandler())
req := httptest.NewRequest(http.MethodPost, "/mcp", nil)
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
assert.Equal(t, http.StatusUnauthorized, rr.Code)
}
func TestBearerAuth_WrongToken(t *testing.T) {
handler := mcp.BearerAuth("secret", nil, okHandler())
req := httptest.NewRequest(http.MethodPost, "/mcp", nil)
req.Header.Set("Authorization", "Bearer wrong")
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
assert.Equal(t, http.StatusUnauthorized, rr.Code)
}
func TestBearerAuth_CorrectToken(t *testing.T) {
called := false
handler := mcp.BearerAuth("secret", nil, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
called = true
w.WriteHeader(http.StatusOK)
}))
req := httptest.NewRequest(http.MethodPost, "/mcp", nil)
req.Header.Set("Authorization", "Bearer secret")
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
assert.Equal(t, http.StatusOK, rr.Code)
assert.True(t, called)
}
func TestBearerAuth_EmptyConfiguredToken(t *testing.T) {
handler := mcp.BearerAuth("", nil, okHandler())
req := httptest.NewRequest(http.MethodPost, "/mcp", nil)
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
assert.Equal(t, http.StatusUnauthorized, rr.Code)
}
// JWT auth tests
func buildOIDCServer(t *testing.T) (*httptest.Server, jwk.Key) {
t.Helper()
raw, err := rsa.GenerateKey(rand.Reader, 2048)
require.NoError(t, err)
priv, err := jwk.FromRaw(raw)
require.NoError(t, err)
require.NoError(t, priv.Set(jwk.KeyIDKey, "k1"))
require.NoError(t, priv.Set(jwk.AlgorithmKey, jwa.RS256))
pub, err := jwk.PublicKeyOf(priv)
require.NoError(t, err)
set := jwk.NewSet()
require.NoError(t, set.AddKey(pub))
jwksBytes, err := json.Marshal(set)
require.NoError(t, err)
muxSrv := http.NewServeMux()
var srv *httptest.Server
muxSrv.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, _ *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]string{
"issuer": srv.URL,
"jwks_uri": srv.URL + "/jwks",
})
})
muxSrv.HandleFunc("/jwks", func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write(jwksBytes)
})
srv = httptest.NewServer(muxSrv)
t.Cleanup(srv.Close)
return srv, priv
}
func signJWT(t *testing.T, priv jwk.Key, issuer, audience string, exp time.Time) string {
t.Helper()
tok, err := jwt.NewBuilder().
Issuer(issuer).Audience([]string{audience}).
Subject("s").Expiration(exp).
Build()
require.NoError(t, err)
signed, err := jwt.Sign(tok, jwt.WithKey(jwa.RS256, priv))
require.NoError(t, err)
return string(signed)
}
func TestBearerAuth_ValidJWT(t *testing.T) {
oidcSrv, priv := buildOIDCServer(t)
v, err := auth.NewValidator(oidcSrv.URL, "brain")
require.NoError(t, err)
called := false
handler := mcp.BearerAuth("static-secret", v, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
called = true
w.WriteHeader(http.StatusOK)
}))
token := signJWT(t, priv, oidcSrv.URL, "brain", time.Now().Add(time.Hour))
req := httptest.NewRequest(http.MethodPost, "/mcp", nil)
req.Header.Set("Authorization", "Bearer "+token)
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
assert.Equal(t, http.StatusOK, rr.Code)
assert.True(t, called)
}
func TestBearerAuth_InvalidJWT_FallsBackToStaticToken(t *testing.T) {
oidcSrv, _ := buildOIDCServer(t)
v, err := auth.NewValidator(oidcSrv.URL, "brain")
require.NoError(t, err)
handler := mcp.BearerAuth("static-secret", v, okHandler())
req := httptest.NewRequest(http.MethodPost, "/mcp", nil)
req.Header.Set("Authorization", "Bearer static-secret")
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
assert.Equal(t, http.StatusOK, rr.Code)
}
func TestBearerAuth_InvalidJWT_WrongStaticToken(t *testing.T) {
oidcSrv, priv := buildOIDCServer(t)
v, err := auth.NewValidator(oidcSrv.URL, "brain")
require.NoError(t, err)
handler := mcp.BearerAuth("static-secret", v, okHandler())
// Expired JWT — JWT fails, static token doesn't match either
token := signJWT(t, priv, oidcSrv.URL, "brain", time.Now().Add(-time.Hour))
req := httptest.NewRequest(http.MethodPost, "/mcp", nil)
req.Header.Set("Authorization", "Bearer "+token)
_ = context.Background() // satisfies import
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
assert.Equal(t, http.StatusUnauthorized, rr.Code)
}
+204
View File
@@ -0,0 +1,204 @@
package mcp_test
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/mcp"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// callResult parses the JSON text payload of a successful tool call.
func callResult(t *testing.T, resp map[string]any) map[string]any {
t.Helper()
require.Nil(t, resp["error"], "tool returned error: %v", resp["error"])
text := resp["result"].(map[string]any)["content"].([]any)[0].(map[string]any)["text"].(string)
var out map[string]any
require.NoError(t, json.Unmarshal([]byte(text), &out))
return out
}
func TestBrainUpdateSupersedesExisting(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
// Seed via brain_write so the note carries real frontmatter.
callResult(t, toolCall(t, srv, "brain_write", map[string]any{
"content": "# Old\n\nold body\n", "filename": "val-vol",
"wing": "jepa-fx", "hall": "facts",
}))
out := callResult(t, toolCall(t, srv, "brain_update", map[string]any{
"wing": "jepa-fx", "hall": "facts", "slug": "val-vol",
"content": "# New\n\nnew body\n", "reason": "facts changed",
}))
assert.Equal(t, "wiki/jepa-fx/facts/val-vol.md", out["path"])
assert.Equal(t, out["path"], out["id"])
assert.NotEmpty(t, out["content_hash"])
assert.Equal(t, true, out["superseded"])
got, err := os.ReadFile(filepath.Join(brainDir, "wiki/jepa-fx/facts/val-vol.md"))
require.NoError(t, err)
s := string(got)
assert.Contains(t, s, "# New")
assert.NotContains(t, s, "old body")
assert.Contains(t, s, "wing: jepa-fx")
assert.Contains(t, s, "supersede_reason: facts changed")
assert.Contains(t, s, "supersedes:")
}
func TestBrainUpdateMissingTargetErrorsNoCreate(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
resp := toolCall(t, srv, "brain_update", map[string]any{
"wing": "jepa-fx", "hall": "facts", "slug": "ghost",
"content": "x\n",
})
require.NotNil(t, resp["error"])
assert.Contains(t, resp["error"].(map[string]any)["message"].(string), "does not exist")
_, statErr := os.Stat(filepath.Join(brainDir, "wiki/jepa-fx/facts/ghost.md"))
assert.True(t, os.IsNotExist(statErr))
}
func TestBrainUpdateByFullPath(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
callResult(t, toolCall(t, srv, "brain_write", map[string]any{
"content": "old\n", "filename": "n", "wing": "a", "hall": "facts",
}))
out := callResult(t, toolCall(t, srv, "brain_update", map[string]any{
"slug": "wiki/a/facts/n.md", "content": "fresh\n",
}))
assert.Equal(t, "wiki/a/facts/n.md", out["path"])
}
func TestBrainGetByIDAndPath(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
w := callResult(t, toolCall(t, srv, "brain_write", map[string]any{
"content": "# Body\n\ntext\n", "filename": "n", "wing": "a", "hall": "facts",
}))
id := w["id"].(string)
hash := w["content_hash"].(string)
require.NotEmpty(t, id)
require.NotEmpty(t, hash)
// by id
g1 := callResult(t, toolCall(t, srv, "brain_get", map[string]any{"id": id}))
assert.Equal(t, id, g1["path"])
assert.Equal(t, hash, g1["content_hash"], "content_hash must round-trip write→get")
assert.Contains(t, g1["body"].(string), "# Body")
fm := g1["frontmatter"].(map[string]any)
assert.Equal(t, "a", fm["wing"])
// by path
g2 := callResult(t, toolCall(t, srv, "brain_get", map[string]any{"path": id}))
assert.Equal(t, hash, g2["content_hash"])
}
func TestBrainGetMissingArgsErrors(t *testing.T) {
srv := mcp.NewServer(t.TempDir(), nil, nil, nil)
resp := toolCall(t, srv, "brain_get", map[string]any{})
require.NotNil(t, resp["error"])
}
func TestBrainWriteReturnsHandle(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
out := callResult(t, toolCall(t, srv, "brain_write", map[string]any{
"content": "# X\n\nbody\n", "filename": "x", "wing": "a", "hall": "facts",
}))
assert.Equal(t, "wiki/a/facts/x.md", out["path"])
assert.Equal(t, out["path"], out["id"])
assert.NotEmpty(t, out["content_hash"])
}
// --- retrieval-reflects-new-content: exercises the real mtime-driven Sync ---
type fakeVecStore struct {
chunks map[string][]float32
deleted []string
}
func (f *fakeVecStore) KnownPathsWithTime(_ context.Context) (map[string]time.Time, error) {
m := make(map[string]time.Time, len(f.chunks))
for p := range f.chunks {
m[p] = time.Unix(0, 0) // always stale → mtime(now) is always newer
}
return m, nil
}
func (f *fakeVecStore) Upsert(_ context.Context, path string, vec []float32) error {
f.chunks[path] = vec
return nil
}
func (f *fakeVecStore) Delete(_ context.Context, path string) error {
delete(f.chunks, path)
f.deleted = append(f.deleted, path)
return nil
}
type fakeEmbedder struct{ seen []string }
func (e *fakeEmbedder) Embed(_ context.Context, text string) ([]float32, error) {
e.seen = append(e.seen, text)
return []float32{1, 0, 0}, nil
}
// TestBrainUpdateReembedsNewContent proves the supersede contract end to
// end against the actual embedding mechanism: brain_update rewrites the
// file, advancing its mtime, and the next vectorstore.Sync pass re-embeds
// the NEW body and drops the stale chunk. No stub of the re-index path.
func TestBrainUpdateReembedsNewContent(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
ctx := context.Background()
callResult(t, toolCall(t, srv, "brain_write", map[string]any{
"content": "# Note\n\nthe OLD distinctive payload\n",
"filename": "n", "wing": "a", "hall": "facts",
}))
store := &fakeVecStore{chunks: map[string][]float32{}}
emb := &fakeEmbedder{}
// First sync embeds the original content.
_, err := vectorstore.Sync(ctx, brainDir, store, emb)
require.NoError(t, err)
require.NotEmpty(t, store.chunks)
require.True(t, anyContains(emb.seen, "OLD distinctive payload"))
callResult(t, toolCall(t, srv, "brain_update", map[string]any{
"wing": "a", "hall": "facts", "slug": "n",
"content": "# Note\n\nthe NEW distinctive payload\n",
}))
emb.seen = nil // only watch what the second pass embeds
_, err = vectorstore.Sync(ctx, brainDir, store, emb)
require.NoError(t, err)
assert.True(t, anyContains(emb.seen, "NEW distinctive payload"),
"Sync must re-embed the superseded body; saw %v", emb.seen)
assert.False(t, anyContains(emb.seen, "OLD distinctive payload"),
"the old body must not be re-embedded")
assert.NotEmpty(t, store.deleted, "stale chunks must be deleted before re-embed")
}
func anyContains(ss []string, sub string) bool {
for _, s := range ss {
if strings.Contains(s, sub) {
return true
}
}
return false
}
+281 -8
View File
@@ -4,12 +4,15 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"fmt" "fmt"
"log/slog"
"path/filepath" "path/filepath"
"strings" "strings"
"time" "time"
"github.com/mathiasbq/hyperguild/ingestion/internal/api" "github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/extract" "github.com/mathiasbq/hyperguild/ingestion/internal/extract"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphsync"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline" "github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
"github.com/mathiasbq/hyperguild/ingestion/internal/search" "github.com/mathiasbq/hyperguild/ingestion/internal/search"
"github.com/mathiasbq/hyperguild/ingestion/internal/session" "github.com/mathiasbq/hyperguild/ingestion/internal/session"
@@ -24,6 +27,10 @@ func (s *Server) tools() []map[string]any {
int_ := func(desc string) map[string]any { int_ := func(desc string) map[string]any {
return map[string]any{"type": "integer", "description": desc} return map[string]any{"type": "integer", "description": desc}
} }
enum := func(desc string, vals ...string) map[string]any {
return map[string]any{"type": "string", "description": desc, "enum": vals}
}
halls := []string{"facts", "decisions", "failures", "hypotheses", "sources"}
schema := func(required []string, props map[string]any) json.RawMessage { schema := func(required []string, props map[string]any) json.RawMessage {
b, _ := json.Marshal(map[string]any{ b, _ := json.Marshal(map[string]any{
"type": "object", "required": required, "properties": props, "type": "object", "required": required, "properties": props,
@@ -34,20 +41,59 @@ func (s *Server) tools() []map[string]any {
return []map[string]any{ return []map[string]any{
{ {
"name": "brain_query", "name": "brain_query",
"description": "BM25 full-text search across brain/knowledge/ and brain/wiki/ markdown files.", "description": "BM25 full-text search across brain/knowledge/ and brain/wiki/ markdown files. Optionally scope by wing (topic domain) and hall (memory type).",
"inputSchema": schema([]string{"query"}, map[string]any{ "inputSchema": schema([]string{"query"}, map[string]any{
"query": str("search terms"), "query": str("search terms"),
"limit": int_("max results, default 5"), "limit": int_("max results, default 5"),
"wing": str("optional wing to scope to, e.g. jepa-fx"),
"hall": enum("optional hall to scope to (requires wing)", halls...),
}), }),
}, },
{ {
"name": "brain_write", "name": "brain_write",
"description": "Write a raw knowledge note to brain/knowledge/.", "description": "Write a markdown note to the brain. With wing+hall set, routes to brain/wiki/<wing>/<hall>/ with wing/hall/created_at frontmatter; otherwise writes to brain/knowledge/ (legacy).",
"inputSchema": schema([]string{"content"}, map[string]any{ "inputSchema": schema([]string{"content"}, map[string]any{
"content": str("markdown content"), "content": str("markdown content"),
"filename": str("optional filename"), "filename": str("optional filename or slug"),
"type": str("optional frontmatter type"), "type": str("optional frontmatter type (legacy)"),
"domain": str("optional frontmatter domain"), "domain": str("optional frontmatter domain (legacy)"),
"wing": str("optional topic domain, e.g. jepa-fx"),
"hall": enum("optional memory type (requires wing)", halls...),
}),
},
{
"name": "brain_update",
"description": "Supersede an existing brain note in place: whole-note body replace + frontmatter re-stamp (updated_at, supersedes=prior content hash, supersede_reason). Errors if the target does not exist — use brain_write to create. Returns {id, path, content_hash, superseded}. Prior version recoverable from git.",
"inputSchema": schema([]string{"content"}, map[string]any{
"content": str("new full body (whole-note replace)"),
"slug": str("target note slug within wing/hall, OR a full brain-relative path (e.g. wiki/jepa-fx/facts/x.md)"),
"wing": str("wing of the target (required unless slug/path is a full path)"),
"hall": enum("hall of the target (required unless slug/path is a full path)", halls...),
"path": str("full brain-relative path to the target; takes precedence over slug/wing/hall"),
"reason": str("optional short note on why superseded — stamped into frontmatter"),
}),
},
{
"name": "brain_get",
"description": "Fetch a single brain note by id or path (both are the brain-relative path — the note handle). Returns {id, path, content_hash, frontmatter, body}. Read-after-write confirmation without a lexical re-query.",
"inputSchema": schema([]string{}, map[string]any{
"id": str("note id (brain-relative path) as returned by brain_write/brain_update"),
"path": str("brain-relative path to the note; equivalent to id"),
}),
},
{
"name": "brain_tunnel",
"description": "Create an explicit bidirectional [[wikilink]] between two notes in different wings. Idempotent.",
"inputSchema": schema([]string{"source", "target"}, map[string]any{
"source": str("path of source note relative to brain dir, e.g. wiki/jepa-fx/decisions/val-vol.md"),
"target": str("path of target note (must be in a different wing)"),
}),
},
{
"name": "brain_index",
"description": "Regenerate _index.md (Map of Content) for one or all wings under brain/wiki/. Auto-called after brain_write with wing+hall.",
"inputSchema": schema([]string{}, map[string]any{
"wing": str("optional wing to index; if absent, rebuilds every wing"),
}), }),
}, },
{ {
@@ -83,6 +129,32 @@ func (s *Server) tools() []map[string]any {
"text": str("raw document text to classify (first 3000 chars used)"), "text": str("raw document text to classify (first 3000 chars used)"),
}), }),
}, },
{
"name": "brain_graph",
"description": "Query the brain knowledge graph (entities + wikilink edges). Op selects the traversal: neighbors (1-hop outgoing from slug), subgraph (every reachable slug within depth hops), or path (shortest directed path src→dst). Returns slug + entity metadata + edge_type + hop distance.",
"inputSchema": schema([]string{"op"}, map[string]any{
"op": enum("traversal kind", "neighbors", "subgraph", "path"),
"slug": str("origin slug for op=neighbors or op=subgraph"),
"src": str("source slug for op=path"),
"dst": str("destination slug for op=path"),
"edge_type": str("optional edge type filter for op=neighbors (e.g. wikilink); empty matches all"),
"limit": int_("max neighbors to return for op=neighbors, default 25"),
"depth": int_("max traversal depth for op=subgraph (default 2, clamped to 6) and op=path (default 4, clamped to 8)"),
}),
},
{
"name": "brain_context",
"description": "Return top-N relevant brain entries for a project context. Use at session start or before a complex task to load prior decisions, corrections, and surprises.",
"inputSchema": schema([]string{"project_root"}, map[string]any{
"project_root": str("absolute path to the project root"),
"recent_files": map[string]any{
"type": "array",
"items": map[string]any{"type": "string"},
"description": "optional: recent file paths in the project to bias relevance",
},
"limit": int_("max entries to return, default 10"),
}),
},
{ {
"name": "session_log", "name": "session_log",
"description": "Append a structured entry to brain/sessions/<session_id>.jsonl.", "description": "Append a structured entry to brain/sessions/<session_id>.jsonl.",
@@ -104,6 +176,8 @@ func (s *Server) tools() []map[string]any {
type brainQueryArgs struct { type brainQueryArgs struct {
Query string `json:"query"` Query string `json:"query"`
Limit int `json:"limit,omitempty"` Limit int `json:"limit,omitempty"`
Wing string `json:"wing,omitempty"`
Hall string `json:"hall,omitempty"`
} }
func (s *Server) brainQuery(ctx context.Context, args json.RawMessage) (json.RawMessage, error) { func (s *Server) brainQuery(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
@@ -117,7 +191,14 @@ func (s *Server) brainQuery(ctx context.Context, args json.RawMessage) (json.Raw
if a.Limit == 0 { if a.Limit == 0 {
a.Limit = 5 a.Limit = 5
} }
results, err := search.Query(s.brainDir, a.Query, a.Limit) results, err := search.QueryContext(ctx, s.brainDir, search.QueryOptions{
Query: a.Query,
Limit: a.Limit,
Wing: a.Wing,
Hall: a.Hall,
Vector: s.vector,
Embedder: s.embedder,
})
if err != nil { if err != nil {
return nil, fmt.Errorf("search: %w", err) return nil, fmt.Errorf("search: %w", err)
} }
@@ -129,6 +210,8 @@ type brainWriteArgs struct {
Filename string `json:"filename,omitempty"` Filename string `json:"filename,omitempty"`
Type string `json:"type,omitempty"` Type string `json:"type,omitempty"`
Domain string `json:"domain,omitempty"` Domain string `json:"domain,omitempty"`
Wing string `json:"wing,omitempty"`
Hall string `json:"hall,omitempty"`
} }
func (s *Server) brainWrite(ctx context.Context, args json.RawMessage) (json.RawMessage, error) { func (s *Server) brainWrite(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
@@ -136,11 +219,191 @@ func (s *Server) brainWrite(ctx context.Context, args json.RawMessage) (json.Raw
if err := json.Unmarshal(args, &a); err != nil { if err := json.Unmarshal(args, &a); err != nil {
return nil, fmt.Errorf("parse args: %w", err) return nil, fmt.Errorf("parse args: %w", err)
} }
relPath, err := api.WriteNote(s.brainDir, a.Content, a.Filename, a.Type, a.Domain) relPath, err := api.WriteNote(s.brainDir, api.WriteNoteOptions{
Content: a.Content,
Filename: a.Filename,
Type: a.Type,
Domain: a.Domain,
Wing: a.Wing,
Hall: a.Hall,
})
if err != nil { if err != nil {
return nil, err return nil, err
} }
return json.Marshal(map[string]string{"path": relPath}) // Auto-regenerate the wing _index.md when the write landed in the
// structured wiki, and auto-tunnel cross-wing matches. Both are
// best-effort: the note is already written.
if a.Wing != "" && a.Hall != "" {
if err := brain.BuildWingIndex(s.brainDir, a.Wing); err != nil {
slog.Warn("brain_write: auto-index failed", "wing", a.Wing, "err", err)
}
if err := brain.AutoTunnel(s.brainDir, relPath, a.Content); err != nil {
slog.Warn("brain_write: auto-tunnel failed", "src", relPath, "err", err)
}
}
s.indexInGraph(ctx, "brain_write", relPath)
// Read-after-write handle: id == relPath, content_hash == sha256 of
// the bytes just written. path is kept for backward compatibility.
_, _, hash, _ := api.ReadNote(s.brainDir, relPath)
return json.Marshal(map[string]string{"id": relPath, "path": relPath, "content_hash": hash})
}
type brainUpdateArgs struct {
Slug string `json:"slug,omitempty"`
Wing string `json:"wing,omitempty"`
Hall string `json:"hall,omitempty"`
Path string `json:"path,omitempty"`
Content string `json:"content"`
Reason string `json:"reason,omitempty"`
}
// brainUpdate supersedes an existing note in place: whole-note body
// replace, frontmatter re-stamp (updated_at/supersedes/supersede_reason),
// graph re-index, and wing _index rebuild. It never creates — a missing
// target is an error so the caller can fall back to brain_write.
//
// Embedding re-sync is delegated to the out-of-band vectorstore.Sync
// ticker: the rewritten file's mtime advances, so the next pass re-embeds
// it. This mirrors brain_write, which likewise does not embed in-handler.
func (s *Server) brainUpdate(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
var a brainUpdateArgs
if err := json.Unmarshal(args, &a); err != nil {
return nil, fmt.Errorf("parse args: %w", err)
}
if a.Content == "" {
return nil, fmt.Errorf("content is required")
}
opts := api.UpdateNoteOptions{Content: a.Content, Reason: a.Reason}
switch {
case a.Path != "":
opts.Path = a.Path
case strings.Contains(a.Slug, "/"):
// slug carries a full path (issue #45: "slug ... OR full path").
opts.Path = a.Slug
default:
opts.Wing, opts.Hall, opts.Slug = a.Wing, a.Hall, a.Slug
}
relPath, hash, _, err := api.UpdateNote(s.brainDir, opts)
if err != nil {
return nil, err
}
// Best-effort wiki upkeep, mirroring brain_write: rebuild the wing
// _index and re-tunnel cross-wing matches against the new body. Both
// are idempotent and never block — the note is already superseded.
if wing := wingFromRelPath(relPath); wing != "" {
if err := brain.BuildWingIndex(s.brainDir, wing); err != nil {
slog.Warn("brain_update: auto-index failed", "wing", wing, "err", err)
}
if err := brain.AutoTunnel(s.brainDir, relPath, a.Content); err != nil {
slog.Warn("brain_update: auto-tunnel failed", "src", relPath, "err", err)
}
}
s.indexInGraph(ctx, "brain_update", relPath)
return json.Marshal(map[string]any{
"id": relPath, "path": relPath, "content_hash": hash, "superseded": true,
})
}
// wingFromRelPath extracts the wing segment from a structured wiki path
// (wiki/<wing>/<hall>/<slug>.md). Returns "" for legacy/non-wiki paths.
func wingFromRelPath(relPath string) string {
parts := strings.Split(relPath, "/")
if len(parts) >= 4 && parts[0] == "wiki" {
return parts[1]
}
return ""
}
type brainGetArgs struct {
ID string `json:"id,omitempty"`
Path string `json:"path,omitempty"`
}
// brainGet fetches a note by id or path (both are the brainDir-relative
// path — the de-facto handle). Read-only; the create-path read-after-
// write primitive that lets callers confirm a write landed without a
// lexical re-query.
func (s *Server) brainGet(_ context.Context, args json.RawMessage) (json.RawMessage, error) {
var a brainGetArgs
if err := json.Unmarshal(args, &a); err != nil {
return nil, fmt.Errorf("parse args: %w", err)
}
target := a.Path
if target == "" {
target = a.ID
}
if target == "" {
return nil, fmt.Errorf("id or path is required")
}
fm, body, hash, err := api.ReadNote(s.brainDir, target)
if err != nil {
return nil, err
}
return json.Marshal(map[string]any{
"id": target, "path": target, "content_hash": hash,
"frontmatter": fm, "body": body,
})
}
// indexInGraph is a best-effort wrapper around graphsync.IndexDoc that
// logs failures but never propagates them — the underlying write/ingest
// has already succeeded and the graph is an augmentation, not a
// correctness invariant.
func (s *Server) indexInGraph(ctx context.Context, op, relPath string) {
if s.graph == nil || relPath == "" {
return
}
if err := graphsync.IndexDoc(ctx, s.graph, s.brainDir, relPath); err != nil {
slog.Warn(op+": graph index failed", "path", relPath, "err", err)
}
}
type brainTunnelArgs struct {
Source string `json:"source"`
Target string `json:"target"`
}
func (s *Server) brainTunnel(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
var a brainTunnelArgs
if err := json.Unmarshal(args, &a); err != nil {
return nil, fmt.Errorf("parse args: %w", err)
}
if a.Source == "" || a.Target == "" {
return nil, fmt.Errorf("source and target are required")
}
if err := brain.WriteTunnel(s.brainDir, a.Source, a.Target); err != nil {
return nil, fmt.Errorf("tunnel: %w", err)
}
s.indexInGraph(ctx, "brain_tunnel", a.Source)
s.indexInGraph(ctx, "brain_tunnel", a.Target)
return json.Marshal(map[string]string{"status": "ok"})
}
type brainIndexArgs struct {
Wing string `json:"wing,omitempty"`
}
func (s *Server) brainIndex(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
var a brainIndexArgs
if len(args) > 0 {
if err := json.Unmarshal(args, &a); err != nil {
return nil, fmt.Errorf("parse args: %w", err)
}
}
if a.Wing == "" {
if err := brain.BuildAllWingIndexes(s.brainDir); err != nil {
return nil, fmt.Errorf("index: %w", err)
}
return json.Marshal(map[string]any{"status": "ok", "scope": "all"})
}
if err := brain.BuildWingIndex(s.brainDir, a.Wing); err != nil {
return nil, fmt.Errorf("index: %w", err)
}
return json.Marshal(map[string]any{"status": "ok", "scope": a.Wing})
} }
type brainIngestRawArgs struct { type brainIngestRawArgs struct {
@@ -172,6 +435,11 @@ func (s *Server) brainIngestRaw(ctx context.Context, args json.RawMessage) (json
if warnings == nil { if warnings == nil {
warnings = []string{} warnings = []string{}
} }
if !a.DryRun {
for _, p := range pages {
s.indexInGraph(ctx, "brain_ingest_raw", p)
}
}
return json.Marshal(map[string]any{"pages": pages, "warnings": warnings}) return json.Marshal(map[string]any{"pages": pages, "warnings": warnings})
} }
@@ -262,6 +530,11 @@ func (s *Server) runIngest(ctx context.Context, content, source string, dryRun b
if pages == nil { if pages == nil {
pages = []string{} pages = []string{}
} }
if !dryRun {
for _, p := range pages {
s.indexInGraph(ctx, "brain_ingest", p)
}
}
warnings := result.Warnings warnings := result.Warnings
if warnings == nil { if warnings == nil {
warnings = []string{} warnings = []string{}
+138
View File
@@ -70,6 +70,144 @@ func TestBrainWriteCreatesFile(t *testing.T) {
assert.Contains(t, string(got), "# Test") assert.Contains(t, string(got), "# Test")
} }
func TestBrainWriteWingHallRoutesToWiki(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
resp := toolCall(t, srv, "brain_write", map[string]any{
"content": "# Val Vol\n\nbody",
"filename": "val-vol-r2",
"wing": "jepa-fx",
"hall": "decisions",
})
require.Nil(t, resp["error"])
got, err := os.ReadFile(filepath.Join(brainDir, "wiki", "jepa-fx", "decisions", "val-vol-r2.md"))
require.NoError(t, err)
assert.Contains(t, string(got), "wing: jepa-fx")
assert.Contains(t, string(got), "hall: decisions")
assert.Contains(t, string(got), "created_at:")
assert.Contains(t, string(got), "# Val Vol")
}
func TestBrainWriteRejectsInvalidHall(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
resp := toolCall(t, srv, "brain_write", map[string]any{
"content": "x",
"wing": "jepa-fx",
"hall": "garbage",
})
require.NotNil(t, resp["error"])
}
func TestBrainQueryWingScope(t *testing.T) {
brainDir := t.TempDir()
for _, p := range []struct{ rel, body string }{
{"wiki/jepa-fx/facts/x.md", "---\nwing: jepa-fx\nhall: facts\n---\nfoo keyword.\n"},
{"wiki/other/facts/y.md", "---\nwing: other\nhall: facts\n---\nfoo keyword.\n"},
} {
full := filepath.Join(brainDir, p.rel)
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
require.NoError(t, os.WriteFile(full, []byte(p.body), 0o644))
}
srv := mcp.NewServer(brainDir, nil, nil, nil)
resp := toolCall(t, srv, "brain_query", map[string]any{
"query": "foo",
"wing": "jepa-fx",
})
require.Nil(t, resp["error"])
text := resp["result"].(map[string]any)["content"].([]any)[0].(map[string]any)["text"].(string)
assert.Contains(t, text, "wiki/jepa-fx/facts/x.md")
assert.NotContains(t, text, "wiki/other/facts/y.md")
}
func TestBrainWriteAutoTunnelsOnExactMatch(t *testing.T) {
brainDir := t.TempDir()
// Seed a pre-existing note in wing "other".
existing := filepath.Join(brainDir, "wiki/other/facts/widget.md")
require.NoError(t, os.MkdirAll(filepath.Dir(existing), 0o755))
require.NoError(t, os.WriteFile(existing,
[]byte("---\nwing: other\nhall: facts\ntitle: Widget\n---\nbody.\n"), 0o644))
srv := mcp.NewServer(brainDir, nil, nil, nil)
// Write a new note in a *different* wing whose content references "Widget".
resp := toolCall(t, srv, "brain_write", map[string]any{
"content": "# Notes\n\nThis note discusses the Widget concept.\n",
"filename": "notes",
"wing": "jepa-fx",
"hall": "facts",
})
require.Nil(t, resp["error"])
newNote := filepath.Join(brainDir, "wiki/jepa-fx/facts/notes.md")
got, err := os.ReadFile(newNote)
require.NoError(t, err)
assert.Contains(t, string(got), "[[other/facts/widget]]", "new note should link to existing")
gotTgt, err := os.ReadFile(existing)
require.NoError(t, err)
assert.Contains(t, string(gotTgt), "[[jepa-fx/facts/notes]]", "existing note should backlink")
}
func TestBrainWriteAutoTunnelSkipsSameWing(t *testing.T) {
brainDir := t.TempDir()
existing := filepath.Join(brainDir, "wiki/jepa-fx/facts/widget.md")
require.NoError(t, os.MkdirAll(filepath.Dir(existing), 0o755))
require.NoError(t, os.WriteFile(existing,
[]byte("---\nwing: jepa-fx\nhall: facts\ntitle: Widget\n---\nbody.\n"), 0o644))
srv := mcp.NewServer(brainDir, nil, nil, nil)
resp := toolCall(t, srv, "brain_write", map[string]any{
"content": "Same wing reference to Widget here.\n",
"filename": "notes",
"wing": "jepa-fx",
"hall": "facts",
})
require.Nil(t, resp["error"])
newNote := filepath.Join(brainDir, "wiki/jepa-fx/facts/notes.md")
got, err := os.ReadFile(newNote)
require.NoError(t, err)
assert.NotContains(t, string(got), "[[jepa-fx/facts/widget]]", "same-wing match must not auto-tunnel")
}
func TestBrainTunnelLinksTwoNotes(t *testing.T) {
brainDir := t.TempDir()
for _, p := range []struct{ rel, body string }{
{"wiki/jepa-fx/decisions/val-vol.md", "---\nwing: jepa-fx\nhall: decisions\n---\n# Val Vol\n"},
{"wiki/hyperguild/decisions/routing.md", "---\nwing: hyperguild\nhall: decisions\n---\n# Routing\n"},
} {
full := filepath.Join(brainDir, p.rel)
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
require.NoError(t, os.WriteFile(full, []byte(p.body), 0o644))
}
srv := mcp.NewServer(brainDir, nil, nil, nil)
resp := toolCall(t, srv, "brain_tunnel", map[string]any{
"source": "wiki/jepa-fx/decisions/val-vol.md",
"target": "wiki/hyperguild/decisions/routing.md",
})
require.Nil(t, resp["error"])
src, err := os.ReadFile(filepath.Join(brainDir, "wiki/jepa-fx/decisions/val-vol.md"))
require.NoError(t, err)
assert.Contains(t, string(src), "[[hyperguild/decisions/routing]]")
tgt, err := os.ReadFile(filepath.Join(brainDir, "wiki/hyperguild/decisions/routing.md"))
require.NoError(t, err)
assert.Contains(t, string(tgt), "[[jepa-fx/decisions/val-vol]]")
}
func TestBrainTunnelRejectsMissing(t *testing.T) {
brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil)
resp := toolCall(t, srv, "brain_tunnel", map[string]any{
"source": "wiki/a/facts/ghost.md",
"target": "wiki/b/facts/ghost.md",
})
require.NotNil(t, resp["error"])
}
func TestBrainWriteRejectsTraversal(t *testing.T) { func TestBrainWriteRejectsTraversal(t *testing.T) {
brainDir := t.TempDir() brainDir := t.TempDir()
srv := mcp.NewServer(brainDir, nil, nil, nil) srv := mcp.NewServer(brainDir, nil, nil, nil)
+54 -1
View File
@@ -1,5 +1,7 @@
// Package mcp implements an MCP HTTP handler for the ingestion service. // Package mcp implements an MCP HTTP handler for the ingestion service.
// Exposed tools: brain_query, brain_write, brain_ingest, brain_ingest_raw, session_log. // Exposed tools: brain_query, brain_write, brain_update, brain_get,
// brain_index, brain_tunnel, brain_ingest, brain_ingest_raw,
// brain_answer, brain_classify, brain_graph, brain_context, session_log.
package mcp package mcp
import ( import (
@@ -8,7 +10,11 @@ import (
"fmt" "fmt"
"net/http" "net/http"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphsync"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline" "github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
"github.com/mathiasbq/hyperguild/ingestion/internal/reranker"
"github.com/mathiasbq/hyperguild/ingestion/internal/search"
) )
type request struct { type request struct {
@@ -36,6 +42,10 @@ type Server struct {
pipeline pipeline.Config pipeline pipeline.Config
llm pipeline.CompleteFunc llm pipeline.CompleteFunc
answerLLM pipeline.CompleteFunc // nil = brain_answer and brain_classify unavailable answerLLM pipeline.CompleteFunc // nil = brain_answer and brain_classify unavailable
reranker *reranker.Client // nil = no rerank, BM25 top-10 → LLM
vector search.VectorSearcher // nil = BM25-only retrieval
embedder search.Embedder // nil = BM25-only retrieval
graph graphsync.Store // nil = brain_graph and GraphRAG augmentation disabled
} }
// NewServer constructs a Server bound to brainDir. pipelineCfg supplies the // NewServer constructs a Server bound to brainDir. pipelineCfg supplies the
@@ -49,6 +59,37 @@ func NewServer(brainDir string, pipelineCfg *pipeline.Config, llm pipeline.Compl
return &Server{brainDir: brainDir, pipeline: cfg, llm: llm, answerLLM: answerLLM} return &Server{brainDir: brainDir, pipeline: cfg, llm: llm, answerLLM: answerLLM}
} }
// WithReranker installs an opt-in cross-encoder reranker. When set,
// brain_answer retrieves a wider BM25 candidate set and prunes it to
// the relevant ones before LLM synthesis. Returns the server for
// fluent chaining.
func (s *Server) WithReranker(r *reranker.Client) *Server {
s.reranker = r
return s
}
// WithHybridRetrieval wires the embedding store and embedder so
// brain_query and brain_answer run BM25 + pgvector merged via RRF
// instead of BM25 alone. Either nil disables hybrid mode.
func (s *Server) WithHybridRetrieval(v search.VectorSearcher, e search.Embedder) *Server {
s.vector = v
s.embedder = e
return s
}
// WithGraph wires the brain entities + edges store so every successful
// brain_write / brain_ingest / brain_tunnel re-indexes its written docs
// into the graph, and so brain_graph + GraphRAG-augmented brain_answer
// are available. nil disables graph features and is the legacy default.
func (s *Server) WithGraph(g *graphstore.PGStore) *Server {
if g == nil {
s.graph = nil
return s
}
s.graph = g
return s
}
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) { func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// MCP streamable HTTP: GET establishes the SSE stream for server-to-client events. // MCP streamable HTTP: GET establishes the SSE stream for server-to-client events.
if r.Method == http.MethodGet { if r.Method == http.MethodGet {
@@ -136,6 +177,14 @@ func (s *Server) handleCall(ctx context.Context, name string, args json.RawMessa
return s.brainQuery(ctx, args) return s.brainQuery(ctx, args)
case "brain_write": case "brain_write":
return s.brainWrite(ctx, args) return s.brainWrite(ctx, args)
case "brain_update":
return s.brainUpdate(ctx, args)
case "brain_get":
return s.brainGet(ctx, args)
case "brain_index":
return s.brainIndex(ctx, args)
case "brain_tunnel":
return s.brainTunnel(ctx, args)
case "brain_ingest_raw": case "brain_ingest_raw":
return s.brainIngestRaw(ctx, args) return s.brainIngestRaw(ctx, args)
case "brain_ingest": case "brain_ingest":
@@ -146,6 +195,10 @@ func (s *Server) handleCall(ctx context.Context, name string, args json.RawMessa
return s.brainAnswer(ctx, args) return s.brainAnswer(ctx, args)
case "brain_classify": case "brain_classify":
return s.brainClassify(ctx, args) return s.brainClassify(ctx, args)
case "brain_graph":
return s.brainGraph(ctx, args)
case "brain_context":
return s.brainContext(ctx, args)
default: default:
return nil, fmt.Errorf("unknown tool: %s", name) return nil, fmt.Errorf("unknown tool: %s", name)
} }
+5 -2
View File
@@ -55,8 +55,11 @@ func TestServerToolsList(t *testing.T) {
names = append(names, t.(map[string]any)["name"].(string)) names = append(names, t.(map[string]any)["name"].(string))
} }
assert.ElementsMatch(t, []string{ assert.ElementsMatch(t, []string{
"brain_query", "brain_write", "brain_ingest_raw", "brain_ingest", "brain_query", "brain_write", "brain_update", "brain_get",
"brain_answer", "brain_classify", "session_log", "brain_index", "brain_tunnel",
"brain_ingest_raw", "brain_ingest",
"brain_answer", "brain_classify", "brain_graph", "brain_context",
"session_log",
}, names) }, names)
} }
+99 -1
View File
@@ -6,9 +6,35 @@ import (
"fmt" "fmt"
"strings" "strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/reranker"
"github.com/mathiasbq/hyperguild/ingestion/internal/search" "github.com/mathiasbq/hyperguild/ingestion/internal/search"
) )
// rerankResults scores each candidate's excerpt against the query and
// returns up to top results whose score is positive, preserving the
// caller's input order (BM25 rank) within the kept set. The reranker is
// a filter: ties are broken by BM25, not by the reranker's binary score.
func rerankResults(ctx context.Context, rr *reranker.Client, query string, results []search.Result, top int) ([]search.Result, error) {
docs := make([]string, len(results))
for i, r := range results {
docs[i] = r.Excerpt
}
scores, err := rr.Score(ctx, query, docs)
if err != nil {
return nil, err
}
kept := make([]search.Result, 0, top)
for i, r := range results {
if scores[i] > 0 {
kept = append(kept, r)
}
if len(kept) == top {
break
}
}
return kept, nil
}
const ( const (
answerSystemPrompt = `You are a knowledge assistant. Answer the question using ONLY the provided sources. answerSystemPrompt = `You are a knowledge assistant. Answer the question using ONLY the provided sources.
Cite source file paths inline when referencing specific content. Cite source file paths inline when referencing specific content.
@@ -35,10 +61,40 @@ func (s *Server) brainAnswer(ctx context.Context, args json.RawMessage) (json.Ra
return nil, fmt.Errorf("query is required") return nil, fmt.Errorf("query is required")
} }
results, err := search.Query(s.brainDir, a.Query, 10) // With reranker disabled: BM25 top-10 straight to the LLM.
// With reranker enabled: BM25 top-20 → cross-encoder filter → top-5.
bm25Limit := 10
if s.reranker != nil {
bm25Limit = 20
}
results, err := search.QueryContext(ctx, s.brainDir, search.QueryOptions{
Query: a.Query,
Limit: bm25Limit,
Vector: s.vector,
Embedder: s.embedder,
})
if err != nil { if err != nil {
return nil, fmt.Errorf("search: %w", err) return nil, fmt.Errorf("search: %w", err)
} }
if s.reranker != nil && len(results) > 0 {
reranked, rerr := rerankResults(ctx, s.reranker, a.Query, results, 5)
if rerr != nil {
return nil, fmt.Errorf("rerank: %w", rerr)
}
// The reranker is a filter, not a gate. The Qwen3-Reranker is a
// web-search cross-encoder: against a conversational / personal-
// intent query ("what am I optimizing toward?") it scores even
// on-topic notes as "no", which would collapse the whole answer to
// "no relevant content" despite BM25 having retrieved relevant
// content. When the reranker keeps nothing, fall back to the
// BM25/vector ordering (capped to the no-reranker depth) rather
// than returning an empty answer.
if len(reranked) > 0 {
results = reranked
} else if len(results) > 10 {
results = results[:10]
}
}
if len(results) == 0 { if len(results) == 0 {
return json.Marshal(map[string]any{ return json.Marshal(map[string]any{
"answer": "No relevant content found in brain.", "answer": "No relevant content found in brain.",
@@ -53,6 +109,29 @@ func (s *Server) brainAnswer(ctx context.Context, args json.RawMessage) (json.Ra
sources = append(sources, r.Path) sources = append(sources, r.Path)
} }
// GraphRAG augmentation: when the graph is wired, attach the 1-hop
// outgoing neighbourhood of the top BM25/rerank hit as an extra
// context block. The LLM can ignore it when irrelevant; when the
// neighbour adds signal we don't need a second retrieval pass.
// Failures are silently skipped — graph is augmentation, not
// correctness.
if reader, ok := s.graph.(graphReader); ok && len(results) > 0 {
topSlug := slugFromPath(results[0].Path)
if topSlug != "" {
if ns, gerr := reader.Subgraph(ctx, topSlug, 1); gerr == nil && len(ns) > 0 {
sb.WriteString("<related>\n")
for _, n := range ns {
label := n.Title
if label == "" {
label = n.Slug
}
fmt.Fprintf(&sb, "- %s (%s) at %s\n", label, n.EdgeType, n.DocPath)
}
sb.WriteString("</related>\n\n")
}
}
}
answer, err := s.answerLLM(ctx, answerSystemPrompt, sb.String()+"Question: "+a.Query) answer, err := s.answerLLM(ctx, answerSystemPrompt, sb.String()+"Question: "+a.Query)
if err != nil { if err != nil {
return nil, fmt.Errorf("llm: %w", err) return nil, fmt.Errorf("llm: %w", err)
@@ -64,6 +143,25 @@ func (s *Server) brainAnswer(ctx context.Context, args json.RawMessage) (json.Ra
}) })
} }
// slugFromPath converts "wiki/concepts/foo.md" → "foo".
// Returns "" when path has no .md suffix or empty basename.
func slugFromPath(path string) string {
if path == "" {
return ""
}
// strip directory
for i := len(path) - 1; i >= 0; i-- {
if path[i] == '/' {
path = path[i+1:]
break
}
}
if !strings.HasSuffix(path, ".md") {
return ""
}
return strings.TrimSuffix(path, ".md")
}
type brainClassifyArgs struct { type brainClassifyArgs struct {
Text string `json:"text"` Text string `json:"text"`
} }
@@ -3,14 +3,17 @@ package mcp_test
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"io"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"os" "os"
"path/filepath" "path/filepath"
"strings"
"testing" "testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/mcp" "github.com/mathiasbq/hyperguild/ingestion/internal/mcp"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline" "github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
"github.com/mathiasbq/hyperguild/ingestion/internal/reranker"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
@@ -46,6 +49,91 @@ func callTool(t *testing.T, ts *httptest.Server, name string, arguments map[stri
return out return out
} }
func TestBrainAnswer_RerankerFiltersBeforeLLM(t *testing.T) {
brainDir := t.TempDir()
wikiDir := filepath.Join(brainDir, "wiki")
require.NoError(t, os.MkdirAll(wikiDir, 0o755))
// Two notes — both BM25-match the query, but only one is truly relevant.
require.NoError(t, os.WriteFile(filepath.Join(wikiDir, "good.md"), []byte(
"---\ntitle: Pass-rate Logging\n---\nPass-rate logging tracks skill invocations.",
), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(wikiDir, "noise.md"), []byte(
"---\ntitle: Pass-rate Tangent\n---\nPass-rate appears here too but as a tangent.",
), 0o644))
// Fake Ollama reranker: yes only when prompt contains "tracks skill invocations".
rrSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
raw, _ := io.ReadAll(r.Body)
yes := strings.Contains(string(raw), "tracks skill invocations")
ans := "no"
if yes {
ans = "yes"
}
_ = json.NewEncoder(w).Encode(map[string]any{"response": ans, "done": true})
}))
defer rrSrv.Close()
// LLM mock captures the rendered sources so we can assert what reached it.
var sawSources string
llm := func(_ context.Context, _, user string) (string, error) {
sawSources = user
return "answer text", nil
}
srv := mcp.NewServer(brainDir, nil, nil, llm).
WithReranker(reranker.New(rrSrv.URL, "qwen3"))
ts := httptest.NewServer(srv)
defer ts.Close()
rpc := callTool(t, ts, "brain_answer", map[string]any{"query": "pass-rate logging"})
require.Nil(t, rpc["error"])
content := rpc["result"].(map[string]any)["content"].([]any)[0].(map[string]any)["text"].(string)
var result map[string]any
require.NoError(t, json.Unmarshal([]byte(content), &result))
sources := result["sources"].([]any)
require.Len(t, sources, 1, "reranker should drop noise.md")
assert.Equal(t, "wiki/good.md", sources[0])
assert.Contains(t, sawSources, "good.md")
assert.NotContains(t, sawSources, "noise.md")
}
func TestBrainAnswer_RerankerKeepsNone_FallsBackToBM25(t *testing.T) {
brainDir := brainDirWithContent(t) // test.md BM25-matches "pass-rate logging"
// Reranker rejects every candidate ("no" to all) — models a
// web-search cross-encoder facing a conversational / personal-intent
// query, which is exactly when it wrongly scores on-topic notes as
// irrelevant. The answer must still synthesize from the BM25 hits, not
// collapse to "no relevant content".
rrSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{"response": "no", "done": true})
}))
defer rrSrv.Close()
var sawSources string
llm := func(_ context.Context, _, user string) (string, error) {
sawSources = user
return "fallback answer", nil
}
srv := mcp.NewServer(brainDir, nil, nil, llm).
WithReranker(reranker.New(rrSrv.URL, "qwen3"))
ts := httptest.NewServer(srv)
defer ts.Close()
rpc := callTool(t, ts, "brain_answer", map[string]any{"query": "pass-rate logging"})
require.Nil(t, rpc["error"])
content := rpc["result"].(map[string]any)["content"].([]any)[0].(map[string]any)["text"].(string)
var result map[string]any
require.NoError(t, json.Unmarshal([]byte(content), &result))
assert.Equal(t, "fallback answer", result["answer"])
assert.NotEmpty(t, result["sources"], "reranker keeping nothing must fall back to BM25, not empty")
assert.Contains(t, sawSources, "test.md")
}
func TestBrainAnswer_NoLLM(t *testing.T) { func TestBrainAnswer_NoLLM(t *testing.T) {
srv := mcp.NewServer(t.TempDir(), nil, nil, nil) srv := mcp.NewServer(t.TempDir(), nil, nil, nil)
ts := httptest.NewServer(srv) ts := httptest.NewServer(srv)
+202
View File
@@ -0,0 +1,202 @@
package mcp
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/search"
)
// brainContextArgs is the input shape of brain_context. project_root is
// required; recent_files biases ranking when provided; limit caps the
// returned set (default 10).
type brainContextArgs struct {
ProjectRoot string `json:"project_root"`
RecentFiles []string `json:"recent_files,omitempty"`
Limit int `json:"limit,omitempty"`
}
// contextEntry is one returned brain entry: the slug, its title,
// frontmatter-stripped excerpt, source (bm25|graph), and a final score
// used for ranking before truncation to Limit.
type contextEntry struct {
Slug string `json:"slug"`
Title string `json:"title"`
DocPath string `json:"doc_path"`
Excerpt string `json:"excerpt"`
EdgeType string `json:"edge_type"`
Score float64 `json:"score"`
}
// brainContext returns top-N brain entries relevant to a project context.
// It runs a BM25 query against the project name, takes the top-3 hits as
// seeds, expands each seed 2 hops in the brain graph (when configured),
// then merges and deduplicates by slug. recent_files optionally boosts
// entries whose doc_path matches a recent file basename.
func (s *Server) brainContext(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
var a brainContextArgs
if err := json.Unmarshal(args, &a); err != nil {
return nil, fmt.Errorf("parse args: %w", err)
}
if a.ProjectRoot == "" {
return nil, fmt.Errorf("project_root is required")
}
limit := a.Limit
if limit <= 0 {
limit = 10
}
projectName := filepath.Base(strings.TrimRight(a.ProjectRoot, "/"))
if projectName == "" || projectName == "." || projectName == "/" {
return nil, fmt.Errorf("project_root has no usable basename: %q", a.ProjectRoot)
}
// Seed BM25 hits on the project name. Take top-3 as graph expansion seeds.
bm25, err := search.QueryContext(ctx, s.brainDir, search.QueryOptions{
Query: projectName,
Limit: 3,
Vector: s.vector,
Embedder: s.embedder,
})
if err != nil {
return nil, fmt.Errorf("search: %w", err)
}
// Dedup by slug while merging BM25 hits and graph neighbours.
bySlug := make(map[string]*contextEntry)
// BM25 score: highest rank gets the largest score, decaying linearly.
// Score 3.0 / 2.0 / 1.0 for ranks 0/1/2 respectively.
for i, r := range bm25 {
slug := slugFromPath(r.Path)
if slug == "" {
continue
}
score := float64(len(bm25) - i)
bySlug[slug] = &contextEntry{
Slug: slug,
Title: r.Title,
DocPath: r.Path,
Excerpt: truncateExcerpt(r.Excerpt, 200),
EdgeType: "bm25",
Score: score,
}
}
// Graph expansion: for each BM25 hit, fetch its 2-hop subgraph and
// merge those neighbours in with a graph score that decays with hop
// distance. Failures are silently dropped — graph augmentation is
// best-effort.
if reader, ok := s.graph.(graphReader); ok {
for _, r := range bm25 {
seed := slugFromPath(r.Path)
if seed == "" {
continue
}
ns, gerr := reader.Subgraph(ctx, seed, 2)
if gerr != nil {
continue
}
for _, n := range ns {
if n.Slug == "" || n.Slug == seed {
continue
}
// Graph score: closer hops carry more signal. Distance 1
// scores 0.6, distance 2 scores 0.3.
gscore := 0.6 / float64(max1(n.Distance))
if existing, ok := bySlug[n.Slug]; ok {
// Already surfaced via BM25 — bump its score so that
// BM25 + graph evidence outranks BM25-only hits.
existing.Score += gscore
continue
}
bySlug[n.Slug] = &contextEntry{
Slug: n.Slug,
Title: n.Title,
DocPath: n.DocPath,
Excerpt: readExcerpt(s.brainDir, n.DocPath, 200),
EdgeType: "graph",
Score: gscore,
}
}
}
}
// Optional recent_files boost: +1 to entries whose doc_path basename
// matches any recent file basename. v1 is intentionally simple.
if len(a.RecentFiles) > 0 {
recent := make(map[string]struct{}, len(a.RecentFiles))
for _, f := range a.RecentFiles {
recent[filepath.Base(f)] = struct{}{}
}
for _, e := range bySlug {
if _, hit := recent[filepath.Base(e.DocPath)]; hit {
e.Score += 1.0
}
}
}
// Flatten and sort by score desc, slug asc as a stable tiebreaker.
entries := make([]contextEntry, 0, len(bySlug))
for _, e := range bySlug {
entries = append(entries, *e)
}
sort.SliceStable(entries, func(i, j int) bool {
if entries[i].Score != entries[j].Score {
return entries[i].Score > entries[j].Score
}
return entries[i].Slug < entries[j].Slug
})
if len(entries) > limit {
entries = entries[:limit]
}
return json.Marshal(map[string]any{"entries": entries})
}
// truncateExcerpt clamps an already-stripped excerpt to maxLen characters
// without re-running the frontmatter parser. The ellipsis suffix matches
// the convention used in search.excerpt.
func truncateExcerpt(s string, maxLen int) string {
if len(s) <= maxLen {
return s
}
return s[:maxLen] + "…"
}
// readExcerpt loads a doc relative to brainDir, strips its frontmatter,
// and returns the first maxLen chars. Returns "" on any error — the
// excerpt is informational, not load-bearing for correctness.
func readExcerpt(brainDir, relPath string, maxLen int) string {
if relPath == "" {
return ""
}
full := filepath.Join(brainDir, filepath.FromSlash(relPath))
content, err := os.ReadFile(full)
if err != nil {
return ""
}
parts := strings.SplitN(string(content), "---", 3)
body := string(content)
if len(parts) == 3 {
body = strings.TrimSpace(parts[2])
}
if len(body) > maxLen {
return body[:maxLen] + "…"
}
return body
}
// max1 returns the maximum of n and 1, used to guard against divide-by-zero
// on graph distance and to give self-references (distance 0) a sensible
// score instead of an infinity.
func max1(n int) int {
if n < 1 {
return 1
}
return n
}
@@ -0,0 +1,212 @@
package mcp
import (
"context"
"encoding/json"
"os"
"path/filepath"
"sort"
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/graph"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphstore"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// fakeGraph implements graphsync.Store + graphReader so it can be
// assigned to Server.graph and downcast by brainContext. Only Subgraph
// is exercised by brain_context today; the rest are no-op satisfiers.
type fakeGraph struct {
subgraph map[string][]graphstore.Neighbor
}
func (f *fakeGraph) UpsertEntity(_ context.Context, _ graph.Entity) error { return nil }
func (f *fakeGraph) ReplaceEdgesForDoc(_ context.Context, _ string, _ []graph.Edge) error {
return nil
}
func (f *fakeGraph) DeleteByDoc(_ context.Context, _ string) error { return nil }
func (f *fakeGraph) Neighbors(_ context.Context, slug, _ string, _ int) ([]graphstore.Neighbor, error) {
return f.subgraph[slug], nil
}
func (f *fakeGraph) Subgraph(_ context.Context, origin string, _ int) ([]graphstore.Neighbor, error) {
return f.subgraph[origin], nil
}
func (f *fakeGraph) Path(_ context.Context, _, _ string, _ int) ([]graphstore.PathStep, error) {
return nil, nil
}
func writeNote(t *testing.T, brainDir, relPath, title, body string) {
t.Helper()
full := filepath.Join(brainDir, filepath.FromSlash(relPath))
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
content := "---\ntitle: " + title + "\n---\n\n" + body
require.NoError(t, os.WriteFile(full, []byte(content), 0o644))
}
// callContext runs brainContext directly and decodes the JSON response.
func callContext(t *testing.T, s *Server, args map[string]any) map[string]any {
t.Helper()
raw, err := json.Marshal(args)
require.NoError(t, err)
out, err := s.brainContext(context.Background(), raw)
require.NoError(t, err)
var resp map[string]any
require.NoError(t, json.Unmarshal(out, &resp))
return resp
}
func sortedSlugs(entries []any) []string {
slugs := make([]string, 0, len(entries))
for _, e := range entries {
slugs = append(slugs, e.(map[string]any)["slug"].(string))
}
sort.Strings(slugs)
return slugs
}
func TestBrainContext_RejectsMissingProjectRoot(t *testing.T) {
s := NewServer(t.TempDir(), nil, nil, nil)
_, err := s.brainContext(context.Background(), json.RawMessage(`{}`))
assert.Error(t, err)
}
func TestBrainContext_RejectsUnusableBasename(t *testing.T) {
s := NewServer(t.TempDir(), nil, nil, nil)
_, err := s.brainContext(context.Background(), json.RawMessage(`{"project_root":"/"}`))
assert.Error(t, err)
}
func TestBrainContext_BM25Only_NoGraph(t *testing.T) {
brainDir := t.TempDir()
// Two notes whose body contains the hyphenated project name. BM25
// uses literal substring matching after whitespace tokenisation, so
// the bodies must carry "azure-tiger" verbatim, not "Azure tiger".
writeNote(t, brainDir, "wiki/finance/decisions/azure-tiger-routing.md",
"Azure Tiger Routing", "azure-tiger payment routing decisions.")
writeNote(t, brainDir, "wiki/finance/facts/iso20022.md",
"Azure Tiger ISO 20022 fields", "azure-tiger maps invoice fields to ISO 20022.")
s := NewServer(brainDir, nil, nil, nil)
// graph is nil — only BM25 hits should appear.
resp := callContext(t, s, map[string]any{
"project_root": "/home/mathias/dev/QKX/azure-tiger",
})
entries := resp["entries"].([]any)
require.NotEmpty(t, entries, "expected at least one BM25 hit on project name")
for _, e := range entries {
entry := e.(map[string]any)
assert.Equal(t, "bm25", entry["edge_type"], "no graph configured, every entry must be BM25")
assert.NotEmpty(t, entry["slug"])
assert.NotEmpty(t, entry["doc_path"])
}
}
func TestBrainContext_BM25PlusGraphExpansion(t *testing.T) {
brainDir := t.TempDir()
// BM25 seed — body carries the hyphenated project name verbatim.
writeNote(t, brainDir, "wiki/finance/decisions/azure-tiger-routing.md",
"Azure Tiger Routing", "azure-tiger payment routing decisions.")
// Graph neighbour — does NOT match BM25 on "azure-tiger" so it can
// only arrive via the graph subgraph traversal.
writeNote(t, brainDir, "wiki/finance/facts/sepa-clearing.md",
"SEPA Clearing", "SEPA payment clearing rules and timing windows.")
graphFake := &fakeGraph{
subgraph: map[string][]graphstore.Neighbor{
"azure-tiger-routing": {
{
Slug: "sepa-clearing",
Title: "SEPA Clearing",
DocPath: "wiki/finance/facts/sepa-clearing.md",
EdgeType: "wikilink",
Distance: 1,
},
},
},
}
s := NewServer(brainDir, nil, nil, nil)
s.graph = graphFake
resp := callContext(t, s, map[string]any{
"project_root": "/home/mathias/dev/QKX/azure-tiger",
})
entries := resp["entries"].([]any)
require.GreaterOrEqual(t, len(entries), 2, "expected BM25 seed plus graph neighbour")
slugs := sortedSlugs(entries)
assert.Contains(t, slugs, "azure-tiger-routing", "BM25 seed must appear")
assert.Contains(t, slugs, "sepa-clearing", "graph neighbour must appear")
// Verify the graph-only entry carries edge_type="graph".
var sepaEntry map[string]any
for _, e := range entries {
m := e.(map[string]any)
if m["slug"] == "sepa-clearing" {
sepaEntry = m
break
}
}
require.NotNil(t, sepaEntry)
assert.Equal(t, "graph", sepaEntry["edge_type"])
assert.NotEmpty(t, sepaEntry["excerpt"], "excerpt should be loaded from disk for graph neighbours")
}
func TestBrainContext_LimitClamps(t *testing.T) {
brainDir := t.TempDir()
// Five notes all matching "azure-tiger".
for i, name := range []string{"a", "b", "c", "d", "e"} {
writeNote(t, brainDir,
"wiki/finance/decisions/azure-tiger-"+name+".md",
"Azure Tiger "+name,
"azure-tiger note "+name+" with index "+string(rune('0'+i)))
}
s := NewServer(brainDir, nil, nil, nil)
resp := callContext(t, s, map[string]any{
"project_root": "/home/mathias/dev/QKX/azure-tiger",
"limit": 2,
})
entries := resp["entries"].([]any)
assert.LessOrEqual(t, len(entries), 2)
}
func TestBrainContext_RecentFilesBoost(t *testing.T) {
brainDir := t.TempDir()
// Both notes BM25-match the project name, but azure-tiger-z has
// twice the term frequency so it naturally ranks above azure-tiger-a.
// The recent_files boost on azure-tiger-a should pull it level on
// score; the alphabetical slug tiebreaker (a < z) then promotes it
// to the top — exercising both the boost and the deterministic
// tiebreak.
writeNote(t, brainDir, "wiki/finance/decisions/azure-tiger-a.md",
"A", "azure-tiger note about a.")
writeNote(t, brainDir, "wiki/finance/decisions/azure-tiger-z.md",
"Z", "azure-tiger azure-tiger note about z.")
s := NewServer(brainDir, nil, nil, nil)
// Baseline ranking: azure-tiger-z must lead (higher term frequency).
baseline := callContext(t, s, map[string]any{
"project_root": "/home/mathias/dev/QKX/azure-tiger",
})
baselineEntries := baseline["entries"].([]any)
require.GreaterOrEqual(t, len(baselineEntries), 2)
baselineTop := baselineEntries[0].(map[string]any)
require.Equal(t, "azure-tiger-z", baselineTop["slug"],
"sanity: higher tf must rank first without a boost")
// With boost on azure-tiger-a — boosted entry must now lead.
boosted := callContext(t, s, map[string]any{
"project_root": "/home/mathias/dev/QKX/azure-tiger",
"recent_files": []string{"/some/where/azure-tiger-a.md"},
})
entries := boosted["entries"].([]any)
require.GreaterOrEqual(t, len(entries), 2)
top := entries[0].(map[string]any)
assert.Equal(t, "azure-tiger-a", top["slug"], "recent_files boost must promote the matching doc")
}
+116
View File
@@ -0,0 +1,116 @@
package mcp
import (
"context"
"encoding/json"
"fmt"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphstore"
)
// graphReader is the read-side surface of graphstore.PGStore the
// brain_graph handler needs. Splitting it out (vs. depending on the
// concrete *PGStore) lets tests inject a fake without standing up
// postgres, and keeps the write-side graphsync.Store interface free
// of query concerns.
type graphReader interface {
Neighbors(ctx context.Context, slug, edgeType string, limit int) ([]graphstore.Neighbor, error)
Subgraph(ctx context.Context, origin string, depth int) ([]graphstore.Neighbor, error)
Path(ctx context.Context, src, dst string, maxDepth int) ([]graphstore.PathStep, error)
}
// Compile-time check that *graphstore.PGStore satisfies graphReader.
var _ graphReader = (*graphstore.PGStore)(nil)
type brainGraphArgs struct {
Op string `json:"op"`
Slug string `json:"slug,omitempty"`
Src string `json:"src,omitempty"`
Dst string `json:"dst,omitempty"`
EdgeType string `json:"edge_type,omitempty"`
Limit int `json:"limit,omitempty"`
Depth int `json:"depth,omitempty"`
}
func (s *Server) brainGraph(ctx context.Context, args json.RawMessage) (json.RawMessage, error) {
reader, ok := s.graph.(graphReader)
if s.graph == nil || !ok {
return nil, fmt.Errorf("brain graph not configured: set BRAIN_GRAPH_ENABLED=true")
}
var a brainGraphArgs
if err := json.Unmarshal(args, &a); err != nil {
return nil, fmt.Errorf("parse args: %w", err)
}
switch a.Op {
case "neighbors":
if a.Slug == "" {
return nil, fmt.Errorf("slug is required for op=neighbors")
}
ns, err := reader.Neighbors(ctx, a.Slug, a.EdgeType, a.Limit)
if err != nil {
return nil, fmt.Errorf("neighbors: %w", err)
}
return json.Marshal(map[string]any{"results": neighborsView(ns)})
case "subgraph":
if a.Slug == "" {
return nil, fmt.Errorf("slug is required for op=subgraph")
}
ns, err := reader.Subgraph(ctx, a.Slug, a.Depth)
if err != nil {
return nil, fmt.Errorf("subgraph: %w", err)
}
return json.Marshal(map[string]any{"results": neighborsView(ns)})
case "path":
if a.Src == "" || a.Dst == "" {
return nil, fmt.Errorf("src and dst are required for op=path")
}
steps, err := reader.Path(ctx, a.Src, a.Dst, a.Depth)
if err != nil {
return nil, fmt.Errorf("path: %w", err)
}
return json.Marshal(map[string]any{"steps": pathView(steps)})
default:
return nil, fmt.Errorf("unknown op %q (want neighbors|subgraph|path)", a.Op)
}
}
type neighborView struct {
Slug string `json:"slug"`
Type string `json:"type,omitempty"`
Wing string `json:"wing,omitempty"`
Hall string `json:"hall,omitempty"`
DocPath string `json:"doc_path,omitempty"`
Title string `json:"title,omitempty"`
EdgeType string `json:"edge_type"`
Distance int `json:"distance"`
}
func neighborsView(ns []graphstore.Neighbor) []neighborView {
out := make([]neighborView, 0, len(ns))
for _, n := range ns {
out = append(out, neighborView{
Slug: n.Slug, Type: n.Type, Wing: n.Wing, Hall: n.Hall,
DocPath: n.DocPath, Title: n.Title,
EdgeType: n.EdgeType, Distance: n.Distance,
})
}
return out
}
type pathStepView struct {
From string `json:"from"`
To string `json:"to"`
EdgeType string `json:"edge_type"`
}
func pathView(steps []graphstore.PathStep) []pathStepView {
out := make([]pathStepView, 0, len(steps))
for _, s := range steps {
out = append(out, pathStepView{From: s.FromSlug, To: s.ToSlug, EdgeType: s.EdgeType})
}
return out
}
+194
View File
@@ -0,0 +1,194 @@
// Package metrics is a tiny Prometheus exposition layer.
//
// Hand-rolled rather than pulling in github.com/prometheus/client_golang
// to keep ingestion's dependency surface minimal (stdlib + jwx + testify
// per the repo CLAUDE.md). The single histogram + counter it emits cover
// the canary alert wired in k3s/apps/monitoring/ — see infra#50.
//
// Wire format follows the OpenMetrics text exposition that
// kube-prometheus-stack scrapes by default.
package metrics
import (
"fmt"
"net/http"
"sort"
"strings"
"sync"
"sync/atomic"
"time"
)
// histogram buckets in seconds. Tuned for in-cluster HTTP API
// latencies: BM25 query is sub-10ms, hybrid retrieval + LLM-synthesis
// can run into seconds. +Inf catch-all is implicit.
var defaultBuckets = []float64{
0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10,
}
// Registry holds one histogram (request latency) labeled by path + status
// and one counter (request total) with the same labels. Concurrent-safe.
type Registry struct {
mu sync.RWMutex
series map[labelKey]*series
buckets []float64
}
type labelKey struct{ path, status string }
type series struct {
// One atomic counter per bucket (counts of observations ≤ bucket).
// counts[len(buckets)] = +Inf bucket (== total observations).
counts []atomic.Uint64
sumNs atomic.Uint64 // sum of durations in nanoseconds
}
// New returns a Registry pre-populated with no series; the first
// observation per (path, status) lazy-creates one.
func New() *Registry {
return &Registry{
series: make(map[labelKey]*series),
buckets: defaultBuckets,
}
}
// Observe records a single request duration for the given path + status.
func (r *Registry) Observe(path, status string, d time.Duration) {
key := labelKey{path: path, status: status}
r.mu.RLock()
s := r.series[key]
r.mu.RUnlock()
if s == nil {
r.mu.Lock()
s = r.series[key]
if s == nil {
s = &series{counts: make([]atomic.Uint64, len(r.buckets)+1)}
r.series[key] = s
}
r.mu.Unlock()
}
secs := d.Seconds()
for i, b := range r.buckets {
if secs <= b {
s.counts[i].Add(1)
}
}
// +Inf bucket always increments.
s.counts[len(r.buckets)].Add(1)
s.sumNs.Add(uint64(d.Nanoseconds()))
}
// Middleware wraps next, observing every request's duration + status.
// The metric label `path` uses the request's Pattern (Go 1.22+ ServeMux),
// falling back to the URL path if no Pattern is set. Pattern keeps
// cardinality bounded (one series per route, not one per unique URL).
func (r *Registry) Middleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
rec := &statusRecorder{ResponseWriter: w, code: http.StatusOK}
start := time.Now()
next.ServeHTTP(rec, req)
path := req.Pattern
if path == "" {
path = req.URL.Path
}
r.Observe(path, statusClass(rec.code), time.Since(start))
})
}
// Handler exposes /metrics in OpenMetrics text format.
func (r *Registry) Handler() http.HandlerFunc {
return func(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
r.write(w)
}
}
func (r *Registry) write(w http.ResponseWriter) {
r.mu.RLock()
defer r.mu.RUnlock()
_, _ = fmt.Fprintln(w, "# HELP brain_query_duration_seconds Brain HTTP API request latency in seconds.")
_, _ = fmt.Fprintln(w, "# TYPE brain_query_duration_seconds histogram")
// Sort keys for stable output (helps diffing scrape responses).
keys := make([]labelKey, 0, len(r.series))
for k := range r.series {
keys = append(keys, k)
}
sort.Slice(keys, func(i, j int) bool {
if keys[i].path != keys[j].path {
return keys[i].path < keys[j].path
}
return keys[i].status < keys[j].status
})
for _, k := range keys {
s := r.series[k]
labels := fmt.Sprintf(`path=%q,status=%q`, k.path, k.status)
for i, b := range r.buckets {
_, _ = fmt.Fprintf(w, "brain_query_duration_seconds_bucket{%s,le=%q} %d\n",
labels, formatBucket(b), s.counts[i].Load())
}
// +Inf bucket
inf := s.counts[len(r.buckets)].Load()
_, _ = fmt.Fprintf(w, "brain_query_duration_seconds_bucket{%s,le=\"+Inf\"} %d\n", labels, inf)
_, _ = fmt.Fprintf(w, "brain_query_duration_seconds_sum{%s} %s\n",
labels, formatSeconds(s.sumNs.Load()))
_, _ = fmt.Fprintf(w, "brain_query_duration_seconds_count{%s} %d\n", labels, inf)
}
}
func formatBucket(b float64) string {
// Match Prometheus convention: no trailing zeros.
s := fmt.Sprintf("%g", b)
if !strings.ContainsAny(s, ".e") {
s = s + ".0"
}
return s
}
func formatSeconds(ns uint64) string {
return fmt.Sprintf("%g", float64(ns)/1e9)
}
func statusClass(code int) string {
switch {
case code >= 200 && code < 300:
return "2xx"
case code >= 300 && code < 400:
return "3xx"
case code >= 400 && code < 500:
return "4xx"
case code >= 500 && code < 600:
return "5xx"
default:
return "xxx"
}
}
// statusRecorder captures the response code so middleware can label
// the histogram by status class without buffering the body.
type statusRecorder struct {
http.ResponseWriter
code int
wroteHeader bool
}
func (r *statusRecorder) WriteHeader(code int) {
if r.wroteHeader {
return
}
r.code = code
r.wroteHeader = true
r.ResponseWriter.WriteHeader(code)
}
func (r *statusRecorder) Write(b []byte) (int, error) {
if !r.wroteHeader {
r.WriteHeader(http.StatusOK)
}
return r.ResponseWriter.Write(b)
}
+119
View File
@@ -0,0 +1,119 @@
package metrics
import (
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
)
func TestRegistry_ObserveAndExpose(t *testing.T) {
t.Parallel()
r := New()
// Three observations on the same series; one falls into each
// representative band.
r.Observe("/query", "2xx", 4*time.Millisecond) // ≤ 5ms
r.Observe("/query", "2xx", 20*time.Millisecond) // ≤ 25ms
r.Observe("/query", "2xx", 600*time.Millisecond) // ≤ 1s
req := httptest.NewRequest(http.MethodGet, "/metrics", nil)
rec := httptest.NewRecorder()
r.Handler().ServeHTTP(rec, req)
body := rec.Body.String()
mustContain := []string{
`# TYPE brain_query_duration_seconds histogram`,
`brain_query_duration_seconds_bucket{path="/query",status="2xx",le="0.005"} 1`,
`brain_query_duration_seconds_bucket{path="/query",status="2xx",le="0.025"} 2`,
`brain_query_duration_seconds_bucket{path="/query",status="2xx",le="1.0"} 3`,
`brain_query_duration_seconds_bucket{path="/query",status="2xx",le="+Inf"} 3`,
`brain_query_duration_seconds_count{path="/query",status="2xx"} 3`,
}
for _, want := range mustContain {
if !strings.Contains(body, want) {
t.Errorf("missing line: %q\n--- body ---\n%s", want, body)
}
}
if got := rec.Header().Get("Content-Type"); !strings.HasPrefix(got, "text/plain") {
t.Errorf("content-type = %q, want text/plain prefix", got)
}
}
func TestRegistry_LabelsByStatus(t *testing.T) {
t.Parallel()
r := New()
r.Observe("/query", "2xx", time.Millisecond)
r.Observe("/query", "5xx", time.Millisecond)
r.Observe("/write", "2xx", time.Millisecond)
rec := httptest.NewRecorder()
r.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/metrics", nil))
body := rec.Body.String()
for _, want := range []string{
`brain_query_duration_seconds_count{path="/query",status="2xx"} 1`,
`brain_query_duration_seconds_count{path="/query",status="5xx"} 1`,
`brain_query_duration_seconds_count{path="/write",status="2xx"} 1`,
} {
if !strings.Contains(body, want) {
t.Errorf("missing %q in body:\n%s", want, body)
}
}
}
func TestMiddleware_RecordsTiming(t *testing.T) {
t.Parallel()
r := New()
handler := r.Middleware(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
time.Sleep(2 * time.Millisecond)
w.WriteHeader(http.StatusOK)
_, _ = io.WriteString(w, "ok")
}))
srv := httptest.NewServer(handler)
defer srv.Close()
resp, err := http.Get(srv.URL + "/query")
if err != nil {
t.Fatalf("get: %v", err)
}
_ = resp.Body.Close()
if resp.StatusCode != http.StatusOK {
t.Fatalf("status %d, want 200", resp.StatusCode)
}
// Exposition should now include /query.
rec := httptest.NewRecorder()
r.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/metrics", nil))
body := rec.Body.String()
if !strings.Contains(body, `path="/query"`) {
t.Errorf("expected /query series, got body:\n%s", body)
}
if !strings.Contains(body, `status="2xx"`) {
t.Errorf("expected 2xx status class, got body:\n%s", body)
}
}
func TestStatusRecorder_DefaultsTo200(t *testing.T) {
t.Parallel()
r := New()
handler := r.Middleware(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte("hello"))
}))
rec := httptest.NewRecorder()
handler.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/x", nil))
if rec.Code != http.StatusOK {
t.Errorf("code %d, want 200", rec.Code)
}
}
+38
View File
@@ -0,0 +1,38 @@
// Package oauth implements a minimal OAuth 2.0 client_credentials flow
// for the brain MCP server. Designed for claude.ai's custom MCP integration
// UI, which only supports OAuth (no static-Bearer field). The flow trades
// a registered client_id + client_secret for the existing BRAIN_MCP_TOKEN —
// no JWTs, no expiry, no refresh — so the rest of the auth middleware is
// unchanged.
package oauth
import (
"encoding/json"
"net/http"
"strings"
)
// MetadataHandler serves RFC 8414 authorization-server metadata at
// GET /.well-known/oauth-authorization-server. issuer must be the public
// origin of the brain MCP (e.g. https://brain-mcp.d-ma.be); the handler
// derives the token endpoint from it.
//
// Mount with no auth — discovery must be reachable to anonymous callers.
func MetadataHandler(issuer string) http.HandlerFunc {
issuer = strings.TrimRight(issuer, "/")
body, _ := json.Marshal(struct {
Issuer string `json:"issuer"`
TokenEndpoint string `json:"token_endpoint"`
GrantTypes []string `json:"grant_types_supported"`
TokenEndpointAuthMeth []string `json:"token_endpoint_auth_methods_supported"`
}{
Issuer: issuer,
TokenEndpoint: issuer + "/oauth/token",
GrantTypes: []string{"client_credentials"},
TokenEndpointAuthMeth: []string{"client_secret_post", "client_secret_basic"},
})
return func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write(body)
}
}
+41
View File
@@ -0,0 +1,41 @@
package oauth_test
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/oauth"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestMetadataHandler_ReturnsJSON(t *testing.T) {
h := oauth.MetadataHandler("https://brain-mcp.d-ma.be")
req := httptest.NewRequest(http.MethodGet, "/.well-known/oauth-authorization-server", nil)
rr := httptest.NewRecorder()
h.ServeHTTP(rr, req)
assert.Equal(t, http.StatusOK, rr.Code)
assert.Equal(t, "application/json", rr.Header().Get("Content-Type"))
var body map[string]any
require.NoError(t, json.Unmarshal(rr.Body.Bytes(), &body))
assert.Equal(t, "https://brain-mcp.d-ma.be", body["issuer"])
assert.Equal(t, "https://brain-mcp.d-ma.be/oauth/token", body["token_endpoint"])
assert.ElementsMatch(t, []any{"client_credentials"}, body["grant_types_supported"])
assert.ElementsMatch(t,
[]any{"client_secret_post", "client_secret_basic"},
body["token_endpoint_auth_methods_supported"])
}
func TestMetadataHandler_StripsTrailingSlashFromIssuer(t *testing.T) {
h := oauth.MetadataHandler("https://brain-mcp.d-ma.be/")
rr := httptest.NewRecorder()
h.ServeHTTP(rr, httptest.NewRequest(http.MethodGet, "/.well-known/oauth-authorization-server", nil))
var body map[string]any
require.NoError(t, json.Unmarshal(rr.Body.Bytes(), &body))
assert.Equal(t, "https://brain-mcp.d-ma.be", body["issuer"])
assert.Equal(t, "https://brain-mcp.d-ma.be/oauth/token", body["token_endpoint"])
}
+87
View File
@@ -0,0 +1,87 @@
package oauth
import (
"crypto/subtle"
"encoding/json"
"net/http"
)
// TokenConfig is the static configuration for the token endpoint. All
// three fields are required.
type TokenConfig struct {
// ClientID and ClientSecret are the single accepted credentials.
// claude.ai's custom-MCP UI persists these on its side.
ClientID string
ClientSecret string
// AccessToken is the bearer value handed back on a successful
// exchange. In this deployment it is BRAIN_MCP_TOKEN — the same
// static token the rest of the auth middleware already accepts —
// so no JWT machinery is needed downstream.
AccessToken string
}
// TokenHandler serves POST /oauth/token. Implements the
// client_credentials grant only, with client_secret_post and
// client_secret_basic auth methods (both advertised by MetadataHandler).
// Errors follow RFC 6749 §5.2 — JSON body with an "error" field.
//
// Mount with no auth — credentials live in the request body / header.
func TokenHandler(cfg TokenConfig) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
w.Header().Set("Allow", http.MethodPost)
writeOAuthError(w, http.StatusMethodNotAllowed, "invalid_request", "POST required")
return
}
if err := r.ParseForm(); err != nil {
writeOAuthError(w, http.StatusBadRequest, "invalid_request", "form parse")
return
}
if r.PostForm.Get("grant_type") != "client_credentials" {
writeOAuthError(w, http.StatusBadRequest, "unsupported_grant_type",
"only client_credentials is supported")
return
}
clientID, clientSecret := extractClientCreds(r)
if !constantTimeEqual(clientID, cfg.ClientID) ||
!constantTimeEqual(clientSecret, cfg.ClientSecret) {
writeOAuthError(w, http.StatusUnauthorized, "invalid_client", "bad credentials")
return
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Cache-Control", "no-store")
_ = json.NewEncoder(w).Encode(struct {
AccessToken string `json:"access_token"`
TokenType string `json:"token_type"`
}{cfg.AccessToken, "bearer"})
}
}
// extractClientCreds returns the client_id and client_secret pair from
// either client_secret_basic (HTTP Basic) or client_secret_post (form
// fields). When both are present, Basic wins per RFC 6749 §2.3.1.
func extractClientCreds(r *http.Request) (string, string) {
if id, secret, ok := r.BasicAuth(); ok {
return id, secret
}
return r.PostForm.Get("client_id"), r.PostForm.Get("client_secret")
}
func constantTimeEqual(a, b string) bool {
if a == "" || b == "" {
return false
}
return subtle.ConstantTimeCompare([]byte(a), []byte(b)) == 1
}
func writeOAuthError(w http.ResponseWriter, status int, code, desc string) {
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Cache-Control", "no-store")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(struct {
Error string `json:"error"`
ErrorDescription string `json:"error_description,omitempty"`
}{code, desc})
}
+134
View File
@@ -0,0 +1,134 @@
package oauth_test
import (
"encoding/base64"
"encoding/json"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/oauth"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func newTokenServer() *httptest.Server {
return httptest.NewServer(oauth.TokenHandler(oauth.TokenConfig{
ClientID: "the-client",
ClientSecret: "the-secret",
AccessToken: "BRAIN_TOKEN_VALUE",
}))
}
func postForm(t *testing.T, srv *httptest.Server, vals url.Values, basic [2]string) *http.Response {
t.Helper()
req, err := http.NewRequest(http.MethodPost, srv.URL+"/oauth/token", strings.NewReader(vals.Encode()))
require.NoError(t, err)
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
if basic[0] != "" {
req.SetBasicAuth(basic[0], basic[1])
}
resp, err := http.DefaultClient.Do(req)
require.NoError(t, err)
return resp
}
func TestTokenHandler_ClientSecretPost_Success(t *testing.T) {
srv := newTokenServer()
defer srv.Close()
resp := postForm(t, srv, url.Values{
"grant_type": {"client_credentials"},
"client_id": {"the-client"},
"client_secret": {"the-secret"},
}, [2]string{})
defer func() { _ = resp.Body.Close() }()
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.Equal(t, "application/json", resp.Header.Get("Content-Type"))
var body map[string]any
require.NoError(t, json.NewDecoder(resp.Body).Decode(&body))
assert.Equal(t, "BRAIN_TOKEN_VALUE", body["access_token"])
assert.Equal(t, "bearer", body["token_type"])
}
func TestTokenHandler_ClientSecretBasic_Success(t *testing.T) {
srv := newTokenServer()
defer srv.Close()
resp := postForm(t, srv,
url.Values{"grant_type": {"client_credentials"}},
[2]string{"the-client", "the-secret"},
)
defer func() { _ = resp.Body.Close() }()
assert.Equal(t, http.StatusOK, resp.StatusCode)
}
func TestTokenHandler_WrongSecret(t *testing.T) {
srv := newTokenServer()
defer srv.Close()
resp := postForm(t, srv, url.Values{
"grant_type": {"client_credentials"},
"client_id": {"the-client"},
"client_secret": {"wrong"},
}, [2]string{})
defer func() { _ = resp.Body.Close() }()
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode)
var body map[string]any
require.NoError(t, json.NewDecoder(resp.Body).Decode(&body))
assert.Equal(t, "invalid_client", body["error"])
}
func TestTokenHandler_BadGrantType(t *testing.T) {
srv := newTokenServer()
defer srv.Close()
resp := postForm(t, srv, url.Values{
"grant_type": {"password"},
"client_id": {"the-client"},
"client_secret": {"the-secret"},
}, [2]string{})
defer func() { _ = resp.Body.Close() }()
assert.Equal(t, http.StatusBadRequest, resp.StatusCode)
var body map[string]any
require.NoError(t, json.NewDecoder(resp.Body).Decode(&body))
assert.Equal(t, "unsupported_grant_type", body["error"])
}
func TestTokenHandler_RejectsGet(t *testing.T) {
srv := newTokenServer()
defer srv.Close()
resp, err := http.Get(srv.URL + "/oauth/token")
require.NoError(t, err)
defer func() { _ = resp.Body.Close() }()
assert.Equal(t, http.StatusMethodNotAllowed, resp.StatusCode)
}
func TestTokenHandler_BasicMalformed_FallsThrough(t *testing.T) {
srv := newTokenServer()
defer srv.Close()
// Malformed (non-base64) Authorization header — handler should treat
// the request as missing creds, not crash.
req, _ := http.NewRequest(http.MethodPost, srv.URL+"/oauth/token",
strings.NewReader("grant_type=client_credentials"))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.Header.Set("Authorization", "Basic ###not-base64###")
resp, err := http.DefaultClient.Do(req)
require.NoError(t, err)
defer func() { _ = resp.Body.Close() }()
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode)
}
func TestTokenHandler_BasicNoColon(t *testing.T) {
srv := newTokenServer()
defer srv.Close()
// "client-only" base64 — missing the `:secret` half.
enc := base64.StdEncoding.EncodeToString([]byte("the-client"))
req, _ := http.NewRequest(http.MethodPost, srv.URL+"/oauth/token",
strings.NewReader("grant_type=client_credentials"))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.Header.Set("Authorization", "Basic "+enc)
resp, err := http.DefaultClient.Do(req)
require.NoError(t, err)
defer func() { _ = resp.Body.Close() }()
assert.Equal(t, http.StatusUnauthorized, resp.StatusCode)
}
+119
View File
@@ -0,0 +1,119 @@
// Package reranker scores (query, document) pairs against a cross-encoder
// served by an Ollama-compatible backend.
//
// Wire format is Ollama's `/api/generate`. The model is prompted with the
// Qwen3-Reranker yes/no template — the canonical interface the model
// itself was trained against — and the first token of the response is
// treated as a binary relevance vote: "yes" → 1.0, anything else → 0.0.
// Ties are expected to be broken by the caller's primary retrieval score
// (e.g. BM25), so the binary signal is a filter rather than a ranking
// substitute.
package reranker
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
"time"
)
// Client posts rerank requests to an Ollama-compatible endpoint.
type Client struct {
URL string
Model string
HTTP *http.Client
}
// New constructs a Client. Returns nil when url is empty so callers can
// treat a missing BRAIN_RERANKER_URL as "feature disabled" with a single
// nil check.
func New(url, model string) *Client {
if url == "" {
return nil
}
return &Client{
URL: strings.TrimRight(url, "/"),
Model: model,
HTTP: &http.Client{Timeout: 30 * time.Second},
}
}
// Score returns one [0, 1] relevance score per input document, parallel
// to the input order. Each (query, doc) pair is scored independently —
// Qwen3-Reranker is a cross-encoder and expects per-pair calls.
func (c *Client) Score(ctx context.Context, query string, docs []string) ([]float64, error) {
out := make([]float64, len(docs))
for i, doc := range docs {
s, err := c.scoreOne(ctx, query, doc)
if err != nil {
return nil, fmt.Errorf("rerank doc %d: %w", i, err)
}
out[i] = s
}
return out, nil
}
func (c *Client) scoreOne(ctx context.Context, query, doc string) (float64, error) {
prompt := buildPrompt(query, doc)
reqBody, _ := json.Marshal(map[string]any{
"model": c.Model,
"prompt": prompt,
"stream": false,
"options": map[string]any{
"num_predict": 4,
"temperature": 0,
},
})
req, err := http.NewRequestWithContext(ctx, http.MethodPost,
c.URL+"/api/generate", bytes.NewReader(reqBody))
if err != nil {
return 0, err
}
req.Header.Set("Content-Type", "application/json")
resp, err := c.HTTP.Do(req)
if err != nil {
return 0, err
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode/100 != 2 {
body, _ := io.ReadAll(resp.Body)
return 0, fmt.Errorf("status %d: %s", resp.StatusCode, string(body))
}
var out struct {
Response string `json:"response"`
}
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
return 0, err
}
return parseYesNo(out.Response), nil
}
// buildPrompt assembles the Qwen3-Reranker chat template. Kept verbatim
// because the model was trained on this exact wording.
func buildPrompt(query, doc string) string {
return "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n" +
"<|im_start|>user\n<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n" +
"<Query>: " + query + "\n" +
"<Document>: " + doc + "<|im_end|>\n" +
"<|im_start|>assistant\n<think>\n\n</think>\n\n"
}
// parseYesNo extracts the first meaningful token from response and
// returns 1.0 when it starts with "yes" (case-insensitive), 0.0 otherwise.
// Any leading whitespace, `<think>` block, or punctuation is skipped.
func parseYesNo(s string) float64 {
s = strings.TrimSpace(s)
// Strip any `<think>…</think>` block the model may emit even with empty thinking.
if idx := strings.Index(s, "</think>"); idx != -1 {
s = strings.TrimSpace(s[idx+len("</think>"):])
}
s = strings.ToLower(s)
if strings.HasPrefix(s, "yes") {
return 1.0
}
return 0.0
}
@@ -0,0 +1,119 @@
package reranker_test
import (
"context"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/reranker"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// fakeOllama responds to /api/generate based on a per-document
// {needle → answer} map: if the prompt contains the needle, returns
// the mapped answer.
type fakeOllama struct {
t *testing.T
answers map[string]string // needle → "yes" or "no"
calls int
lastBody map[string]any
}
func (f *fakeOllama) handler() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
require.Equal(f.t, http.MethodPost, r.Method)
require.Equal(f.t, "/api/generate", r.URL.Path)
body, err := io.ReadAll(r.Body)
require.NoError(f.t, err)
var p map[string]any
require.NoError(f.t, json.Unmarshal(body, &p))
f.calls++
f.lastBody = p
prompt := p["prompt"].(string)
answer := "no"
for needle, a := range f.answers {
if strings.Contains(prompt, needle) {
answer = a
break
}
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{
"model": p["model"], "response": answer, "done": true,
})
})
}
func TestNew_EmptyURLReturnsNil(t *testing.T) {
assert.Nil(t, reranker.New("", "model"))
}
func TestScore_YesAndNoOrdered(t *testing.T) {
f := &fakeOllama{t: t, answers: map[string]string{
"alpha doc": "yes",
"beta doc": "no",
"gamma doc": "yes",
}}
srv := httptest.NewServer(f.handler())
defer srv.Close()
c := reranker.New(srv.URL, "test-model")
require.NotNil(t, c)
scores, err := c.Score(context.Background(), "what is alpha",
[]string{"alpha doc body", "beta doc body", "gamma doc body"})
require.NoError(t, err)
require.Len(t, scores, 3)
assert.Equal(t, 1.0, scores[0])
assert.Equal(t, 0.0, scores[1])
assert.Equal(t, 1.0, scores[2])
assert.Equal(t, 3, f.calls)
}
func TestScore_SendsCorrectShape(t *testing.T) {
f := &fakeOllama{t: t, answers: map[string]string{"hello": "yes"}}
srv := httptest.NewServer(f.handler())
defer srv.Close()
c := reranker.New(srv.URL, "qwen3-rerank")
_, err := c.Score(context.Background(), "greeting", []string{"hello world"})
require.NoError(t, err)
assert.Equal(t, "qwen3-rerank", f.lastBody["model"])
prompt := f.lastBody["prompt"].(string)
assert.Contains(t, prompt, "greeting")
assert.Contains(t, prompt, "hello world")
assert.Contains(t, prompt, `"yes" or "no"`)
}
func TestScore_HandlesAmbiguousResponse(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{"response": "maybe — unclear", "done": true})
}))
defer srv.Close()
c := reranker.New(srv.URL, "m")
scores, err := c.Score(context.Background(), "q", []string{"d"})
require.NoError(t, err)
// Anything that does not start with "yes" (case-insensitive, after
// whitespace/think trim) is treated as "no" = 0.
assert.Equal(t, []float64{0}, scores)
}
func TestScore_EmptyDocsReturnsEmpty(t *testing.T) {
c := reranker.New("http://127.0.0.1:1", "m")
scores, err := c.Score(context.Background(), "q", nil)
require.NoError(t, err)
assert.Empty(t, scores)
}
func TestScore_UpstreamErrorPropagates(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
}))
defer srv.Close()
c := reranker.New(srv.URL, "m")
_, err := c.Score(context.Background(), "q", []string{"d"})
require.Error(t, err)
}
+325 -17
View File
@@ -3,38 +3,117 @@ package search
import ( import (
"bufio" "bufio"
"context"
"fmt" "fmt"
"log/slog" "log/slog"
"os" "os"
"path/filepath" "path/filepath"
"sort" "sort"
"strings" "strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
) )
// VectorSearcher returns the top-limit nearest paths by cosine
// distance. The vectorstore package implements this against pgvector.
type VectorSearcher interface {
Search(ctx context.Context, query []float32, limit int) ([]VectorHit, error)
}
// VectorHit is a single path + distance pair from a vector search.
// Re-declared here (rather than imported) to keep search package
// free of vectorstore/embed deps and to make stubbing trivial in tests.
type VectorHit struct {
Path string
Distance float64
}
// Embedder turns a query string into a dense vector. The embed package
// implements this against Ollama's /api/embed.
type Embedder interface {
Embed(ctx context.Context, text string) ([]float32, error)
}
// Result is a single search hit from the brain wiki. // Result is a single search hit from the brain wiki.
type Result struct { type Result struct {
Path string `json:"path"` Path string `json:"path"`
Title string `json:"title"` Title string `json:"title"`
Excerpt string `json:"excerpt"` Excerpt string `json:"excerpt"`
Score int `json:"score"` Score int `json:"score"`
Wing string `json:"wing,omitempty"`
Hall string `json:"hall,omitempty"`
// Tier is the DIKW classification used for retrieval weighting
// (infra#72). Read from frontmatter when present, otherwise
// inferred from the parent directory.
Tier string `json:"tier,omitempty"`
} }
// Query searches all .md files under brainDir/wiki/ for pages containing // tierWeight maps the DIKW tier to a score multiplier applied right
// any of the whitespace-separated terms in query. Returns up to limit results // before the final truncation. Knowledge entries (focused lessons that
// sorted by score descending. // age well) get boosted; inbox entries (raw captures, sessions, clips)
func Query(brainDir, query string, limit int) ([]Result, error) { // get demoted. Empty / unknown tiers keep the original BM25 score
if limit <= 0 { // (multiplier 1.0). See infra#72 for the failure mode this addresses:
limit = 5 // short focused entries lose to long aggregate dump-files under
// raw BM25 ranking.
func tierWeight(tier string) float64 {
switch tier {
case "knowledge":
return 1.5
case "note":
return 1.0
case "inbox":
return 0.3
default:
return 1.0
} }
terms := strings.Fields(strings.ToLower(query)) }
// QueryOptions configures a search.
//
// When Wing is set, the walk is restricted to brain/wiki/<wing>/.
// When Hall is additionally set, the walk is restricted to
// brain/wiki/<wing>/<hall>/. Without either, the legacy walk over
// brain/knowledge/ and brain/wiki/ is used.
//
// When both Vector and Embedder are non-nil, results are computed
// hybridly: BM25 and vector candidate lists are merged via Reciprocal
// Rank Fusion. With either nil the function falls back to BM25 only,
// keeping behaviour unchanged for callers that have not opted in.
type QueryOptions struct {
Query string
Limit int
Wing string
Hall string
Vector VectorSearcher
Embedder Embedder
}
// Query searches the brain. Returns up to opts.Limit results sorted by
// score descending. Empty query returns nil.
func Query(brainDir string, opts QueryOptions) ([]Result, error) {
return QueryContext(context.Background(), brainDir, opts)
}
// QueryContext is the cancellable variant of Query. Hybrid retrieval
// requires a context because both the embedder and the vector store are
// network calls.
func QueryContext(ctx context.Context, brainDir string, opts QueryOptions) ([]Result, error) {
if opts.Limit <= 0 {
opts.Limit = 5
}
terms := strings.Fields(strings.ToLower(opts.Query))
if len(terms) == 0 { if len(terms) == 0 {
return nil, nil return nil, nil
} }
var results []Result roots, err := resolveRoots(brainDir, opts.Wing, opts.Hall)
if err != nil {
return nil, err
}
for _, subdir := range []string{"knowledge", "wiki"} { var results []Result
dir := filepath.Join(brainDir, subdir) for _, dir := range roots {
if _, statErr := os.Stat(dir); os.IsNotExist(statErr) { if _, statErr := os.Stat(dir); os.IsNotExist(statErr) {
continue continue
} }
@@ -46,13 +125,11 @@ func Query(brainDir, query string, limit int) ([]Result, error) {
if d.IsDir() || !strings.HasSuffix(path, ".md") { if d.IsDir() || !strings.HasSuffix(path, ".md") {
return nil return nil
} }
content, err := os.ReadFile(path) content, err := os.ReadFile(path)
if err != nil { if err != nil {
slog.Warn("search: skipping unreadable file", "path", path, "err", err) slog.Warn("search: skipping unreadable file", "path", path, "err", err)
return nil return nil
} }
lower := strings.ToLower(string(content)) lower := strings.ToLower(string(content))
score := 0 score := 0
for _, term := range terms { for _, term := range terms {
@@ -61,18 +138,21 @@ func Query(brainDir, query string, limit int) ([]Result, error) {
if score == 0 { if score == 0 {
return nil return nil
} }
rel, err := filepath.Rel(brainDir, path) rel, err := filepath.Rel(brainDir, path)
if err != nil { if err != nil {
return fmt.Errorf("rel path: %w", err) return fmt.Errorf("rel path: %w", err)
} }
rel = filepath.ToSlash(rel) rel = filepath.ToSlash(rel)
wing, hall := extractWingHall(string(content), rel)
tier := extractTier(string(content), rel)
results = append(results, Result{ results = append(results, Result{
Path: rel, Path: rel,
Title: extractTitle(string(content), d.Name()), Title: extractTitle(string(content), d.Name()),
Excerpt: excerpt(string(content), 300), Excerpt: excerpt(string(content), 300),
Score: score, Score: score,
Wing: wing,
Hall: hall,
Tier: tier,
}) })
return nil return nil
}) })
@@ -84,12 +164,241 @@ func Query(brainDir, query string, limit int) ([]Result, error) {
sort.Slice(results, func(i, j int) bool { sort.Slice(results, func(i, j int) bool {
return results[i].Score > results[j].Score return results[i].Score > results[j].Score
}) })
if len(results) > limit {
results = results[:limit] // Hybrid scoring kicks in only when both the embedder and the
// vector store are wired and BM25 actually returned candidates.
if opts.Vector != nil && opts.Embedder != nil && len(results) > 0 {
merged, err := hybridMerge(ctx, brainDir, opts, results)
if err != nil {
slog.Warn("search: hybrid merge failed, falling back to BM25", "err", err)
} else {
results = merged
}
}
// Tier-weighted final re-rank (infra#72). Knowledge tier entries
// boost ×1.5, inbox demote ×0.3, note stays at ×1.0. Applied after
// hybridMerge so RRF ranking still drives candidate generation;
// the tier weight only re-orders the merged set.
sort.SliceStable(results, func(i, j int) bool {
return float64(results[i].Score)*tierWeight(results[i].Tier) >
float64(results[j].Score)*tierWeight(results[j].Tier)
})
if len(results) > opts.Limit {
results = results[:opts.Limit]
} }
return results, nil return results, nil
} }
// rrfK is the constant in the Reciprocal Rank Fusion formula. 60 is
// standard (Cormack et al. 2009) and parameter-free in practice.
const rrfK = 60.0
// hybridMerge embeds the query, runs a vector search, and merges its
// candidates with the BM25 list via Reciprocal Rank Fusion. Results
// that came only from the vector side are hydrated by reading the
// note's frontmatter for title/wing/hall and excerpting the body.
//
// rrf(d) = sum_r 1 / (k + rank_r(d)) over rankers r ∈ {BM25, vector}.
func hybridMerge(ctx context.Context, brainDir string, opts QueryOptions, bm25 []Result) ([]Result, error) {
q, err := opts.Embedder.Embed(ctx, opts.Query)
if err != nil {
return nil, fmt.Errorf("embed query: %w", err)
}
vectorLimit := opts.Limit * 4
if vectorLimit < 20 {
vectorLimit = 20
}
hits, err := opts.Vector.Search(ctx, q, vectorLimit)
if err != nil {
return nil, fmt.Errorf("vector search: %w", err)
}
rrf := make(map[string]float64)
byPath := make(map[string]Result)
for rank, r := range bm25 {
rrf[r.Path] += 1.0 / (rrfK + float64(rank+1))
byPath[r.Path] = r
}
for rank, h := range hits {
// Vector store keys are chunk paths ("wiki/foo.md#0001"); collapse
// back to the parent so multiple chunk hits from the same file
// score against a single result row.
parent := vectorstore.ParentPath(h.Path)
if opts.Wing != "" && !pathInScope(parent, opts.Wing, opts.Hall) {
continue
}
rrf[parent] += 1.0 / (rrfK + float64(rank+1))
if _, seen := byPath[parent]; !seen {
r, err := hydrate(brainDir, parent)
if err != nil {
slog.Warn("search: hydrate failed for vector hit", "path", parent, "err", err)
continue
}
byPath[parent] = r
}
}
merged := make([]Result, 0, len(byPath))
for p, r := range byPath {
r.Score = int(rrf[p] * 1e6) // scale to int for stable JSON; relative order is what matters
merged = append(merged, r)
}
sort.Slice(merged, func(i, j int) bool {
return merged[i].Score > merged[j].Score
})
return merged, nil
}
// pathInScope reports whether a wiki path satisfies the wing/hall filter.
func pathInScope(relPath, wing, hall string) bool {
prefix := "wiki/" + brain.Sanitise(wing) + "/"
if hall != "" {
prefix += hall + "/"
}
return strings.HasPrefix(relPath, prefix)
}
// hydrate reads a single note from disk and returns a Result with title,
// excerpt, wing, and hall populated. Used for paths that surface only
// via vector search.
func hydrate(brainDir, relPath string) (Result, error) {
full := filepath.Join(brainDir, filepath.FromSlash(relPath))
content, err := os.ReadFile(full)
if err != nil {
return Result{}, err
}
wing, hall := extractWingHall(string(content), relPath)
tier := extractTier(string(content), relPath)
return Result{
Path: relPath,
Title: extractTitle(string(content), filepath.Base(relPath)),
Excerpt: excerpt(string(content), 300),
Wing: wing,
Hall: hall,
Tier: tier,
}, nil
}
// resolveRoots returns the directories to walk for the given wing/hall
// filters. Validates hall against the closed vocabulary when set.
func resolveRoots(brainDir, wing, hall string) ([]string, error) {
if hall != "" && !brain.IsValidHall(hall) {
return nil, fmt.Errorf("invalid hall %q", hall)
}
if wing != "" {
w := brain.Sanitise(wing)
if w == "" {
return nil, fmt.Errorf("invalid wing %q", wing)
}
if hall != "" {
return []string{filepath.Join(brainDir, "wiki", w, hall)}, nil
}
return []string{filepath.Join(brainDir, "wiki", w)}, nil
}
if hall != "" {
return nil, fmt.Errorf("hall filter requires wing")
}
return []string{
filepath.Join(brainDir, "knowledge"),
filepath.Join(brainDir, "wiki"),
}, nil
}
// extractTier reads the DIKW tier from frontmatter first, falling back
// to the path prefix mapping (infra#72). Mirrors graph.inferTierFromPath
// so the two callers stay in lockstep — frontmatter is canonical,
// path inference is the migration-window fallback.
func extractTier(content, relPath string) string {
scanner := bufio.NewScanner(strings.NewReader(content))
inFrontmatter := false
for scanner.Scan() {
line := scanner.Text()
if strings.TrimSpace(line) == "---" {
if !inFrontmatter {
inFrontmatter = true
continue
}
break
}
if !inFrontmatter {
continue
}
key, val, ok := strings.Cut(line, ":")
if !ok {
continue
}
if strings.TrimSpace(key) == "tier" {
return strings.Trim(strings.TrimSpace(val), `"'`)
}
}
parts := strings.Split(relPath, "/")
if len(parts) == 0 {
return ""
}
switch parts[0] {
case "inbox", "raw", "sessions", "clips":
return "inbox"
case "notes":
return "note"
case "wiki":
// wiki/entities/ anchor pages map to knowledge (see
// graph.inferTierFromPath for the rationale).
if len(parts) >= 2 && parts[1] == "entities" {
return "knowledge"
}
return "note"
case "knowledge":
return "knowledge"
}
return ""
}
// extractWingHall reads wing/hall from frontmatter first, falling back to
// path segments brain/wiki/<wing>/<hall>/.
func extractWingHall(content, relPath string) (wing, hall string) {
scanner := bufio.NewScanner(strings.NewReader(content))
inFrontmatter := false
for scanner.Scan() {
line := scanner.Text()
if strings.TrimSpace(line) == "---" {
if !inFrontmatter {
inFrontmatter = true
continue
}
break
}
if !inFrontmatter {
continue
}
key, val, ok := strings.Cut(line, ":")
if !ok {
continue
}
v := strings.Trim(strings.TrimSpace(val), `"'`)
switch strings.TrimSpace(key) {
case "wing":
wing = v
case "hall":
hall = v
}
}
if wing != "" && hall != "" {
return wing, hall
}
parts := strings.Split(relPath, "/")
if len(parts) >= 4 && parts[0] == "wiki" {
if wing == "" {
wing = parts[1]
}
if hall == "" && brain.IsValidHall(parts[2]) {
hall = parts[2]
}
}
return wing, hall
}
func extractTitle(content, filename string) string { func extractTitle(content, filename string) string {
scanner := bufio.NewScanner(strings.NewReader(content)) scanner := bufio.NewScanner(strings.NewReader(content))
inFrontmatter := false inFrontmatter := false
@@ -113,7 +422,6 @@ func extractTitle(content, filename string) string {
} }
func excerpt(content string, maxLen int) string { func excerpt(content string, maxLen int) string {
// Skip frontmatter, return first maxLen chars of body.
parts := strings.SplitN(content, "---", 3) parts := strings.SplitN(content, "---", 3)
body := content body := content
if len(parts) == 3 { if len(parts) == 3 {
+163 -2
View File
@@ -2,9 +2,11 @@
package search_test package search_test
import ( import (
"context"
"fmt" "fmt"
"os" "os"
"path/filepath" "path/filepath"
"strings"
"testing" "testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/search" "github.com/mathiasbq/hyperguild/ingestion/internal/search"
@@ -12,6 +14,99 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
type stubEmbedder struct{ vec []float32 }
func (s stubEmbedder) Embed(_ context.Context, _ string) ([]float32, error) { return s.vec, nil }
type stubVector struct{ hits []search.VectorHit }
func (s stubVector) Search(_ context.Context, _ []float32, _ int) ([]search.VectorHit, error) {
return s.hits, nil
}
func TestSearch_HybridRRFPromotesVectorOnlyHit(t *testing.T) {
dir := t.TempDir()
for _, p := range []struct{ rel, body string }{
// BM25-keyword note (matches "lejpa" once)
{"wiki/jepa-fx/facts/foo.md", "---\ntitle: Foo\n---\nlejpa keyword\n"},
// Semantically related note that does NOT contain the keyword.
{"wiki/jepa-fx/facts/semantic.md", "---\ntitle: Semantic\n---\nNo keyword in body.\n"},
} {
full := filepath.Join(dir, p.rel)
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
require.NoError(t, os.WriteFile(full, []byte(p.body), 0o644))
}
embedder := stubEmbedder{vec: []float32{0.1}}
vector := stubVector{hits: []search.VectorHit{
{Path: "wiki/jepa-fx/facts/semantic.md", Distance: 0.05}, // best vector match
{Path: "wiki/jepa-fx/facts/foo.md", Distance: 0.10},
}}
got, err := search.Query(dir, search.QueryOptions{
Query: "lejpa",
Limit: 5,
Vector: vector,
Embedder: embedder,
})
require.NoError(t, err)
require.Len(t, got, 2, "vector-only hit should be hydrated into results")
paths := []string{got[0].Path, got[1].Path}
assert.Contains(t, paths, "wiki/jepa-fx/facts/foo.md")
assert.Contains(t, paths, "wiki/jepa-fx/facts/semantic.md")
}
func TestSearch_HybridDedupesChunkPathsToParent(t *testing.T) {
dir := t.TempDir()
full := filepath.Join(dir, "knowledge", "long.md")
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
// Body contains the BM25 keyword "alpaca" so hybridMerge actually runs
// (it only kicks in when BM25 returns at least one candidate).
require.NoError(t, os.WriteFile(full, []byte("---\ntitle: Long\n---\nalpaca content.\n"), 0o644))
embedder := stubEmbedder{vec: []float32{0.1}}
// Vector store returns three chunk-path hits all pointing at the same
// parent file. The merged result must surface ONE row per parent — not
// three rows with chunk-suffixed paths.
vector := stubVector{hits: []search.VectorHit{
{Path: "knowledge/long.md#0001", Distance: 0.05},
{Path: "knowledge/long.md#0002", Distance: 0.07},
{Path: "knowledge/long.md#0003", Distance: 0.09},
}}
got, err := search.Query(dir, search.QueryOptions{
Query: "alpaca",
Limit: 5,
Vector: vector,
Embedder: embedder,
})
require.NoError(t, err)
require.Len(t, got, 1, "three chunk hits for one parent must merge to one result")
assert.Equal(t, "knowledge/long.md", got[0].Path)
assert.Equal(t, "Long", got[0].Title)
}
func TestSearch_HybridFallsBackOnEmbedderError(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "wiki", "x.md"), []byte("keyword foo"), 0o644))
embedder := errorEmbedder{}
vector := stubVector{}
got, err := search.Query(dir, search.QueryOptions{
Query: "keyword", Limit: 5, Vector: vector, Embedder: embedder,
})
require.NoError(t, err)
require.Len(t, got, 1, "BM25 result should still come back when embedder fails")
assert.Equal(t, "wiki/x.md", got[0].Path)
}
type errorEmbedder struct{}
func (errorEmbedder) Embed(_ context.Context, _ string) ([]float32, error) {
return nil, assert.AnError
}
func TestSearch_ReturnsMatchingPages(t *testing.T) { func TestSearch_ReturnsMatchingPages(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "knowledge"), 0o755)) require.NoError(t, os.MkdirAll(filepath.Join(dir, "knowledge"), 0o755))
@@ -27,7 +122,7 @@ func TestSearch_ReturnsMatchingPages(t *testing.T) {
0o644, 0o644,
)) ))
results, err := search.Query(dir, "retry transient", 5) results, err := search.Query(dir, search.QueryOptions{Query: "retry transient", Limit: 5})
require.NoError(t, err) require.NoError(t, err)
require.Len(t, results, 1) require.Len(t, results, 1)
assert.Equal(t, "knowledge/retry-logic.md", results[0].Path) assert.Equal(t, "knowledge/retry-logic.md", results[0].Path)
@@ -36,6 +131,72 @@ func TestSearch_ReturnsMatchingPages(t *testing.T) {
assert.Contains(t, results[0].Excerpt, "Retry") assert.Contains(t, results[0].Excerpt, "Retry")
} }
func TestSearch_TierWeightingReordersResults(t *testing.T) {
dir := t.TempDir()
// A long note-tier dump mentions the keyword many times (high raw
// BM25 score); a short knowledge entry mentions it three times.
// Raw BM25 prefers the dump; tier weighting (knowledge ×1.5 vs
// note ×1.0) flips the order if the score gap is within reach.
// note raw = 5 × 2 terms = 10 hits, weight 1.0 → 10
// knowledge raw = 4 × 2 terms = 8 hits, weight 1.5 → 12 (overtakes)
noteBody := "---\ntier: note\n---\n" + strings.Repeat("scram trap. ", 5)
knowledgeBody := "---\ntier: knowledge\n---\n" + strings.Repeat("scram trap. ", 4)
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki", "sources"), 0o755))
require.NoError(t, os.MkdirAll(filepath.Join(dir, "knowledge"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "wiki", "sources", "dump.md"), []byte(noteBody), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(dir, "knowledge", "trap.md"), []byte(knowledgeBody), 0o644))
results, err := search.Query(dir, search.QueryOptions{Query: "scram trap", Limit: 5})
require.NoError(t, err)
require.GreaterOrEqual(t, len(results), 2)
assert.Equal(t, "knowledge/trap.md", results[0].Path, "knowledge tier weight should beat note tier")
assert.Equal(t, "knowledge", results[0].Tier)
assert.Equal(t, "note", results[1].Tier)
}
func TestSearch_WingHallScoping(t *testing.T) {
dir := t.TempDir()
for _, p := range []struct{ rel, body string }{
{"wiki/jepa-fx/decisions/val-vol.md", "---\nwing: jepa-fx\nhall: decisions\n---\nval-vol-r2 keyword.\n"},
{"wiki/jepa-fx/facts/architecture.md", "---\nwing: jepa-fx\nhall: facts\n---\nval-vol-r2 keyword in facts.\n"},
{"wiki/hyperguild/decisions/routing.md", "---\nwing: hyperguild\nhall: decisions\n---\nval-vol-r2 reference.\n"},
{"knowledge/loose.md", "---\n---\nval-vol-r2 in knowledge.\n"},
} {
full := filepath.Join(dir, p.rel)
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
require.NoError(t, os.WriteFile(full, []byte(p.body), 0o644))
}
// No filter: walk both knowledge/ and wiki/ — all 4 match.
got, err := search.Query(dir, search.QueryOptions{Query: "val-vol-r2", Limit: 10})
require.NoError(t, err)
assert.Len(t, got, 4)
// Wing scope: 2 jepa-fx hits, no hyperguild, no knowledge.
got, err = search.Query(dir, search.QueryOptions{Query: "val-vol-r2", Limit: 10, Wing: "jepa-fx"})
require.NoError(t, err)
require.Len(t, got, 2)
for _, r := range got {
assert.Equal(t, "jepa-fx", r.Wing)
}
// Wing+Hall scope: 1 hit.
got, err = search.Query(dir, search.QueryOptions{Query: "val-vol-r2", Limit: 10, Wing: "jepa-fx", Hall: "decisions"})
require.NoError(t, err)
require.Len(t, got, 1)
assert.Equal(t, "jepa-fx", got[0].Wing)
assert.Equal(t, "decisions", got[0].Hall)
assert.Equal(t, "wiki/jepa-fx/decisions/val-vol.md", got[0].Path)
// Invalid hall rejected.
_, err = search.Query(dir, search.QueryOptions{Query: "x", Wing: "jepa-fx", Hall: "garbage"})
require.Error(t, err)
// Hall without wing rejected.
_, err = search.Query(dir, search.QueryOptions{Query: "x", Hall: "facts"})
require.Error(t, err)
}
func TestSearch_RespectsLimit(t *testing.T) { func TestSearch_RespectsLimit(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "knowledge"), 0o755)) require.NoError(t, os.MkdirAll(filepath.Join(dir, "knowledge"), 0o755))
@@ -46,7 +207,7 @@ func TestSearch_RespectsLimit(t *testing.T) {
0o644, 0o644,
)) ))
} }
results, err := search.Query(dir, "retry", 3) results, err := search.Query(dir, search.QueryOptions{Query: "retry", Limit: 3})
require.NoError(t, err) require.NoError(t, err)
assert.LessOrEqual(t, len(results), 3) assert.LessOrEqual(t, len(results), 3)
} }
+202
View File
@@ -0,0 +1,202 @@
package vectorstore
import (
"fmt"
"strings"
"unicode/utf8"
)
// NumberedChunk pairs a chunk's body with the storage path it will use
// in brain_embeddings. Path format: "<parent>#NNNN" where NNNN is the
// 1-based chunk index zero-padded to 4 digits.
type NumberedChunk struct {
Path string
Content string
}
// ParentPath returns the file path with any "#NNNN" chunk suffix removed.
// Inputs without a "#" are returned unchanged. Used by search to dedupe
// chunk-level hits back to a single document per result.
func ParentPath(p string) string {
if i := strings.Index(p, "#"); i >= 0 {
return p[:i]
}
return p
}
// NumberChunks assigns "<parent>#NNNN" storage paths to a slice of chunk
// bodies, indexed from 0001. Empty chunks are dropped.
func NumberChunks(parent string, chunks []string) []NumberedChunk {
out := make([]NumberedChunk, 0, len(chunks))
idx := 1
for _, c := range chunks {
if strings.TrimSpace(c) == "" {
continue
}
out = append(out, NumberedChunk{
Path: fmt.Sprintf("%s#%04d", parent, idx),
Content: c,
})
idx++
}
return out
}
// ChunkMarkdown splits a markdown document into embedding-sized pieces.
// Strategy:
// 1. Split at H1/H2 headings (top-of-line "#" or "##"). The intro before
// the first heading is its own chunk.
// 2. Any section larger than maxBytes is further split at paragraph
// boundaries (blank lines), packing paragraphs greedily under the
// byte budget.
//
// The function aims for "fits comfortably under nomic-embed-text's 2048-
// token context" — at ~4 chars/token for English markdown, maxBytes ≈ 4000
// is a safe call-site default.
func ChunkMarkdown(content string, maxBytes int) []string {
if maxBytes <= 0 {
maxBytes = 4000
}
sections := splitAtHeadings(content)
out := make([]string, 0, len(sections))
for _, s := range sections {
if len(s) <= maxBytes {
out = append(out, s)
continue
}
out = append(out, splitAtParagraphs(s, maxBytes)...)
}
// Final guarantee: no chunk exceeds maxBytes. A single heading-less,
// paragraph-less block (JSON-lines, minified content) survives the two
// passes above whole — splitAtParagraphs emits an over-budget paragraph
// rather than truncating prose. Hard-split any such chunk at line/rune
// boundaries so the embedder never rejects an over-context chunk.
final := make([]string, 0, len(out))
for _, c := range out {
if len(c) <= maxBytes {
final = append(final, c)
continue
}
final = append(final, hardSplit(c, maxBytes)...)
}
return final
}
// hardSplit slices s into pieces no larger than maxBytes, breaking at line
// boundaries where possible and otherwise mid-line at a UTF-8 rune boundary.
// Last resort for content that has neither headings nor blank-line paragraphs.
func hardSplit(s string, maxBytes int) []string {
var out []string
var cur strings.Builder
flush := func() {
if cur.Len() > 0 {
out = append(out, cur.String())
cur.Reset()
}
}
for _, line := range strings.SplitAfter(s, "\n") {
if line == "" {
continue
}
if len(line) > maxBytes {
flush()
out = append(out, runeSplit(line, maxBytes)...)
continue
}
if cur.Len() > 0 && cur.Len()+len(line) > maxBytes {
flush()
}
cur.WriteString(line)
}
flush()
return out
}
// runeSplit slices s into <=maxBytes pieces without splitting a UTF-8 rune.
func runeSplit(s string, maxBytes int) []string {
var out []string
for len(s) > maxBytes {
cut := maxBytes
for cut > 0 && !utf8.RuneStart(s[cut]) {
cut--
}
if cut == 0 { // single rune wider than the budget; emit it whole
cut = maxBytes
}
out = append(out, s[:cut])
s = s[cut:]
}
if len(s) > 0 {
out = append(out, s)
}
return out
}
// splitAtHeadings cuts content into sections that each start with an
// "# " or "## " line (intro before any heading is the leading section).
func splitAtHeadings(content string) []string {
lines := strings.Split(content, "\n")
var sections []string
var cur strings.Builder
flush := func() {
if cur.Len() == 0 {
return
}
// Trim all trailing whitespace then re-add a single newline so a
// single-paragraph file round-trips to its original content rather
// than accumulating extra newlines from the empty-line split.
s := strings.TrimRight(cur.String(), "\n")
sections = append(sections, s+"\n")
cur.Reset()
}
for _, ln := range lines {
trimmed := strings.TrimLeft(ln, " ")
isH := strings.HasPrefix(trimmed, "# ") || strings.HasPrefix(trimmed, "## ")
if isH && cur.Len() > 0 {
flush()
}
cur.WriteString(ln)
cur.WriteByte('\n')
}
flush()
// Drop empty / whitespace-only trailing section (common when content
// itself ends with a "\n" — Split leaves a final empty element).
if n := len(sections); n > 0 && strings.TrimSpace(sections[n-1]) == "" {
sections = sections[:n-1]
}
return sections
}
// splitAtParagraphs packs paragraphs (blank-line separated blocks) into
// sub-chunks of at most maxBytes. A single paragraph that itself exceeds
// maxBytes is emitted as one over-budget chunk rather than being split
// mid-sentence — better to over-spend a little than truncate prose.
func splitAtParagraphs(section string, maxBytes int) []string {
paras := strings.Split(section, "\n\n")
var out []string
var cur strings.Builder
for _, p := range paras {
if p == "" {
continue
}
// +2 for the "\n\n" rejoin if cur isn't empty
need := len(p)
if cur.Len() > 0 {
need += 2
}
if cur.Len() > 0 && cur.Len()+need > maxBytes {
out = append(out, cur.String())
cur.Reset()
}
if cur.Len() > 0 {
cur.WriteString("\n\n")
}
cur.WriteString(p)
}
if cur.Len() > 0 {
out = append(out, cur.String())
}
return out
}
@@ -0,0 +1,89 @@
package vectorstore_test
import (
"strings"
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestChunkMarkdown_ShortFileFitsInOne(t *testing.T) {
out := vectorstore.ChunkMarkdown("Just a short paragraph.\n", 4000)
require.Len(t, out, 1)
assert.Equal(t, "Just a short paragraph.\n", out[0])
}
func TestChunkMarkdown_SplitsAtHeadings(t *testing.T) {
src := "# Top\n\nintro\n\n## A\n\nbody a\n\n## B\n\nbody b\n"
out := vectorstore.ChunkMarkdown(src, 50) // tiny limit forces per-section split
assert.GreaterOrEqual(t, len(out), 2, "should split at H2 boundaries")
// Each chunk should start with a heading (top-level intro chunk OK without one)
for i, c := range out {
if i == 0 {
continue
}
assert.True(t, strings.HasPrefix(strings.TrimSpace(c), "#"),
"non-first chunk %d should start with heading: %q", i, c)
}
}
func TestChunkMarkdown_HardSplitsHeadinglessOversizedBlock(t *testing.T) {
// A document with no headings and no blank-line paragraph breaks (e.g.
// JSON-lines like wiki/telos/decisions/human-intent-column.md). The old
// chunker emitted it as one over-budget chunk → nomic-embed returned
// "input length exceeds the context length" (400). Every chunk must now
// fit the budget, with no content lost.
maxBytes := 200
src := strings.Repeat("x", 1000) // one 1000-byte blob, no headings, no \n\n
out := vectorstore.ChunkMarkdown(src, maxBytes)
require.Greater(t, len(out), 1, "oversized blob must be split")
for i, c := range out {
assert.LessOrEqual(t, len(c), maxBytes, "chunk %d over budget: %d bytes", i, len(c))
}
assert.Equal(t, 1000, strings.Count(strings.Join(out, ""), "x"), "no content lost")
}
func TestChunkMarkdown_FurtherSplitsOversizedSection(t *testing.T) {
// One H2 section with 4 paragraphs of ~80 chars each, limit 100.
src := "## big\n\n" +
strings.Repeat("paragraph one is moderately long.\n\n", 1) +
strings.Repeat("paragraph two also moderately long.\n\n", 1) +
strings.Repeat("paragraph three is moderately long.\n\n", 1) +
strings.Repeat("paragraph four is moderately long.\n\n", 1)
out := vectorstore.ChunkMarkdown(src, 100)
assert.Greater(t, len(out), 1, "oversized section should sub-split at paragraph boundaries")
for i, c := range out {
assert.LessOrEqual(t, len(c), 200,
"chunk %d exceeds 2x maxBytes: %d", i, len(c))
}
}
func TestChunkMarkdown_PreservesContent(t *testing.T) {
src := "# H1\n\nfirst section body.\n\n## H2a\n\nsecond section body.\n\n## H2b\n\nthird section body.\n"
out := vectorstore.ChunkMarkdown(src, 50)
joined := strings.Join(out, "")
// All non-whitespace tokens from src must appear in the joined output
for _, token := range []string{"H1", "first", "H2a", "second", "H2b", "third"} {
assert.Contains(t, joined, token, "token %q missing after chunking", token)
}
}
func TestChunkMarkdown_NumberedSuffix(t *testing.T) {
out := vectorstore.NumberChunks("knowledge/foo.md", []string{"a", "b", "c"})
require.Len(t, out, 3)
assert.Equal(t, "knowledge/foo.md#0001", out[0].Path)
assert.Equal(t, "knowledge/foo.md#0002", out[1].Path)
assert.Equal(t, "knowledge/foo.md#0003", out[2].Path)
assert.Equal(t, "a", out[0].Content)
}
func TestParentPath_StripsChunkSuffix(t *testing.T) {
assert.Equal(t, "knowledge/foo.md", vectorstore.ParentPath("knowledge/foo.md#0001"))
assert.Equal(t, "knowledge/foo.md", vectorstore.ParentPath("knowledge/foo.md"))
assert.Equal(t, "wiki/a/b.md", vectorstore.ParentPath("wiki/a/b.md#9999"))
}
+161
View File
@@ -0,0 +1,161 @@
// Package vectorstore stores brain note embeddings in pgvector on the
// shared postgres18 instance. One row per markdown path, cosine-distance
// indexed via HNSW for sub-millisecond top-k retrieval.
package vectorstore
import (
"context"
"errors"
"fmt"
"strings"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
)
// Hit is a single result from a cosine-distance search.
type Hit struct {
Path string
Distance float64 // 0 = identical, 2 = opposite
}
// PGStore is a pgvector-backed embeddings store. Construct with New and
// call Init once to create the table + HNSW index. Use Close to release
// the underlying pool.
type PGStore struct {
pool *pgxpool.Pool
}
// New opens a connection pool against dsn (a libpq-style URL). Caller
// owns the resulting *PGStore and must invoke Close.
func New(ctx context.Context, dsn string) (*PGStore, error) {
pool, err := pgxpool.New(ctx, dsn)
if err != nil {
return nil, fmt.Errorf("pgxpool: %w", err)
}
if err := pool.Ping(ctx); err != nil {
pool.Close()
return nil, fmt.Errorf("ping: %w", err)
}
return &PGStore{pool: pool}, nil
}
// Close releases the underlying connection pool.
func (s *PGStore) Close() {
if s.pool != nil {
s.pool.Close()
}
}
// Init creates the brain_embeddings table and its HNSW index if they
// don't already exist. Safe to call on every startup. Assumes the
// `vector` extension is already installed (one-time DBA setup; see
// scripts/brain-embeddings-init.sql).
func (s *PGStore) Init(ctx context.Context) error {
const ddl = `
CREATE TABLE IF NOT EXISTS brain_embeddings (
path TEXT PRIMARY KEY,
embedding vector(768) NOT NULL,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS brain_embeddings_embedding_idx
ON brain_embeddings USING hnsw (embedding vector_cosine_ops);
`
_, err := s.pool.Exec(ctx, ddl)
return err
}
// Upsert inserts or replaces the embedding for path. Embedding must be
// 768-dim (nomic-embed-text). Caller is responsible for normalising
// paths to forward-slash form.
func (s *PGStore) Upsert(ctx context.Context, path string, embedding []float32) error {
if len(embedding) != 768 {
return fmt.Errorf("expected 768-dim embedding, got %d", len(embedding))
}
_, err := s.pool.Exec(ctx, `
INSERT INTO brain_embeddings (path, embedding, updated_at)
VALUES ($1, $2, now())
ON CONFLICT (path) DO UPDATE
SET embedding = EXCLUDED.embedding, updated_at = now()
`, path, vectorLiteral(embedding))
return err
}
// Delete removes the row at path. No-op when the row doesn't exist.
func (s *PGStore) Delete(ctx context.Context, path string) error {
_, err := s.pool.Exec(ctx, `DELETE FROM brain_embeddings WHERE path = $1`, path)
return err
}
// Search returns the top-limit nearest paths by cosine distance.
func (s *PGStore) Search(ctx context.Context, query []float32, limit int) ([]Hit, error) {
if len(query) != 768 {
return nil, fmt.Errorf("expected 768-dim query, got %d", len(query))
}
if limit <= 0 {
limit = 10
}
rows, err := s.pool.Query(ctx, `
SELECT path, embedding <=> $1 AS distance
FROM brain_embeddings
ORDER BY embedding <=> $1
LIMIT $2
`, vectorLiteral(query), limit)
if err != nil {
return nil, fmt.Errorf("query: %w", err)
}
defer rows.Close()
var hits []Hit
for rows.Next() {
var h Hit
if err := rows.Scan(&h.Path, &h.Distance); err != nil {
return nil, fmt.Errorf("scan: %w", err)
}
hits = append(hits, h)
}
if err := rows.Err(); err != nil && !errors.Is(err, pgx.ErrNoRows) {
return nil, err
}
return hits, nil
}
// KnownPathsWithTime returns every embedded chunk path paired with the
// row's updated_at. Sync uses the timestamps to decide whether a file
// has been edited since its chunks were last embedded — when the file's
// mtime exceeds the oldest chunk's updated_at, the file is re-embedded.
func (s *PGStore) KnownPathsWithTime(ctx context.Context) (map[string]time.Time, error) {
rows, err := s.pool.Query(ctx, `SELECT path, updated_at FROM brain_embeddings`)
if err != nil {
return nil, fmt.Errorf("query paths: %w", err)
}
defer rows.Close()
out := make(map[string]time.Time)
for rows.Next() {
var (
p string
t time.Time
)
if err := rows.Scan(&p, &t); err != nil {
return nil, err
}
out[p] = t
}
return out, rows.Err()
}
// vectorLiteral renders a Go float32 slice as the literal representation
// pgvector accepts as a parametric input: `[v1,v2,...,vN]`.
func vectorLiteral(v []float32) string {
var b strings.Builder
b.WriteByte('[')
for i, x := range v {
if i > 0 {
b.WriteByte(',')
}
fmt.Fprintf(&b, "%g", x)
}
b.WriteByte(']')
return b.String()
}
+94
View File
@@ -0,0 +1,94 @@
package vectorstore_test
import (
"context"
"os"
"testing"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// integration tests run against a real postgres18 + pgvector. Gated by
// BRAIN_PG_TEST_DSN so `task check` stays hermetic on hosts without a
// reachable database.
//
// To run:
// BRAIN_PG_TEST_DSN='postgres://brain_app:pwd@127.0.0.1:5432/brain' \
// go test ./internal/vectorstore/... -run Integration
func dsn(t *testing.T) string {
t.Helper()
v := os.Getenv("BRAIN_PG_TEST_DSN")
if v == "" {
t.Skip("BRAIN_PG_TEST_DSN not set; skipping pgvector integration tests")
}
return v
}
func freshStore(t *testing.T) (*vectorstore.PGStore, context.Context) {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
t.Cleanup(cancel)
s, err := vectorstore.New(ctx, dsn(t))
require.NoError(t, err)
t.Cleanup(s.Close)
require.NoError(t, s.Init(ctx))
// Clean slate per test.
_, _ = s.KnownPathsWithTime(ctx)
require.NoError(t, s.Delete(ctx, "%test-fixture%"))
return s, ctx
}
func vec(dim int, fill float32) []float32 {
v := make([]float32, dim)
for i := range v {
v[i] = fill
}
return v
}
func TestIntegration_UpsertAndSearch(t *testing.T) {
s, ctx := freshStore(t)
require.NoError(t, s.Upsert(ctx, "wiki/a.md", vec(768, 1.0)))
require.NoError(t, s.Upsert(ctx, "wiki/b.md", vec(768, -1.0)))
hits, err := s.Search(ctx, vec(768, 1.0), 2)
require.NoError(t, err)
require.GreaterOrEqual(t, len(hits), 1)
assert.Equal(t, "wiki/a.md", hits[0].Path)
assert.InDelta(t, 0.0, hits[0].Distance, 1e-5)
t.Cleanup(func() {
_ = s.Delete(ctx, "wiki/a.md")
_ = s.Delete(ctx, "wiki/b.md")
})
}
func TestIntegration_KnownPathsWithTime(t *testing.T) {
s, ctx := freshStore(t)
before := time.Now()
require.NoError(t, s.Upsert(ctx, "wiki/k.md", vec(768, 0.5)))
t.Cleanup(func() { _ = s.Delete(ctx, "wiki/k.md") })
paths, err := s.KnownPathsWithTime(ctx)
require.NoError(t, err)
at, ok := paths["wiki/k.md"]
require.True(t, ok)
assert.False(t, at.IsZero(), "updated_at must not be zero")
assert.WithinDuration(t, before, at, 5*time.Second, "updated_at must be recent")
}
func TestUpsert_RejectsWrongDimension(t *testing.T) {
s := &vectorstore.PGStore{}
err := s.Upsert(context.Background(), "x", vec(100, 0))
require.Error(t, err)
}
func TestSearch_RejectsWrongDimension(t *testing.T) {
s := &vectorstore.PGStore{}
_, err := s.Search(context.Background(), vec(100, 0), 5)
require.Error(t, err)
}
+205
View File
@@ -0,0 +1,205 @@
package vectorstore
import (
"context"
"fmt"
"log/slog"
"os"
"path/filepath"
"strings"
"time"
)
// Embedder produces dense vectors. The embed package's Client satisfies
// this; it's declared locally so vectorstore doesn't depend on embed.
type Embedder interface {
Embed(ctx context.Context, text string) ([]float32, error)
}
// Store is the subset of PGStore that Sync needs. Lets tests stub it.
type Store interface {
// KnownPathsWithTime returns every embedded chunk path paired with the
// row's updated_at. Sync uses the timestamp to detect edits — a file
// whose mtime is newer than ANY of its chunks' updated_at is re-embedded
// from scratch (old chunks deleted, fresh chunks upserted).
KnownPathsWithTime(ctx context.Context) (map[string]time.Time, error)
Upsert(ctx context.Context, path string, embedding []float32) error
Delete(ctx context.Context, path string) error
}
// SyncResult tallies what Sync did. Returned for logs / metrics; callers
// generally don't act on the fields directly.
type SyncResult struct {
Added int
Updated int
Deleted int
Errors []error
}
// scanDirs is the set of brainDir subdirectories whose .md files are
// embedded for vector retrieval. wiki/ holds LLM-extracted entity and
// source pages; knowledge/ holds curated hand-written entries.
var scanDirs = []string{"wiki", "knowledge"}
// maxChunkBytes is the per-chunk byte budget passed to ChunkMarkdown.
// Sized to fit comfortably under nomic-embed-text's 2048-token default
// context (~4 chars/token for English markdown → ~8 KB ceiling; we sit
// at 4 KB to leave headroom for unicode, code blocks, and tokenizer
// variance).
const maxChunkBytes = 4000
// Sync brings the embedding store in line with brain/{wiki,knowledge}/
// on disk:
// - new files (in the tree, not in the store) get embedded + upserted
// - files whose mtime exceeds the store's updated_at get re-embedded
// - files no longer on disk get deleted from the store
//
// Designed to be called on a ticker. Best-effort: per-file errors are
// collected into SyncResult.Errors and do not abort the run.
func Sync(ctx context.Context, brainDir string, store Store, embedder Embedder) (SyncResult, error) {
var res SyncResult
if store == nil || embedder == nil {
return res, nil
}
known, err := store.KnownPathsWithTime(ctx)
if err != nil {
return res, fmt.Errorf("known paths: %w", err)
}
// Group known chunks by parent path and remember the EARLIEST
// updated_at per parent. A file is considered stale if its mtime is
// after the oldest of its chunk rows — i.e. at least one chunk hasn't
// been refreshed since the last edit. Also keep the full chunk-path
// list per parent so we can delete every old chunk before re-embedding
// (handles "file shrunk → fewer chunks → orphan rows" cleanly).
type parentState struct {
minUpdatedAt time.Time
chunkPaths []string
}
parents := make(map[string]*parentState, len(known))
for p, t := range known {
parent := ParentPath(p)
ps, ok := parents[parent]
if !ok {
ps = &parentState{minUpdatedAt: t}
parents[parent] = ps
} else if t.Before(ps.minUpdatedAt) {
ps.minUpdatedAt = t
}
ps.chunkPaths = append(ps.chunkPaths, p)
}
seenParents := make(map[string]struct{})
for _, sub := range scanDirs {
root := filepath.Join(brainDir, sub)
if _, err := os.Stat(root); os.IsNotExist(err) {
continue
}
err = filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() || !strings.HasSuffix(path, ".md") || d.Name() == "_index.md" {
return nil
}
rel, err := filepath.Rel(brainDir, path)
if err != nil {
return err
}
relSlash := filepath.ToSlash(rel)
seenParents[relSlash] = struct{}{}
if ps, ok := parents[relSlash]; ok {
// File already has chunks in the store. Re-embed only when
// the file has been edited since the oldest chunk was
// written. Tolerate clock skew with a sub-second grace.
info, statErr := d.Info()
if statErr != nil {
res.Errors = append(res.Errors, fmt.Errorf("stat %s: %w", relSlash, statErr))
return nil
}
if !info.ModTime().After(ps.minUpdatedAt) {
return nil
}
// Stale: delete old chunks before re-embedding so a shrunk
// file doesn't leave orphan rows at higher #NNNN indexes.
for _, oldPath := range ps.chunkPaths {
if delErr := store.Delete(ctx, oldPath); delErr != nil {
res.Errors = append(res.Errors, fmt.Errorf("delete %s for re-embed: %w", oldPath, delErr))
return nil
}
}
}
content, readErr := os.ReadFile(path)
if readErr != nil {
res.Errors = append(res.Errors, fmt.Errorf("read %s: %w", relSlash, readErr))
return nil
}
chunks := NumberChunks(relSlash, ChunkMarkdown(string(content), maxChunkBytes))
for _, ch := range chunks {
vec, embErr := embedder.Embed(ctx, ch.Content)
if embErr != nil {
res.Errors = append(res.Errors, fmt.Errorf("embed %s: %w", ch.Path, embErr))
continue
}
if upErr := store.Upsert(ctx, ch.Path, vec); upErr != nil {
res.Errors = append(res.Errors, fmt.Errorf("upsert %s: %w", ch.Path, upErr))
continue
}
res.Added++
}
return nil
})
if err != nil {
return res, fmt.Errorf("walk %s: %w", sub, err)
}
}
// Drop chunk rows whose parent file is gone.
for path := range known {
if _, ok := seenParents[ParentPath(path)]; ok {
continue
}
if err := store.Delete(ctx, path); err != nil {
res.Errors = append(res.Errors, fmt.Errorf("delete %s: %w", path, err))
continue
}
res.Deleted++
}
return res, nil
}
// StartSync launches Sync on a ticker in a background goroutine. The
// goroutine exits when ctx is cancelled. Failures are logged via slog.
func StartSync(ctx context.Context, brainDir string, store Store, embedder Embedder, interval time.Duration) {
if interval <= 0 {
interval = 5 * time.Minute
}
go func() {
t := time.NewTicker(interval)
defer t.Stop()
// Run once immediately so first-boot doesn't wait a full tick.
if r, err := Sync(ctx, brainDir, store, embedder); err != nil {
slog.Error("embed sync failed", "err", err)
} else if r.Added+r.Deleted > 0 || len(r.Errors) > 0 {
slog.Info("embed sync", "added", r.Added, "deleted", r.Deleted, "errors", len(r.Errors))
for _, e := range r.Errors {
slog.Warn("embed sync item failed", "err", e)
}
}
for {
select {
case <-ctx.Done():
return
case <-t.C:
if r, err := Sync(ctx, brainDir, store, embedder); err != nil {
slog.Error("embed sync failed", "err", err)
} else if r.Added+r.Deleted > 0 || len(r.Errors) > 0 {
slog.Info("embed sync", "added", r.Added, "deleted", r.Deleted, "errors", len(r.Errors))
}
}
}
}()
}
+274
View File
@@ -0,0 +1,274 @@
package vectorstore_test
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
type stubStore struct {
// known maps chunk-path → updated_at. Tests that don't care about
// re-embed-on-mtime use a far-future time so the Sync skip path
// always wins. Tests that do exercise the mtime path set the
// updated_at explicitly.
known map[string]time.Time
upserts map[string][]float32
deletes []string
failNext error
}
// farFuture is "newer than any file mtime", used as the default
// updated_at in stubs that don't care about re-embed behavior.
var farFuture = time.Now().Add(24 * time.Hour)
func (s *stubStore) KnownPathsWithTime(_ context.Context) (map[string]time.Time, error) {
out := make(map[string]time.Time, len(s.known))
for k, t := range s.known {
if t.IsZero() {
t = farFuture
}
out[k] = t
}
return out, nil
}
func (s *stubStore) Upsert(_ context.Context, path string, v []float32) error {
if s.failNext != nil {
err := s.failNext
s.failNext = nil
return err
}
if s.upserts == nil {
s.upserts = make(map[string][]float32)
}
s.upserts[path] = v
return nil
}
func (s *stubStore) Delete(_ context.Context, path string) error {
s.deletes = append(s.deletes, path)
return nil
}
type stubEmbedder struct {
vec []float32
err error
}
func (e stubEmbedder) Embed(_ context.Context, _ string) ([]float32, error) {
return e.vec, e.err
}
func writeNote(t *testing.T, dir, rel, body string) {
t.Helper()
full := filepath.Join(dir, rel)
require.NoError(t, os.MkdirAll(filepath.Dir(full), 0o755))
require.NoError(t, os.WriteFile(full, []byte(body), 0o644))
}
func TestSync_AddsNewFiles(t *testing.T) {
dir := t.TempDir()
writeNote(t, dir, "wiki/jepa-fx/facts/x.md", "body of x")
writeNote(t, dir, "wiki/jepa-fx/facts/y.md", "body of y")
store := &stubStore{known: map[string]time.Time{}}
emb := stubEmbedder{vec: make([]float32, 768)}
res, err := vectorstore.Sync(context.Background(), dir, store, emb)
require.NoError(t, err)
assert.Equal(t, 2, res.Added)
assert.Empty(t, res.Deleted)
assert.Contains(t, store.upserts, "wiki/jepa-fx/facts/x.md#0001")
assert.Contains(t, store.upserts, "wiki/jepa-fx/facts/y.md#0001")
}
func TestSync_SkipsAlreadyKnown(t *testing.T) {
dir := t.TempDir()
writeNote(t, dir, "wiki/a/facts/x.md", "x")
store := &stubStore{known: map[string]time.Time{"wiki/a/facts/x.md#0001": {}}}
emb := stubEmbedder{vec: make([]float32, 768)}
res, err := vectorstore.Sync(context.Background(), dir, store, emb)
require.NoError(t, err)
assert.Equal(t, 0, res.Added)
assert.Empty(t, store.upserts)
}
func TestSync_DeletesDisappearedFiles(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki"), 0o755))
// store has a path that doesn't exist on disk anymore
store := &stubStore{known: map[string]time.Time{"wiki/old/facts/ghost.md#0001": {}}}
res, err := vectorstore.Sync(context.Background(), dir, &stubStoreWithDelete{stubStore: store}, stubEmbedder{vec: make([]float32, 768)})
require.NoError(t, err)
assert.Equal(t, 1, res.Deleted)
}
// stubStoreWithDelete is a thin wrapper to capture Delete calls;
// stubStore already implements Delete but we need the wrapper to mix
// store interfaces with sync-specific expectations.
type stubStoreWithDelete struct {
*stubStore
}
func TestSync_SkipsIndexFiles(t *testing.T) {
dir := t.TempDir()
writeNote(t, dir, "wiki/a/_index.md", "moc")
writeNote(t, dir, "wiki/a/facts/real.md", "body")
store := &stubStore{known: map[string]time.Time{}}
res, err := vectorstore.Sync(context.Background(), dir, store, stubEmbedder{vec: make([]float32, 768)})
require.NoError(t, err)
assert.Equal(t, 1, res.Added)
assert.NotContains(t, store.upserts, "wiki/a/_index.md#0001")
}
func TestSync_ScansKnowledgeDir(t *testing.T) {
dir := t.TempDir()
writeNote(t, dir, "wiki/a/facts/x.md", "x")
writeNote(t, dir, "knowledge/2026-05-19-koala-gpu-setup.md", "knowledge body")
store := &stubStore{known: map[string]time.Time{}}
emb := stubEmbedder{vec: make([]float32, 768)}
res, err := vectorstore.Sync(context.Background(), dir, store, emb)
require.NoError(t, err)
assert.Equal(t, 2, res.Added)
assert.Contains(t, store.upserts, "wiki/a/facts/x.md#0001")
assert.Contains(t, store.upserts, "knowledge/2026-05-19-koala-gpu-setup.md#0001")
}
func TestSync_ChunksLongFiles(t *testing.T) {
dir := t.TempDir()
// Build a file that's well over the chunk byte budget. Multi-section
// markdown so the chunker has heading boundaries to cut on.
body := "# Doc\n\nintro line.\n\n"
for i := 0; i < 10; i++ {
body += "## Section " + string(rune('A'+i)) + "\n\n"
body += strings.Repeat("This section has a fair amount of content. ", 50) + "\n\n"
}
writeNote(t, dir, "knowledge/long.md", body)
store := &stubStore{known: map[string]time.Time{}}
emb := stubEmbedder{vec: make([]float32, 768)}
res, err := vectorstore.Sync(context.Background(), dir, store, emb)
require.NoError(t, err)
assert.Greater(t, res.Added, 1, "long file should produce multiple chunk rows")
// Every upserted path for this file must be a chunk path.
chunkCount := 0
for p := range store.upserts {
if strings.HasPrefix(p, "knowledge/long.md#") {
chunkCount++
}
}
assert.Equal(t, res.Added, chunkCount, "all rows for long file should be chunk-suffixed")
// The bare parent path must NOT be upserted directly.
assert.NotContains(t, store.upserts, "knowledge/long.md")
}
func TestSync_ShortFileGetsSingleChunkRow(t *testing.T) {
dir := t.TempDir()
writeNote(t, dir, "wiki/short.md", "tiny body\n")
store := &stubStore{known: map[string]time.Time{}}
emb := stubEmbedder{vec: make([]float32, 768)}
res, err := vectorstore.Sync(context.Background(), dir, store, emb)
require.NoError(t, err)
assert.Equal(t, 1, res.Added)
assert.Contains(t, store.upserts, "wiki/short.md#0001")
}
func TestSync_SkipsFileIfAnyChunkAlreadyKnown(t *testing.T) {
dir := t.TempDir()
writeNote(t, dir, "wiki/foo.md", "body\n")
store := &stubStore{known: map[string]time.Time{
"wiki/foo.md#0001": {},
}}
emb := stubEmbedder{vec: make([]float32, 768)}
res, err := vectorstore.Sync(context.Background(), dir, store, emb)
require.NoError(t, err)
assert.Equal(t, 0, res.Added)
assert.Empty(t, store.upserts)
}
func TestSync_DeletesAllChunksOfDisappearedFile(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki"), 0o755))
store := &stubStore{known: map[string]time.Time{
"wiki/ghost.md#0001": {},
"wiki/ghost.md#0002": {},
"wiki/ghost.md#0003": {},
}}
res, err := vectorstore.Sync(context.Background(), dir, store, stubEmbedder{vec: make([]float32, 768)})
require.NoError(t, err)
assert.Equal(t, 3, res.Deleted)
}
func TestSync_ReembedsFileWhenMtimeNewer(t *testing.T) {
dir := t.TempDir()
writeNote(t, dir, "wiki/edited.md", "original body\n")
// Force the file's mtime ahead of any plausible store updated_at.
future := time.Now().Add(1 * time.Hour)
require.NoError(t, os.Chtimes(filepath.Join(dir, "wiki/edited.md"), future, future))
store := &stubStore{
known: map[string]time.Time{
// Existing chunk row pre-dates the file's mtime.
"wiki/edited.md#0001": time.Now().Add(-1 * time.Hour),
},
}
emb := stubEmbedder{vec: make([]float32, 768)}
res, err := vectorstore.Sync(context.Background(), dir, store, emb)
require.NoError(t, err)
assert.Equal(t, 1, res.Added, "file with newer mtime should be re-embedded")
assert.Contains(t, store.upserts, "wiki/edited.md#0001")
// Old chunks of the same parent must be deleted before re-embed so
// shrunk files don't leave orphan rows at higher #NNNN indexes.
assert.Contains(t, store.deletes, "wiki/edited.md#0001")
}
func TestSync_SkipsFileWhenMtimeOlder(t *testing.T) {
dir := t.TempDir()
writeNote(t, dir, "wiki/stable.md", "body\n")
// Backdate mtime to before the store's recorded updated_at.
past := time.Now().Add(-2 * time.Hour)
require.NoError(t, os.Chtimes(filepath.Join(dir, "wiki/stable.md"), past, past))
store := &stubStore{
known: map[string]time.Time{
"wiki/stable.md#0001": time.Now(),
},
}
emb := stubEmbedder{vec: make([]float32, 768)}
res, err := vectorstore.Sync(context.Background(), dir, store, emb)
require.NoError(t, err)
assert.Equal(t, 0, res.Added)
assert.Empty(t, store.upserts)
assert.Empty(t, store.deletes)
}
func TestSync_NoOpWhenComponentsNil(t *testing.T) {
dir := t.TempDir()
writeNote(t, dir, "wiki/a/facts/x.md", "x")
res, err := vectorstore.Sync(context.Background(), dir, nil, nil)
require.NoError(t, err)
assert.Equal(t, 0, res.Added)
}
func TestSync_CollectsEmbedderErrors(t *testing.T) {
dir := t.TempDir()
writeNote(t, dir, "wiki/a/facts/x.md", "x")
store := &stubStore{known: map[string]time.Time{}}
emb := stubEmbedder{err: errors.New("upstream down")}
res, err := vectorstore.Sync(context.Background(), dir, store, emb)
require.NoError(t, err)
assert.Equal(t, 0, res.Added)
assert.Len(t, res.Errors, 1)
}
+2 -2
View File
@@ -11,7 +11,7 @@ import (
type RoutingConfig struct { type RoutingConfig struct {
Port string // ROUTING_PORT, default 3210 Port string // ROUTING_PORT, default 3210
MCPAuthToken string // ROUTING_MCP_TOKEN, optional bearer token MCPAuthToken string // ROUTING_MCP_TOKEN, optional bearer token
LiteLLMBaseURL string // LITELLM_BASE_URL, default http://piguard:4000 LiteLLMBaseURL string // LITELLM_BASE_URL, default https://llm-api.d-ma.be
LiteLLMAPIKey string // LITELLM_API_KEY LiteLLMAPIKey string // LITELLM_API_KEY
BrainURL string // BRAIN_URL, default http://ingestion.supervisor:3300 BrainURL string // BRAIN_URL, default http://ingestion.supervisor:3300
FastModel string // HYPERGUILD_FAST_MODEL, default koala/qwen35-9b-fast FastModel string // HYPERGUILD_FAST_MODEL, default koala/qwen35-9b-fast
@@ -41,7 +41,7 @@ func LoadRouting() (RoutingConfig, error) {
cfg := RoutingConfig{ cfg := RoutingConfig{
Port: envOr("ROUTING_PORT", "3210"), Port: envOr("ROUTING_PORT", "3210"),
MCPAuthToken: os.Getenv("ROUTING_MCP_TOKEN"), MCPAuthToken: os.Getenv("ROUTING_MCP_TOKEN"),
LiteLLMBaseURL: envOr("LITELLM_BASE_URL", "http://piguard:4000"), LiteLLMBaseURL: envOr("LITELLM_BASE_URL", "https://llm-api.d-ma.be"),
LiteLLMAPIKey: os.Getenv("LITELLM_API_KEY"), LiteLLMAPIKey: os.Getenv("LITELLM_API_KEY"),
BrainURL: envOr("BRAIN_URL", "http://ingestion.supervisor:3300"), BrainURL: envOr("BRAIN_URL", "http://ingestion.supervisor:3300"),
FastModel: envOr("HYPERGUILD_FAST_MODEL", "koala/qwen35-9b-fast"), FastModel: envOr("HYPERGUILD_FAST_MODEL", "koala/qwen35-9b-fast"),
+1 -1
View File
@@ -22,7 +22,7 @@ func TestLoadRoutingDefaults(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, "3210", cfg.Port) assert.Equal(t, "3210", cfg.Port)
assert.Equal(t, "", cfg.MCPAuthToken) assert.Equal(t, "", cfg.MCPAuthToken)
assert.Equal(t, "http://piguard:4000", cfg.LiteLLMBaseURL) assert.Equal(t, "https://llm-api.d-ma.be", cfg.LiteLLMBaseURL)
assert.Equal(t, "http://ingestion.supervisor:3300", cfg.BrainURL) assert.Equal(t, "http://ingestion.supervisor:3300", cfg.BrainURL)
assert.Equal(t, "koala/qwen35-9b-fast", cfg.FastModel) assert.Equal(t, "koala/qwen35-9b-fast", cfg.FastModel)
assert.Equal(t, "iguana/gemma4-26b", cfg.ThinkingModel) assert.Equal(t, "iguana/gemma4-26b", cfg.ThinkingModel)
+18 -3
View File
@@ -7,6 +7,7 @@ import (
"bytes" "bytes"
"context" "context"
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"io" "io"
"net/http" "net/http"
@@ -20,13 +21,27 @@ type Client struct {
http *http.Client http *http.Client
} }
// New returns a Client. token may be empty for unauthenticated servers. // ErrTokenRequired is returned by New when token is empty. Empty token
func New(url, token string) *Client { // causes mcpclient to omit the Authorization header at request time,
// which is silently misread as 401 by bearer-auth servers — see
// hyperguild #13 and the brain entry on the failure mode.
var ErrTokenRequired = errors.New("mcpclient: token required")
// New returns a Client. Returns ErrTokenRequired when token is empty:
// every MCP server we talk to today is bearer-protected, and an empty
// token is always a configuration bug (typically a Kubernetes Secret
// missing the expected key, see hyperguild #13). Callers that genuinely
// need an unauthenticated client should construct &Client{} directly in
// tests, not call New.
func New(url, token string) (*Client, error) {
if token == "" {
return nil, ErrTokenRequired
}
return &Client{ return &Client{
url: url, url: url,
token: token, token: token,
http: &http.Client{Timeout: 60 * time.Second}, http: &http.Client{Timeout: 60 * time.Second},
} }, nil
} }
// WithHTTPClient overrides the underlying HTTP client (test injection). // WithHTTPClient overrides the underlying HTTP client (test injection).
+18 -8
View File
@@ -14,6 +14,13 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
) )
func TestNew_EmptyTokenFailsFast(t *testing.T) {
c, err := mcpclient.New("http://example.invalid", "")
require.Error(t, err)
require.Nil(t, c)
require.ErrorIs(t, err, mcpclient.ErrTokenRequired)
}
func TestCallTool_Success(t *testing.T) { func TestCallTool_Success(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, http.MethodPost, r.Method) assert.Equal(t, http.MethodPost, r.Method)
@@ -30,13 +37,13 @@ func TestCallTool_Success(t *testing.T) {
})) }))
defer srv.Close() defer srv.Close()
c := mcpclient.New(srv.URL, "tok") c, err := mcpclient.New(srv.URL, "tok")
require.NoError(t, err)
var out struct { var out struct {
OK bool `json:"ok"` OK bool `json:"ok"`
N int `json:"n"` N int `json:"n"`
} }
err := c.CallTool(context.Background(), "x_y", map[string]any{"a": 1}, &out) require.NoError(t, c.CallTool(context.Background(), "x_y", map[string]any{"a": 1}, &out))
require.NoError(t, err)
assert.True(t, out.OK) assert.True(t, out.OK)
assert.Equal(t, 7, out.N) assert.Equal(t, 7, out.N)
} }
@@ -48,8 +55,9 @@ func TestCallTool_RPCError(t *testing.T) {
})) }))
defer srv.Close() defer srv.Close()
c := mcpclient.New(srv.URL, "") c, err := mcpclient.New(srv.URL, "test")
err := c.CallTool(context.Background(), "x", nil, nil) require.NoError(t, err)
err = c.CallTool(context.Background(), "x", nil, nil)
require.Error(t, err) require.Error(t, err)
var me *mcpclient.Error var me *mcpclient.Error
require.True(t, errors.As(err, &me)) require.True(t, errors.As(err, &me))
@@ -64,8 +72,9 @@ func TestCallTool_HTTPError(t *testing.T) {
})) }))
defer srv.Close() defer srv.Close()
c := mcpclient.New(srv.URL, "") c, err := mcpclient.New(srv.URL, "test")
err := c.CallTool(context.Background(), "x", nil, nil) require.NoError(t, err)
err = c.CallTool(context.Background(), "x", nil, nil)
require.Error(t, err) require.Error(t, err)
assert.Contains(t, err.Error(), "401") assert.Contains(t, err.Error(), "401")
} }
@@ -77,6 +86,7 @@ func TestCallTool_NilResult(t *testing.T) {
})) }))
defer srv.Close() defer srv.Close()
c := mcpclient.New(srv.URL, "") c, err := mcpclient.New(srv.URL, "test")
require.NoError(t, err)
require.NoError(t, c.CallTool(context.Background(), "x", nil, nil)) require.NoError(t, c.CallTool(context.Background(), "x", nil, nil))
} }
+43 -32
View File
@@ -13,12 +13,13 @@ import (
) )
type createArgs struct { type createArgs struct {
Name string `json:"name"` Name string `json:"name"`
Description string `json:"description"` Description string `json:"description"`
Hypothesis string `json:"hypothesis"` Hypothesis string `json:"hypothesis"`
Folder string `json:"folder"` Folder string `json:"folder"`
Stack string `json:"stack"` Stack string `json:"stack"`
Private bool `json:"private"` Private bool `json:"private"`
MirrorToGitHub bool `json:"mirror_to_github,omitempty"`
} }
type createResult struct { type createResult struct {
@@ -59,11 +60,12 @@ func (s *Skill) handleCreate(ctx context.Context, raw json.RawMessage) (json.Raw
tmpl := templateFor(args.Stack) tmpl := templateFor(args.Stack)
giteaURL := fmt.Sprintf("http://gitea.d-ma.be/%s/%s", s.cfg.GiteaOwner, args.Name) giteaURL := fmt.Sprintf("http://gitea.d-ma.be/%s/%s", s.cfg.GiteaOwner, args.Name)
githubURL := fmt.Sprintf("https://github.com/%s/%s", s.cfg.GitHubOwner, args.Name)
res := createResult{ res := createResult{
GiteaURL: giteaURL, GiteaURL: giteaURL,
GitHubURL: githubURL, }
if args.MirrorToGitHub {
res.GitHubURL = fmt.Sprintf("https://github.com/%s/%s", s.cfg.GitHubOwner, args.Name)
} }
// Step 1: create_project_from_template. If the repo already exists, // Step 1: create_project_from_template. If the repo already exists,
@@ -75,30 +77,36 @@ func (s *Skill) handleCreate(ctx context.Context, raw json.RawMessage) (json.Raw
} }
res.Reached = append(res.Reached, stepCreateRepo) res.Reached = append(res.Reached, stepCreateRepo)
// Step 2: create empty GitHub repo. Gitea's push-mirror cannot push // Steps 2+3 are skipped when MirrorToGitHub is false. Default per
// to a non-existent remote, so the destination must exist before // infra ADR (Gitea as true master, GitHub as optional opt-in): keep
// step 3 configures the mirror. Skipped when GitHub client is unset // client / business-logic / personal repos Gitea-only. Set
// (degraded mode — see Config.GitHub doc). // `mirror_to_github: true` for open-source projects that want a
if s.cfg.GitHub != nil { // public GitHub mirror (hyperguild, gitea-mcp, template-*).
if err := s.callCreateGitHubRepo(ctx, args); err != nil && !errors.Is(err, githubclient.ErrAlreadyExists) { if args.MirrorToGitHub {
return marshalPartial(res, stepCreateGitHub, err) // Step 2: create empty GitHub repo. Gitea's push-mirror cannot push
// to a non-existent remote, so the destination must exist before
// step 3 configures the mirror. Skipped when GitHub client is unset
// (degraded mode — see Config.GitHub doc).
if s.cfg.GitHub != nil {
if err := s.callCreateGitHubRepo(ctx, args); err != nil && !errors.Is(err, githubclient.ErrAlreadyExists) {
return marshalPartial(res, stepCreateGitHub, err)
}
res.Reached = append(res.Reached, stepCreateGitHub)
} }
res.Reached = append(res.Reached, stepCreateGitHub)
}
// Step 3: configure push mirror to GitHub. Idempotent: if a mirror with // Step 3: configure push mirror to GitHub. Idempotent: if a mirror with
// the same remote already exists, gitea-mcp returns Conflict; we swallow it. // the same remote already exists, gitea-mcp returns Conflict; we swallow it.
if err := s.callMirror(ctx, args.Name); err != nil { if err := s.callMirror(ctx, args.Name); err != nil {
if !isConflict(err) { if !isConflict(err) {
return marshalPartial(res, stepMirror, err) return marshalPartial(res, stepMirror, err)
}
} }
res.Reached = append(res.Reached, stepMirror)
} }
res.Reached = append(res.Reached, stepMirror)
// Step 3: commit staging namespace manifest to infra repo. Done before // Step 3: commit staging namespace manifest to infra repo. Done before
// the issue so the staging env is reconciling by the time the issue lands. // the issue so the staging env is reconciling by the time the issue lands.
branch := fmt.Sprintf("staging/%s", args.Name) if err := s.callInfraCommit(ctx, args.Name); err != nil {
if err := s.callInfraCommit(ctx, args.Name, branch); err != nil {
if !isConflict(err) { if !isConflict(err) {
return marshalPartial(res, stepInfraCommit, err) return marshalPartial(res, stepInfraCommit, err)
} }
@@ -171,17 +179,16 @@ func (s *Skill) callMirror(ctx context.Context, name string) error {
}, nil) }, nil)
} }
// callInfraCommit writes the staging namespace manifest into the infra repo // callInfraCommit writes the staging namespace manifest directly to infra
// on a dedicated branch. Flux picks it up after merge. // main. Flux reconciles within ~60s. See DECISIONS.md 2026-05-18.
func (s *Skill) callInfraCommit(ctx context.Context, name, branch string) error { func (s *Skill) callInfraCommit(ctx context.Context, name string) error {
manifest := stagingNamespaceManifest(name, time.Now().UTC().Format(time.RFC3339)) manifest := stagingNamespaceManifest(name, time.Now().UTC().Format(time.RFC3339))
return s.cfg.Client.CallTool(ctx, "file_write_branch", map[string]any{ return s.cfg.Client.CallTool(ctx, "file_write_branch", map[string]any{
"owner": s.cfg.GiteaOwner, "owner": s.cfg.GiteaOwner,
"name": s.cfg.InfraRepo, "name": s.cfg.InfraRepo,
"path": fmt.Sprintf("k3s/staging/%s/namespace.yaml", name), "path": fmt.Sprintf("k3s/staging/%s/namespace.yaml", name),
"content": manifest, "content": manifest,
"branch": branch, "branch": "main",
"base": "main",
"message": fmt.Sprintf("feat(staging): add namespace for %s\n\nGenerated by hyperguild project_create.", name), "message": fmt.Sprintf("feat(staging): add namespace for %s\n\nGenerated by hyperguild project_create.", name),
}, nil) }, nil)
} }
@@ -230,7 +237,11 @@ func experimentBrief(args createArgs, existed bool) string {
b.WriteString("- Repo created from `template-") b.WriteString("- Repo created from `template-")
b.WriteString(args.Stack) b.WriteString(args.Stack)
b.WriteString("` on Gitea.\n") b.WriteString("` on Gitea.\n")
b.WriteString("- Push-mirror configured to GitHub.\n") if args.MirrorToGitHub {
b.WriteString("- Push-mirror configured to GitHub.\n")
} else {
b.WriteString("- Gitea-only (no GitHub mirror — set `mirror_to_github: true` to opt in).\n")
}
b.WriteString("- Staging namespace manifest committed to infra repo.\n\n") b.WriteString("- Staging namespace manifest committed to infra repo.\n\n")
if existed { if existed {
b.WriteString("> Note: this repo already existed when `project_create` ran — provisioning steps were re-applied idempotently.\n") b.WriteString("> Note: this repo already existed when `project_create` ran — provisioning steps were re-applied idempotently.\n")
+79 -9
View File
@@ -125,7 +125,7 @@ func newSkill(t *testing.T, f *fakeGiteaMCP) (*project.Skill, *fakeGitHub) {
t.Cleanup(ghSrv.Close) t.Cleanup(ghSrv.Close)
return project.New(project.Config{ return project.New(project.Config{
Client: mcpclient.New(srv.URL, ""), Client: mustClient(t, srv.URL),
GitHub: githubclient.New("ghp_test").WithBaseURL(ghSrv.URL), GitHub: githubclient.New("ghp_test").WithBaseURL(ghSrv.URL),
GiteaOwner: "mathias", GiteaOwner: "mathias",
GitHubOwner: "mathiasb", GitHubOwner: "mathiasb",
@@ -141,13 +141,26 @@ func newSkillNoGitHub(t *testing.T, f *fakeGiteaMCP) *project.Skill {
srv := httptest.NewServer(f.handler()) srv := httptest.NewServer(f.handler())
t.Cleanup(srv.Close) t.Cleanup(srv.Close)
return project.New(project.Config{ return project.New(project.Config{
Client: mcpclient.New(srv.URL, ""), Client: mustClient(t, srv.URL),
GiteaOwner: "mathias", GiteaOwner: "mathias",
GitHubOwner: "mathiasb", GitHubOwner: "mathiasb",
InfraRepo: "infra", InfraRepo: "infra",
}) })
} }
// mustClient builds an mcpclient against an httptest server. Uses a
// non-empty dummy token because httptest servers don't enforce bearer
// auth, but mcpclient.New now requires non-empty token (see #13).
func mustClient(t *testing.T, url string) *mcpclient.Client {
t.Helper()
c, err := mcpclient.New(url, "test-token")
require.NoError(t, err)
return c
}
// happyArgs returns the minimal valid request. With the Gitea-as-true-master
// ADR shipped, this defaults to Gitea-only (mirror_to_github omitted = false).
// Tests that need the full Gitea + GitHub mirror flow use mirroredArgs().
func happyArgs() json.RawMessage { func happyArgs() json.RawMessage {
return json.RawMessage(`{ return json.RawMessage(`{
"name":"my-experiment", "name":"my-experiment",
@@ -159,6 +172,20 @@ func happyArgs() json.RawMessage {
}`) }`)
} }
// mirroredArgs is happyArgs + mirror_to_github=true — the explicit opt-in
// path. Equivalent to the pre-ADR default.
func mirroredArgs() json.RawMessage {
return json.RawMessage(`{
"name":"my-experiment",
"description":"One-line desc",
"hypothesis":"We believe X produces Y",
"folder":"AGENTS",
"stack":"go-agent",
"private":true,
"mirror_to_github":true
}`)
}
func TestProjectCreate_HappyPath(t *testing.T) { func TestProjectCreate_HappyPath(t *testing.T) {
f := &fakeGiteaMCP{ f := &fakeGiteaMCP{
Responses: map[string]any{ Responses: map[string]any{
@@ -167,7 +194,7 @@ func TestProjectCreate_HappyPath(t *testing.T) {
} }
skill, gh := newSkill(t, f) skill, gh := newSkill(t, f)
out, err := skill.Handle(context.Background(), "project_create", happyArgs()) out, err := skill.Handle(context.Background(), "project_create", mirroredArgs())
require.NoError(t, err) require.NoError(t, err)
var res map[string]any var res map[string]any
@@ -218,7 +245,7 @@ func TestProjectCreate_GitHubExists_Idempotent(t *testing.T) {
skill, gh := newSkill(t, f) skill, gh := newSkill(t, f)
gh.ReturnError = 422 // already exists gh.ReturnError = 422 // already exists
_, err := skill.Handle(context.Background(), "project_create", happyArgs()) _, err := skill.Handle(context.Background(), "project_create", mirroredArgs())
require.NoError(t, err, "422 already-exists should be idempotent") require.NoError(t, err, "422 already-exists should be idempotent")
require.Len(t, f.Calls, 4, "all gitea steps still run despite github 422") require.Len(t, f.Calls, 4, "all gitea steps still run despite github 422")
} }
@@ -228,7 +255,7 @@ func TestProjectCreate_GitHubFails(t *testing.T) {
skill, gh := newSkill(t, f) skill, gh := newSkill(t, f)
gh.ReturnError = 401 // bad PAT gh.ReturnError = 401 // bad PAT
out, err := skill.Handle(context.Background(), "project_create", happyArgs()) out, err := skill.Handle(context.Background(), "project_create", mirroredArgs())
require.Error(t, err) require.Error(t, err)
var res map[string]any var res map[string]any
require.NoError(t, json.Unmarshal(out, &res)) require.NoError(t, json.Unmarshal(out, &res))
@@ -245,7 +272,11 @@ func TestProjectCreate_NoGitHubClient_DegradedMode(t *testing.T) {
} }
skill := newSkillNoGitHub(t, f) skill := newSkillNoGitHub(t, f)
out, err := skill.Handle(context.Background(), "project_create", happyArgs()) // Use mirroredArgs so we exercise the GitHub-mirror path. With the
// GitHub client nil, the create_github_repo step is skipped but the
// mirror step still attempts to configure the push-mirror remote
// (degraded mode preserves the prior contract for opted-in projects).
out, err := skill.Handle(context.Background(), "project_create", mirroredArgs())
require.NoError(t, err) require.NoError(t, err)
var res map[string]any var res map[string]any
require.NoError(t, json.Unmarshal(out, &res)) require.NoError(t, json.Unmarshal(out, &res))
@@ -265,7 +296,7 @@ func TestProjectCreate_Idempotent_RepoExists(t *testing.T) {
} }
skill, _ := newSkill(t, f) skill, _ := newSkill(t, f)
out, err := skill.Handle(context.Background(), "project_create", happyArgs()) out, err := skill.Handle(context.Background(), "project_create", mirroredArgs())
require.NoError(t, err) require.NoError(t, err)
var res map[string]any var res map[string]any
@@ -285,7 +316,7 @@ func TestProjectCreate_MirrorFails(t *testing.T) {
} }
skill, _ := newSkill(t, f) skill, _ := newSkill(t, f)
out, err := skill.Handle(context.Background(), "project_create", happyArgs()) out, err := skill.Handle(context.Background(), "project_create", mirroredArgs())
require.Error(t, err) require.Error(t, err)
assert.Contains(t, err.Error(), `"mirror" failed`) assert.Contains(t, err.Error(), `"mirror" failed`)
@@ -307,7 +338,7 @@ func TestProjectCreate_InfraCommitFails(t *testing.T) {
} }
skill, _ := newSkill(t, f) skill, _ := newSkill(t, f)
out, err := skill.Handle(context.Background(), "project_create", happyArgs()) out, err := skill.Handle(context.Background(), "project_create", mirroredArgs())
require.Error(t, err) require.Error(t, err)
var res map[string]any var res map[string]any
@@ -341,6 +372,45 @@ func TestProjectCreate_ValidationErrors(t *testing.T) {
assert.Empty(t, f.Calls, "no upstream calls should occur on validation failure") assert.Empty(t, f.Calls, "no upstream calls should occur on validation failure")
} }
func TestProjectCreate_DefaultSkipsGitHubMirror(t *testing.T) {
// Default (mirror_to_github omitted) skips create_github_repo + mirror
// per the Gitea-as-true-master ADR. Gitea repo + staging namespace
// + issue still run; github_url is empty in the response.
f := &fakeGiteaMCP{
Responses: map[string]any{
"issue_create": map[string]any{"html_url": "http://gitea.d-ma.be/mathias/my-experiment/issues/1"},
},
}
skill, gh := newSkill(t, f)
out, err := skill.Handle(context.Background(), "project_create", happyArgs())
require.NoError(t, err)
var res map[string]any
require.NoError(t, json.Unmarshal(out, &res))
assert.Equal(t, "http://gitea.d-ma.be/mathias/my-experiment", res["gitea_url"])
assert.Equal(t, "", res["github_url"], "github_url must be empty when mirror not opted in")
assert.Equal(t, "http://gitea.d-ma.be/mathias/my-experiment/issues/1", res["issue_url"])
// 3 gitea-mcp calls: template create, staging file write, issue. NO mirror call.
require.Len(t, f.Calls, 3)
assert.Equal(t, "create_project_from_template", f.Calls[0].Tool)
assert.Equal(t, "file_write_branch", f.Calls[1].Tool)
assert.Equal(t, "issue_create", f.Calls[2].Tool)
// Zero GitHub API calls.
assert.Empty(t, gh.Calls, "no GitHub repo created when mirror_to_github is false")
// reached lists the Gitea-only path.
reached := res["reached"].([]any)
assert.Equal(t, []any{"create_repo", "infra_commit", "issue"}, reached)
// experiment-brief body reflects Gitea-only provisioning.
require.Contains(t, f.Calls[2].Args["body"], "Gitea-only")
require.NotContains(t, f.Calls[2].Args["body"], "Push-mirror configured")
}
func TestProjectCreate_UnknownTool(t *testing.T) { func TestProjectCreate_UnknownTool(t *testing.T) {
f := &fakeGiteaMCP{} f := &fakeGiteaMCP{}
skill, _ := newSkill(t, f) skill, _ := newSkill(t, f)
+11 -2
View File
@@ -79,13 +79,22 @@ func (s *Skill) Tools() []registry.ToolDef {
"description": "Selects template-go-agent or template-go-web.", "description": "Selects template-go-agent or template-go-web.",
}, },
"private": map[string]any{"type": "boolean"}, "private": map[string]any{"type": "boolean"},
"mirror_to_github": map[string]any{
"type": "boolean",
"description": "Default false. When true, also create an empty GitHub repo " +
"and configure a push-mirror from Gitea. Opt-in per the Gitea-as-true-master " +
"ADR — only set true for open-source projects (hyperguild, gitea-mcp, template-*). " +
"Never set true for client projects, business logic, or personal experiments.",
},
}, },
"required": []string{"name", "description", "hypothesis", "stack"}, "required": []string{"name", "description", "hypothesis", "stack"},
}) })
return []registry.ToolDef{ return []registry.ToolDef{
{ {
Name: "project_create", Name: "project_create",
Description: "Bootstrap a new project: Gitea repo from template, GitHub push-mirror, staging namespace manifest, experiment-brief issue. Idempotent — re-running with an existing repo returns the existing URLs.", Description: "Bootstrap a new project: Gitea repo from template, staging namespace manifest, " +
"experiment-brief issue. Optionally mirrors to GitHub when `mirror_to_github: true` " +
"(default false). Idempotent — re-running with an existing repo returns the existing URLs.",
InputSchema: schema, InputSchema: schema,
}, },
} }
+46
View File
@@ -0,0 +1,46 @@
-- One-time DBA setup for the brain vector store on postgres18.
--
-- Creates the `brain` database, the `brain_app` role, and the pgvector
-- extension. The ingestion service connects as brain_app and creates
-- the table + HNSW index idempotently at startup (see
-- internal/vectorstore.PGStore.Init).
--
-- Run from koala as the postgres superuser:
--
-- kubectl exec -n databases postgres18-0 -- \
-- psql -U postgres -f /tmp/brain-embeddings-init.sql
--
-- Or apply with:
--
-- PASSWORD='<sops-generated>' \
-- kubectl exec -i -n databases postgres18-0 -- \
-- psql -U postgres -v password="'$PASSWORD'" \
-- < scripts/brain-embeddings-init.sql
--
-- Idempotent: rerunning is safe.
\set ON_ERROR_STOP on
-- CREATE DATABASE cannot run inside a DO block (transactional limitation).
-- Use \gexec to emit the statement conditionally instead.
SELECT 'CREATE DATABASE brain'
WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'brain')
\gexec
-- DO blocks can't see psql `:'password'` substitutions (those resolve
-- client-side). Use \if to branch at psql level instead.
SELECT EXISTS (SELECT FROM pg_roles WHERE rolname = 'brain_app') AS role_exists \gset
\if :role_exists
ALTER ROLE brain_app WITH PASSWORD :'password';
\else
CREATE ROLE brain_app LOGIN PASSWORD :'password';
\endif
GRANT ALL PRIVILEGES ON DATABASE brain TO brain_app;
\c brain
CREATE EXTENSION IF NOT EXISTS vector;
GRANT ALL ON SCHEMA public TO brain_app;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO brain_app;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO brain_app;
+135
View File
@@ -0,0 +1,135 @@
#!/usr/bin/env bash
# migrate-brain-halls.sh — move flat brain/wiki/{concepts,entities}/ notes
# into the structured brain/wiki/<wing>/<hall>/ layout introduced by
# hyperguild#1.
#
# Reads each note's YAML frontmatter:
# type: maps to hall (decision, hypothesis, failure, source → eponymous;
# concept, entity, anything else → facts)
# domain: maps to wing (sanitised: lowercase, alphanumerics + hyphens);
# empty → "general"
#
# Dry-run by default. Pass --commit to actually move files. Idempotent:
# already-migrated notes (already under a Wing dir) are left alone.
#
# Usage:
# scripts/migrate-brain-halls.sh /path/to/brain # dry-run
# scripts/migrate-brain-halls.sh --commit /path/to/brain # apply
set -euo pipefail
COMMIT=0
BRAIN=""
for arg in "$@"; do
case "$arg" in
--commit) COMMIT=1 ;;
-h|--help)
sed -n '2,18p' "$0"
exit 0
;;
*) BRAIN="$arg" ;;
esac
done
if [[ -z "$BRAIN" ]]; then
echo "error: brain directory required" >&2
echo "usage: $0 [--commit] <brain-dir>" >&2
exit 2
fi
if [[ ! -d "$BRAIN" ]]; then
echo "error: $BRAIN is not a directory" >&2
exit 2
fi
WIKI="$BRAIN/wiki"
if [[ ! -d "$WIKI" ]]; then
echo "no $WIKI/ — nothing to migrate"
exit 0
fi
sanitise() {
# lowercase, replace non-alnum with hyphen, collapse hyphens, trim
local s
s=$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]' \
| sed -E 's/[^a-z0-9]+/-/g; s/^-+//; s/-+$//; s/-+/-/g')
printf '%s' "$s"
}
# extract_frontmatter_value <file> <key>
# Echoes the value (trimmed, unquoted) of `key:` from a leading YAML
# frontmatter block. Empty if absent or no frontmatter.
extract_frontmatter_value() {
awk -v key="$2" '
BEGIN { in_fm = 0; first = 1 }
/^---[[:space:]]*$/ {
if (first) { in_fm = 1; first = 0; next }
if (in_fm) { exit }
}
in_fm {
idx = index($0, ":")
if (idx == 0) next
k = substr($0, 1, idx-1)
v = substr($0, idx+1)
gsub(/^[[:space:]]+|[[:space:]]+$/, "", k)
gsub(/^[[:space:]]+|[[:space:]]+$/, "", v)
gsub(/^["'\'']|["'\'']$/, "", v)
if (k == key) { print v; exit }
}
' "$1"
}
hall_for_type() {
case "$1" in
decision|decisions) echo "decisions" ;;
hypothesis|hypotheses) echo "hypotheses" ;;
failure|failures) echo "failures" ;;
source|sources) echo "sources" ;;
*) echo "facts" ;;
esac
}
declare -i moved=0 skipped=0
migrate_source_dir() {
local src="$1"
[[ -d "$src" ]] || return 0
while IFS= read -r -d '' f; do
local typ domain wing hall slug dest
typ=$(extract_frontmatter_value "$f" type)
domain=$(extract_frontmatter_value "$f" domain)
hall=$(hall_for_type "$typ")
wing=$(sanitise "${domain:-general}")
[[ -z "$wing" ]] && wing="general"
slug=$(basename "$f" .md)
dest="$WIKI/$wing/$hall/$slug.md"
if [[ "$f" == "$dest" ]]; then
skipped=$((skipped + 1))
continue
fi
if [[ -e "$dest" ]]; then
echo "skip (target exists): $f$dest"
skipped=$((skipped + 1))
continue
fi
if [[ "$COMMIT" -eq 1 ]]; then
mkdir -p "$(dirname "$dest")"
git -C "$BRAIN" mv "$f" "$dest" 2>/dev/null || mv "$f" "$dest"
fi
echo "move: $f$dest"
moved=$((moved + 1))
done < <(find "$src" -maxdepth 1 -type f -name '*.md' -print0)
}
migrate_source_dir "$WIKI/concepts"
migrate_source_dir "$WIKI/entities"
echo
if [[ "$COMMIT" -eq 1 ]]; then
echo "moved=$moved skipped=$skipped (committed)"
else
echo "moved=$moved skipped=$skipped (dry-run — pass --commit to apply)"
fi
+1 -1
View File
@@ -4,7 +4,7 @@ set -euo pipefail
# Boot the routing binary and exercise its four tools against live deps. # Boot the routing binary and exercise its four tools against live deps.
# Skipped when LITELLM_BASE_URL or BRAIN_URL is unreachable. # Skipped when LITELLM_BASE_URL or BRAIN_URL is unreachable.
LITELLM_BASE_URL="${LITELLM_BASE_URL:-http://piguard:4000}" LITELLM_BASE_URL="${LITELLM_BASE_URL:-https://llm-api.d-ma.be}"
BRAIN_URL="${BRAIN_URL:-http://koala:30330}" BRAIN_URL="${BRAIN_URL:-http://koala:30330}"
if ! curl -sS --max-time 2 "${LITELLM_BASE_URL}/v1/models" >/dev/null 2>&1; then if ! curl -sS --max-time 2 "${LITELLM_BASE_URL}/v1/models" >/dev/null 2>&1; then