hyperguild#81. brain_pending lists brain/raw/ notes awaiting review;
brain_promote moves one into the wiki. There was no way to say "reviewed,
not worth promoting" — the only options were promote it or let it sit
forever. Live consequence, confirmed this session: the pending queue has
grown to 11 items, the oldest 152 days old, several of which are generic
web scrapes nobody is ever going to promote but nobody could clear either
except an un-audited rm bypassing the tool surface entirely.
RejectNote (internal/api/promote.go) mirrors PromoteNote's exact safety
shape: same basename-only filename validation, same write-then-delete
ordering (the source survives any write failure), same collision check
(never silently overwrite an existing rejected note under the same
name). Stamps rejected_at (+ reject_reason if given) into frontmatter,
moves brain/raw/<filename> -> brain/raw/rejected/<filename>. No index
rebuild or auto-tunnel -- a rejected note never entered the wiki, so
there's nothing to index. raw/rejected/ is a subdirectory ListPending's
non-recursive os.ReadDir never walks into, so a rejected note is never
re-surfaced as pending -- the human decision, once made, stays made.
brain_reject wired into the MCP tool list and dispatch, mirroring
brain_promote's schema shape (filename required, reason optional).
Human-invoked only, same as promote -- never called automatically.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
brain_edit_root was gated but unobserved: on success it emitted a slog.Warn
and nothing else. That proves a rewrite was allowed, not that one happened —
and brain-sync commits it within 15 minutes authored `brain-sync`, so git
history cannot tell a CLAUDE.md rewrite apart from routine note traffic.
A successful brain_edit_root now emits an ntfy notification and an AuditEntry
through the same sink a capture already uses. notes.Result gains
PrevContentHash so both ends of the change are identifiable.
Path and hashes only in the notification body. Reserved at Internal, not
Confidential, because Confidential makes DegradingSink refuse to record while
loki is down rather than buffer. Observation is best-effort and never fails
the call; a failure to observe logs at ERROR.
Ordinary brain_edit on knowledge//wiki/ is deliberately not audited — git
history already carries the same before/after bytes with a diff.
Closes#97.
Co-authored-by: Mathias <mthbqv@gmail.com>
brain_read returns a note verbatim; brain_edit is str_replace with an
exactly-once match, operating on the tree brain-sync commits from.
An adversarial design review before implementation changed three things:
1. Governance files are not in the model-facing tool. Brain content is
ingested from the outside world and reaches a model's context, and that
model would hold the editor — one poisoned page could rewrite the
CLAUDE.md a .dispatch-allow agent later loads as instructions. And
brain-sync merges with --strategy-option=ours, so the live tree beats a
human's corrective push on the conflicting lines. CLAUDE.md now goes
through brain_edit_root: de-surfaced from tools/list and gated on the
local static principal. The scopes are disjoint — the privileged door
has the smaller reach.
2. Reads needed the same allowlist, and brain_get already lacked it: it
returned verbatim session transcripts under archive/ by path. A test
proved the exposure live before the fix.
3. Canonicalise before allowlisting — knowledge/../.git/hooks/post-commit
would otherwise reach a hook that brain-sync executes in a pod holding
the Gitea push credentials.
Plus overlapping match counting, UTF-8 validation both ways, refusal while
git is mid-operation, flock with the read inside the lock, atomic
same-dir rename, and reindex only when frontmatter is touched.
Refs brain#4.
Co-authored-by: Mathias <mthbqv@gmail.com>
brain_index is auto-called after every write, and its own description
said so. A verb surfaced to callers that callers should never call is
cognitive load with no decision behind it — ADR-0012 rule 4.
Removed from tools/list only. The dispatch case stays, so maintenance
can still rebuild a Map of Content by hand: de-surfacing hides the verb,
not the operation. A test pins both halves, since removing the descriptor
and removing the capability look identical from the tool list.
Nothing called it directly — grepped the skills, specs and docs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
Wires the curation primitives into the MCP surface (all three sites:
tools() descriptors, handleCall dispatch, package doc) and registers the
GET /pending + POST /promote REST routes. brain_promote additionally
re-indexes the promoted note into the graph (best-effort), matching the
other write paths. brain_pending is the human-review-queue complement to
the agent-facing brain_write.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the `capture` MCP tool: the #55 relay for harnesses that cannot run
the use-case in-process (claude.ai Chat/Cowork/Design, Crush, Pi, LLM
Council). They reach it through the existing /mcp OAuth connector.
- Thin: forwards to the SAME CaptureService as POST /capture; holds no
state and retains nothing beyond the I5 audit record. The containment
properties accepted in infra security-baseline (I2 ledger) hold by
construction.
- Per-principal: ServeHTTP re-derives the caller's principal from the
Bearer header (the chassis middleware gates but discards it) and stashes
it in context; the tool resolves the trust-zone origin from it. A
caller-asserted harness/origin in the body is ignored — origin is
server-derived, so the I1 confidential refusal still fires for us-nexus
callers (claude.ai), and sovereign-allowlisted JWT principals pass.
- Registered only when WithCapture is wired (all three sites: tools(),
handleCall, package doc); main wires REST + MCP from the same service,
resolver, and credentials.
Tests: listed-only-when-wired, forwards-via-static-principal,
confidential-via-us-nexus-refused, confidential-via-sovereign-allowed,
unauthenticated-rejected, caller-cannot-forge-origin.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The HTTP door for the capture capability. Thin: authenticate → derive
trust-zone origin → decode → capture.Service → map receipt to status.
- Auth mirrors the chassis Bearer precedence (static token wins, then Dex
JWT) but returns the resolved principal + auth path, which the chassis
middleware hides — capture needs the principal to derive the origin.
Depends on a small Validator interface (the chassis *JWTValidator
satisfies it) so the JWT/origin path is testable without a live JWKS.
- OriginResolver maps principal → trust zone: static-token caller and
allowlisted JWT subjects → sovereign; every other principal → us-nexus
(fail safe, so the I1 gate refuses confidential by default). Principal
and origin are server-set on the input, overwriting any body the caller
sent.
- HTTP status: 200 all-ok / dry-run, 207 partial, 502 all-failed, 403 on
the I1 refusal, 400 on fail-closed validation.
- Wired in main behind the same static+JWT credentials as /mcp, reusing
the MCP server's graph-wired brain store (one implementation) and the
classification tags (#50). Mounts only when a Gitea tracker is
configured. Sovereign JWT principals via BRAIN_CAPTURE_SOVEREIGN_PRINCIPALS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements the IssueTracker port as a real Gitea REST client (#49c) — the
new outbound dependency the brain server gains for capture.
- CreateIssue / CommentIssue / CloseIssue(+optional closing comment) over
the Gitea API. Owner is the const "mathias", never caller-supplied, so
a caller cannot redirect a write to another owner's repo.
- Token read once at construction (BRAIN_GITEA_TOKEN), held in the struct,
travels only in the Authorization header — never logged or in argv.
Error messages carry status + truncated body, never the token
(regression-tested). gitea.New returns nil when URL or token is unset,
so missing config = tracker disabled via one nil check.
- Injected into the MCP server behind the capture.IssueTracker interface
via WithIssueTracker (constructor injection, swappable/testable); main
wires it from BRAIN_GITEA_URL (default https://git.d-ma.be) +
BRAIN_GITEA_TOKEN. Consumed by the capture use-case in #53.
Tests use httptest transports: create (owner+auth header asserted),
comment, close with/without comment, error path that proves the token
never leaks into an error string.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extracts the #45 write/update/get logic + the wiki upkeep that must
accompany a write (wing _index rebuild, cross-wing auto-tunnel, graph
re-index) into a single concrete brainstore.Store implementing
capture.BrainStore. The MCP brain_write/brain_update/brain_get handlers
are re-pointed at it, so there is one implementation, not two — the DRY
payoff #51 is named for. capture and MCP now share the exact same brain
write path and read-after-write contract.
The Server gains a *brainstore.Store, constructed in NewServer and given
the graph store in WithGraph. Embedding refresh stays out-of-band
(mtime-driven vectorstore.Sync), unchanged. Existing MCP brain_update/
brain_get/brain_write tests pass unmodified — behaviour and the
{id, path, content_hash} response contract are preserved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires the #45 verbs into the MCP surface (all three sites: tools()
descriptors, handleCall dispatch, package doc comment).
- brain_update: supersede-by-slug or full path; rebuilds wing _index and
re-tunnels cross-wing matches against the new body (idempotent,
best-effort), re-indexes the graph, returns {id, path, content_hash,
superseded}.
- brain_get: fetch by id or path (both are the brain-relative handle);
returns {id, path, content_hash, frontmatter, body}.
- brain_write: return contract extended from {path} to {id, path,
content_hash} — path kept for backward compat — so the create path
also yields a stable handle.
id == relPath; content_hash == sha256 of the file bytes. Tests cover the
supersede happy path, missing-target error + no-create, get by id/path,
write handle, and an end-to-end re-embed test that drives the real
vectorstore.Sync re-index after an update.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Returns top-N relevant brain entries for a project context. Combines
BM25 hits on project name with 2-hop graph expansion via Track A's
graphstore (when BRAIN_GRAPH_ENABLED). Closes hyperguild#28.
Notes on implementation choices that deviate slightly from the spec:
- Excerpt length: 200 chars per spec (vs the 300 used by search.Result).
truncateExcerpt clamps the already-stripped BM25 excerpt; graph-only
neighbours load their excerpt from disk via a private readExcerpt
helper (search.hydrate is unexported).
- Graph scoring: 0.6 / max(1, distance) per neighbour, so distance-1
contributes 0.6 and distance-2 contributes 0.3. BM25 hits decay
linearly from 3.0 (rank-0) to 1.0 (rank-2), giving BM25 hits a
natural ceiling above pure-graph hits while still letting a doc
surfaced via both edge types outrank a BM25-only one.
- Test placement: package mcp (internal) rather than mcp_test, because
graphReader is unexported and WithGraph only accepts *PGStore; an
internal test can install a dual-interface fake directly on s.graph
without spinning up postgres.
Bump-Type: minor
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Commit 3 of Track A. The MCP server now publishes a new tool that
opens the brain knowledge graph (entities + wikilink edges) for
external consumers (claude.ai connectors, gitea-mcp, agentsquad).
- tools_graph.go: brain_graph handler dispatches by op:
neighbors — 1-hop outgoing from slug, optional edge_type filter
subgraph — every reachable slug within depth hops (≤6)
path — shortest directed path src→dst within depth (≤8)
Returns slug + entity metadata + edge_type + hop distance.
- server.go: handleCall routes "brain_graph" to brainGraph.
- handlers.go: tool descriptor with the op enum + per-op required
fields documented in the description.
- server_test.go: TestServerToolsList expects brain_graph in the
listing.
The tool returns an error when BRAIN_GRAPH_ENABLED is unset — same
shape as brain_answer when the answer LLM is unconfigured.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Commit 2 of Track A. Service stays a no-op until BRAIN_GRAPH_ENABLED=
true; flipping it on creates the schema (idempotent), starts indexing
every successful write, and optionally backfills the existing brain
dir.
- internal/graphsync: best-effort wrapper around graph.Extract +
graphstore. IndexDoc reads docPath under brainDir, parses, upserts
entity + replaces edges. BackfillFromBrainDir walks wiki/ +
knowledge/. Both are no-ops on nil store so callers wire
unconditionally.
- mcp.Server gains WithGraph builder + graphsync.Store field.
brain_write, brain_ingest, brain_ingest_raw, brain_tunnel call
indexInGraph after success — failures slog.Warn but never
propagate (graph is augmentation, not correctness).
- cmd/server gates the wiring on BRAIN_GRAPH_ENABLED=true (default
off so first rollout doesn't surprise). BRAIN_GRAPH_BACKFILL=true
triggers a one-shot walk of the brain dir on boot.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires nomic-embed-text (iguana ollama) + pgvector on the shared
postgres18 into brain_query / brain_answer via Reciprocal Rank Fusion.
Pure BM25 stays the default; setting BRAIN_PG_DSN and BRAIN_EMBED_URL
together opts in. Setting one without the other is misconfiguration →
exit 1.
New packages:
- internal/embed
Client.Embed(ctx, text) → []float32 via POST {URL}/api/embed.
Defaults to nomic-embed-text:latest (768 dim). nil-on-empty-URL so
callers gate on a single nil check.
- internal/vectorstore
PGStore wraps a pgxpool against postgres18. Init creates
brain_embeddings(path PK, vector(768), updated_at) + HNSW cosine
index idempotently. Upsert / Delete / Search / KnownPaths.
Sync(brainDir, store, embedder) diffs brain/wiki/ against the store
and upserts new files / deletes removed ones; StartSync runs it on
a ticker (default 300s). Integration tests gated by BRAIN_PG_TEST_DSN.
- scripts/brain-embeddings-init.sql
One-time DBA setup: brain DB, brain_app role, vector extension,
GRANTs. Idempotent.
Search layer:
- search.QueryOptions gains Vector + Embedder fields.
- QueryContext is the cancellable variant; Query stays for callers.
- When both are set, BM25 (top-N) and pgvector (top-4N) candidates
merge via Reciprocal Rank Fusion (k=60, Cormack et al. 2009 — no
tuning knob, robust to scale differences between rankers).
- Vector-only hits are hydrated from disk so callers see uniform
Result records (path, title, excerpt, wing, hall, score).
- Wing/hall filters still apply to vector candidates via path-prefix.
- On embedder/vector errors the search falls back to BM25 — embedding
outage degrades quality but doesn't take the brain offline.
MCP wiring:
- mcp.Server.WithHybridRetrieval(v, e) opt-in setter, same shape as
WithReranker.
- brainQuery and brainAnswer pass the wired vector/embedder through
to search.QueryContext.
REST:
- POST /backfill-embeddings drives Sync synchronously. Returns
{added, deleted, errors[]}. 503 when feature is unconfigured.
cmd/server/main.go:
- BRAIN_PG_DSN + BRAIN_EMBED_URL together enable hybrid; one alone
→ exit 1.
- vectorAdapter bridges *PGStore (returns []Hit) to
search.VectorSearcher (which takes []VectorHit) without either
package importing the other.
- BRAIN_EMBED_SYNC_INTERVAL (default 300s) controls the background
Sync ticker.
Backend pivot from Qdrant to pgvector recorded in DECISIONS.md
2026-05-18 (supersedes 2026-04-08): postgres18 already runs in
databases/ ns, Qdrant was never deployed, one engine beats two.
Dependency: github.com/jackc/pgx/v5 — modern, native pgvector via
parametric vector literals.
Tests:
- embed.Client: empty-URL nil, request shape, dimension, upstream
error propagation, empty-text rejection.
- vectorstore.PGStore: dimension validation (unit); upsert/search/
KnownPaths (integration, BRAIN_PG_TEST_DSN-gated).
- vectorstore.Sync: adds new files, skips known, deletes
disappeared, skips _index.md, no-op when nil, collects embedder
errors.
- search.Query: hybrid promotes vector-only hits via RRF; falls
back to BM25 on embedder error.
Closes hyperguild#8.
Adds an opt-in cross-encoder rerank step between BM25 retrieval and LLM
synthesis. With BRAIN_RERANKER_URL set, brain_answer retrieves BM25
top-20, scores each excerpt against the query via Qwen3-Reranker on
Ollama, drops the "no" answers, and forwards up to 5 surviving sources
to the LLM. Unset, behaviour is unchanged (BM25 top-10 → LLM).
The reranker is a *filter*, not a re-ranker: Qwen3-Reranker emits a
binary yes/no token under its native chat template, and ties within the
"yes" set are broken by BM25 rank — what got retrieved first stays
ahead.
New package ingestion/internal/reranker:
- Client with URL, Model, HTTP fields.
- New(url, model) returns nil on empty url so callers can treat
"feature disabled" as a single nil check.
- Score(ctx, query, docs) issues one /api/generate call per doc using
the Qwen3-Reranker yes/no chat template (verbatim, because the model
was trained on this exact wording). Parses the first non-think token.
Wiring:
- mcp.Server gains a WithReranker fluent setter to keep NewServer
signature stable.
- brain_answer's BM25 limit jumps to 20 only when a reranker is wired,
to give the filter something to do.
- cmd/server/main.go reads BRAIN_RERANKER_URL (+ optional
BRAIN_RERANKER_MODEL, default dengcao/Qwen3-Reranker-0.6B:F16).
Tests cover: nil-on-empty-url, ordered yes/no scoring, request shape
(model, prompt contents, yes/no template), ambiguous response → 0,
empty doc slice, upstream-error propagation, plus an end-to-end
brain_answer integration that proves only the relevant note reaches the
LLM when noise.md is rejected.
Closes hyperguild#7.
Adds the `brain_tunnel` MCP tool and auto-tunnel behaviour for
`brain_write`, so concepts that appear in multiple wings become
navigable from any of them.
New surface in package brain:
- WriteTunnel(brainDir, src, tgt) — appends a `## See also` bidirectional
wikilink between two notes in different wings. Idempotent (link not
duplicated on re-call) and reuses an existing See also section.
- DetectTunnels(brainDir, content) — walks brain/wiki/, returns
TunnelCandidates for notes whose title appears in content. Tags
whole-word case-insensitive hits as Exact=true and substring-only hits
as Exact=false.
- AutoTunnel(brainDir, src, content) — wraps DetectTunnels: writes
cross-wing exact matches, stages fuzzy matches into
brain/raw/tunnel-candidates-<YYYY-MM-DD>.md for human review.
MCP wiring:
- `brain_tunnel` tool: explicit manual link (source, target).
- `brain_write` with wing+hall now triggers AutoTunnel on the new
content. Failures are logged and never abort the primary write.
readTitleAndCreated also humanises the slug fallback (hyphens → spaces)
so titleless notes participate in content matching.
Closes hyperguild#16.
Tests: idempotency, same-wing rejection, missing-note rejection,
See-also reuse, exact/fuzzy detection, slug fallback, MCP tool happy
path, auto-tunnel hook (cross-wing exact → linked; same-wing → skipped;
fuzzy → candidates file).
Adds a two-dimensional address (wing, hall) to brain notes. A wing is a
topic domain (e.g. jepa-fx, hyperguild); a hall is one of a closed
vocabulary of memory types (facts, decisions, failures, hypotheses,
sources). Notes route to brain/wiki/<wing>/<hall>/<slug>.md with
wing/hall/created_at YAML frontmatter, making the directory a valid
Obsidian vault.
Changes:
- new package ingestion/internal/brain (NotePath, ValidHalls, Sanitise,
BuildWingIndex, BuildAllWingIndexes)
- api.WriteNote refactored to WriteNoteOptions; wing+hall routes to
brain/wiki/, otherwise falls back to brain/knowledge/ (legacy)
- search.Query → QueryOptions with optional Wing/Hall filtering; Result
carries wing/hall extracted from frontmatter or path segments
- MCP tools brain_write and brain_query gain optional wing/hall params
(hall enum-validated); new brain_index tool regenerates _index.md MOC
- POST /index REST endpoint mirrors brain_index
- brain_write auto-rebuilds the wing's _index.md after a wing+hall write
- scripts/migrate-brain-halls.sh migrates flat brain/wiki/{concepts,entities}/
into the new layout (dry-run by default, --commit applies)
All existing tests pass; new tests cover wing/hall write routing, scope
filtering, invalid hall rejection, _index.md generation, and migration
script paths.
Closes hyperguild#1.
Adds two new LLM-backed MCP tools to the ingestion service:
- brain_answer(query): BM25 retrieval + LLM synthesis → answer + sources
- brain_classify(text): classifies doc into type/title/tags via LLM
Adds llm.Router for primary→fallback routing (berget.ai → iguana).
Wired via BRAIN_LLM_PRIMARY_URL/BRAIN_LLM_FALLBACK_URL env vars;
no-op when unset so existing deployments are unaffected.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Appends a JSON line to brainDir/sessions/<session_id>.jsonl using the
session package copied in Task 2. Required for upcoming pass-rate
logging.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wraps pipeline.Run with the existing LLM client. Mirrors the HTTP
/ingest and /ingest-path semantics — accepts either path or
content+source, validates mutual exclusion, surfaces an explicit error
when the LLM client is not configured (test-mode).
ctx is threaded through to pipeline.Run for cancellation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wraps pipeline.RunRaw directly. Same dry-run semantics as the HTTP
/ingest-raw endpoint. Test exercises a single concept page; asserts
returned path and that no file is written under dry_run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
api.WriteNote captures the file-write logic that was previously inline
in Handler.Write. The existing HTTP endpoint now delegates to it; the
new MCP brain_write tool reuses the same function. Path-traversal
guard is strengthened to explicitly reject filenames containing path
separators or "..", so the rejection is surfaced before filepath.Base
strips the suspicious component (the previous defense-in-depth prefix
check became unreachable for these inputs after Base normalisation).
HTTP error code for caller-input errors shifts from 500 to 400, which
is semantically correct and not exercised by any existing test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wraps the existing search.Query function. Same BM25 over
brain/knowledge/ and brain/wiki/ that the HTTP /query endpoint serves.
Plan note: handleCall switch replaces the single-line stub from Task 1
— no unknownToolError type to remove since Task 1 inlined the error.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an MCP HTTP handler under ingestion/internal/mcp. Implements
initialize, tools/list, and the JSON-RPC notification skip from prior
work. Tool dispatch is stubbed (returns unknown-tool error) and will be
filled in by subsequent tasks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mathiasandClaude Sonnet 5