feat(mcp): brain_read + brain_edit, contained to the sync source (brain#4) #96

2026-08-18T15:02:14Z

mathias commented

2026-08-18 15:02:14 +00:00

Implements mathias/brain#4. Code lands here per the #91 part-2 decision (brain-mcp stays inside ingestion).

task check green: 19 tests in internal/notes under -race, plus JSON-RPC round-trip tests in internal/mcp, lint clean, gitleaks clean.

An adversarial design review ran before any code, and changed the design three times

1. Governance files are not in the model-facing tool

I originally planned brain_edit to accept root *.md, so CLAUDE.md would have an MCP path at all — its absence is what pushed a session to edit via the gitea REST API on 2026-06-16, which never reached the live tree and wedged brain-sync for hours.

The review traced the full chain and it holds, verified end to end:

Outside-world content lands in wiki/ via brain_ingest/brain_promote.
It reaches a later model's context via brain_query / brain_context.
That model holds brain_edit. It writes CLAUDE.md.
brain-sync commits and pushes within 15 minutes, unattended.
/var/lib/supervisor/brain/.dispatch-allow exists — a dispatch agent clones this repo and loads that CLAUDE.md as instructions, with a Gitea token.

And the kicker I confirmed in the CronJob script: git merge --strategy-option=ours, whose own comment says "ours" is the current checked-out branch — i.e. the local live tree. So when a human pushes a corrective CLAUDE.md and the live tree has a conflicting hunk, the injected version wins.

Also, "root *.md" resolves today to exactly {CLAUDE.md, log.md} — and log.md is gitignored. That allowlist entry would have re-created brain#2 by hand: edit succeeds, never syncs, vanishes on re-clone.

So: brain_edit is knowledge/ + wiki/, .md, no _index.md. CLAUDE.md goes through brain_edit_root — de-surfaced from tools/list (the brain_index precedent, ADR-0012 rule 4) and gated on the local static principal. The scopes are disjoint, not nested: the privileged door has the smaller reach.

Consequence worth knowing: a claude.ai-origin session — including mine — cannot edit CLAUDE.md through MCP. That is the design working, and it means brain#3's CLAUDE.md change needs the local CLI path, not this tool.

2. `brain_get` was already leaking, and this PR closes it

Not introduced here: api.ReadNote resolves any brain-relative path, and parseFrontmatter returns the whole file as body when there is no --- fence. So brain_get(path="archive/claude-sessions/koala/session-1.md") returns a verbatim session transcript — up to 500 KB, never BM25-indexed, so invisible to search and trivially readable by path. CLAUDE.md documents the layout.

There is a test in this PR that proved the exposure live before the fix. brain_get now goes through the same Resolve.

3. Canonicalise before allowlisting

A prefix check on the raw argument accepts knowledge/../.git/hooks/post-commit, which Clean collapses to a git hook that brain-sync executes inside a pod holding the Gitea push credentials. Order is now canonicalise → static denials → allowlist → lexical containment → symlink re-check. There were zero traversal tests in this repo before; there is a table now.

Correctness details, each with a test

Overlapping matches. bytes.Count says "aa" occurs once in "aaa", so a Count-based uniqueness check calls an ambiguous edit unique.
UTF-8 both ways. encoding/json substitutes U+FFFD for invalid bytes, so brain_read would return content the file does not hold and every derived old_str would be permanently unmatchable. Refused loudly instead. (This repo has had one truncated-UTF-8 incident already — #93.)
Refuse mid-git-operation. .git/MERGE_HEAD, REBASE_HEAD, CHERRY_PICK_HEAD, index.lock, or conflict markers in the target. An edit racing the merge is hard-reset away or committed as a wrongly resolved conflict, with the caller told it succeeded.
Concurrency. flock + per-path mutex, with the read taken inside the lock — two pods share the hostPath during a RollingUpdate, and old_str uniqueness evaluated against a stale read loses an update while both callers see success. A test runs two concurrent edits and asserts both land.
Atomicity. Same-directory *.tmp (rename across the hostPath is EXDEV; *.tmp is already gitignored so a leak is not committed), chmod 0644 because CreateTemp gives 0600 and other uids write this tree, fsync of file and parent.
CRLF hint. Never normalises — that breaks byte-identity — but a zero-match error says so when the only difference is \r, because an opaque "no match" is unfixable from the caller's side.
Reindex only when frontmatter is touched. BM25 walks the filesystem live and embeddings are mtime-driven, so an unconditional rebuild buys nothing and dirties _index.md — precisely the file class brain-sync already needs a bespoke conflict resolver for. Best-effort, surfaced as reindex_error.

Not done

The issue's end-to-end acceptance (adding the Superseded by [[adr-gitea-as-true-master]] marker to knowledge/adr-github-as-primary-remote.md entirely through MCP) needs this deployed. Ready to run once merged.
ntfy on root edits. The review wanted a notification per governance-file edit; there is a slog.Warn today. Worth adding alongside brain-sync's existing ntfy wiring.
Audit record. brain_edit logs but does not write to the capture AuditSink. A byte-level mutation primitive on a pushed tree arguably deserves one.

Implements `mathias/brain#4`. Code lands here per the #91 part-2 decision (brain-mcp stays inside `ingestion`). `task check` green: 19 tests in `internal/notes` under `-race`, plus JSON-RPC round-trip tests in `internal/mcp`, lint clean, gitleaks clean. ## An adversarial design review ran before any code, and changed the design three times ### 1. Governance files are not in the model-facing tool I originally planned `brain_edit` to accept root `*.md`, so `CLAUDE.md` would have an MCP path at all — its absence is what pushed a session to edit via the gitea REST API on 2026-06-16, which never reached the live tree and wedged `brain-sync` for hours. The review traced the full chain and it holds, verified end to end: 1. Outside-world content lands in `wiki/` via `brain_ingest`/`brain_promote`. 2. It reaches a later model's context via `brain_query` / `brain_context`. 3. That model holds `brain_edit`. It writes `CLAUDE.md`. 4. `brain-sync` commits and pushes within 15 minutes, unattended. 5. `/var/lib/supervisor/brain/.dispatch-allow` exists — a dispatch agent clones this repo and loads that `CLAUDE.md` as instructions, with a Gitea token. And the kicker I confirmed in the CronJob script: `git merge --strategy-option=ours`, whose own comment says *"ours" is the current checked-out branch — i.e. the local live tree*. So when a human pushes a corrective `CLAUDE.md` and the live tree has a conflicting hunk, **the injected version wins**. Also, "root `*.md`" resolves today to exactly `{CLAUDE.md, log.md}` — and `log.md` is **gitignored**. That allowlist entry would have re-created brain#2 by hand: edit succeeds, never syncs, vanishes on re-clone. So: `brain_edit` is `knowledge/` + `wiki/`, `.md`, no `_index.md`. `CLAUDE.md` goes through **`brain_edit_root`** — de-surfaced from `tools/list` (the `brain_index` precedent, ADR-0012 rule 4) and gated on the local static principal. The scopes are **disjoint**, not nested: the privileged door has the smaller reach. **Consequence worth knowing:** a claude.ai-origin session — including mine — cannot edit `CLAUDE.md` through MCP. That is the design working, and it means brain#3's `CLAUDE.md` change needs the local CLI path, not this tool. ### 2. `brain_get` was already leaking, and this PR closes it Not introduced here: `api.ReadNote` resolves any brain-relative path, and `parseFrontmatter` returns the **whole file** as `body` when there is no `---` fence. So `brain_get(path="archive/claude-sessions/koala/session-1.md")` returns a verbatim session transcript — up to 500 KB, never BM25-indexed, so invisible to search and trivially readable by path. `CLAUDE.md` documents the layout. There is a test in this PR that **proved the exposure live** before the fix. `brain_get` now goes through the same `Resolve`. ### 3. Canonicalise before allowlisting A prefix check on the raw argument accepts `knowledge/../.git/hooks/post-commit`, which `Clean` collapses to a git hook that `brain-sync` executes **inside a pod holding the Gitea push credentials**. Order is now canonicalise → static denials → allowlist → lexical containment → symlink re-check. There were **zero traversal tests** in this repo before; there is a table now. ## Correctness details, each with a test - **Overlapping matches.** `bytes.Count` says `"aa"` occurs once in `"aaa"`, so a Count-based uniqueness check calls an ambiguous edit unique. - **UTF-8 both ways.** `encoding/json` substitutes U+FFFD for invalid bytes, so `brain_read` would return content the file does not hold and every derived `old_str` would be permanently unmatchable. Refused loudly instead. (This repo has had one truncated-UTF-8 incident already — #93.) - **Refuse mid-git-operation.** `.git/MERGE_HEAD`, `REBASE_HEAD`, `CHERRY_PICK_HEAD`, `index.lock`, or conflict markers in the target. An edit racing the merge is hard-reset away or committed as a wrongly resolved conflict, with the caller told it succeeded. - **Concurrency.** `flock` + per-path mutex, with the read taken **inside** the lock — two pods share the hostPath during a `RollingUpdate`, and `old_str` uniqueness evaluated against a stale read loses an update while both callers see success. A test runs two concurrent edits and asserts both land. - **Atomicity.** Same-directory `*.tmp` (rename across the hostPath is `EXDEV`; `*.tmp` is already gitignored so a leak is not committed), `chmod 0644` because `CreateTemp` gives 0600 and other uids write this tree, `fsync` of file and parent. - **CRLF hint.** Never normalises — that breaks byte-identity — but a zero-match error says so when the only difference is `\r`, because an opaque "no match" is unfixable from the caller's side. - **Reindex only when frontmatter is touched.** BM25 walks the filesystem live and embeddings are mtime-driven, so an unconditional rebuild buys nothing and dirties `_index.md` — precisely the file class `brain-sync` already needs a bespoke conflict resolver for. Best-effort, surfaced as `reindex_error`. ## Not done - **The issue's end-to-end acceptance** (adding the `Superseded by [[adr-gitea-as-true-master]]` marker to `knowledge/adr-github-as-primary-remote.md` entirely through MCP) needs this deployed. Ready to run once merged. - **ntfy on root edits.** The review wanted a notification per governance-file edit; there is a `slog.Warn` today. Worth adding alongside `brain-sync`'s existing ntfy wiring. - **Audit record.** `brain_edit` logs but does not write to the capture `AuditSink`. A byte-level mutation primitive on a pushed tree arguably deserves one.

mathias added 2 commits 2026-08-18 15:02:14 +00:00

fix(capture): name what escalated the classification in an I1 refusal (#91 )

CI / Lint / Test / Vet (pull_request) Successful in 30s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

db00644cab

infra#265 reported "capture with a tickets array fails": 20 tickets
errored, 5 errored, the caller fell back to 20 individual issue_create
calls and filed a bug against the array.

The array was never the failing dimension. Every ticket in that batch
targeted a client repo, and the I1 sovereignty gate refuses a
confidential capture through a us-nexus origin at ANY length — reproduced
live today against brain-mcp: length 1 succeeds for an internal repo,
length 20 validates clean, and a single ticket targeting
trucker-ecommerce is refused.

What made it look like an array bug is that the refusal named neither the
offending target nor the fact that a policy said no:

    capture refused by I1 sovereignty gate: effective classification
    confidential through us-nexus origin

With twenty tickets a caller cannot tell which repo tripped it, so
"the array is broken" is a reasonable conclusion from the evidence
available. resolveClassification now also returns the targets carrying
the effective level, and the refusal names them:

    ... through us-nexus origin (required by: trucker-ecommerce)

Only targets that actually reach the effective level are named — at
twenty tickets, listing the innocent ones hides the one that matters —
and a caller who declared confidential themselves is told exactly that,
because "you asked for this" and "a repo you touched required it" call
for different fixes.

Regression tests at length 1 and 20 pin the array path that was never
broken, and a third pins per-item partial success: infra#265 asked "if
ticket 3 of 5 fails, what happened to 1 and 2?" — they land, and the
receipt says so per item.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m

feat(mcp): add brain_read + brain_edit, contained to the sync source (brain#4)

CI / Lint / Test / Vet (pull_request) Successful in 31s

Details

CI / Mirror to GitHub (pull_request) Skipped

Details

36b757fabf

brain_read returns a note verbatim; brain_edit is str_replace with an
exactly-once match. Both operate on INGEST_BRAIN_DIR, which is the pod's
mount of /var/lib/supervisor/brain — the tree brain-sync commits from, so
an edit reaches gitea on the next cycle rather than landing somewhere the
sync cannot see it (brain#2).

An adversarial design review before implementation changed three things,
each of which would otherwise have shipped:

1. Governance files are NOT in the model-facing tool. The original plan
   allowed root *.md so CLAUDE.md had an MCP path at all — the absence of
   one is what pushed a session to edit via the gitea REST API on
   2026-06-16 and wedge the sync. But brain content is ingested from the
   outside world, reaches a model's context via brain_query, and that
   model would hold the editor: one poisoned page could rewrite the
   CLAUDE.md a .dispatch-allow agent later loads as instructions. Worse,
   brain-sync merges with --strategy-option=ours, so the live tree wins
   against a human's corrective push on exactly the conflicting lines.
   CLAUDE.md now goes through brain_edit_root: de-surfaced from
   tools/list (the brain_index precedent, ADR-0012 rule 4) and gated on
   the local static principal, so a claude.ai-origin session is refused.
   The two scopes are disjoint — the privileged door has the smaller reach.

2. Reads needed the same allowlist, and brain_get already lacked it.
   api.ReadNote resolves any brain-relative path and parseFrontmatter
   returns the whole file when there is no fence, so brain_get reads
   verbatim session transcripts under archive/ (up to 500 KB, never
   BM25-indexed, so nobody thinks of them as reachable). A test proves
   the exposure was live; the same Resolve now gates brain_get.

3. Canonicalise before allowlisting. A prefix check on the raw argument
   accepts knowledge/../.git/hooks/post-commit, which Clean collapses to
   a git hook that brain-sync executes inside a pod holding the Gitea
   push credentials. Traversal, symlink escape, dotfiles, .git/, archive/,
   sessions/ and raw/ are refused by a static backstop that does not
   depend on the allowlist being right. There were no traversal tests in
   this repo before.

Also from the review: overlapping match counting (bytes.Count says "aa"
occurs once in "aaa", so a Count-based uniqueness check accepts an
ambiguous edit); UTF-8 validation on both sides, because encoding/json
substitutes U+FFFD and would hand back bytes the file does not contain,
making every derived old_str unmatchable; refusal while .git/MERGE_HEAD
or index.lock exists, since an edit racing brain-sync's merge is either
hard-reset away or committed as a wrongly resolved conflict; flock plus a
per-path mutex with the read taken inside the lock, because two pods
share the hostPath during a RollingUpdate and uniqueness evaluated
against a stale read loses an update while reporting success; same-dir
*.tmp + rename + chmod 0644 + fsync of file and parent; and reindex only
when the edit touches frontmatter, since BM25 walks live and embeddings
are mtime-driven.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m

mathias merged commit 055af368cd into main

2026-08-18 18:14:05 +00:00

mathias referenced this pull request

2026-08-18 18:39:52 +00:00

brain_edit_root: alert and audit a governance-file rewrite, don't just slog it #97

mathias referenced this issue from a commit

2026-08-20 15:30:25 +00:00

feat(mcp): record and alert on a governance-file rewrite (#97)

mathias referenced this pull request

2026-08-22 06:22:50 +00:00

The audit ntfy notifier is dark in production — BRAIN_NTFY_URL is unset on ingestion #100

mathias referenced this pull request