Compare commits

...
Author SHA1 Message Date
mathiasandClaude Opus 5.5 22d0653b61 test(watcher): a marker link planted during the run is refused and not retried (#158)
CI / Lint / Test / Vet (pull_request) Successful in 1m3s
CI / Mirror to GitHub (pull_request) Skipped
Review round 2 on PR #195: the marker writes had no test, so reverting
them to by-path writes, or the refusal to log-only, stayed green. The
link is planted from inside the fake model call, between the walk's
Lstat and the marker write, for both markers and both an outside and
an in-brain relative target; red on all four flips. The marker-entry
skip is now checked for .failed too.

Also pins cleanRel's acceptance of a backslash as a filename byte in
brainfs itself, and stops the buildAndWrite control claiming writes it
does not count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 23:47:24 +02:00
mathiasandClaude Opus 5.5 1a2a93d2f4 fix(watcher): refusals end in a terminal state; pin every converted writer (#158)
CI / Lint / Test / Vet (pull_request) Successful in 1m6s
CI / Mirror to GitHub (pull_request) Skipped
Review round 1 on PR #195:
- A refused marker or archive write left no marker, so the note was
  reprocessed every interval (archive copies piling up; an LLM call per
  round on the failure path). The marker check now uses Lstat (any entry
  at the marker path counts, so a dangling marker link skips the note
  instead of looping), and a note whose archive copy or marker write is
  refused is remembered (path, size, mtime) and not retried until it
  changes.
- cleanRel no longer refuses a backslash: it is a legal filename byte on
  Linux, not a traversal, and a raw note named with one was never
  archived or marked.
- Tests now reach every converted writer: the log.md append (real
  raw/failed so the log is reached, symlinked log.md), a dangling marker
  link, the refused-retry loop with a positive control, a backslash name,
  a buildAndWrite swap race, and the claude-session archive with in-brain
  and outside symlinked archive dirs. The stale lint allowlist entry for
  appendWatcherLog is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 23:40:19 +02:00
mathiasandClaude Opus 5.5 77c44f298c fix: pipeline, backfill, watcher and archive writes go through the brain root (#158 part B)
CI / Lint / Test / Vet (pull_request) Successful in 1m13s
CI / Mirror to GitHub (pull_request) Skipped
The remaining writers created or rewrote brain files by path:
- pipeline buildAndWrite: os.MkdirAll + os.WriteFile of every page.
- BackfillRefs: os.WriteFile of each concept page; a page swapped for a
  symlink after the read was written through (the race test overwrote an
  in-brain target before this change).
- watcher: the raw/processed and raw/failed archive copies (os.Create),
  the .processed/.failed markers and the log.md entry (os.OpenFile).
- cmd/server: the claude-session archive.

All now go through brainfs PutRegular/AppendRegular (new package-level
PutFile/AppendFile for one-off writers: the watcher's call-site lint bans
.Open calls in that file so raw/ reads stay on OpenRegular).

Stacked on #189 (needs PutRegular/AppendRegular).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 23:26:02 +02:00
mathiasandClaude Opus 5.5 f4636c7ef0 fix: creating and generated writes go through the brain root (#188, #158 part A) (#189)
CI / Lint / Test / Vet (push) Failing after 1m2s
CI / Mirror to GitHub (push) Skipped
New brainfs PutRegular (create or replace), CreateRegular (create only), AppendRegular, Remove and RealParent: missing parents created one component at a time through the root, every existing component a real directory, new files created with O_EXCL and confirmed before writing. brain_write, promote/reject (O_EXCL create, source kept on refusal), StampProvenanceFile, BuildWingIndex, RebuildIndex, AppendLog and the tunnel candidate log use them; provenance.ApplyFile removed.

Fixes #188. Refs #158. Adversarial review: BLOCKED, BLOCKED, PASS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-05 21:25:54 +00:00
mathiasandClaude Opus 5.5 bd989f887f fix(linkcheck): keep the write when the link index cannot be built (#178) (#191)
CI / Lint / Test / Vet (push) Successful in 1m4s
CI / Mirror to GitHub (push) Successful in 4s
StripUnresolvedLinks logs a warning and returns the content unchanged when the link index cannot be built (e.g. an unreadable subdirectory), instead of failing every brain_write/brain_update and losing the learning. Nothing is unwrapped on a partial index.

Fixes #178.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-05 21:16:04 +00:00
mathiasandClaude Opus 5.5 4770f97ffd fix(symlinkguard): report and remove FIFOs, sockets and devices too (#179) (#190)
CI / Lint / Test / Vet (push) Successful in 1m5s
CI / Mirror to GitHub (push) Successful in 4s
Scan reports special files (not regular, directory or symlink) in Result.Special, so the startup check refuses with a message naming them; Watch removes them through an os.Root after re-checking the type, and reports any it cannot remove (never as removed).

Fixes #179. Adversarial review: BLOCKED, BLOCKED, PASS. Flake on main tracked in #193.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-05 21:15:57 +00:00
mathiasandClaude Opus 5.5 ddd4bdc2ac fix(brainfs): refuse a file behind a symlinked directory for reads and writes (#173) (#187)
CI / Lint / Test / Vet (push) Successful in 1m2s
CI / Mirror to GitHub (push) Successful in 3s
ReadRegular, ReadRegularMax and WriteRegular require a clean, local path and a real directory at every level above the file, so an in-brain directory symlink (or a ".." after one) can no longer redirect brain_update, brain_tunnel, brain_read/get or the other readers elsewhere in the brain. session.Append opens through an os.Root on the brain dir, refuses a non-directory sessions dir, and writes only after confirming the log path is the regular file it opened.

Fixes #173. Adversarial review: BLOCKED, PASS. Follow-up: #188 (brain_write create path).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-05 20:17:37 +00:00
mathiasandClaude Opus 5.5 1fff54769d fix(tunnel): DetectTunnels walks and reads through brainfs (#174, #157 slice 5b) (#186)
CI / Lint / Test / Vet (push) Successful in 1m2s
CI / Mirror to GitHub (push) Successful in 4s
DetectTunnels (inside every brain_write/brain_update via AutoTunnel) walks with brainfs.WalkRegular and reads titles with ReadRegular: a FIFO under wiki/ no longer hangs the write, and a symlinked note's title no longer becomes a tunnel candidate. A wiki/ that is not a real directory yields no candidates; other walk errors still abort.

Fixes #174. Refs #157 (last reader slice). Adversarial review: BLOCKED, PASS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-05 20:12:44 +00:00
mathias 0c3e82ca0a test(brainfs): per-caller in-brain symlink targets for slices 1-3 (#171) (#172)
CI / Lint / Test / Vet (push) Successful in 1m2s
CI / Mirror to GitHub (push) Successful in 4s
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-05 19:36:31 +00:00
mathiasandClaude Opus 5.5 eedb110be5 fix(pipeline,classification): read the brain through brainfs (#157, slice 6) (#176)
CI / Lint / Test / Vet (push) Successful in 1m1s
CI / Mirror to GitHub (push) Successful in 4s
Ingest pipeline readers (readSourcePage and loadSchema feed the LLM prompt; reviseOrCreate; injectSourceRefs; BackfillRefs list and reads) and classification.Load read through brainfs: symlinked or special files no longer reach the prompt, written pages or the I1 sovereignty config, and a FIFO no longer hangs ingest or startup. A symlinked/special classification.yaml is a hard error; an absent one still yields defaults.

Refs #157. Adversarial review: BLOCKED, BLOCKED, PASS. Follow-ups: #158 (write paths), #179 (FIFO tripwire).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-05 18:54:42 +00:00
mathiasandClaude Opus 5.5 febab3e45d fix: brain_tunnel, brain_update and session_log confine caller-supplied paths (#169) (#170)
CI / Lint / Test / Vet (push) Successful in 1m2s
CI / Mirror to GitHub (push) Successful in 4s
brain_tunnel accepts only clean wiki/.../*.md paths and reads/rewrites both notes through brainfs; new brainfs.WriteRegular never creates, never writes through a final symlink (Lstat + O_NOFOLLOW|O_NONBLOCK + os.SameFile on the handle, since os.Root follows in-root final symlinks even with O_NOFOLLOW). brain_update resolves like brain_edit (knowledge/ and wiki/ notes only) and reads/rewrites through brainfs. session_log requires a plain-name session id and opens with O_NOFOLLOW|O_NONBLOCK.

Fixes #169. Adversarial review: BLOCKED, BLOCKED, PASS (round 3 by the estate session's reviewer). Follow-ups: #173, #174.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-05 18:47:07 +00:00
mathias bdc1e51a1a fix(ingestion): build /mcp's bearer and the metrics wrap inside buildMux (#154) (#183)
CI / Lint / Test / Vet (push) Successful in 1m1s
CI / Mirror to GitHub (push) Successful in 4s
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-05 17:48:44 +00:00
mathias 8c48e2e343 fix(symlinkguard): deliver alerts beside the scan, not inside it (#164) (#180)
CI / Lint / Test / Vet (push) Successful in 54s
CI / Mirror to GitHub (push) Successful in 4s
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-05 14:48:16 +00:00
mathias c0be5c473c fix(ingestion): bump mcp-chassis to v0.7.1; log auth rejections; label refused REST requests by route (#181)
CI / Lint / Test / Vet (push) Successful in 1m4s
CI / Mirror to GitHub (push) Successful in 4s
Closes #155

Reviewed by pr-gatekeeper: #181 (comment) (block, round 1) and re-review at e05a13a (approve).

Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-05 14:40:20 +00:00
mathiasandClaude Opus 5.5 0bd0fbf03e test(brainfstest): swap-race asserts fail when nothing was tested
CI / Lint / Test / Vet (push) Successful in 55s
CI / Mirror to GitHub (push) Successful in 4s
AssertNoLeakFrom and the search Query race test passed vacuously when
every symlink swap failed or every read errored. Both now count
successful symlink swaps and reads that returned the inside note and
fail if either is zero; the Query race reader no longer swallows
Query's error.

Verified by flipping: a swapper whose os.Symlink always fails turns
notes, wiki and search race tests red; Query on a missing brain dir
turns the search race test red.

Closes #177

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 09:32:25 +02:00
mathiasandClaude Opus 5.5 ef39435252 fix(brain): wing indexes and the link check read through brainfs (#157, slice 5a) (#175)
CI / Lint / Test / Vet (push) Successful in 55s
CI / Mirror to GitHub (push) Successful in 5s
BuildWingIndex/BuildAllWingIndexes list wings and halls with the new brainfs.ReadDirDirs and ReadDirRegular and read titles with ReadRegular: a symlinked note no longer reaches _index.md, a symlinked wing is refused (nothing written into its target), a symlinked hall is skipped. linkcheck entryIndex walks with WalkRegular, so a symlinked .md cannot make a link resolve, and a linked tree that is not a real directory is skipped with a warning (brain_write still lands).

Refs #157. Adversarial review: BLOCKED, PASS. Follow-up: #178.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-05 07:26:12 +00:00
mathiasandClaude Opus 5.5 b6ef11600b fix(deps): bump x/net and x/text past called vulns
CI / Lint / Test / Vet (push) Successful in 1m45s
CI / Mirror to GitHub (push) Successful in 4s
govulncheck 2026-10-04 found called vulns in ingestion (brain-mcp, public),
reached via audit.LokiCentral.Push: golang.org/x/net v0.26.0
(GO-2026-4918 HTTP/2 infinite loop, GO-2026-5026) and golang.org/x/text
v0.29.0 (GO-2026-5970). x/net -> v0.55.0, x/text -> v0.39.0; tidy also
moved x/sync, x/sys, x/term. govulncheck ./... now: 0 called findings
(19 in required modules, none reachable).

Refs: govulncheck 2026-10-04

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 06:59:31 +02:00
mathiasandClaude Opus 5.5 432e09d122 test(ingestion): OIDC discovery 502s then 200 against a real issuer (#83)
The retry landed in 9a232ad but was tested only through startupretry with
fake funcs; nothing exercised chassisauth.NewJWTValidator's real 502 path.
Extracts the main() closure into buildJWTValidator (behaviour unchanged)
and tests it against an httptest issuer: 3x502 then 200 succeeds with 4
discoveries; a persistent 502 stops at exactly the 8-attempt budget with
the last error; cancelling ctx mid-backoff returns context.Canceled.
Flip-verified: Attempts=1 turns all three red.

Lane: risk/medium issue, committed to main because the behaviour change
already shipped in 9a232ad; this is a test plus a pure extraction.

Closes #83

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 06:57:45 +02:00
mathiasandClaude Opus 5.5 77e32de33c fix(graph,wiki): read the brain through brainfs (#157, slice 4) (#168)
CI / Lint / Test / Vet (push) Successful in 52s
CI / Mirror to GitHub (push) Successful in 4s
graphsync, wiki inventory/index read through brainfs; brainfs refuses an in-brain file swapped in behind a symlink (os.SameFile). Gatekeeper round 2 PASS on d90ce66, CI run 9180 green.

Refs #157

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-05 04:53:34 +00:00
mathiasandClaude Opus 5.5 57d0fb0fae fix(vectorstore): walk and read the brain through brainfs (#157, slice 3) (#166)
CI / Lint / Test / Vet (push) Successful in 48s
CI / Mirror to GitHub (push) Successful in 3s
vectorstore.Sync walks and reads via brainfs (regular files only, handle-checked), so symlinked notes are no longer embedded and their old chunks are purged. brainfs.WalkRegular requires every component of its root to be a real directory (fixes in-brain root aliasing in Sync and search Wing/Wing+Hall). Walk errors, including a symlinked scan root, abort Sync before the delete pass. brainfs gains Lstat. Directory-swap race recorded as accepted residual (#167).

Refs #157. Adversarial review: BLOCKED, BLOCKED, PASS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-04 06:35:17 +00:00
mathiasandClaude Opus 5.5 27a01dd2af fix(brain): context, brain_get and brain_read read through brainfs (#157, slice 2) (#165)
CI / Lint / Test / Vet (push) Successful in 46s
CI / Mirror to GitHub (push) Successful in 4s
readExcerpt, ReadNote and ReadScoped now read through brainfs (os.Root, regular files only, type checked on the opened handle), so a symlink swapped in after a check is not followed and a FIFO does not hang the reader. ReadRegularMax bounds the read itself with io.LimitReader(max+1). notes.Target carries the brain root.

Refs #157. Adversarial review round 2: PASS (round 1 BLOCKED: unbounded read, rule-9 test names).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-04 05:59:55 +00:00
mathias eb77305580 fix(search): walk and read the brain through an os.Root (#157, slice 1) (#161)
CI / Lint / Test / Vet (push) Successful in 37s
CI / Mirror to GitHub (push) Successful in 4s
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-03 19:34:46 +00:00
mathias c986df6799 feat(ingestion): runtime symlink tripwire removes what it finds, 30s scans (#157) (#160)
CI / Lint / Test / Vet (push) Failing after 11m16s
CI / Mirror to GitHub (push) Skipped
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-03 19:34:19 +00:00
mathias 687afa0145 fix(ingestion): brain/raw/ readers go through an os.Root; symlinks are inert (#153) (#156)
CI / Lint / Test / Vet (push) Successful in 36s
CI / Mirror to GitHub (push) Successful in 4s
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-03 15:30:49 +00:00
mathias cad389f62f feat(ingestion): symlink tripwire for the Syncthing-shared brain dir (#157) (#159)
CI / Lint / Test / Vet (push) Successful in 35s
CI / Mirror to GitHub (push) Successful in 3s
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-03 15:23:06 +00:00
mathias cfe3f24e25 fix(ingestion): authenticate the REST API; confine ingest paths to brain/raw/ (#149) (#152)
CI / Lint / Test / Vet (push) Successful in 34s
CI / Mirror to GitHub (push) Successful in 4s
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-03 14:38:07 +00:00
mathiasandClaude Opus 5.5 a6da93332d feat(brain): audit-log every tools/call; server-owned write provenance (#147)
CI / Lint / Test / Vet (push) Successful in 39s
CI / Mirror to GitHub (push) Successful in 4s
Closes #144 (audit line per tools/call) and #145 (origin/written_by stamped on every model-reachable write, surfaced in brain_query/brain_get).

Passed pr-gatekeeper after four rounds of adversarial review, verified by execution: #147

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-10-01 21:06:36 +00:00
mathiasandClaude Opus 5.5 f21be1aef6 chore(context): re-sync adapters after root brain-scoping correction
CI / Lint / Test / Vet (push) Successful in 43s
CI / Mirror to GitHub (push) Successful in 4s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 20:42:10 +02:00
mathiasandClaude Opus 5 3c70b6f367 fix(claudewatcher): scrub credentials embedded in URL userinfo
CI / Lint / Test / Vet (push) Successful in 33s
CI / Mirror to GitHub (push) Successful in 5s
The scrubber had a rule for `postgres://user:pass@` and none for the
general `scheme://user:secret@host` form — which is exactly what a git
credential store holds:

    https://<user>:<token>@git.d-ma.be

Verified against all 13 DefaultRules with an invented value: it matched
none of them. Not authorization-header, not bearer-token, and not
homelab-env-token, which needs a `_TOKEN=`-style prefix a bare URL does
not have. So such a turn was ingested rather than dropped, against the
stated intent in this file's own doc comment:

    Fail-closed by design: we'd rather lose signal than ingest
    credentials into a public-readable brain.

It failed closed for the shapes it named and open for this one.

How it surfaced: a test suite in mathias/infra executed a script that
configures git while inheriting the operator's real $HOME, read koala's
live ~/.git-credentials, and printed two live values. ~/.claude/projects
is this watcher's own input, so the scrubber was the last line of defence
for that mistake. A sweep of that tree found 12 occurrences of 2 distinct
credential-shaped URLs across 4 transcripts in 3 sessions — so the shape
had been passing for a while, not just that once. Both credentials are
being rotated (infra#466).

The generalisable point is not "do not read the credential file". That
mistake will recur. It is that this shape must fail closed, because the
scrubber is what decides whether such a mistake is survivable or
permanent.

Ordering: the new rule sits after postgres-uri-with-password so a DSN
still reports the more specific name, the same convention as
authorization-header before bearer-token.

Both directions are tested, because a rule that dropped every URL with a
colon would discard most of the signal this watcher exists to keep. The
{8,} floor on the secret half keeps `https://host:443/path` and
`https://user:@host` out; the `[^:\s/@]+` user half stops a host:port
pair being read as userinfo. Must-not-scrub cases include a host with a
port, a password-less DSN, a colon in a query string, and the literal
printf format string `https://oauth2:%s@%s`.

Closes #142
Refs mathias/infra#466

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-30 16:08:17 +02:00
mathiasandClaude Opus 5 b3b555fa23 chore(context): adopt canonical context-sync.sh and re-sync adapters
CI / Lint / Test / Vet (push) Successful in 1m46s
CI / Mirror to GitHub (push) Successful in 4s
infra's scripts/context-sync.sh is the estate canonical, and it moved in infra
262369b (a SIGPIPE fix on the placeholder-reporting path: `printf | grep -o |
head -1` became `head -1 < <(grep -o ...)`). Every other repo still carried the
old line, so ContextSyncScriptDrift and ContextAdapterDrift were both firing.

Also picks up the canonical AGENT.md change from local-dev a01ee80 — rule 5's
GitOps review lane and the new precedence rule — which every repo's adapters
need regenerated for regardless of the script.

Refs mathias/infra#470

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-30 10:36:40 +02:00
mathiasandClaude Opus 5 e9805c7eb6 chore(context): regenerate derived adapters from the canonical root context
CI / Lint / Test / Vet (push) Successful in 56s
CI / Mirror to GitHub (push) Successful in 5s
ContextAdapterDrift was firing for 13 repos. Picks up behaviour rule 7
("Finish what's closable"), which landed in ~/dev/.context/AGENT.md and never
reached any repo's derived files.

Where mcp.json changes, it also picks up root MCP config the repo was stale on:
brain/gitea/infra move from public hostnames to tailnet Serve URLs, and gitea's
bearer changes from GITEA_MCP_TOKEN to DMABE_GITEA_API_TOKEN — gitea-mcp
authenticates by pass-through and 401s on the former, so a harness syncing from
the old file had a dead gitea server and no signal saying so.

No literal secrets: mcp.json carries ${VAR} placeholders only.

CLAUDE.md deliberately unchanged — context-sync writes it from PROJECT.md alone,
since Claude Code walks up to ~/dev/CLAUDE.md for root context.

Refs mathias/infra ContextAdapterDrift

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-25 07:34:25 +02:00
mathias 3d5af85713 fix(frontmatter): consolidate 3 independent fence-parsers into one (#139)
CI / Lint / Test / Vet (push) Successful in 32s
CI / Mirror to GitHub (push) Successful in 4s
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-09-23 13:07:32 +00:00
mathiasandClaude Sonnet 5 c6341f020a feat(server): say at startup when alerting is unconfigured (#107) (#140)
CI / Lint / Test / Vet (push) Successful in 39s
CI / Mirror to GitHub (push) Successful in 5s
feat(server): say at startup when alerting is unconfigured (#107)

WARN once at startup when BRAIN_NTFY_URL is unset, naming what will not fire (brain_edit_root governance rewrites, capture audit degradation). Rebase of PR #108 (never CI-tested) onto current main, no functional change to alertingWarnings.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-09-23 13:02:40 +00:00
mathiasandClaude Sonnet 5 e157311cd8 fix(mcp): validate the claim envelope before brain_edit writes too (#136)
CI / Lint / Test / Vet (push) Successful in 39s
CI / Mirror to GitHub (push) Successful in 4s
7 review rounds closed 8 distinct bypasses across two mechanism families: fence-detection (field-name substring, block-containment, fence-stripping, CRLF, UTF-8 BOM) and value-extraction (folded YAML block scalar hiding an invalid value). A known v1 gap remains in the block-scalar fold's handling of blank-line-interrupted continuations — documented on checkClaimEnvelope, accepted deliberately (narrow, requires an already-privileged caller deliberately constructing a specific malformed-YAML shape) rather than chased past diminishing returns. Real fix (shared, tested, encoding-normalized frontmatter parser) tracked separately as hyperguild#138.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-09-23 10:28:07 +00:00
mathias a5f502eb9c fix(api): UpdateNote reserves updated_at/supersedes/supersede_reason from the caller's fragment (#137)
CI / Mirror to GitHub (push) Successful in 3s
CI / Lint / Test / Vet (push) Successful in 39s
Closes #125. Skips updated_at/supersedes/supersede_reason in the caller fragment merge loop so they cannot be set via the fragment, only via the intended entry points.
delete_branch_after_merge=true

Co-authored-by: Mathias <mthbqv@gmail.com>
2026-09-23 08:18:30 +00:00
mathias 7ed2a7d3b4 fix(api): frontmatter parser must not misparse indented list items as keys (#135)
CI / Mirror to GitHub (push) Successful in 4s
CI / Lint / Test / Vet (push) Successful in 40s
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-09-23 08:12:06 +00:00
mathiasandClaude Sonnet 5 674858fe5e fix(watcher): never walk into raw/rejected/ — brain_reject (closes #81)
CI / Lint / Test / Vet (push) Successful in 33s
CI / Mirror to GitHub (push) Successful in 4s
Adds brain_reject (RejectNote + MCP tool) and fixes the watcher to skip raw/rejected/, preventing declined notes from being silently re-ingested and republished.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-09-23 06:06:35 +00:00
mathias 9a37881441 feat(pipeline): feed the current source page into the extraction prompt (#106) (#132)
CI / Lint / Test / Vet (push) Successful in 39s
CI / Mirror to GitHub (push) Successful in 4s
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-09-23 05:45:47 +00:00
mathias f324f04944 fix: hyperguild brain write requires wing/hall, not legacy type/slug (#79)
CI / Lint / Test / Vet (push) Successful in 34s
CI / Mirror to GitHub (push) Successful in 4s
Removes brain_write legacy domain/type/knowledge-path params (ADR-0012) and migrates the previously-missed cmd/hyperguild CLI caller (hyperguild brain write) from <type> <slug> to <wing> <hall> <slug>, fixing a 100%% regression the first review round caught. Adds TestWrite_CLIPayloadRoundTrips exercising the real Handler.Write over HTTP to pin the wire contract.

Closes #79.

Co-authored-by: Mathias <mthbqv@gmail.com>
2026-09-23 05:43:28 +00:00
mathiasandClaude Sonnet 5 cdafdf6e0f fix(wiki): surface merge conflicts as textual-diff warnings (closes #104)
CI / Lint / Test / Vet (push) Successful in 33s
CI / Mirror to GitHub (push) Successful in 4s
Textual-diff conflict detection only (semantic contradiction detection via brain-gardener Judge deferred, confirmed by Mathias, comment 12594). Includes canonicalBlock whitespace-run normalization fix for the disagreement comparison.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-09-23 05:22:27 +00:00
mathias b340f86a45 feat(pipeline): skip extraction when the source has not changed (#105) (#126)
CI / Lint / Test / Vet (push) Successful in 40s
CI / Mirror to GitHub (push) Successful in 4s
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-09-23 05:09:30 +00:00
mathias f450a6c84f feat(pipeline): ingest revises the existing page instead of clobbering it (#103)
CI / Lint / Test / Vet (push) Successful in 32s
CI / Mirror to GitHub (push) Successful in 5s
fix(wiki): give the preamble its own field instead of overloading heading=="" (#103)

Round 3 review: verified structurally that the preamble is no longer in the sections slice or the idx map, so it cannot collide with a degenerate `## ` heading. Flip-tested by reverting to the heading=="" model and confirming TestMerge_DegenerateEmptyHeadingDoesNotCollideWithPreamble goes red. Adversarial pass (multiple degenerate headings, frontmatter-only page, unicode around the ## boundary, 3x self-merge idempotency with preamble+degenerate) all held. CI green on 09aeaa9 (run 7820).

Co-authored-by: Mathias <mthbqv@gmail.com>
2026-09-22 21:43:04 +00:00
mathias 5a5d938e8e Brain-quality sweep, batch 1: brain_edit/brain_update correctness + close-session docs (#124)
CI / Lint / Test / Vet (push) Successful in 32s
CI / Mirror to GitHub (push) Successful in 4s
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-09-22 21:27:11 +00:00
mathiasandClaude Opus 5 cc86a74dc5 fix(ci): stop the tag push from re-deploying, and make the infra bump idempotent
CI / Lint / Test / Vet (push) Successful in 34s
CI / Mirror to GitHub (push) Successful in 4s
Two runs of the full build+deploy fired for every push, and the second died:

    nothing to commit, working tree clean
    ❌  Failure - Main Update infra repo

The deploy was fine both times — the second run had nothing left to do. The
cause is the pair of triggers meeting: ci.yml ran on `push` to main AND on a
`v*` tag, and cd.yml keys off "CI succeeded", so `git push origin main` followed
by `git push origin refs/tags/vX` produced two CI runs for one sha and two
deploys behind them. The tag run carries the same github.sha, so it rebuilt the
same image and then found the infra manifest already pointing at it.

Nothing consumed the tag-triggered run: there is no release workflow, and cd.yml
reads github.sha rather than the tag. Dropping `tags:` removes the duplicate
build outright.

The idempotency guard stays anyway, because a manual re-run has the same shape
and should not ring the CI-red alarm for work that is already done — infra#366
turns a red run into a tracked issue, so a false red costs a real triage.

Closes #118

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 07:09:14 +02:00
mathiasandClaude Opus 5 6e449b464c fix(taskfile): remove build targets for binaries deleted months ago, and make check build (#121)
CI / Lint / Test / Vet (push) Successful in 32s
CI / Mirror to GitHub (push) Successful in 4s
Removes supervisor:dev, supervisor:build, supervisor:test:smoke, smoke:routing and scripts/smoke-routing.sh -- all pointed at cmd/supervisor (deleted c117634, 2026-05-12) or cmd/routing (iceboxed fb59c39, 2026-07-01). Repoints `build` at hyperguild:build + ingestion:build and adds `build` to `check`, so a task target naming a deleted package fails the gate instead of staying invisible to lint/test/vet.

Refs #120. Item 2 of #120 (internal/config has no non-test callers) is deliberately NOT addressed -- #120 stays open for it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Mathias <mthbqv@gmail.com>
2026-09-16 22:36:49 +00:00
mathias 868fe28c96 fix(config): delete the model-routing chain — nothing has called it since April (#119)
CI / Lint / Test / Vet (push) Successful in 31s
CI / Mirror to GitHub (push) Successful in 4s
Closes #116.

Deletes config/models.yaml, internal/config/models.go (LoadModels, ModelFor),
its test, the ModelsFile field on Config, and SUPERVISOR_MODELS_FILE from
.env.example. Option A from the issue.

Merge-gate evidence: negative claim independently re-established (no Go, shell,
Dockerfile, k8s, CI or infra-repo caller); task check exit 0 across both
modules; CI run 7374 green on the PR head.

Co-authored-by: Mathias <mthbqv@gmail.com>
2026-09-16 22:24:47 +00:00
mathiasandClaude Opus 5 6e69698343 fix(brain): a one-word tunnel title must match the prose case-exactly
CI / Mirror to GitHub (push) Successful in 4s
CI / Lint / Test / Vet (push) Successful in 31s
AutoTunnel matched note titles case-insensitively, so a one-word title linked
every entry that happened to use that word. Measured on the live corpus: 185
entries carry an auto-written See-also link to wiki/telos/decisions/wrong.md
(title "WRONG") and 136 to .../status.md — the two most-linked nodes in the
whole graph, and both of them artifacts of the English language rather than
references. telos's projects/problems/strategies/goals/beliefs entries account
for another ~60 between them.

That pollution is not cosmetic: inbound-link count is what the duplicate
Fixer picks survivors by, what orphan detection reads, and what brain_graph
shows.

One-word titles now need the title's own casing in the prose ("WRONG", "BERT",
"NLP" — an entity name in a sentence still links). Multi-word titles keep the
case-insensitive match: "risk tier middleware" is a real mention however it is
capitalised, and a three-word collision is not an accident.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-17 00:08:40 +02:00
mathiasandClaude Opus 5 eb2acfeeb8 fix(brain): move the wikilink strip into WriteNote/UpdateNote
CI / Mirror to GitHub (push) Successful in 4s
CI / Lint / Test / Vet (push) Successful in 32s
The BrainStore is not the only door. POST /write — the HTTP path CLAUDE.md
hands to curl and scripts — calls api.WriteNote directly, so the strip
shipped one commit ago covered brain_write and capture while leaving the raw
door ungated. A gate one door wide is not a gate.

The BrainStore keeps its own call so it can still report Ref.Unlinked to the
MCP caller; running the strip twice is a no-op on content the primitive has
already cleaned.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-16 23:59:44 +02:00
mathiasandClaude Opus 5 6e5a9cfce3 fix(brain): unwrap wikilinks that name no entry at the write boundary
CI / Mirror to GitHub (push) Successful in 4s
CI / Lint / Test / Vet (push) Successful in 35s
Every brain write accepted [[targets]] that resolve to nothing — skill
names, repo names, agent-local memory-file slugs — and each one became a
permanent [high] broken-link finding. The brain-audit digest grew 19 -> 63
high findings between 2026-08-26 and 2026-09-16; 53 of the 63 were exactly
this, and none are fixable by brain-gardener's BrokenLinkFixer, which only
repoints edit-distance-1 renames. There is no target to repoint to.

StripUnresolvedLinks resolves each link against knowledge/ + wiki/ using
brain-gardener's own resolver semantics (bare slug or wing/hall/slug path
form, code spans excluded) and unwraps the misses to plain text, reporting
them as Ref.Unlinked / the MCP `unlinked` field.

Unwrap rather than reject: a refused write on claudewatcher's unattended
ingest path is a lost learning, and the author's words are worth keeping
even when the link is not. An empty corpus is a no-op, so a misconfigured
brainDir cannot strip every link in a note.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-16 23:53:42 +02:00
mathiasandClaude Opus 5 25b79f057e feat(claudewatcher): exclude whole project directories from ingestion
CI / Mirror to GitHub (push) Successful in 4s
CI / Lint / Test / Vet (push) Successful in 1m20s
CLAUDE_INGEST_SKIP_PROJECTS takes filepath.Match globs tested against
the project directory NAME, and a matching directory is skipped during
the directory WALK — so its files are never opened and their content
never enters the process. That is a stronger property than filtering
after reading, and the difference is observable: without SkipDir,
WalkDir lists the directory's entries, and an unreadable one aborts the
entire walk, silently stopping ingestion for every other project.

Globs rather than exact names because a project directory is derived
from the session's working directory, so a git worktree yields an
ephemeral name no exact-match list could keep up with.

An unparseable pattern refuses to run. filepath.Match returns an error
ALONGSIDE a false match, so a bad pattern checked per-file would ingest
the directory while the log complained — the failure nobody notices.
Validated once, up front, in both entry points.

This complements the client-name scrubber rather than replacing it. The
scrubber drops individual turns whose content matches a client
identity; this excludes whole workspaces whose content is confidential
in ways no pattern reliably recognises. Learnings from excluded
sessions are captured deliberately through the session-close ritual
instead.

Recorded caveat: the project directory reflects the session's working
directory, NOT the repositories it touches. A session run from ~/dev
that reads an excluded repo is not excluded by this. The scrubber
remains the control for content; this is a control for workspaces.

A mutation test earned the strongest assertion here: the original tests
could not distinguish SkipDir from filtering after the walk, because
neither opens files. The unreadable-directory test can.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-15 08:09:58 +02:00
mathias 47bbac4b9f revert(lint): back out the canonical .golangci.yml adoption — repo has a nested module
CI / Lint / Test / Vet (push) Successful in 30s
CI / Mirror to GitHub (push) Successful in 3s
Adopting it turned CI red, and my pre-push verification missed why:
`golangci-lint run ./...` does NOT traverse into nested Go modules. This repo
has two — ./go.mod and ./ingestion/go.mod. I linted the root (0 issues, which
CI reproduces exactly) and never linted ingestion/, which has 3 findings:

  ingestion/internal/audit/degrading.go:81   missing case capture.AuditCentral   (exhaustive)
  ingestion/internal/capture/entities.go:37  missing case capture.ZoneUnknown    (exhaustive)
  ingestion/internal/mcp/handlers.go:29      var int_ should be int              (revive)

Those are behavioural questions and a rename, not formatting — the same class
of finding that kept cad, dispatch, ingestion-svc, dma-cli and
email-triage-agent out of the earlier waves. Consistent with that, this repo
goes back in the queue rather than being force-fitted now.

The gofumpt formatting from the same commit is kept: it is independently
correct and already green.

Verification method corrected going forward: a repo with nested modules must
be linted per module, not once from the root.
2026-09-05 14:47:39 +02:00
mathias 8a27ead91c chore(lint): adopt the canonical .golangci.yml from infra/ci-templates
CI / Mirror to GitHub (push) Skipped
CI / Lint / Test / Vet (push) Failing after 12s
Canonical lives at infra ci-templates/golangci.yml, distributed by the
infra#309 pattern: byte-identical copies, checksum drift gate in infra. Do
not edit this file here -- edit the canonical and run `task ci:sync`.

Verified locally before pushing, not pushed-and-hoped: golangci-lint v2.11.4
against a Go 1.26.1 toolchain (matching CI) reports 0 issues here.

Also applies gofumpt, which this config enables and the golangci-lint v2
default set does not check at all. That gap is why the drift was invisible:
earlier passes ran gofmt, which is weaker. Mechanical, deterministic,
build+vet verified.
2026-09-05 14:42:59 +02:00
mathias 1a9be0cfb1 style: apply gofmt
CI / Lint / Test / Vet (push) Successful in 30s
CI / Mirror to GitHub (push) Successful in 4s
Estate-wide consolidation pass. golangci-lint v2 default set enables NO
formatter, so gofmt drift accumulated silently across the estate -- 8 repos
were carrying it, this one included. Same hole that let two files rot in
dma-cli until they were found by hand (dma-cli#12).

Mechanical and deterministic: `gofmt -w`, nothing hand-edited. Verified
before commit -- gofmt -l now reports 0, go build and go vet both pass.

Clears this repo to adopt the canonical .golangci.yml (infra
ci-templates/golangci.yml, infra#309 pattern) in a later wave.
2026-09-05 12:40:34 +02:00
mathias dc2265da0b fix(ci): skip context:sync's drift check under CI, matching dma-cli/gitea-mcp
CI / Mirror to GitHub (push) Successful in 4s
CI / Lint / Test / Vet (push) Successful in 1m22s
hyperguild's Taskfile ran context:sync unconditionally in 'check', with no
guard for the CI runner having no ancestor .context/AGENT.md. That made it
silently regenerate AGENTS.md and .context/system-prompt.txt from project
context alone (82 lines vs. the committed, correct 430) and then fail its
own drift assertion against that wrong baseline every run -- a false
positive, not real drift. Verified locally: syncing with the real root
context reproduces the committed 430-line files byte for byte; syncing
without it (as CI does) produces the shorter fallback that triggered this.

Same guard dma-cli and gitea-mcp already carry, same rationale: drift is
now caught either on a developer machine or by infra's estate-wide
context-sync-drift CronJob (ADR-0019), which does have canonical access.
2026-09-05 11:36:45 +02:00
mathias c176b71b3d fix(context): AGENTS.md/system-prompt.txt committed in 4e8bdc1 were missing root content
CI / Lint / Test / Vet (push) Failing after 2s
CI / Mirror to GitHub (push) Skipped
The previous commit's regeneration produced AGENTS.md byte-identical to
CLAUDE.md (82 lines, project-only) instead of the expected root+project
concatenation (430 lines) -- root_block() rendered empty despite the run's
own log claiming "Root context: .../AGENT.md" and "AGENTS.md (root +
project...)". Not reproduced: re-running the identical command
(ROOT_CONTEXT=.../AGENT.md bash scripts/context-sync.sh) immediately after
produces the correct 430-line output every time. Whatever caused the
original miss was transient and is not diagnosed here -- flagging rather
than asserting a cause I have not confirmed.

Caught by the estate-wide context-sync-drift gate's own verification run,
not by trusting the fix commit. Regenerated fresh and confirmed root
content is present before committing this time.
2026-09-05 08:36:03 +02:00
mathias 4e8bdc137e fix(context): sync to the estate canonical context-sync.sh (ADR-0019)
CI / Lint / Test / Vet (push) Successful in 1m5s
CI / Mirror to GitHub (push) Successful in 4s
Caught by the new estate-wide context-sync-drift gate (infra, ADR-0019).
This repo's copy had diverged from the canonical established in dma-cli/
infra -- synced to it, dead Cursor/Aider artifacts removed (2026-09-01
harness decision), Taskfile's drift assertion narrowed to match, adapters
regenerated against the current canonical AGENT.md.

Changed:
M  .context/system-prompt.txt
M  AGENTS.md
M  scripts/context-sync.sh

Refs: ADR-0019 (infra)
2026-09-05 08:30:04 +02:00
mathias 42235dcb6a fix(scan): an allowlist was hiding a real Gitea PAT on a false justification
CI / Lint / Test / Vet (push) Successful in 32s
CI / Mirror to GitHub (push) Successful in 5s
infra#344. The removed block suppressed
docs/superpowers/plans/2026-04-22-phase4-attempt-wiring.md, and its description
stated why:

    "Literal placeholder word REDACTED matched as if it were a token
     (verified 2026-08-04: extracted Secret == 'REDACTED' exactly, gitleaks'
      curl-auth-header rule matched the placeholder text itself, not a real
      credential)"

That is not what the file contained. Line 1054 carried a real 40-character hex
Gitea PAT in an `Authorization: token ...` header — not the eight-character word
REDACTED. Whether the file changed after the check or the check was wrong, the
justification has been guarding the wrong thing since.

Confirmed dead by side-effect before touching it, never by printing it: `GET
/api/v1/user` with that token returns 401. So there is nothing to rotate. It is
redacted here anyway, because a dead credential in a doc is still a working
example of how to write a live one, and git history keeps the original either
way.

Two independent failures had to line up, which is the same pair as infra#256:

1. The `paths` clause made the block far broader than its description. gitleaks
   ORs global allowlist clauses and ignores `condition = "AND"`, so the ENTIRE
   file was immune — a live token pasted into that doc tomorrow would have been
   suppressed on the strength of a note about a different value.
2. The justification was never re-checked against the file it justifies.

With the token redacted the allowlist has nothing left to do, so it is deleted
rather than narrowed. Repo scans clean, and a planted live-shaped credential in
that same file is now caught.

Refs: mathias/infra#344, mathias/infra#256, mathias/infra#317
2026-08-31 07:24:07 +02:00
mathiasandClaude Sonnet 5 05330ad54a feat(brain): add Gemini as brain_answer's third LLM tier
CI / Lint / Test / Vet (push) Successful in 36s
CI / Mirror to GitHub (push) Successful in 4s
llm.Router was primary-then-fallback (berget -> iguana ollama). Both
have failed together at least once in production (berget 5xx outage +
a cold fallback model exceeding the shared timeout budget), leaving
brain_answer/brain_classify fully down.

Router gains an optional Tertiary *Client, tried only if both Primary
and Fallback fail. Nil-safe — existing 2-tier deployments (no
BRAIN_LLM_TERTIARY_URL) behave exactly as before.

main.go wires it from BRAIN_LLM_TERTIARY_URL/MODEL + GEMINI_API_KEY,
pointed at Google's OpenAI-compatible Gemini endpoint
(generativelanguage.googleapis.com/v1beta/openai/) — llm.Client already
speaks OpenAI-shaped chat-completions, so no new client type needed.

Tests: TestRouter_FallsBackToTertiaryWhenPrimaryAndFallbackFail,
TestRouter_TertiaryNotCalledWhenFallbackSucceeds, TestRouter_AllThreeFail,
TestRouter_NilTertiary (2-tier backward-compat). Full suite green
(both modules, -race), golangci-lint clean.

Deployment env vars (infra repo, separate commit — GitOps, TBD there
too): BRAIN_LLM_TERTIARY_URL, BRAIN_LLM_TERTIARY_MODEL, GEMINI_API_KEY
secretRef.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-30 14:43:25 +02:00
mathias 692244d213 security: disposition the one historical gitleaks finding (infra#326)
CI / Lint / Test / Vet (push) Successful in 31s
CI / Mirror to GitHub (push) Successful in 3s
A Gitea PAT was pasted into a CD-watching curl command in a 2026-04-22
planning doc (c9310b10, last_eight 3105a670).

Verified dead before dispositioning it, rather than assuming: authenticating
GET /api/v1/user with that exact value returns 401. It was almost certainly
invalidated by the early-August rotation of the `mathias` PATs -- the same
rotation whose fan-out caused infra#260, where CI's stale copies kept
failing. So it was killed for unrelated reasons, by luck rather than by the
leak being noticed.

No rotation outstanding. `task security:gitleaks:history` now exits 0 here.

Used .gitleaksignore (per-finding fingerprint) rather than a path allowlist
in .gitleaks.toml: a path allowlist would also blind the working-tree scan
that gates commits. Verified both directions -- history 1 -> 0, working tree
still clean.
2026-08-29 10:04:11 +02:00
mathias d98a305985 security(hooks): add local pre-commit gitleaks scan (infra#326)
CI / Lint / Test / Vet (push) Successful in 1m33s
CI / Mirror to GitHub (push) Successful in 4s
task check -> security:gitleaks was already CI-enforced here, but that
catches a leak after the push, not before -- exactly the gap infra#39's
prevention checklist called out ("gitleaks pre-commit hook in both infra
and hyperguild"), true for infra but not this repo.

Mirrors infra's own .githooks/pre-commit, minus the YAML/image-tag
checks that are infra-manifest-specific and don't apply here.

Verified functionally, not just wired: staged a real high-entropy fake
key and confirmed `git commit` is blocked by the hook
(RuleID=generic-api-key); staged a well-known documented-example AWS
key (AKIAIOSFODNN7EXAMPLE) first and confirmed gitleaks correctly
passes it (it's not a real leak, and gitleaks' own default rules know
that) -- worth recording since a naive fixture check would have looked
like detection was broken.

Enable per clone: git config core.hooksPath .githooks
2026-08-28 23:58:05 +02:00
mathiasandClaude Opus 5 670325e519 ci(deploy): refuse to patch a manifest with an empty IMAGE_TAG (infra#311)
CI / Lint / Test / Vet (push) Successful in 39s
CI / Mirror to GitHub (push) Successful in 4s
On 2026-08-25 a deploy job ran with an empty IMAGE_TAG, wrote
`image: localhost:5000/gitea-mcp:` into the infra repo and pushed it.
That is invalid YAML, so Flux failed the ENTIRE apps Kustomization --
every app stopped reconciling for about an hour, including three
unrelated deploys queued behind it.

This job's existing check could not catch it. It greps for
`image: <repo>:${IMAGE_TAG}` after the sed, and with IMAGE_TAG empty that
pattern matches the corrupted line itself, so the verification passes
because of the bug rather than despite it.

Asserting the input up front is the fix: an empty tag now fails the job
before anything is written, let alone committed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011PL5XcG68SvQqz3vGVbWbf
2026-08-25 17:25:25 +02:00
mathias 54237af7ad feat(brain): validate the claim-property envelope on write (brain#21) (#102)
CI / Lint / Test / Vet (push) Successful in 31s
CI / Mirror to GitHub (push) Skipped
First half of brain#21: half_life (volatile|seasonal|durable), applies
(local|homelab|universal), claim_provenance (asserted|inferred).

writeHallNote already passed unknown frontmatter keys through verbatim, so
preservation was free — a test now pins it so a refactor of the merge loop
cannot silently drop the fields. What was missing is validation, and it
fails closed: a value outside the enum is rejected (a lint skips what it
cannot parse, so the note would read as classified while nothing acts on
it); an envelope field without claim_provenance is rejected (a value a
model inferred must never be indistinguishable from one a human asserted,
or a single classifier pass launders guesses into apparent judgement
across the corpus); provenance alone is rejected too.

Validation runs before MkdirAll, so a rejected write leaves nothing behind.

The field is `applies`, not `scope`: scope: has been in use since brain#5
for glossary terms (homelab-core, project:<repo>), a disjoint value space
on the same key. The shipped schema keeps the name.

Notes with none of these fields are byte-identical to before. Bulk
backfill stays out of scope.

Schema documented in brain/CLAUDE.md via brain_edit_root, which produced
the hyperguild#97 audit record for the rewrite.

Refs brain#21, brain#19 (superseded), brain#5, hyperguild#97.

Co-authored-by: Mathias <mthbqv@gmail.com>
2026-08-22 08:58:34 +00:00
mathias b6ceccca26 docs: name brain-mcp and say where its code lives (#94) (#99)
CI / Lint / Test / Vet (push) Successful in 1m43s
CI / Mirror to GitHub (push) Skipped
A forge search for "brain-mcp" returned nothing for this repo: the description
mentioned neither the brain nor brain-mcp, and the README never used the string
or the hostname. Both appeared only in the generated CLAUDE.md/AGENTS.md. Since
#91 part 2 decided the code stays in ingestion, discoverability is answered by
naming.

Adds a "Where brain-mcp lives" section: the ingress, the package, a table
mapping what-you-are-looking-for to where-it-is (including that the brain
content is a separate repo), and why it is not standalone, with a pointer to
the ADR.

Also repoints the README's three forge links from the decommissioned
gitea.d-ma.be alias to git.d-ma.be. Dated files under docs/superpowers are
left alone — they record what was true when written.

Repo description and topics were set through the forge API, and NPM's
default-site was changed from congratulations to 404 so a retired hostname
reads as absent rather than as a moved repo.

Closes #94.

Co-authored-by: Mathias <mthbqv@gmail.com>
2026-08-22 06:10:06 +00:00
mathias 36a45068ab feat(mcp): record and alert on a governance-file rewrite (#97) (#98)
CI / Lint / Test / Vet (push) Successful in 1m23s
CI / Mirror to GitHub (push) Successful in 4s
brain_edit_root was gated but unobserved: on success it emitted a slog.Warn
and nothing else. That proves a rewrite was allowed, not that one happened —
and brain-sync commits it within 15 minutes authored `brain-sync`, so git
history cannot tell a CLAUDE.md rewrite apart from routine note traffic.

A successful brain_edit_root now emits an ntfy notification and an AuditEntry
through the same sink a capture already uses. notes.Result gains
PrevContentHash so both ends of the change are identifiable.

Path and hashes only in the notification body. Reserved at Internal, not
Confidential, because Confidential makes DegradingSink refuse to record while
loki is down rather than buffer. Observation is best-effort and never fails
the call; a failure to observe logs at ERROR.

Ordinary brain_edit on knowledge//wiki/ is deliberately not audited — git
history already carries the same before/after bytes with a diff.

Closes #97.

Co-authored-by: Mathias <mthbqv@gmail.com>
2026-08-22 06:09:53 +00:00
mathias 055af368cd feat(mcp): brain_read + brain_edit, contained to the sync source (brain#4) (#96)
CI / Lint / Test / Vet (push) Successful in 29s
CI / Mirror to GitHub (push) Skipped
brain_read returns a note verbatim; brain_edit is str_replace with an
exactly-once match, operating on the tree brain-sync commits from.

An adversarial design review before implementation changed three things:

1. Governance files are not in the model-facing tool. Brain content is
   ingested from the outside world and reaches a model's context, and that
   model would hold the editor — one poisoned page could rewrite the
   CLAUDE.md a .dispatch-allow agent later loads as instructions. And
   brain-sync merges with --strategy-option=ours, so the live tree beats a
   human's corrective push on the conflicting lines. CLAUDE.md now goes
   through brain_edit_root: de-surfaced from tools/list and gated on the
   local static principal. The scopes are disjoint — the privileged door
   has the smaller reach.

2. Reads needed the same allowlist, and brain_get already lacked it: it
   returned verbatim session transcripts under archive/ by path. A test
   proved the exposure live before the fix.

3. Canonicalise before allowlisting — knowledge/../.git/hooks/post-commit
   would otherwise reach a hook that brain-sync executes in a pod holding
   the Gitea push credentials.

Plus overlapping match counting, UTF-8 validation both ways, refusal while
git is mid-operation, flock with the read inside the lock, atomic
same-dir rename, and reindex only when frontmatter is touched.

Refs brain#4.

Co-authored-by: Mathias <mthbqv@gmail.com>
2026-08-18 18:14:05 +00:00
mathias eb9e3e69a9 fix(capture): name what escalated the classification in an I1 refusal (#91 part 1) (#95)
CI / Lint / Test / Vet (push) Successful in 31s
CI / Mirror to GitHub (push) Successful in 4s
The tickets array was never broken. Every ticket in the reported batch
targeted a client repo, and the I1 gate refuses a confidential capture
through a us-nexus origin at any length — reproduced live: length 1 and
20 both fine for an internal repo, length 1 refused for a client one.

What made it look like an array bug is that the refusal named neither the
offending target nor that a policy had said no, so with 20 tickets in
flight "the array is broken" was a reasonable conclusion. The refusal now
names the targets carrying the effective classification, and tells a
caller who declared confidential themselves exactly that.

Regression tests at length 1 and 20 pin the path that was already
correct; a third pins per-item partial success.

Closes #91 part 1.

Co-authored-by: Mathias <mthbqv@gmail.com>
2026-08-18 18:13:48 +00:00
mathias b7cf0efb9d chore: opt in to headless dispatch (dispatch#3)
CI / Lint / Test / Vet (push) Successful in 29s
CI / Mirror to GitHub (push) Successful in 4s
2026-08-18 06:05:18 +00:00
mathiasandClaude Opus 5 125c99bb46 fix(text): truncate on rune boundaries, not byte offsets (#93)
CI / Mirror to GitHub (push) Successful in 4s
CI / Lint / Test / Vet (push) Successful in 31s
Four call sites did `s[:limit] + "…"`. Both halves are byte operations:
len() counts bytes and a slice cuts bytes. On Swedish text — å ä ö are
two bytes, an em dash three — the cut lands mid-character and leaves an
orphaned lead byte, producing invalid UTF-8.

It failed loudly in exactly one place: wiki/index.md is written raw to
disk, where every strict decoder then rejects the whole file. The other
three (brain_context excerpts x2, brain_query excerpts) feed MCP
responses, where encoding/json silently substitutes U+FFFD — so those had
been quietly eating the last character of truncated excerpts, with no
error anywhere and nothing to notice.

textutil.Truncate counts runes and appends the marker only when content
was actually dropped; an ellipsis on complete text is a lie about the
content. The test is a property, not an example: it walks the limit
across every offset of a multi-byte string and asserts utf8.Valid at each
one, so a future off-by-one cannot pass by picking a lucky boundary.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
2026-08-18 07:13:15 +02:00
mathiasandClaude Opus 5 3e537bd582 feat(brain): surface an aging review queue in brain_context (#82)
CI / Lint / Test / Vet (push) Successful in 30s
CI / Mirror to GitHub (push) Successful in 4s
brain_pending existed the whole time four real items sat unreviewed from
2026-04-23 to 2026-07-15. The tool was never the gap — nothing ever
prompted anyone to call it. The system should be what remembers the queue
exists.

Option 2 from the issue: ride along with brain_context, which sessions
already call, so noticing an aging note costs nobody a decision to go
looking. No scheduled job, no ntfy wiring, nothing new to keep running.

Threshold-gated at 14 days, and silent below it — a notice on every
context call is one people learn to skip, which reproduces the original
failure with extra steps. It reports the whole queue once it speaks: the
threshold decides whether to talk, not what to count.

It never promotes, rejects or reorders. Age is a reason to ask a human,
not a verdict; an old note may be exactly right to promote once someone
looks. A test asserts the queue is untouched and that the payload carries
a hint rather than an action.

An unreadable raw/ dir degrades to silence — brain_context is a hot path
and must not fail because the review queue is broken.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
2026-08-18 00:37:48 +02:00
mathiasandClaude Opus 5 4e6fc7aeee chore(brain): de-surface brain_index from the model-facing toolset (#78)
CI / Lint / Test / Vet (push) Successful in 29s
CI / Mirror to GitHub (push) Successful in 4s
brain_index is auto-called after every write, and its own description
said so. A verb surfaced to callers that callers should never call is
cognitive load with no decision behind it — ADR-0012 rule 4.

Removed from tools/list only. The dispatch case stays, so maintenance
can still rebuild a Map of Content by hand: de-surfacing hides the verb,
not the operation. A test pins both halves, since removing the descriptor
and removing the capability look identical from the tool list.

Nothing called it directly — grepped the skills, specs and docs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
2026-08-18 00:31:06 +02:00
mathiasandClaude Opus 5 9a232ad444 fix(ingestion): retry OIDC discovery instead of exiting on cold-boot 502 (#83)
CI / Lint / Test / Vet (push) Successful in 36s
CI / Mirror to GitHub (push) Successful in 4s
On a cluster restart this pod can beat dex to serving, gets a 502 from
ingress, and exits — after connecting postgres, backfilling 794 graph
nodes and ingesting three session batches, all of which the restart
redoes. Nothing was misconfigured; it started first.

The validator now retries with bounded exponential backoff (8 attempts,
1s doubling to 30s) and is built immediately after ctx, before the
watcher and the backfill. That second part is the other half of the
issue: a genuine config error now fails in seconds rather than after all
that work.

Bounded on purpose. Retrying forever would convert a wrong issuer URL
into a process that looks alive and never serves, which is harder to
diagnose than a crash loop.

startupretry.Do is generic and tested directly: transient-then-success,
budget exhaustion surfacing the last error via errors.Is, prompt return
on context cancellation, growing-and-capped delays, and a zero-value
policy still making one attempt rather than silently succeeding.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
2026-08-18 00:21:56 +02:00
mathiasandClaude Opus 5 3da0eca9b4 fix(ingestion): decide quoting with YAML itself, not a colon heuristic (#90)
CI / Lint / Test / Vet (push) Successful in 29s
CI / Mirror to GitHub (push) Successful in 4s
`title: "Foo" — bar` opens with a quote and is not quoted; the previous
prefix check read it as already-quoted and left two notes broken. Probing
the value as the right-hand side of a mapping answers the real question —
does this load as written — and leaves anything that already parses alone,
whatever it parses into.

Repairs the last two quotable files in mathias/brain. Only wiki/index.md
remains unparseable, for an unrelated reason: a truncated UTF-8 sequence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
2026-08-18 00:10:11 +02:00
mathiasandClaude Opus 5 7d1213451c chore: re-sync context adapters
CI / Lint / Test / Vet (push) Successful in 29s
CI / Mirror to GitHub (push) Successful in 3s
Root .context/AGENT.md gained the CAD sprint patterns section; the derived
adapters had not been regenerated since. Pure `task context:sync` output.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
2026-08-18 00:08:07 +02:00
mathiasandClaude Opus 5 7c9293f4ed feat(ingestion): lint and repair unparseable note frontmatter (#90)
Hand-authored and promoted notes skip pipeline.BuildPages, so nothing
quotes their scalars. A title carrying an unescaped ": " yields a block
yaml.Unmarshal rejects, and every consumer that reads frontmatter drops
the note silently — 18 files repo-wide today, up from the 8 in #90.

frontmatter.Lint reports it; frontmatter.Fix quotes only what quoting can
rescue and refuses to return a document that still fails to parse, so an
unfixable file comes back byte-identical rather than half-repaired. Scan
walks a tree and reports an unreadable file as a finding instead of
aborting, so one poisoned note cannot hide the rest.

Sequence items are deliberately left alone: "- foo: bar" is valid YAML
that silently becomes a map, which is a type corruption rather than a
parse failure, and repairing it would rewrite notes that currently load.

    task lint:frontmatter          # report
    task lint:frontmatter FIX=1    # repair in place

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeAp5LTscnz2W7Ubt4eJ6m
2026-08-18 00:07:27 +02:00
mathiasandClaude Sonnet 5 6e0155a2ab feat(security): add gitleaks pre-commit scan, wire into task check (infra#39)
CI / Lint / Test / Vet (push) Successful in 1m41s
CI / Mirror to GitHub (push) Successful in 5s
Ported from mathias/infra's .gitleaks.toml (2026-08-04) -- same
homelab bearer/MCP-token rules, plus a BRAIN_PG_DSN pattern for the
hyperguild#20 leak class. Triaged 10 initial findings: 9 were
claudewatcher's own scrubber test fixtures (deliberately fake), 1 was
gitleaks matching the literal placeholder word "REDACTED" in a plan
doc. Both allowlisted with rationale, clean scan confirmed before
wiring into `task check`.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016FQX4RwxyxuQY4yf4XaACL
2026-08-04 08:32:42 +02:00
mathiasandClaude Sonnet 5 1cea2c9f78 feat(capture): remove summary->ai-sessions write path
CI / Lint / Test / Vet (push) Successful in 29s
CI / Mirror to GitHub (push) Successful in 4s
ai-sessions#13: capture's `summary` field wrote a frontmatter shape
(title/harness/fidelity/captured_at/repos_touched) that ai-sessions'
own extract/audit pipeline can't parse -- silently invisible to that
repo's own audit tooling, and bypassing its redaction + Stage2
completeness gates by construction. Only 5 files ever landed this way
over 3 weeks; the summary capability's whole value (speed) fights
ai-sessions' whole value (redacted, audited, complete), so kill it
rather than build a second parse branch.

capture now persists insights -> brain and action items -> Gitea
tickets only. Removes Summary/SummaryResult/SummaryWriter and all
wiring (service, REST body, MCP tool schema); close-session updated to
stop assembling a summary payload. specs/capture-*.md kept as historical
record with a superseded note -- the feature shipped and is documented,
just no longer current behaviour.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WPdSHbp9Utb2hPFm9wDG59
2026-07-27 23:17:56 +02:00
mathiasandClaude Sonnet 5 9dcd60931a chore: retrigger CI after gitea-runner RBAC fix (round 6 of the CI/CD outage chain)
CI / Lint / Test / Vet (push) Successful in 30s
CI / Mirror to GitHub (push) Has been skipped
cd.yml's build+push+ssh chain fully worked last round (infra 707a3c0
landed via CD Bot). Final gap was RBAC: kubectl (new in baked-v6) had
no ServiceAccount permissions. Fixed in infra@949b333 (dedicated SA +
scoped Role/RoleBinding for flux-system + supervisor, verified live
via kubectl auth can-i before this retrigger). No code change here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 22:29:57 +02:00
mathiasandClaude Sonnet 5 1fac90ed2a fix(cd): pass git.d-ma.be SSH override as -o flags, not an appended ~/.ssh/config line
CI / Lint / Test / Vet (push) Successful in 29s
CI / Mirror to GitHub (push) Successful in 4s
cd.yml's own $HOME (/data) is a PVC that persists across job runs on
this runner. The previous fix (cb9c251) appended the corrected
HostName to ~/.ssh/config, but a stale 127.0.0.1:30022 entry from an
earlier failed run was already there — ssh_config is first-match-wins,
so the stale entry silently shadowed the fix and the very next run
failed identically. Cleaned the persisted stale entry on the runner
directly; this commit removes the append entirely so the step no
longer depends on file state surviving (or not surviving) between
runs — CLI -o overrides always win. Verified end-to-end against the
real infra repo from inside the runner pod (git ls-remote, correctly
reached gitea and got a clean auth rejection, not a connection error).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 22:16:00 +02:00
mathiasandClaude Sonnet 5 cb9c2513a4 fix(cd): use in-cluster DNS for git.d-ma.be SSH, not 127.0.0.1:30022
CI / Lint / Test / Vet (push) Successful in 28s
CI / Mirror to GitHub (push) Successful in 4s
act_runner moved from koala's bare host network into a k3s pod
(2026-07-26); loopback inside that pod's own network namespace never
reaches the host, so the git.d-ma.be SSH override (used to push the
infra repo update) failed with "Connection refused". Verified
TCP+SSH-handshake reachability to gitea-ssh-nodeport.gitea.svc.cluster.local:22
from inside the actual runner pod before committing this.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 22:09:52 +02:00
mathiasandClaude Sonnet 5 3617a6c386 chore: retrigger CI after gitea-runner baked-v6 (ssh/skopeo/buildctl/kubectl)
CI / Lint / Test / Vet (push) Successful in 28s
CI / Mirror to GitHub (push) Successful in 4s
check job passed clean on baked-v5; mirror job then exposed missing
ssh/ssh-keyscan. Fixed in infra@707a3c0, which also preemptively adds
skopeo/buildctl/kubectl for cd.yml's deploy job. No code change here —
retriggering for the full CI->CD end-to-end proof.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 22:03:07 +02:00
mathiasandClaude Sonnet 5 1938170131 chore: retrigger CI after gitea-runner gcc/libc6-dev fix (baked-v5)
CI / Lint / Test / Vet (push) Successful in 1m12s
CI / Mirror to GitHub (push) Failing after 1s
Runner was missing gcc/libc6-dev too (-race requires cgo). Added in
infra@0f88aea. No code change here — retriggering CI.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 21:55:20 +02:00
mathiasandClaude Sonnet 5 b34717e6b8 chore: retrigger CI after gitea-runner image fix (infra#task-baked-v4)
CI / Lint / Test / Vet (push) Failing after 34s
CI / Mirror to GitHub (push) Has been skipped
Runner was missing task (taskfile.dev), added in infra@686c90f. No
code change here — retriggering CI to confirm the fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 21:49:33 +02:00
mathiasandClaude Sonnet 5 d8d7e9a307 chore(context): re-sync derived adapters (Python fallback stack, error-handling note)
CI / Lint / Test / Vet (push) Failing after 1s
CI / Mirror to GitHub (push) Has been skipped
Canonical ~/dev/.context/AGENT.md drifted (Python fallback stack table
row, Python error-handling convention) without a hyperguild commit
re-running task context:sync since — task check's drift gate has been
failing CI on every push since the sprint-1 commits landed, blocking
CD entirely (unrelated to those commits' own content: lint/test/vet
all pass clean, including -race).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Roq1ajWKR5f1hG5Df9wC6A
2026-07-27 15:18:38 +02:00
182 changed files with 15729 additions and 1688 deletions
+5 -1
View File
@@ -64,7 +64,11 @@ the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients.
for shell scripts and non-MCP clients, **with the same bearer token as `/mcp`**
(`BRAIN_MCP_TOKEN`; hyperguild#149). It is not served on the public host
(infra#517): use the tailnet (`koala:8443`) or in-cluster
`ingestion.supervisor.svc:3300`. `/ingest-path` and `brain_ingest` `path`
accept only paths inside `brain/raw/`.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
+23
View File
@@ -7,6 +7,29 @@
"supervisor": {
"url": "http://localhost:3200/mcp",
"description": "Hyperguild SDO — skill workers (tdd, retrospective), brain tools (brain_query, brain_write), session logging, tier detection"
},
"brain": {
"type": "http",
"url": "https://koala.tail30472e.ts.net:8443/mcp",
"headers": {
"Authorization": "Bearer ${BRAIN_MCP_TOKEN}"
}
},
"gitea": {
"_comment": "gitea-mcp authenticates by PASS-THROUGH: the bearer is a Gitea PAT it validates against Gitea and then acts as. GITEA_MCP_TOKEN -- which this entry used until 2026-09-08 -- 401s against it, so every harness synced from this file had a dead gitea server and no signal saying so. Probed 2026-09-08: GITEA_MCP_TOKEN 401, DMABE_GITEA_API_TOKEN 200, on both the public and the Serve URL.",
"type": "http",
"url": "https://koala.tail30472e.ts.net:8446/mcp",
"headers": {
"Authorization": "Bearer ${DMABE_GITEA_API_TOKEN}"
}
},
"infra": {
"_comment": "The bearer is belt-and-braces: this endpoint answers 200 with or without it, since Serve already limits reach to the tailnet. Sending it anyway means the day the server starts enforcing is not the day every harness breaks.",
"type": "http",
"url": "https://koala.tail30472e.ts.net:8444/mcp",
"headers": {
"Authorization": "Bearer ${INFRA_MCP_TOKEN}"
}
}
}
}
+175 -26
View File
@@ -53,14 +53,51 @@ These rules apply to every task across every project, regardless of harness.
logical change (one tool, one fix, one test) with passing tests. Main is always
deployable. Never create long-lived feature branches.
**Exception — parallel agents on same repo:** If another agent is known to be
actively working on the same repo simultaneously, create a short-lived branch
(`agent/<description>`), finish the task, and merge to main within the same
session. Do not leave agent branches open between sessions.
**The PR trigger is RISK, not concurrency.** `risk/low` work commits straight to
main with CI as the gate — that is most work and is unchanged. `risk/medium` and
above opens a PR, even solo, even with nobody waiting to review it. This mirrors
the tiered merge model in `assessor-loop`
`method/transfer-to-a-regulated-organisation.md` §4.1, so the estate practises
what it sells instead of merging everything on green and claiming otherwise.
**Exception — external contributor or client four-eyes requirement:** Use
PR flow only when a human reviewer outside the project is required. Document
the reason in PROJECT.md.
**Concurrency is NOT a reason to branch.** Measured 2026-08-29 across three real
collisions between parallel sessions: two were not branching problems at all — a
generated artifact written to the same path by both (branching turns that into a
merge conflict on a 33k-line JSON), and uncommitted work in a shared checkout
(a per-session worktree fixes it) — and the third was an ordinary non-fast-forward
push resolved by rebase in seconds. Use a worktree per session on shared repos and
timestamp generated artifacts. Do not reach for a branch.
Branch+PR flow carries documented costs on solo repos: merging auto-closes gated
issues, review comments live on a thread separate from the issue's, and a
merged-and-deleted branch cannot be amended (brain:
`gitea-pr-merge-autoclose-and-review-comment-footguns`). Those are the price of
the `risk/medium`+ lane — paid deliberately, not incurred by accident.
**Exception — external contributor or client four-eyes requirement:** Use PR flow
when a human reviewer outside the project is required. Document the reason in
PROJECT.md.
**Exception — a GitOps repo where a branch cannot be reconciled.** In a repo whose
deployment watches one branch (`mathias/infra`: Flux watches `main`), a branch is
invisible to the thing that applies it. A `risk/medium` change to `alerts.yaml` or
`litellm-config.yaml` can be linted on a branch but never *exercised* there, so a
PR would buy a review of a state that was never deployed.
There, `risk/medium`+ work stays on `main` and instead requires a **recorded
adversarial review pass before the push** — `skills:pr-gatekeeper` or equivalent,
reviewing the staged diff. **The reviewer's verdict goes in the issue, not a PR
thread.** No verdict recorded means the lane was not used, and saying "the gate was
green" is not the same claim.
Be honest about what this is: a recorded *independent* pass, not four-eyes in the
sense `assessor-loop` §4.1 means for a regulated organisation, because the reviewer
is an agent. It is more than the repo had and less than the model it mirrors; do not
describe it as the stronger property. (infra#470, decided 2026-09-29.)
This lane depends on the `risk/` label being right. A label that triggers a heavier
lane is load-bearing, not cosmetic — if work that belongs in it is unlabelled, the
lane silently does not apply.
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
the code is not the end of the task; capturing it is. Run this unprompted:
@@ -78,6 +115,63 @@ These rules apply to every task across every project, regardless of harness.
- Surface the brain entries and issue numbers in the closing summary so the
trail is auditable.
**Chat is not a durable store. Never rely on Mathias remembering anything.**
Every gap, caveat, risk, open question and next step must land in one of four
places before the turn ends: **the brain**, **a repo** (code, docs, ADR),
**a tracker issue**, or **a plan file**. A finding stated only in conversation is
lost the moment the session closes — and this estate has repeatedly proved that a
remembered convention is an unreliable control.
This applies hardest to what is tempting to leave as a closing remark: "worth
knowing X isn't enforced yet", "this is a convention, not a gate", "we should
revisit Y". Those are exactly the items that need a locator. If it is worth
saying in a summary, it is worth writing where it can be found again without you.
7. **Finish what's closable — only involve me for a genuine call.** If a
tracker issue, PR, or checklist item is demonstrably resolved — the fix
landed, the state changed, you verified it live — close/merge/check it off
in the same turn. Don't leave it open "for the record" or park it waiting
on me to confirm something that already happened; that's bookkeeping, not
a decision. Reserve asking me (`AskUserQuestion`, a `DECISION NEEDED`
comment, or otherwise pausing) for when it's genuinely mine to call:
irreversible or prod/host-impacting actions, a real judgment call with no
clearly-better option, ambiguous intent, or anything secret-writes/
destructive-git already gate. Before claiming something is closable,
verify it live (re-check the issue, re-run the command) rather than
trusting your own earlier summary (brain: `filed-issue-not-same-as-fixed`).
8. **Control code needs an adversary, not a checklist.** Code whose job is to
refuse, deny, gate or contain is not adequately verified by a green gate.
Rule 5 makes CI the quality gate and that holds for ordinary work — but a
passing suite only asserts what its author already thought of, and for a
control the author is precisely the person whose imagination just failed.
Measured on `dispatch#75`: five review rounds, five real escapes verified by
execution, and **CI was green on every one of them** while `git rm
././.dispatch-allow` deleted the file the control existed to protect. So
route this work through a review with authority to block, instructed to
verify by execution rather than by reading the implementation. Expect to be
blocked more than once; that is the control working, not the reviewer being
difficult. (brain: `a-green-gate-is-not-evidence-for-control-code`; the
practices live in the `pr-gatekeeper` agent.)
9. **A test's name must not claim more than its assertions prove.** Name a test
after the mechanism it observes, not the goal it serves.
`DoesNotInheritEnv` is honest; `IsContained` is a claim the test cannot
support. This is not pedantry about naming — when name and assertion
diverge, the name wins in the reader's head and the gap becomes invisible,
which is what lets the defect survive review. Found three times in one PR,
twice in tests written to fix the previous instance of it; each passed while
the thing it was named for was exploitable. If you cannot name the mechanism,
you do not yet know what the test observes.
## CAD sprint patterns
Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`):
- **Inception Sprint** — birth a new repo. Oath-gated close; tag only once every clause holds. Brain: `.../inception-sprint-and-oath`.
- **Sweep Sprint** — clear an existing backlog. Parallel read-only research per issue → sequential one-issue-per-commit execution. Brain: `.../sweep-sprint-and-oath`.
Shared: decisions that are yours, not the agent's, surface as `DECISION NEEDED` (question/options/recommendation/blocks) — ask live if you're present, else comment and wait, never act past it. Live-breaking regressions get `git revert`, never forward-fixed; host-network work skips even read-only research (solo, synchronous, you present). Rule 6's closing ritual only fires once every Oath clause genuinely holds — blocked = descoped-and-tracked, never marked done.
## Default stack
| Layer | Default | Fallback | Last resort |
@@ -88,7 +182,7 @@ These rules apply to every task across every project, regardless of harness.
| Containers | Docker Compose (dev), k3s (prod) | — | — |
| DB | PostgreSQL + sqlc | SQLite | — |
| Search | pgvector (vector), BM25 | Qdrant (when >1M vectors or hybrid retrieval) | — |
| Logging | slog (structured) | — | — |
| Logging | slog (structured) | stdlib `logging` w/ structured `extra=` (or structlog) | — |
| Testing | Table-driven, testify | — | — |
| Agents (Go) | google.golang.org/adk + pkg/litellm adapter | — | — |
@@ -97,7 +191,12 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
## Code conventions
- **Go style**: golines, gofumpt, golangci-lint
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return
- **Python style** (fallback language): ruff (format+lint, one tool), mypy --strict (non-negotiable,
matches Go's static typing discipline), pytest + pytest-cov (table-driven via
`@pytest.mark.parametrize`), uv (venv+deps+lock, one tool), pydantic-settings (typed env-var config
— same principle as Go's typed structs), src-layout + `pyproject.toml` only (no `setup.py`)
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return.
Python: `raise X from e` (exception chaining, same principle) — never bare `except`, never silent `pass`
- **Naming**: stdlib conventions, no stuttering
- **Architecture**: prefer stdlib over frameworks, constructor injection, env-var config parsed into typed structs
- **Git**: conventional commits (`feat:`, `fix:`, `chore:`), commit directly to main,
@@ -135,10 +234,10 @@ Three machines on Tailscale:
|---------|------|-----------|
| koala | GPU inference, heavy compute | RTX 5070, runs k3s + llama-swap + shared postgres18/pgvector |
| iguana | Services, builds | M2 Ultra Mac |
| flamingo | Daily driver, edge | Mac mini, ~/dev is here |
| flamingo | Daily driver, edge | Apple Silicon Mac, ~/dev is here |
- **Model routing**: LiteLLM in front of llama-swap (local) + cloud APIs (when permitted)
- **Orchestration**: k3s cluster across all three machines
- **Orchestration**: k3s on koala only — single control-plane node, no workers (verified 2026-08-30). iguana and flamingo are on the tailnet but are not cluster members
- **Networking**: Tailscale mesh
## Project landscape
@@ -205,10 +304,16 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
|---------|-------|-------|
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` |
| **Anything HTTP-only (curl, scripts)** | Tailnet only: `POST https://koala.tail30472e.ts.net:8443/query` with `{"query":"..."}`, sending `BRAIN_MCP_TOKEN` (not yet enforced on REST, hyperguild#149). The public `brain-mcp.d-ma.be` serves `/mcp` only (infra#517). | `POST .../write` with `{"content":"...","wing":"...","hall":"..."}` |
| **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files |
- **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`.
- **Scoping — the brain holds no confidential client content.** Nothing on the read path
filters by client: every token holder can read every wing, and `wing` on `brain_query`
is an optional argument, not an access control. Client confidentiality is enforced only
at write time (`capture`'s I1 gate refuses internal/confidential). So never `brain_write`
client material; generalise it first or keep it in the client's own repo. (hyperguild#144,
decided 2026-10-01. This line previously claimed client projects were filtered; they
were not.)
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
fallback. Both are configurable in the `supervisor/ingestion-deployment.yaml`
on the koala k3s cluster; don't hardcode local-only model names into the
@@ -234,10 +339,11 @@ When working on a project tagged with a client name:
## Harness-agnostic principles
This context is designed to work with any AI coding tool:
- Claude Code, Cursor, Aider, Open WebUI, Charmbracelet Mods/Crush
- Pi Coding Agent, Mistral Vibe, Antigravity
- Any tool that accepts a system prompt or reads a markdown context file
Supported harnesses (2026-09-01) — Cursor and Aider were dropped; do not
re-add adapters for them:
- Claude Code, Antigravity, OpenCode, Mistral Vibe, Crush, Pi Coding Agent
- Open WebUI, Charmbracelet Mods, or any tool that takes a system prompt string
- Any other tool that reads `AGENTS.md`, which is the de facto convention
The canonical source is always `.context/AGENT.md` (root) and `.context/PROJECT.md` (per-project).
Derived files are committed (see *How context propagates* below) so a `git pull` on any host yields full agent context with no setup.
@@ -249,26 +355,65 @@ Canonical sources of truth:
- Project: `<repo>/.context/PROJECT.md` (per-repo)
Derived files (committed, regenerated by `task context:sync`):
- `CLAUDE.md`, `AGENTS.md`, `.cursorrules`, `.aider.conventions.md`,
`.context/system-prompt.txt`
- `CLAUDE.md`, `AGENTS.md`, `.context/system-prompt.txt`
Repos still carrying `.cursorrules`, `.aider.conventions.md` or
`.aider.conf.yml` are stale: nothing regenerates them. Delete them the next time
you touch the repo, along with their entries in that repo's `task check`
assertion.
Workflow:
1. Edit a canonical file. Run `task context:sync`. Commit canonical and
derived together. Push.
2. On any other host, `git pull` brings both. Claude Code (tree-walking)
uses `CLAUDE.md`; Crush / Pi / Antigravity (cwd-only) use `AGENTS.md`;
Cursor uses `.cursorrules`; Aider uses `.aider.conventions.md`.
2. On any other host, `git pull` brings both. Which file a harness reads does
NOT split cleanly by lookup strategy, so do not reason from the strategy:
- `CLAUDE.md` — Claude Code (walks up the tree)
- `AGENTS.md` — Crush, Pi, Antigravity (cwd-only), OpenCode (project root,
global at `~/.config/opencode/AGENTS.md`), and Mistral Vibe, which **walks
up** from cwd and takes the first `AGENTS.md` it finds
(user-level `~/.vibe/AGENTS.md`)
- `.context/system-prompt.txt` — Open WebUI, Mods, raw API calls; not read by
any harness, it is for pasting
Vibe tree-walks for `AGENTS.md`, so "tree-walkers use CLAUDE.md" is false.
Each repo's `AGENTS.md` embeds the canonical and is the first one found from
inside that repo, so this works — but by construction, not by luck.
3. `task check` runs `context:sync` then asserts `git status --porcelain`
is empty over the derived files (catches both modified-tracked drift
and missing-untracked adapters). A drift fails the check with a
message telling you to stage the regenerated files.
Behavior rules in this file and per-project rules in `PROJECT.md` apply
unconditionally on every host, every harness.
Behavior rules in this file and per-project rules in `PROJECT.md` apply on every
host and every harness.
**Where a project rule contradicts a rule here, the project rule governs — and it
must state its reason.** Both halves matter: the project file wins because it knows
something this file cannot (infra's Flux constraint is real and not visible from
here), and it must say why because a bare override is indistinguishable from drift.
This replaces the previous wording, which said rules here and in `PROJECT.md` both
apply *"unconditionally"*. That was not a tiebreak but a contradiction, and it went
unnoticed until rule 5 and infra's "never create feature branches" had to be obeyed
at the same time (infra#470). A standard can hold two individually sensible clauses
that are jointly unsatisfiable, and survive for as long as everyone reads it
charitably — the same failure the Inception Sprint Oath's G6 had.
## Engineering Skills
Shared engineering skills live in the **`mathias/skills`** repo (`git.d-ma.be/mathias/skills`). Clone it to `~/dev/skills/` and run `SKILLS_CHECKOUT_DIR="$PWD" bash install.sh` there to wire every skill into your harnesses (Claude Code, Crush, Antigravity, Mistral Vibe) as native, on-demand skills. (Use `install.sh`, not `task install` — the latter is currently broken, skills#7.) Load at task start — not "on demand" but on schedule, before writing code. Browse `~/dev/skills/SKILLS_INDEX.md` for the full list.
Shared engineering skills live in the **`mathias/skills`** repo (`git.d-ma.be/mathias/skills`). Load at task start — not "on demand" but on schedule, before writing code. Browse `SKILLS_INDEX.md` for the full list.
**Claude Code** installs the repo as a plugin from the marketplace the same repo hosts. That path also carries the `secret-guard` and `pre-task-ritual` hooks and the brain/gitea/infra MCP servers, which symlinks cannot:
```
/plugin marketplace add https://git.d-ma.be/mathias/skills.git
/plugin install skills@mathias
```
Skills are **namespaced** under Claude Code: load `tdd` as `skills:tdd`. The trigger table below uses bare names — prefix them.
**Crush, Pi and OpenCode** have no plugin system and are wired by `install.sh` as one directory symlink per harness via `~/.agents/skills`. Do not clone the checkout into `~/dev`: that folder is iCloud-shared between the Macs and spans two path roots (`/Users/mathias` vs `/home/mathias` on koala), so a checkout there is a git working tree with two concurrent writers, and any absolute symlink stored in it is broken on one side by construction (skills#23). Leave `SKILLS_CHECKOUT_DIR` at its host-local default.
Check a host with `task doctor` in the skills repo; it also runs automatically at session start via the plugin.
**Skill trigger table — load before starting, not after getting stuck:**
@@ -348,7 +493,11 @@ the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients.
for shell scripts and non-MCP clients, **with the same bearer token as `/mcp`**
(`BRAIN_MCP_TOKEN`; hyperguild#149). It is not served on the public host
(infra#517): use the tailnet (`koala:8443`) or in-cluster
`ingestion.supervisor.svc:3300`. `/ingest-path` and `brain_ingest` `path`
accept only paths inside `brain/raw/`.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
+2
View File
@@ -0,0 +1,2 @@
# Presence of this file marks this repo as opt-in for headless dispatch.
# See dispatch#3.
-1
View File
@@ -1,7 +1,6 @@
# Supervisor MCP server
SUPERVISOR_PORT=3200
SUPERVISOR_CONFIG_DIR=./config/supervisor
SUPERVISOR_MODELS_FILE=./config/models.yaml
# LiteLLM gateway (iguana)
LITELLM_BASE_URL=http://iguana:4000
+36 -8
View File
@@ -45,12 +45,32 @@ jobs:
set -e
trap 'rm -rf /tmp/infra-update; rm -f ~/.ssh/infra_deploy_key' EXIT
IMAGE_TAG="${{ github.sha }}"
# infra#311: github.sha is never empty in practice, but the manifest
# patch below is the same shape that took Flux down estate-wide on
# 2026-08-25 -- an empty tag yields `image: <repo>:`, which is invalid
# YAML and fails the ENTIRE Kustomization, not just this app. The
# assert costs nothing and removes the class.
[ -n "${IMAGE_TAG:-}" ] || { echo "IMAGE_TAG is empty - refusing to patch"; exit 1; }
mkdir -p ~/.ssh
echo "${{ secrets.INFRA_DEPLOY_KEY }}" > ~/.ssh/infra_deploy_key
chmod 600 ~/.ssh/infra_deploy_key
printf 'Host git.d-ma.be\n HostName 127.0.0.1\n Port 30022\n StrictHostKeyChecking no\n' >> ~/.ssh/config
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \
# In-cluster DNS to gitea's SSH NodePort service, not 127.0.0.1:30022
# (that only worked when act_runner ran on koala's bare host network;
# from inside the containerized runner's own pod netns, loopback
# never reaches the host — "Connection refused", found 2026-07-27).
#
# Pass as -o overrides on the ssh invocation itself, NOT appended to
# ~/.ssh/config: $HOME (/data) is a PVC that persists across job
# runs on this runner (same "workspace not ephemeral" class as
# brain: act-runner-host-executor-tmp-persists), so an appended
# line here would pile up duplicate `Host git.d-ma.be` blocks
# across every run — ssh_config is first-match-wins, so a stale
# entry from an earlier failed run would silently shadow this
# fix forever (exactly what happened once already: this fix's
# own first attempt got appended AFTER an already-stale entry
# and lost). CLI -o options always win regardless of file state,
# so this step is safe to re-run any number of times.
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes -o HostName=gitea-ssh-nodeport.gitea.svc.cluster.local -o Port=22 -o StrictHostKeyChecking=no" \
git clone "${INFRA_REPO}" /tmp/infra-update
cd /tmp/infra-update
@@ -61,11 +81,19 @@ jobs:
git config user.email "cd-bot@d-ma.be"
git config user.name "CD Bot"
git add "k3s/apps/supervisor/ingestion-deployment.yaml"
git commit -m "chore(deploy): ingestion → ${IMAGE_TAG}"
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \
git push
echo "Infra repo updated: ingestion → ${IMAGE_TAG}"
# Idempotent: `git commit` exits 1 on an empty index, so a run that
# finds the bump already applied failed the whole job and rang the
# CI-red alarm for a deploy that was fine (hyperguild#118). Two
# sources of a no-op run: a manual re-run, and the tag-push CI run
# that used to shadow the branch-push one for the same sha.
if git diff --cached --quiet; then
echo "Infra already at ${IMAGE_TAG} — nothing to commit"
else
git commit -m "chore(deploy): ingestion → ${IMAGE_TAG}"
GIT_SSH_COMMAND="ssh -i ~/.ssh/infra_deploy_key -o IdentitiesOnly=yes" \
git push
echo "Infra repo updated: ingestion → ${IMAGE_TAG}"
fi
- name: Trigger Flux reconcile (immediate)
run: |
+6 -1
View File
@@ -2,8 +2,13 @@ name: CI
"on":
push:
# Branch only. A `v*` tag here points at a commit main already tested, so
# the tag push re-ran the whole gate and — because cd.yml keys off "CI
# succeeded" — triggered a SECOND full image build and deploy for the same
# sha. That second deploy then died on "nothing to commit" (hyperguild#118).
# Nothing consumes a tag-triggered run: there is no release workflow, and
# cd.yml reads github.sha, not the tag.
branches: [main]
tags: ["v*"]
pull_request:
branches: [main]
+34
View File
@@ -0,0 +1,34 @@
#!/usr/bin/env bash
# Pre-commit secret scan for the hyperguild repo.
#
# task check -> security:gitleaks enforces this in CI, but CI is after the
# push. This catches a staged secret before it ever leaves the machine --
# the infra#39 rotation pass found exactly this class of leak (a printed
# credential, committed, archived into git) and there was no local gate to
# have caught it. Mirrors infra's own .githooks/pre-commit (infra#326).
#
# Enable once per clone: git config core.hooksPath .githooks
# Bypass if absolutely needed: git commit --no-verify
set -euo pipefail
GITLEAKS="$(command -v gitleaks || true)"
if [[ -z "$GITLEAKS" && -x "$(go env GOPATH 2>/dev/null)/bin/gitleaks" ]]; then
GITLEAKS="$(go env GOPATH)/bin/gitleaks"
fi
if [[ -z "$GITLEAKS" ]]; then
echo "⚠ gitleaks not installed — skipping local secret scan (CI still enforces it)."
echo " Install: go install github.com/zricethezav/gitleaks/v8@latest"
exit 0
fi
echo "→ Scanning staged changes for secrets..."
if ! "$GITLEAKS" protect --staged --redact --config .gitleaks.toml >/dev/null 2>&1; then
echo " ✗ gitleaks found a secret in the staged changes."
echo " Review with: gitleaks protect --staged --redact --config .gitleaks.toml -v"
echo " If it is a false positive, allowlist it in .gitleaks.toml or add a"
echo " '# gitleaks:allow' comment. Do NOT --no-verify a real secret."
exit 1
fi
echo " ✓ no staged secrets"
+71
View File
@@ -0,0 +1,71 @@
# gitleaks config for the hyperguild repo (infra#39 — leak prevention pass,
# Phase 3 checklist item: "gitleaks pre-commit hook in infra AND hyperguild").
#
# Ported from mathias/infra's .gitleaks.toml (2026-08-04) — same homelab
# token-shape rules, minus the SOPS/searxng allowlists infra needed (this
# repo doesn't use SOPS).
title = "hyperguild gitleaks config"
[extend]
useDefault = true
# --- Homelab-specific rules -------------------------------------------------
[[rules]]
id = "homelab-static-bearer"
description = "Homelab MCP/LLM static bearer or API key assigned a long literal value"
regex = '''(?i)\b(DMABE_[A-Z0-9_]+|[A-Z0-9_]*MCP_TOKEN|ROUTING_MCP_TOKEN|INFRA_MCP_TOKEN|BRAIN_MCP_TOKEN|GITEA_MCP_TOKEN|LITELLM_MASTER_KEY|LITELLM_SALT_KEY|DMABE_LLMAPI_KEY|BRAIN_PG_DSN)\s*[:=]\s*['"]?([A-Za-z0-9/_+.\-]{16,})['"]?'''
keywords = ["dmabe_", "mcp_token", "litellm_master_key", "litellm_salt_key", "llmapi_key", "brain_pg_dsn"]
[[rules.allowlists]]
description = "Env indirection is not a literal secret"
regexes = [
'''os\.environ''',
'''valueFrom''',
'''secretKeyRef''',
'''\$\{?[A-Za-z_][A-Za-z0-9_]*\}?''',
'''REDACTED''',
'''<[A-Z_]+>''',
]
[[rules]]
id = "homelab-authorization-bearer"
description = "Hardcoded Authorization: Bearer header"
regex = '''(?i)authorization['"]?\s*[:=]\s*['"]?bearer\s+([A-Za-z0-9/_+.\-=]{16,})'''
keywords = ["authorization", "bearer"]
[[rules.allowlists]]
description = "Env indirection is not a literal secret"
regexes = [
'''\$\{?[A-Za-z_][A-Za-z0-9_]*\}?''',
'''os\.environ''',
'''REDACTED''',
'''<[A-Z_]+>''',
]
# --- Global allowlist: claudewatcher's own scrubber test fixtures ------------
# ingestion/internal/claudewatcher/{scrubber,watcher}_test.go deliberately
# contain fake secret-shaped literals to test that the scrubber detects and
# redacts them. Verified 2026-08-04: all 9 findings here are test fixtures
# (github-pat, jwt, generic-api-key, homelab-authorization-bearer rules) plus
# 1 doc finding that was gitleaks matching the literal placeholder word
# "REDACTED" in a plan doc — not a real secret in either case.
[[allowlists]]
description = "claudewatcher scrubber test fixtures — deliberately fake secrets"
paths = [
'''ingestion/internal/claudewatcher/scrubber_test\.go$''',
'''ingestion/internal/claudewatcher/watcher_test\.go$''',
]
# infra#344: the allowlist that used to sit here is gone, and so is the reason
# for it. It suppressed docs/superpowers/plans/2026-04-22-phase4-attempt-wiring.md
# on the stated grounds that "extracted Secret == 'REDACTED' exactly ... the
# placeholder text itself, not a real credential" (verified 2026-08-04). That was
# not what the file contained: line 1054 carried a real 40-char hex Gitea PAT in
# an `Authorization: token ...` header. Confirmed dead by side-effect — GET
# /api/v1/user with it returns 401 — and redacted in that file, so there is now
# nothing to allowlist.
#
# The `paths` entry also made the block far broader than its description: gitleaks
# ORs global allowlist clauses and ignores `condition = "AND"`, so the whole file
# was immune, not just lines matching /REDACTED/. A live token added to that doc
# would have been suppressed on the strength of a note about a different value.
+33
View File
@@ -0,0 +1,33 @@
# Historical gitleaks findings, dispositioned (infra#326 / infra#317).
#
# WHY THIS FILE AND NOT .gitleaks.toml:
# This is a finding in git HISTORY, surfaced by the full-history audit
# (`task security:gitleaks:history`). The working-tree scan that gates commits
# (`task security:gitleaks`, the pre-commit hook, CI) is clean. Suppressing by
# PATH in .gitleaks.toml would also blind the working-tree scan for that file
# forever; a fingerprint pins one finding in one commit, so a NEW secret in the
# same file still fires.
#
# History is not rewritten here, so the remediation for a real leak is
# ROTATION -- making the committed value worthless -- never scrubbing.
#
# Dispositioned 2026-08-29. Re-check with: task security:gitleaks:history
# --- Real credential, ALREADY DEAD -------------------------------------------
# A Gitea PAT pasted into a CD-watching command in a planning doc
# (`Authorization: token <40-hex>`, last_eight 3105a670).
#
# Verified dead 2026-08-29: authenticating against
# GET https://git.d-ma.be/api/v1/user with this exact value returns 401, so the
# committed token no longer grants anything. It was almost certainly invalidated
# by the early-August rotation of the `mathias` PATs (the same rotation whose
# fan-out caused infra#260, where CI copies of the old value kept failing).
#
# No rotation action outstanding. Kept as a fingerprint rather than removed from
# history because history cannot be rewritten here.
#
# The generalisable lesson is recorded separately: a credential printed into a
# doc/transcript is permanent, so the only remedy is rotating it -- which in
# this case had already happened for unrelated reasons, by luck rather than by
# the leak being noticed.
c9310b1079478bf4430b0a2dfb25445d7ccc9b59:docs/superpowers/plans/2026-04-22-phase4-attempt-wiring.md:curl-auth-header:1053
+175 -26
View File
@@ -48,14 +48,51 @@ These rules apply to every task across every project, regardless of harness.
logical change (one tool, one fix, one test) with passing tests. Main is always
deployable. Never create long-lived feature branches.
**Exception — parallel agents on same repo:** If another agent is known to be
actively working on the same repo simultaneously, create a short-lived branch
(`agent/<description>`), finish the task, and merge to main within the same
session. Do not leave agent branches open between sessions.
**The PR trigger is RISK, not concurrency.** `risk/low` work commits straight to
main with CI as the gate — that is most work and is unchanged. `risk/medium` and
above opens a PR, even solo, even with nobody waiting to review it. This mirrors
the tiered merge model in `assessor-loop`
`method/transfer-to-a-regulated-organisation.md` §4.1, so the estate practises
what it sells instead of merging everything on green and claiming otherwise.
**Exception — external contributor or client four-eyes requirement:** Use
PR flow only when a human reviewer outside the project is required. Document
the reason in PROJECT.md.
**Concurrency is NOT a reason to branch.** Measured 2026-08-29 across three real
collisions between parallel sessions: two were not branching problems at all — a
generated artifact written to the same path by both (branching turns that into a
merge conflict on a 33k-line JSON), and uncommitted work in a shared checkout
(a per-session worktree fixes it) — and the third was an ordinary non-fast-forward
push resolved by rebase in seconds. Use a worktree per session on shared repos and
timestamp generated artifacts. Do not reach for a branch.
Branch+PR flow carries documented costs on solo repos: merging auto-closes gated
issues, review comments live on a thread separate from the issue's, and a
merged-and-deleted branch cannot be amended (brain:
`gitea-pr-merge-autoclose-and-review-comment-footguns`). Those are the price of
the `risk/medium`+ lane — paid deliberately, not incurred by accident.
**Exception — external contributor or client four-eyes requirement:** Use PR flow
when a human reviewer outside the project is required. Document the reason in
PROJECT.md.
**Exception — a GitOps repo where a branch cannot be reconciled.** In a repo whose
deployment watches one branch (`mathias/infra`: Flux watches `main`), a branch is
invisible to the thing that applies it. A `risk/medium` change to `alerts.yaml` or
`litellm-config.yaml` can be linted on a branch but never *exercised* there, so a
PR would buy a review of a state that was never deployed.
There, `risk/medium`+ work stays on `main` and instead requires a **recorded
adversarial review pass before the push** — `skills:pr-gatekeeper` or equivalent,
reviewing the staged diff. **The reviewer's verdict goes in the issue, not a PR
thread.** No verdict recorded means the lane was not used, and saying "the gate was
green" is not the same claim.
Be honest about what this is: a recorded *independent* pass, not four-eyes in the
sense `assessor-loop` §4.1 means for a regulated organisation, because the reviewer
is an agent. It is more than the repo had and less than the model it mirrors; do not
describe it as the stronger property. (infra#470, decided 2026-09-29.)
This lane depends on the `risk/` label being right. A label that triggers a heavier
lane is load-bearing, not cosmetic — if work that belongs in it is unlabelled, the
lane silently does not apply.
6. **Close the loop — every substantive task ends with the same ritual.** Shipping
the code is not the end of the task; capturing it is. Run this unprompted:
@@ -73,6 +110,63 @@ These rules apply to every task across every project, regardless of harness.
- Surface the brain entries and issue numbers in the closing summary so the
trail is auditable.
**Chat is not a durable store. Never rely on Mathias remembering anything.**
Every gap, caveat, risk, open question and next step must land in one of four
places before the turn ends: **the brain**, **a repo** (code, docs, ADR),
**a tracker issue**, or **a plan file**. A finding stated only in conversation is
lost the moment the session closes — and this estate has repeatedly proved that a
remembered convention is an unreliable control.
This applies hardest to what is tempting to leave as a closing remark: "worth
knowing X isn't enforced yet", "this is a convention, not a gate", "we should
revisit Y". Those are exactly the items that need a locator. If it is worth
saying in a summary, it is worth writing where it can be found again without you.
7. **Finish what's closable — only involve me for a genuine call.** If a
tracker issue, PR, or checklist item is demonstrably resolved — the fix
landed, the state changed, you verified it live — close/merge/check it off
in the same turn. Don't leave it open "for the record" or park it waiting
on me to confirm something that already happened; that's bookkeeping, not
a decision. Reserve asking me (`AskUserQuestion`, a `DECISION NEEDED`
comment, or otherwise pausing) for when it's genuinely mine to call:
irreversible or prod/host-impacting actions, a real judgment call with no
clearly-better option, ambiguous intent, or anything secret-writes/
destructive-git already gate. Before claiming something is closable,
verify it live (re-check the issue, re-run the command) rather than
trusting your own earlier summary (brain: `filed-issue-not-same-as-fixed`).
8. **Control code needs an adversary, not a checklist.** Code whose job is to
refuse, deny, gate or contain is not adequately verified by a green gate.
Rule 5 makes CI the quality gate and that holds for ordinary work — but a
passing suite only asserts what its author already thought of, and for a
control the author is precisely the person whose imagination just failed.
Measured on `dispatch#75`: five review rounds, five real escapes verified by
execution, and **CI was green on every one of them** while `git rm
././.dispatch-allow` deleted the file the control existed to protect. So
route this work through a review with authority to block, instructed to
verify by execution rather than by reading the implementation. Expect to be
blocked more than once; that is the control working, not the reviewer being
difficult. (brain: `a-green-gate-is-not-evidence-for-control-code`; the
practices live in the `pr-gatekeeper` agent.)
9. **A test's name must not claim more than its assertions prove.** Name a test
after the mechanism it observes, not the goal it serves.
`DoesNotInheritEnv` is honest; `IsContained` is a claim the test cannot
support. This is not pedantry about naming — when name and assertion
diverge, the name wins in the reader's head and the gap becomes invisible,
which is what lets the defect survive review. Found three times in one PR,
twice in tests written to fix the previous instance of it; each passed while
the thing it was named for was exploitable. If you cannot name the mechanism,
you do not yet know what the test observes.
## CAD sprint patterns
Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`):
- **Inception Sprint** — birth a new repo. Oath-gated close; tag only once every clause holds. Brain: `.../inception-sprint-and-oath`.
- **Sweep Sprint** — clear an existing backlog. Parallel read-only research per issue → sequential one-issue-per-commit execution. Brain: `.../sweep-sprint-and-oath`.
Shared: decisions that are yours, not the agent's, surface as `DECISION NEEDED` (question/options/recommendation/blocks) — ask live if you're present, else comment and wait, never act past it. Live-breaking regressions get `git revert`, never forward-fixed; host-network work skips even read-only research (solo, synchronous, you present). Rule 6's closing ritual only fires once every Oath clause genuinely holds — blocked = descoped-and-tracked, never marked done.
## Default stack
| Layer | Default | Fallback | Last resort |
@@ -83,7 +177,7 @@ These rules apply to every task across every project, regardless of harness.
| Containers | Docker Compose (dev), k3s (prod) | — | — |
| DB | PostgreSQL + sqlc | SQLite | — |
| Search | pgvector (vector), BM25 | Qdrant (when >1M vectors or hybrid retrieval) | — |
| Logging | slog (structured) | — | — |
| Logging | slog (structured) | stdlib `logging` w/ structured `extra=` (or structlog) | — |
| Testing | Table-driven, testify | — | — |
| Agents (Go) | google.golang.org/adk + pkg/litellm adapter | — | — |
@@ -92,7 +186,12 @@ Exploratory: Rust, Zig — I'll tell you when I want these.
## Code conventions
- **Go style**: golines, gofumpt, golangci-lint
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return
- **Python style** (fallback language): ruff (format+lint, one tool), mypy --strict (non-negotiable,
matches Go's static typing discipline), pytest + pytest-cov (table-driven via
`@pytest.mark.parametrize`), uv (venv+deps+lock, one tool), pydantic-settings (typed env-var config
— same principle as Go's typed structs), src-layout + `pyproject.toml` only (no `setup.py`)
- **Errors**: `fmt.Errorf("operation: %w", err)` — never naked, never log-and-return.
Python: `raise X from e` (exception chaining, same principle) — never bare `except`, never silent `pass`
- **Naming**: stdlib conventions, no stuttering
- **Architecture**: prefer stdlib over frameworks, constructor injection, env-var config parsed into typed structs
- **Git**: conventional commits (`feat:`, `fix:`, `chore:`), commit directly to main,
@@ -130,10 +229,10 @@ Three machines on Tailscale:
|---------|------|-----------|
| koala | GPU inference, heavy compute | RTX 5070, runs k3s + llama-swap + shared postgres18/pgvector |
| iguana | Services, builds | M2 Ultra Mac |
| flamingo | Daily driver, edge | Mac mini, ~/dev is here |
| flamingo | Daily driver, edge | Apple Silicon Mac, ~/dev is here |
- **Model routing**: LiteLLM in front of llama-swap (local) + cloud APIs (when permitted)
- **Orchestration**: k3s cluster across all three machines
- **Orchestration**: k3s on koala only — single control-plane node, no workers (verified 2026-08-30). iguana and flamingo are on the tailnet but are not cluster members
- **Networking**: Tailscale mesh
## Project landscape
@@ -200,10 +299,16 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
|---------|-------|-------|
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` |
| **Anything HTTP-only (curl, scripts)** | Tailnet only: `POST https://koala.tail30472e.ts.net:8443/query` with `{"query":"..."}`, sending `BRAIN_MCP_TOKEN` (not yet enforced on REST, hyperguild#149). The public `brain-mcp.d-ma.be` serves `/mcp` only (infra#517). | `POST .../write` with `{"content":"...","wing":"...","hall":"..."}` |
| **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files |
- **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`.
- **Scoping — the brain holds no confidential client content.** Nothing on the read path
filters by client: every token holder can read every wing, and `wing` on `brain_query`
is an optional argument, not an access control. Client confidentiality is enforced only
at write time (`capture`'s I1 gate refuses internal/confidential). So never `brain_write`
client material; generalise it first or keep it in the client's own repo. (hyperguild#144,
decided 2026-10-01. This line previously claimed client projects were filtered; they
were not.)
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
fallback. Both are configurable in the `supervisor/ingestion-deployment.yaml`
on the koala k3s cluster; don't hardcode local-only model names into the
@@ -229,10 +334,11 @@ When working on a project tagged with a client name:
## Harness-agnostic principles
This context is designed to work with any AI coding tool:
- Claude Code, Cursor, Aider, Open WebUI, Charmbracelet Mods/Crush
- Pi Coding Agent, Mistral Vibe, Antigravity
- Any tool that accepts a system prompt or reads a markdown context file
Supported harnesses (2026-09-01) — Cursor and Aider were dropped; do not
re-add adapters for them:
- Claude Code, Antigravity, OpenCode, Mistral Vibe, Crush, Pi Coding Agent
- Open WebUI, Charmbracelet Mods, or any tool that takes a system prompt string
- Any other tool that reads `AGENTS.md`, which is the de facto convention
The canonical source is always `.context/AGENT.md` (root) and `.context/PROJECT.md` (per-project).
Derived files are committed (see *How context propagates* below) so a `git pull` on any host yields full agent context with no setup.
@@ -244,26 +350,65 @@ Canonical sources of truth:
- Project: `<repo>/.context/PROJECT.md` (per-repo)
Derived files (committed, regenerated by `task context:sync`):
- `CLAUDE.md`, `AGENTS.md`, `.cursorrules`, `.aider.conventions.md`,
`.context/system-prompt.txt`
- `CLAUDE.md`, `AGENTS.md`, `.context/system-prompt.txt`
Repos still carrying `.cursorrules`, `.aider.conventions.md` or
`.aider.conf.yml` are stale: nothing regenerates them. Delete them the next time
you touch the repo, along with their entries in that repo's `task check`
assertion.
Workflow:
1. Edit a canonical file. Run `task context:sync`. Commit canonical and
derived together. Push.
2. On any other host, `git pull` brings both. Claude Code (tree-walking)
uses `CLAUDE.md`; Crush / Pi / Antigravity (cwd-only) use `AGENTS.md`;
Cursor uses `.cursorrules`; Aider uses `.aider.conventions.md`.
2. On any other host, `git pull` brings both. Which file a harness reads does
NOT split cleanly by lookup strategy, so do not reason from the strategy:
- `CLAUDE.md` — Claude Code (walks up the tree)
- `AGENTS.md` — Crush, Pi, Antigravity (cwd-only), OpenCode (project root,
global at `~/.config/opencode/AGENTS.md`), and Mistral Vibe, which **walks
up** from cwd and takes the first `AGENTS.md` it finds
(user-level `~/.vibe/AGENTS.md`)
- `.context/system-prompt.txt` — Open WebUI, Mods, raw API calls; not read by
any harness, it is for pasting
Vibe tree-walks for `AGENTS.md`, so "tree-walkers use CLAUDE.md" is false.
Each repo's `AGENTS.md` embeds the canonical and is the first one found from
inside that repo, so this works — but by construction, not by luck.
3. `task check` runs `context:sync` then asserts `git status --porcelain`
is empty over the derived files (catches both modified-tracked drift
and missing-untracked adapters). A drift fails the check with a
message telling you to stage the regenerated files.
Behavior rules in this file and per-project rules in `PROJECT.md` apply
unconditionally on every host, every harness.
Behavior rules in this file and per-project rules in `PROJECT.md` apply on every
host and every harness.
**Where a project rule contradicts a rule here, the project rule governs — and it
must state its reason.** Both halves matter: the project file wins because it knows
something this file cannot (infra's Flux constraint is real and not visible from
here), and it must say why because a bare override is indistinguishable from drift.
This replaces the previous wording, which said rules here and in `PROJECT.md` both
apply *"unconditionally"*. That was not a tiebreak but a contradiction, and it went
unnoticed until rule 5 and infra's "never create feature branches" had to be obeyed
at the same time (infra#470). A standard can hold two individually sensible clauses
that are jointly unsatisfiable, and survive for as long as everyone reads it
charitably — the same failure the Inception Sprint Oath's G6 had.
## Engineering Skills
Shared engineering skills live in the **`mathias/skills`** repo (`git.d-ma.be/mathias/skills`). Clone it to `~/dev/skills/` and run `SKILLS_CHECKOUT_DIR="$PWD" bash install.sh` there to wire every skill into your harnesses (Claude Code, Crush, Antigravity, Mistral Vibe) as native, on-demand skills. (Use `install.sh`, not `task install` — the latter is currently broken, skills#7.) Load at task start — not "on demand" but on schedule, before writing code. Browse `~/dev/skills/SKILLS_INDEX.md` for the full list.
Shared engineering skills live in the **`mathias/skills`** repo (`git.d-ma.be/mathias/skills`). Load at task start — not "on demand" but on schedule, before writing code. Browse `SKILLS_INDEX.md` for the full list.
**Claude Code** installs the repo as a plugin from the marketplace the same repo hosts. That path also carries the `secret-guard` and `pre-task-ritual` hooks and the brain/gitea/infra MCP servers, which symlinks cannot:
```
/plugin marketplace add https://git.d-ma.be/mathias/skills.git
/plugin install skills@mathias
```
Skills are **namespaced** under Claude Code: load `tdd` as `skills:tdd`. The trigger table below uses bare names — prefix them.
**Crush, Pi and OpenCode** have no plugin system and are wired by `install.sh` as one directory symlink per harness via `~/.agents/skills`. Do not clone the checkout into `~/dev`: that folder is iCloud-shared between the Macs and spans two path roots (`/Users/mathias` vs `/home/mathias` on koala), so a checkout there is a git working tree with two concurrent writers, and any absolute symlink stored in it is broken on one side by construction (skills#23). Leave `SKILLS_CHECKOUT_DIR` at its host-local default.
Check a host with `task doctor` in the skills repo; it also runs automatically at session start via the plugin.
**Skill trigger table — load before starting, not after getting stuck:**
@@ -343,7 +488,11 @@ the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients.
for shell scripts and non-MCP clients, **with the same bearer token as `/mcp`**
(`BRAIN_MCP_TOKEN`; hyperguild#149). It is not served on the public host
(infra#517): use the tailnet (`koala:8443`) or in-cluster
`ingestion.supervisor.svc:3300`. `/ingest-path` and `brain_ingest` `path`
accept only paths inside `brain/raw/`.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
+5 -1
View File
@@ -64,7 +64,11 @@ the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients.
for shell scripts and non-MCP clients, **with the same bearer token as `/mcp`**
(`BRAIN_MCP_TOKEN`; hyperguild#149). It is not served on the public host
(infra#517): use the tailnet (`koala:8443`) or in-cluster
`ingestion.supervisor.svc:3300`. `/ingest-path` and `brain_ingest` `path`
accept only paths inside `brain/raw/`.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
-1
View File
@@ -1,2 +1 @@
ingestion: cd ingestion && INGEST_BRAIN_DIR=../brain INGEST_PORT=3300 INGEST_WATCH_INTERVAL=30 go run ./cmd/server/
supervisor: SUPERVISOR_CONFIG_DIR=./config/supervisor SUPERVISOR_MODELS_FILE=./config/models.yaml SUPERVISOR_SESSIONS_DIR=./brain/sessions INGEST_BASE_URL=http://localhost:3300 INGEST_SVC_URL=http://localhost:3300 go run ./cmd/supervisor/
+93 -3
View File
@@ -5,6 +5,11 @@ Instead of letting Claude Code do whatever it wants, hyperguild enforces structu
workflows (TDD red/green/refactor), logs every session, and accumulates learnings
into a searchable brain.
**This repo is also where `brain-mcp` lives** — the MCP server at
`brain-mcp.d-ma.be` that `brain_query`, `brain_write`, `brain_read`, `brain_edit`
and `capture` are served from. It is not a separate deployable; see
[Where brain-mcp lives](#where-brain-mcp-lives).
## Hypothesis
> We believe routing skill tasks through local models, backed by brain context,
@@ -20,7 +25,7 @@ usage, the fast-model routing path should be reconsidered.
**hyperguild = Claude Code + MCP.** This is a supervisor for Claude Code sessions specifically.
For multi-agent orchestration (OpenCode + LiteLLM, executor/reviewer pipelines), see
[agentsquad](http://gitea.d-ma.be/mathias/agentsquad) — a separate harness for a different
[agentsquad](https://git.d-ma.be/mathias/agentsquad) — a separate harness for a different
orchestration model. Skills (mathias/skills) are shared between both.
## How it works
@@ -44,6 +49,91 @@ brain/
└── knowledge/ — legacy flat notes (migration pending: hyperguild#22)
```
## Where brain-mcp lives
`brain-mcp` is served from **`ingestion/internal/mcp`** in this repo, at
**`brain-mcp.d-ma.be`**. It is a component *inside* the `ingestion` service, not
a standalone deployable — one pod in the `supervisor` namespace serves the MCP
endpoint, the REST API, and the claudewatcher together.
| You are looking for | It is here |
|---|---|
| The MCP tool surface (`brain_query`, `brain_write`, `brain_read`, `brain_edit`, `capture`, …) | `ingestion/internal/mcp` |
| The HTTP/REST API (`/query`, `/write`, `/ingest`) | `ingestion/internal/api` |
| Session ingestion from Claude Code transcripts | `ingestion/internal/claudewatcher` |
| The brain *content* (notes, wings, halls) | [`mathias/brain`](https://git.d-ma.be/mathias/brain) — a separate repo |
| Deployment manifests | `config/supervisor` |
### Why it is not its own repo
Asked and decided in [#91](https://git.d-ma.be/mathias/hyperguild/issues/91)
(part 2). `internal/mcp` imports 17 of ingestion's other internal packages and
has exactly one importer — it is the MCP facade over the service's own
internals, so "move it out" resolves to renaming `ingestion`, publishing 17
packages as a shared library, or maintaining a shim. The discoverability problem
that prompted the question is answered by naming it here instead.
Full reasoning, including the reversal trigger: brain
`wiki/hyperguild/decisions/brain-mcp-stays-in-ingestion-fix-discoverability-by-name`.
## Ingest revises, it does not overwrite
Re-ingesting a source **merges into the page already on disk** rather than
replacing it. Previously the write was unconditional, so a second ingest of the
same source discarded everything accumulated since — including hand-written
sections.
The merge is deliberately asymmetric in the existing page's favour:
| On the page | In the fresh extraction | Result |
|---|---|---|
| a section | the same section | **the on-disk version wins** |
| — | a new section | added |
| a bullet list (`Sources`, `Key Claims`, …) | more bullets | unioned, no duplicates |
| an append section (`Open Questions`, `Updates`, …) | more lines | only genuinely new lines appended |
| frontmatter | frontmatter | **the on-disk frontmatter wins** |
So an extraction can add to a note but never silently rewrite what a human put
there, which is what makes it safe to run on every ingest rather than behind a
review gate.
Re-ingesting unchanged content is **byte-identical** — no diff, so brain-sync
does not commit noise that hides real edits.
Two fields are **generator-owned** and do get refreshed on a revise:
`last_updated` and `source_hash`. Everything else on disk wins. The boundary is
an explicit allowlist, so a field nobody has thought about stays human-owned.
### Disagreement is reported, not resolved
When the fresh extraction produces a *different version* of a section that
already exists, the on-disk version still wins — but the run now says so:
```
wiki/sources/shape-up.md: source now disagrees on section "Summary" — on-disk version kept
```
That line lands in `log.md` with the rest of the run's warnings. Without it, "the
source is unchanged" and "the source changed its mind and we ignored it" look
identical from the outside.
This is deliberately **textual** disagreement, not semantic contradiction: cheap,
deterministic, no model on the write path. Deciding whether a disagreement is a
genuine contradiction stays with the Judge-gated `brain-enrich` pass, which is
approval-gated by design. Detection here, adjudication there.
## Re-ingesting an unchanged source is skipped
Ingest stores `source_hash` — sha256 of the ingested **body**, frontmatter
excluded — on the source page. A later ingest of the same source compares it
first and skips extraction entirely when it matches, reporting the skip rather
than doing it silently.
Body-only is deliberate: a re-crawl that stamps a new fetch date has not changed
the source, and treating that as drift would re-extract the corpus every time a
crawler ran. A missing or unreadable hash always means "extract" — a missed skip
costs one extraction, a wrong skip loses an update.
## Phase 1 tools (available now)
| Tool | What it does |
@@ -58,7 +148,7 @@ brain/
| `tier` | Returns the current connectivity tier (1=cloud, 2=LAN, 3=offline) |
> **Note:** `tdd_red/green/refactor` and `spec` were retired in Plan 7 (2026-05-12).
> They are now SKILL.md files in [mathias/skills](http://gitea.d-ma.be/mathias/skills).
> They are now SKILL.md files in [mathias/skills](https://git.d-ma.be/mathias/skills).
## Connect a project
@@ -147,7 +237,7 @@ The routing pod selects models per skill call based on historical pass rate:
## Open issues
See [issues](http://gitea.d-ma.be/mathias/hyperguild/issues) — key open items:
See [issues](https://git.d-ma.be/mathias/hyperguild/issues) — key open items:
- **#25** — skills platform overhaul (audit first, then lazy loading + brain feedback loop)
- **#24** — reduce context burn from skill listing
+74 -36
View File
@@ -21,7 +21,7 @@ tasks:
# ── Development ────────────────────────────────────────────────────────────
start:
desc: Start ingestion + supervisor (requires goreman — go install github.com/mattn/goreman@latest)
desc: Start ingestion (requires goreman — go install github.com/mattn/goreman@latest)
cmds:
- goreman start
@@ -32,11 +32,6 @@ tasks:
- lsof -ti:3200 | xargs kill -9 2>/dev/null || true
- echo "hyperguild stopped"
supervisor:dev:
desc: Run supervisor MCP server (development)
cmds:
- go run ./cmd/supervisor
hyperguild:dev:
desc: Run hyperguild CLI from source (e.g. task hyperguild:dev -- tier)
cmds:
@@ -67,14 +62,9 @@ tasks:
build:
desc: Build all binaries
cmds:
- task: supervisor:build
- task: hyperguild:build
- task: ingestion:build
supervisor:build:
desc: Build supervisor binary
cmds:
- go build -trimpath -ldflags="-s -w -X main.version={{.VERSION}}" -o bin/supervisor ./cmd/supervisor
ingestion:build:
desc: Build ingestion server binary
dir: ingestion
@@ -84,29 +74,90 @@ tasks:
# ── Quality ────────────────────────────────────────────────────────────────
check:
desc: Run all checks (context freshness + lint + test + vet) across all modules
desc: Run all checks (context freshness + lint + test + vet + build) across all modules
cmds:
- task: context:sync
# Context sync is skipped under CI, matching dma-cli, gitea-mcp, and the
# rest of the estate. It is not a preference: context-sync.sh walks UP
# the tree for the canonical .context/AGENT.md, which lives in local-dev
# and does not exist on a CI runner. Without this guard, the script
# silently falls back to "project context only" and regenerates AGENTS.md
# and .context/system-prompt.txt shorter than their committed, correct
# versions -- so the drift assertion fails every CI run for a purely
# environmental reason, not real drift. Verified, not assumed: this is
# exactly what was happening here (AGENTS.md 430 lines committed vs. an
# 82-line regeneration on the runner).
#
# The cost is real and worth stating: drift is therefore only ever caught
# on a developer machine, or by infra's estate-wide context-sync-drift
# CronJob (ADR-0019), which does have canonical access and is the actual
# enforcement mechanism now.
- cmd: |
drift=$(git status --porcelain -- AGENTS.md CLAUDE.md .context/system-prompt.txt 2>/dev/null)
if [ -n "$drift" ]; then
echo "ERROR: derived adapters drifted from canonical context." >&2
echo "$drift" >&2
echo "" >&2
echo "Run: git add AGENTS.md CLAUDE.md .context/system-prompt.txt" >&2
echo " git commit -m 'chore: re-sync context adapters'" >&2
exit 1
if [ -n "${CI:-}" ]; then
echo "✓ context sync: skipped in CI (no canonical AGENT.md on the runner)"
else
bash scripts/context-sync.sh
drift=$(git status --porcelain -- AGENTS.md CLAUDE.md .context/system-prompt.txt 2>/dev/null)
if [ -n "$drift" ]; then
echo "ERROR: derived adapters drifted from canonical context." >&2
echo "$drift" >&2
echo "" >&2
echo "Run: git add AGENTS.md CLAUDE.md .context/system-prompt.txt" >&2
echo " git commit -m 'chore: re-sync context adapters'" >&2
exit 1
fi
echo "✓ context: canonical and adapters are in sync"
fi
echo "✓ context: canonical and adapters are in sync"
- task: lint
- task: test
- task: vet
# `build` is in the gate deliberately (hyperguild#120). vet and test walk
# ./... , which enumerates the packages that EXIST — so a build target
# naming a DELETED package is invisible to both. `task build` pointed at
# ./cmd/supervisor for months after that directory was removed, and CI
# stayed green the whole time because nothing in check ever built.
- task: build
- task: security:gitleaks
# ── Security ─────────────────────────────────────────────
security:gitleaks:
desc: Scan the working tree for secrets (gitleaks, fail-closed; skipped if gitleaks absent)
dir: '{{.ROOT_DIR}}'
cmds:
- |
GL="$(command -v gitleaks || true)"
[ -z "$GL" ] && [ -x "$(go env GOPATH 2>/dev/null)/bin/gitleaks" ] && GL="$(go env GOPATH)/bin/gitleaks"
if [ -z "$GL" ]; then
echo "⚠ gitleaks not installed — skipping secret scan (CI enforces it)."
echo " Install: go install github.com/zricethezav/gitleaks/v8@latest"
exit 0
fi
"$GL" detect --no-git --redact --config .gitleaks.toml --source .
security:gitleaks:history:
desc: "One-time FULL-HISTORY secret audit (infra#39 rotation pass; not a per-push gate)"
dir: '{{.ROOT_DIR}}'
cmds:
- |
GL="$(command -v gitleaks || true)"
[ -z "$GL" ] && [ -x "$(go env GOPATH 2>/dev/null)/bin/gitleaks" ] && GL="$(go env GOPATH)/bin/gitleaks"
if [ -z "$GL" ]; then
echo "gitleaks not installed: go install github.com/zricethezav/gitleaks/v8@latest" >&2
exit 2
fi
echo "Scanning FULL git history (redacted). Known historical leaks are expected"
echo "until the infra#39 rotation pass completes — triage against the rotation list."
"$GL" detect --redact --config .gitleaks.toml
lint:
cmds:
- golangci-lint run ./...
- cd ingestion && golangci-lint run ./...
lint:frontmatter:
desc: Report brain notes whose YAML frontmatter does not parse (BRAIN=path, FIX=1 repairs in place)
cmds:
- cd ingestion && go run ./cmd/frontmatterlint {{if .FIX}}-fix{{end}} {{.BRAIN | default "../../brain"}}
test:
cmds:
- go test -race -count=1 ./...
@@ -118,19 +169,6 @@ tasks:
- govulncheck ./... || true
- cd ingestion && go vet ./...
supervisor:test:smoke:
desc: Smoke test supervisor via MCP (requires start running)
cmds:
- |
curl -s -X POST http://localhost:${SUPERVISOR_PORT:-3200}/mcp \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' | jq .
smoke:routing:
desc: Boot the routing pod against live LiteLLM + brain and verify _routing logs land
cmds:
- bash scripts/smoke-routing.sh
# ── Git / Release ──────────────────────────────────────────────────────────
tag:
+7 -4
View File
@@ -55,18 +55,21 @@ Flags:
- `--limit N` — max results (default 5)
- `--json` — emit the raw response envelope
### `hyperguild brain write <type> <slug>`
### `hyperguild brain write <wing> <hall> <slug>`
Reads markdown from stdin, writes a knowledge entry.
Reads markdown from stdin, writes it to `brain/wiki/<wing>/<hall>/<slug>.md`
with `wing`/`hall`/`created_at` frontmatter injected (there is no legacy
`brain/knowledge/` fallback — ADR-0012 / #79, both wing and hall are
required).
```bash
$ cat <<EOF | hyperguild brain write knowledge example-lesson
$ cat <<EOF | hyperguild brain write homelab facts example-lesson
# Example lesson
## Lesson
...
EOF
knowledge/example-lesson.md
wiki/homelab/facts/example-lesson.md
```
### `hyperguild brain pass-rate <skill>`
+6 -5
View File
@@ -60,13 +60,14 @@ func runBrainWrite(ctx context.Context, args []string, stdin io.Reader, stdout,
if err := fs.Parse(args); err != nil {
return fmt.Errorf("parse flags: %w", err)
}
if fs.NArg() < 2 {
return errors.New("type and slug required (e.g. brain write knowledge my-slug)")
if fs.NArg() < 3 {
return errors.New("wing, hall and slug required (e.g. brain write homelab facts my-slug)")
}
kind := fs.Arg(0)
slug := fs.Arg(1)
wing := fs.Arg(0)
hall := fs.Arg(1)
slug := fs.Arg(2)
res, err := newBrainClient().Write(ctx, kind, slug, stdin)
res, err := newBrainClient().Write(ctx, wing, hall, slug, stdin)
if err != nil {
return err
}
+5 -5
View File
@@ -101,7 +101,7 @@ func TestRunBrainWrite_Success(t *testing.T) {
var out, errBuf bytes.Buffer
err := runBrain(
context.Background(),
[]string{"write", "knowledge", "test-slug"},
[]string{"write", "homelab", "facts", "test-slug"},
strings.NewReader("# Test\n\nSome body content.\n"),
&out, &errBuf,
)
@@ -111,9 +111,9 @@ func TestRunBrainWrite_Success(t *testing.T) {
func TestRunBrainWrite_MissingArgs(t *testing.T) {
var out, errBuf bytes.Buffer
err := runBrain(context.Background(), []string{"write", "knowledge"}, strings.NewReader("x"), &out, &errBuf)
err := runBrain(context.Background(), []string{"write", "homelab"}, strings.NewReader("x"), &out, &errBuf)
assert.Error(t, err)
assert.Contains(t, err.Error(), "type and slug required")
assert.Contains(t, err.Error(), "wing, hall and slug required")
}
func TestRunBrainWrite_BackendError(t *testing.T) {
@@ -127,7 +127,7 @@ func TestRunBrainWrite_BackendError(t *testing.T) {
var out, errBuf bytes.Buffer
err := runBrain(
context.Background(),
[]string{"write", "knowledge", "bad slug"},
[]string{"write", "homelab", "facts", "bad slug"},
strings.NewReader("body"),
&out, &errBuf,
)
@@ -150,7 +150,7 @@ func TestRunBrainWrite_EmptyStdin(t *testing.T) {
t.Setenv("BRAIN_URL", srv.URL)
var out, errBuf bytes.Buffer
err := runBrain(context.Background(), []string{"write", "knowledge", "empty"}, strings.NewReader(""), &out, &errBuf)
err := runBrain(context.Background(), []string{"write", "homelab", "facts", "empty"}, strings.NewReader(""), &out, &errBuf)
require.NoError(t, err)
assert.Equal(t, 0, gotLen, "empty stdin should produce empty content payload")
}
+21 -10
View File
@@ -20,6 +20,7 @@ const defaultBrainURL = "http://koala:30330"
// shell-friendly client; MCP framing is unnecessary.
type brainClient struct {
baseURL string
token string // BRAIN_MCP_TOKEN; the REST API requires it since hyperguild#149
http *http.Client
}
@@ -30,6 +31,7 @@ func newBrainClient() *brainClient {
}
return &brainClient{
baseURL: u,
token: os.Getenv("BRAIN_MCP_TOKEN"),
http: &http.Client{Timeout: 5 * time.Second},
}
}
@@ -62,8 +64,7 @@ func (c *brainClient) Query(ctx context.Context, topic string, limit int) (*Quer
return nil, fmt.Errorf("build request: %w", err)
}
req.Header.Set("Content-Type", "application/json")
resp, err := c.http.Do(req)
resp, err := c.do(req)
if err != nil {
return nil, fmt.Errorf("brain POST /query: %w", err)
}
@@ -84,16 +85,17 @@ type WriteResult struct {
Path string `json:"path"`
}
func (c *brainClient) Write(ctx context.Context, kind, slug string, content io.Reader) (*WriteResult, error) {
func (c *brainClient) Write(ctx context.Context, wing, hall, filename string, content io.Reader) (*WriteResult, error) {
body, err := io.ReadAll(content)
if err != nil {
return nil, fmt.Errorf("read content: %w", err)
}
payload, err := json.Marshal(struct {
Type string `json:"type"`
Slug string `json:"slug"`
Content string `json:"content"`
}{Type: kind, Slug: slug, Content: string(body)})
Wing string `json:"wing"`
Hall string `json:"hall"`
Filename string `json:"filename"`
Content string `json:"content"`
}{Wing: wing, Hall: hall, Filename: filename, Content: string(body)})
if err != nil {
return nil, fmt.Errorf("marshal payload: %w", err)
}
@@ -104,8 +106,7 @@ func (c *brainClient) Write(ctx context.Context, kind, slug string, content io.R
return nil, fmt.Errorf("build request: %w", err)
}
req.Header.Set("Content-Type", "application/json")
resp, err := c.http.Do(req)
resp, err := c.do(req)
if err != nil {
return nil, fmt.Errorf("brain POST /write: %w", err)
}
@@ -142,7 +143,7 @@ func (c *brainClient) PassRate(ctx context.Context, skill, window string) (*Pass
if err != nil {
return nil, fmt.Errorf("build request: %w", err)
}
resp, err := c.http.Do(req)
resp, err := c.do(req)
if err != nil {
return nil, fmt.Errorf("brain GET /pass-rate: %w", err)
}
@@ -157,3 +158,13 @@ func (c *brainClient) PassRate(ctx context.Context, skill, window string) (*Pass
}
return &out, nil
}
// do sends every brain request, adding the bearer token when one is
// configured. All methods go through here so none can forget it (PassRate
// did, PR #152 review).
func (c *brainClient) do(req *http.Request) (*http.Response, error) {
if c.token != "" {
req.Header.Set("Authorization", "Bearer "+c.token)
}
return c.http.Do(req)
}
+57
View File
@@ -0,0 +1,57 @@
package main
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// hyperguild#149: the ingestion REST API now requires a bearer token; the CLI
// sends BRAIN_MCP_TOKEN, or it gets 401 on every brain call.
func TestBrainClientSendsBrainMCPTokenOnQueryWriteAndPassRate(t *testing.T) {
var got []string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
got = append(got, r.URL.Path+" "+r.Header.Get("Authorization"))
switch r.URL.Path {
case "/write":
_, _ = w.Write([]byte(`{"path":"wiki/x/facts/y.md"}`))
return
case "/pass-rate":
_, _ = w.Write([]byte(`{}`))
return
}
_, _ = w.Write([]byte(`{"results":[]}`))
}))
defer srv.Close()
t.Setenv("BRAIN_URL", srv.URL)
t.Setenv("BRAIN_MCP_TOKEN", "cli-test-token")
c := newBrainClient()
_, err := c.Query(context.Background(), "x", 1)
require.NoError(t, err)
_, err = c.Write(context.Background(), "x", "facts", "y", strings.NewReader("# y"))
require.NoError(t, err)
// PR #152 review: PassRate sent no token and would 401 in production.
_, _ = c.PassRate(context.Background(), "tdd", "7d")
assert.Equal(t, []string{"/query Bearer cli-test-token", "/write Bearer cli-test-token",
"/pass-rate Bearer cli-test-token"}, got)
}
func TestBrainClientSendsNoAuthorizationWhenTokenUnset(t *testing.T) {
var auth string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
auth = r.Header.Get("Authorization")
_, _ = w.Write([]byte(`{"results":[]}`))
}))
defer srv.Close()
t.Setenv("BRAIN_URL", srv.URL)
t.Setenv("BRAIN_MCP_TOKEN", "")
_, err := newBrainClient().Query(context.Background(), "x", 1)
require.NoError(t, err)
assert.Empty(t, auth)
}
+10 -8
View File
@@ -65,23 +65,25 @@ func TestBrainClient_Write_Success(t *testing.T) {
assert.Equal(t, http.MethodPost, r.Method)
body, _ := io.ReadAll(r.Body)
var got struct {
Type string `json:"type"`
Slug string `json:"slug"`
Content string `json:"content"`
Wing string `json:"wing"`
Hall string `json:"hall"`
Filename string `json:"filename"`
Content string `json:"content"`
}
require.NoError(t, json.Unmarshal(body, &got))
assert.Equal(t, "knowledge", got.Type)
assert.Equal(t, "find-h", got.Slug)
assert.Equal(t, "homelab", got.Wing)
assert.Equal(t, "facts", got.Hall)
assert.Equal(t, "find-h", got.Filename)
assert.Equal(t, "# body\n", got.Content)
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"path":"knowledge/find-h.md"}`))
_, _ = w.Write([]byte(`{"path":"wiki/homelab/facts/find-h.md"}`))
}))
defer srv.Close()
c := &brainClient{baseURL: srv.URL, http: srv.Client()}
res, err := c.Write(context.Background(), "knowledge", "find-h", strings.NewReader("# body\n"))
res, err := c.Write(context.Background(), "homelab", "facts", "find-h", strings.NewReader("# body\n"))
require.NoError(t, err)
assert.Equal(t, "knowledge/find-h.md", res.Path)
assert.Equal(t, "wiki/homelab/facts/find-h.md", res.Path)
}
func TestBrainClient_PassRate_Success(t *testing.T) {
-26
View File
@@ -1,26 +0,0 @@
# Model selection — first entry per skill is used.
# Override per-call by passing model in the MCP tool args.
# Model names come from LiteLLM /v1/models (host/name format).
default_chain:
- iguana/qwen3-coder-next
skills:
tdd:
chain:
- koala/qwen3-coder-30b
review:
chain:
- iguana/devstral
debug:
chain:
- iguana/deepseek-r1-14b
spec:
chain:
- koala/phi4-14b
retrospective:
chain:
- iguana/qwen3-coder-next
trainer:
chain:
- iguana/qwen3-coder-next
@@ -24,7 +24,7 @@ Only a `tools/call` to the routing pod records a pass/fail.
|---------|-----|------|
| Routing MCP (local, Tailscale) | `http://koala:30310/mcp` | Bearer `ROUTING_MCP_TOKEN` |
| Routing MCP (remote) | `https://routing-mcp.d-ma.be/mcp` | OAuth via `auth.d-ma.be` (audience `claude-ai`) |
| Pass-rate readout | `http://koala:30330/pass-rate?skill=<name>` | none (read-only) |
| Pass-rate readout | `http://koala:30330/pass-rate?skill=<name>` | Bearer `BRAIN_MCP_TOKEN` (since hyperguild#149) |
Tools advertised: **`review`**, **`debug`** (the two the #35 gate measures),
plus `session_log`, `retrospective`, `trainer`.
@@ -1051,7 +1051,7 @@ git push
```bash
# Poll until complete (replace RUN_ID with the new run):
curl -s "https://gitea.d-ma.be/api/v1/repos/mathias/hyperguild/actions/runs?limit=2" \
-H "Authorization: token 736a8c36adcc6ecb41fff56e5ae4d0eb3105a670" \
-H "Authorization: token <REDACTED-dead-gitea-pat-infra#344>" \
| python3 -c "import sys,json; [print(r['id'],r['status'],r.get('conclusion','—')) for r in json.load(sys.stdin)['workflow_runs']]"
```
@@ -16,7 +16,7 @@ Three needs converge on a single small Go binary:
- [ ] `hyperguild tier` returns the same `tier.Info` that `internal/tier.Detect` produces for the same probe URLs, in < 3 s under all three tier conditions, with both human-readable and `--json` output.
- [ ] `hyperguild brain query <topic>` returns BM25 results from the brain HTTP REST `/query` endpoint, exit 0 on success and non-zero on transport failure.
- [ ] `hyperguild brain write <type> <slug>` reads markdown content from stdin, posts to `/write` with the type and slug, and creates `brain/knowledge/<slug>.md`. A round-trip (`hyperguild brain query <slug>` immediately after) finds the entry.
- [ ] `hyperguild brain write <wing> <hall> <slug>` reads markdown content from stdin, posts to `/write` with wing/hall/filename, and creates `brain/wiki/<wing>/<hall>/<slug>.md`. A round-trip (`hyperguild brain query <slug>` immediately after) finds the entry. (Updated post-ADR-0012/#79: the legacy `<type> <slug>` shape and the `brain/knowledge/` fallback were removed; wing and hall are now required.)
- [ ] `hyperguild mode <cloud|client-local|sovereign>` writes a parseable JSON file at the target path with the per-mode `mcpServers` entries; `jq -e .mcpServers` succeeds on the output.
- [ ] All commands print usage on `--help`, exit 2 on unknown flags, exit non-zero on operational errors.
- [ ] `task check` passes (lint + test + vet) on each task and on the merged branch.
@@ -50,7 +50,7 @@ Three needs converge on a single small Go binary:
- **`brain` subcommand calls the HTTP REST API.** Two nested subcommands:
- `brain query <topic>` issues `POST /query` with JSON body `{query, limit}` (default `--limit 5`), prints results in human-readable form by default and with `--json` for machine consumption.
- `brain write <type> <slug>` reads stdin, posts `POST /write` with JSON body `{type, slug, content}`, prints the resulting path on success.
- `brain write <wing> <hall> <slug>` reads stdin, posts `POST /write` with JSON body `{wing, hall, filename, content}`, prints the resulting path on success. (Updated post-ADR-0012/#79: wing and hall are required, replacing the old `<type> <slug>` shape and the `brain/knowledge/` legacy fallback.)
Rationale: HTTP REST is simpler than MCP framing for a CLI. Per CLAUDE.md, the REST endpoints are documented as the official non-MCP interface.
- **`mode <name>` writes a per-mode `.mcp.json` template.** Defaults to writing `./.mcp.json` (cwd); accepts `--out <path>`. Per-mode bodies:
+53
View File
@@ -0,0 +1,53 @@
// Command frontmatterlint reports markdown notes whose YAML frontmatter does
// not parse, and optionally repairs the ones that only need quoting.
//
// go run ./cmd/frontmatterlint ../../brain # report
// go run ./cmd/frontmatterlint -fix ../../brain # repair in place
//
// Exit codes: 0 clean (or everything repaired), 1 findings remain, 2 the scan
// itself failed.
package main
import (
"flag"
"fmt"
"os"
"github.com/mathiasbq/hyperguild/ingestion/internal/frontmatter"
)
func main() {
fix := flag.Bool("fix", false, "repair files that quoting can rescue, in place")
flag.Parse()
root := "."
if flag.NArg() > 0 {
root = flag.Arg(0)
}
findings, err := frontmatter.Scan(root, *fix)
if err != nil {
fmt.Fprintln(os.Stderr, "frontmatterlint:", err)
os.Exit(2)
}
var unresolved int
for _, f := range findings {
if f.Repaired {
fmt.Printf("repaired %s\n", f.Path)
continue
}
unresolved++
fmt.Printf("BROKEN %s: %v\n", f.Path, f.Err)
}
if unresolved > 0 {
fmt.Fprintf(os.Stderr, "\n%d file(s) with unparseable frontmatter", unresolved)
if !*fix {
fmt.Fprint(os.Stderr, " — re-run with -fix")
}
fmt.Fprintln(os.Stderr)
os.Exit(1)
}
fmt.Printf("✓ frontmatter: %d file(s) repaired, none broken\n", len(findings))
}
+26
View File
@@ -0,0 +1,26 @@
package main
import (
"context"
"fmt"
"log/slog"
chassisauth "git.d-ma.be/mathias/mcp-chassis/auth"
)
// buildJWTValidator builds the JWT validator for issuer (nil when issuer is
// empty: JWT auth disabled). Since mcp-chassis v0.7 construction no longer
// fails when OIDC discovery does (mcp-chassis#4): the issuer resolves lazily,
// and until it does a JWT is answered 503-and-retry, not 401. That replaces
// #83's boot-time retry loop (hyperguild#155). An issuer still unresolved at
// boot is logged, so a misconfigured issuer is visible rather than silent.
func buildJWTValidator(ctx context.Context, logger *slog.Logger, issuer, audience string) (*chassisauth.JWTValidator, error) {
v, err := chassisauth.NewJWTValidator(ctx, issuer, audience)
if err != nil {
return nil, fmt.Errorf("build jwt validator: %w", err)
}
if ok, rerr := v.Ready(ctx); !ok {
logger.Warn("jwt issuer not resolved at boot; JWTs answer 503 until it is", "issuer", issuer, "err", rerr)
}
return v, nil
}
+111
View File
@@ -0,0 +1,111 @@
package main
import (
"bytes"
"context"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/metrics"
)
// fakeIssuer answers OIDC discovery with 502 for the first fail requests, as
// ingress does while dex is not yet serving on a cold boot (#83), then with a
// real discovery document and an empty JWKS.
func fakeIssuer(t *testing.T, fail int32) (*httptest.Server, *atomic.Int32) {
t.Helper()
var discoveries atomic.Int32
mux := http.NewServeMux()
var srv *httptest.Server
mux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, _ *http.Request) {
if discoveries.Add(1) <= fail {
w.WriteHeader(http.StatusBadGateway)
return
}
w.Header().Set("Content-Type", "application/json")
_, _ = io.WriteString(w, `{"issuer":"`+srv.URL+`","jwks_uri":"`+srv.URL+`/keys"}`)
})
mux.HandleFunc("/keys", func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = io.WriteString(w, `{"keys":[]}`)
})
srv = httptest.NewServer(mux)
t.Cleanup(srv.Close)
return srv, &discoveries
}
// #83 retried discovery at boot because chassis v0.2.0 failed construction on
// a 502. Since chassis v0.7 (mcp-chassis#4) the validator is built anyway and
// resolves the issuer lazily, so boot no longer waits or exits on it
// (hyperguild#155). An unresolved issuer is logged at boot.
func TestBuildJWTValidatorReturnsAValidatorWhileDiscovery502s(t *testing.T) {
srv, _ := fakeIssuer(t, 1000)
var buf bytes.Buffer
v, err := buildJWTValidator(context.Background(), slog.New(slog.NewTextHandler(&buf, nil)), srv.URL, "brain")
require.NoError(t, err)
assert.NotNil(t, v)
assert.Contains(t, buf.String(), "jwt issuer not resolved at boot")
}
func TestBuildJWTValidatorLogsNothingAgainstAHealthyIssuer(t *testing.T) {
srv, _ := fakeIssuer(t, 0)
var buf bytes.Buffer
v, err := buildJWTValidator(context.Background(), slog.New(slog.NewTextHandler(&buf, nil)), srv.URL, "brain")
require.NoError(t, err)
assert.NotNil(t, v)
assert.NotContains(t, buf.String(), "not resolved")
}
func TestBuildJWTValidatorIsNilWithoutAnIssuer(t *testing.T) {
v, err := buildJWTValidator(context.Background(), slog.New(slog.DiscardHandler), "", "brain")
require.NoError(t, err)
assert.Nil(t, v)
}
// With the issuer unresolved, a JWT on a REST route answers 503 (retry), not
// 401: the token may be fine, the server just cannot check it yet.
func TestBuildMuxAnswers503ToAJWTWhileTheIssuerIsUnresolved(t *testing.T) {
srv, _ := fakeIssuer(t, 1000)
v, err := buildJWTValidator(context.Background(), slog.New(slog.DiscardHandler), srv.URL, "brain")
require.NoError(t, err)
mux, err := buildMux(restAuth{token: routesTok, validator: v, realm: "brain"}, testHandler(t),
okHandler("mcp"), metrics.New(), nil)
require.NoError(t, err)
code, _ := do(t, mux, "GET", "/pending", "Bearer aaa.bbb.ccc")
assert.Equal(t, http.StatusServiceUnavailable, code)
}
// PR #183 review: /mcp shares REST's bearer, but only the static token was
// asserted on it. With the issuer unresolved, a JWT on /mcp answers 503 too,
// so /mcp's middleware was built with the JWT validator.
func TestBuildMuxAnswers503ToAJWTOnMCPWhileTheIssuerIsUnresolved(t *testing.T) {
srv, _ := fakeIssuer(t, 1000)
v, err := buildJWTValidator(context.Background(), slog.New(slog.DiscardHandler), srv.URL, "brain")
require.NoError(t, err)
mux, err := buildMux(restAuth{token: routesTok, validator: v, realm: "brain"}, testHandler(t),
okHandler("mcp"), metrics.New(), nil)
require.NoError(t, err)
code, _ := do(t, mux, "POST", "/mcp", "Bearer aaa.bbb.ccc")
assert.Equal(t, http.StatusServiceUnavailable, code)
}
// PR #183 review: a 401 on /mcp names the protected-resource metadata in
// WWW-Authenticate (what the claude.ai OAuth connector discovers from), so
// /mcp's middleware was built with resourceMetadataURL.
func TestBuildMuxMCP401CarriesResourceMetadataInWWWAuthenticate(t *testing.T) {
const rm = "https://brain.example/.well-known/oauth-protected-resource"
mux, err := buildMux(restAuth{token: routesTok, realm: "brain", resourceMetadataURL: rm}, testHandler(t),
okHandler("mcp"), metrics.New(), nil)
require.NoError(t, err)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, httptest.NewRequest(http.MethodPost, "/mcp", nil))
require.Equal(t, http.StatusUnauthorized, rec.Code)
assert.Contains(t, rec.Header().Get("WWW-Authenticate"), `resource_metadata="`+rm+`"`)
}
+149 -59
View File
@@ -17,6 +17,7 @@ import (
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/audit"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/capturehttp"
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
@@ -32,6 +33,7 @@ import (
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
"github.com/mathiasbq/hyperguild/ingestion/internal/reranker"
"github.com/mathiasbq/hyperguild/ingestion/internal/search"
"github.com/mathiasbq/hyperguild/ingestion/internal/symlinkguard"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/watcher"
"github.com/mathiasbq/hyperguild/ingestion/internal/webhook"
@@ -88,11 +90,15 @@ func (s *claudeSink) Ingest(ctx context.Context, b claudewatcher.Batch) error {
sb.WriteString("\n\n")
}
slug := "session-" + b.Host + "-" + b.SessionID
dest := filepath.Join(s.brainDir, "archive", "claude-sessions", b.Host, slug+".md")
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
return fmt.Errorf("create claude-sessions archive dir: %w", err)
dest := "archive/claude-sessions/" + b.Host + "/" + slug + ".md"
// Written through the brain root, never through a symlinked directory
// or file (hyperguild#158).
bfs, err := brainfs.Open(s.brainDir)
if err != nil {
return fmt.Errorf("open brain: %w", err)
}
if err := os.WriteFile(dest, []byte(sb.String()), 0o644); err != nil {
defer func() { _ = bfs.Close() }()
if err := bfs.PutRegular(dest, []byte(sb.String())); err != nil {
return fmt.Errorf("write claude session archive: %w", err)
}
s.logger.Debug("claude session archived (non-indexed)", "path", dest)
@@ -143,6 +149,48 @@ func envInt(key string, fallback int) int {
return fallback
}
// buildNotifier returns the ntfy notifier, or a nil interface when
// BRAIN_NTFY_URL is unset. A nil *interface* (not a typed-nil) is required so
// DegradingSink/Reconcile and the governance observer skip it cleanly.
// alertingWarnings names every alert that will not fire under the current
// configuration (#107).
//
// buildNotifier returns a nil interface when BRAIN_NTFY_URL is unset, and every
// consumer skips a nil notifier silently — right for an unconfigured homelab,
// wrong for an alert whose entire value is that someone finds out. The cost of
// the silence is that "configured and working" and "never configured" are
// byte-identical in behaviour: no error, no warning, only a message that never
// arrives, which nobody notices because nobody is waiting for an alert they
// have never seen. Two alerts were dark in production for exactly that reason
// (#100), found by firing a real event rather than by reading anything.
//
// WARN rather than fatal, deliberately. The brain sibling rule
// (`empty-credential-must-refuse-to-start-not-switch-modes`) says an empty
// credential must be a startup error, never a silent mode switch — but that is
// about a control being removed. Here observability is degraded, not a gate
// bypassed, and refusing to boot would break the unconfigured-homelab case the
// nil notifier exists to support. What the rule does demand, and what this
// provides, is that the mode switch stops being silent.
//
// Pure and env-injected so the message is testable without a running server:
// the reason this gap survived is that nothing about it was observable.
func alertingWarnings(getenv func(string) string) []string {
var out []string
if getenv("BRAIN_NTFY_URL") == "" {
out = append(out, "alerting DISABLED: BRAIN_NTFY_URL unset — "+
"brain_edit_root governance rewrites, capture audit degradation and the "+
"brain symlink tripwire (hyperguild#157) will not notify")
}
return out
}
func buildNotifier() audit.Notifier {
if n := audit.NewNtfyNotifier(os.Getenv("BRAIN_NTFY_URL"), os.Getenv("BRAIN_NTFY_TOKEN")); n != nil {
return n
}
return nil
}
// buildAuditSink selects the capture audit sink. When BRAIN_LOKI_URL is
// set it builds the classification-aware DegradingSink (loki central +
// durable file buffer + optional ntfy) and starts the reconcile loop;
@@ -160,12 +208,7 @@ func buildAuditSink(ctx context.Context, brainDir string, logger *slog.Logger) c
logger.Error("capture audit buffer init", "err", err)
os.Exit(1)
}
// Keep notifier as a nil interface (not a typed-nil) when unconfigured
// so DegradingSink/Reconcile skip it cleanly.
var notifier audit.Notifier
if n := audit.NewNtfyNotifier(os.Getenv("BRAIN_NTFY_URL"), os.Getenv("BRAIN_NTFY_TOKEN")); n != nil {
notifier = n
}
notifier := buildNotifier()
reconcileInterval := time.Duration(envInt("BRAIN_AUDIT_RECONCILE_INTERVAL", 60)) * time.Second
audit.StartReconcile(ctx, central, buffer, notifier, reconcileInterval)
logger.Info("capture audit: loki+buffer sink", "loki", lokiURL, "reconcile_s", int(reconcileInterval.Seconds()))
@@ -197,7 +240,15 @@ func systemHostname() string {
func main() {
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
for _, w := range alertingWarnings(os.Getenv) {
logger.Warn(w)
}
brainDir := envOr("INGEST_BRAIN_DIR", "../brain")
if err := startupSymlinkCheck(brainDir); err != nil {
logger.Error("brain symlink tripwire", "err", err)
os.Exit(1)
}
port := envOr("INGEST_PORT", "3300")
llmURL := envOr("INGEST_LLM_URL", "http://iguana:4000/v1")
@@ -230,6 +281,11 @@ func main() {
fallbackModel := envOr("BRAIN_LLM_FALLBACK_MODEL", "gemma4:31b")
router.Fallback = llm.New(fallbackURL, "", fallbackModel, timeout)
}
if tertiaryURL := os.Getenv("BRAIN_LLM_TERTIARY_URL"); tertiaryURL != "" {
tertiaryModel := envOr("BRAIN_LLM_TERTIARY_MODEL", "gemini-2.5-flash")
tertiaryKey := os.Getenv("GEMINI_API_KEY")
router.Tertiary = llm.New(tertiaryURL, tertiaryKey, tertiaryModel, timeout)
}
answerComplete = router.Complete
logger.Info("brain answer LLM configured", "primary", primaryURL, "model", primaryModel)
}
@@ -314,6 +370,26 @@ func main() {
}
ctx := context.Background()
// hyperguild#157: report a symlink that appears in the brain while running.
go symlinkguard.Watch(ctx, brainDir, symlinkScanInterval(os.Getenv), buildNotifier())
// Never fatal on an unreachable issuer (#83): on a cold cluster boot this
// pod can beat dex to serving and get a 502 from ingress, and exiting meant
// CrashLoopBackOff redoing the backfill. Since mcp-chassis v0.7 the
// validator resolves the issuer lazily and answers JWTs 503-and-retry until
// it does (hyperguild#155), so no boot retry is needed. A misconfigured
// issuer no longer exits the process: it is logged at boot and every JWT
// gets 503; the static bearer keeps working.
jwtValidator, err := buildJWTValidator(ctx, logger,
os.Getenv("DEX_ISSUER_URL"), os.Getenv("MCP_AUDIENCE"))
if err != nil {
logger.Error("build jwt validator", "err", err)
os.Exit(1)
}
if jwtValidator != nil {
logger.Info("jwt auth enabled", "issuer", os.Getenv("DEX_ISSUER_URL"))
}
if watchInterval > 0 {
watcher.Start(ctx, watcher.Config{
BrainDir: brainDir,
@@ -355,15 +431,39 @@ func main() {
}
host := envOr("CLAUDE_INGEST_HOST", systemHostname())
interval := time.Duration(envInt("CLAUDE_INGEST_INTERVAL", 60)) * time.Second
// Project directories excluded from ingestion entirely, as globs
// against the project directory name — e.g.
// "*cobalt-dingo*,*claude-worktrees-bridge-cse*".
//
// This is a veto list, deliberately named as one. It complements the
// client-name scrubber rather than replacing it: the scrubber drops
// individual TURNS whose content matches, while this never opens a
// project's files at all. Sessions working on live company financials
// are excluded here because their content is confidential in ways no
// pattern reliably recognises, and their learnings are captured
// deliberately through the session-close ritual instead.
//
// Caveat worth knowing: a project directory is derived from the
// session's WORKING DIRECTORY, not from the repositories it touches.
// A session run from ~/dev that happens to read a skipped repo will
// NOT be excluded by this. The scrubber remains the control for
// content; this is a control for whole workspaces.
skipProjects := splitList(os.Getenv("CLAUDE_INGEST_SKIP_PROJECTS"))
if len(skipProjects) > 0 {
logger.Info("claudewatcher: projects excluded from ingestion", "patterns", skipProjects)
}
sink := &claudeSink{brainDir: brainDir, logger: logger}
go func() {
if err := claudewatcher.Watch(ctx, claudewatcher.Config{
SessionsDir: claudeDir,
Host: host,
Interval: interval,
Sink: sink,
Cursors: cursorStore,
Logger: logger,
SessionsDir: claudeDir,
Host: host,
Interval: interval,
Sink: sink,
Cursors: cursorStore,
Logger: logger,
SkipProjects: skipProjects,
}); err != nil && err != context.Canceled {
logger.Error("claudewatcher exited", "err", err)
}
@@ -403,45 +503,24 @@ func main() {
logger.Info("embed sync started", "interval_s", embedSyncInterval)
}
mux := http.NewServeMux()
mux.HandleFunc("POST /query", h.Query)
mux.HandleFunc("POST /write", h.Write)
mux.HandleFunc("POST /index", h.Index)
mux.HandleFunc("POST /ingest", h.Ingest)
mux.HandleFunc("POST /ingest-path", h.IngestPath)
mux.HandleFunc("POST /ingest-raw", h.IngestRaw)
mux.HandleFunc("POST /backfill-refs", h.BackfillRefs)
mux.HandleFunc("GET /pending", h.Pending)
mux.HandleFunc("POST /promote", h.Promote)
mux.HandleFunc("POST /backfill-embeddings", h.BackfillEmbeddings)
mux.HandleFunc("GET /pass-rate", h.PassRate)
// Public routes only; buildMux mounts the authenticated REST API (#149).
public := map[string]http.Handler{}
if webhookHandler != nil {
mux.Handle("POST /webhooks/brain-sync", webhookHandler)
public["POST /webhooks/brain-sync"] = webhookHandler
}
jwtValidator, err := chassisauth.NewJWTValidator(ctx, os.Getenv("DEX_ISSUER_URL"), os.Getenv("MCP_AUDIENCE"))
if err != nil {
logger.Error("build jwt validator", "err", err)
os.Exit(1)
}
if jwtValidator != nil {
logger.Info("jwt auth enabled", "issuer", os.Getenv("DEX_ISSUER_URL"))
}
// Resource-metadata URL is only emitted on 401 when Dex OAuth is
// configured. Static-Bearer-only deployments leave this empty so
// clients never see an OAuth challenge.
var resourceMetadataURL string
if dexURL := os.Getenv("DEX_ISSUER_URL"); dexURL != "" {
resourceURL := os.Getenv("MCP_RESOURCE_URL")
mux.HandleFunc("GET /.well-known/oauth-protected-resource",
public["GET /.well-known/oauth-protected-resource"] = http.HandlerFunc(
chassisauth.ProtectedResourceHandler(resourceURL, dexURL))
if resourceURL != "" {
resourceMetadataURL = strings.TrimRight(resourceURL, "/") + "/.well-known/oauth-protected-resource"
}
}
mux.Handle("/mcp", chassisauth.BearerMiddleware(mcpToken, jwtValidator, "brain", resourceMetadataURL, mcpSrv))
// POST /capture (#53/#54): the uniform capture REST door. Needs a ticket
// tracker to file action items, so it only mounts when Gitea is
// configured. It reuses the MCP server's graph-wired brain store (one
@@ -457,22 +536,24 @@ func main() {
os.Exit(1)
}
auditSink := buildAuditSink(ctx, brainDir, logger)
// The Gitea client also satisfies SummaryWriter (#66): session
// summaries are written to mathias/ai-sessions over the same API
// token. nil only if a future tracker impl lacks file writes.
summaryWriter, _ := tracker.(capture.SummaryWriter)
captureSvc := capture.NewService(
mcpSrv.BrainStore(), tracker, summaryWriter, classCfg, auditSink)
mcpSrv.BrainStore(), tracker, classCfg, auditSink)
sovereign := splitList(os.Getenv("BRAIN_CAPTURE_SOVEREIGN_PRINCIPALS"))
resolver := capturehttp.NewOriginResolver(sovereign)
captureH := capturehttp.New(captureSvc, jwtValidator, mcpToken, "local-cli", resolver)
mux.Handle("POST /capture", captureH)
public["POST /capture"] = captureH
// Same use-case behind the MCP `capture` tool (#55 relay) so MCP-native
// harnesses (claude.ai, Crush, Pi, LLM Council) reach capture through
// the existing /mcp OAuth connector. mcpSrv is already wrapped above;
// the existing /mcp OAuth connector. buildMux wraps this same mcpSrv;
// WithCapture mutates the same instance, so the tool appears live.
mcpSrv.WithCapture(captureSvc, jwtValidator, mcpToken, "local-cli", resolver)
logger.Info("capture enabled (REST + MCP tool)", "sovereign_principals", len(sovereign))
// hyperguild#97: a governance-file rewrite gets the same trail a
// capture already gets, plus an out-of-band alert. Wired here because
// brain_edit_root needs the capture plumbing anyway — that is what
// re-derives the static principal its gate checks.
mcpSrv.WithGovernanceObserver(auditSink, buildNotifier())
logger.Info("capture enabled (REST + MCP tool)", "sovereign_principals", len(sovereign),
"governance_observer", true)
} else {
logger.Info("capture endpoint disabled (BRAIN_GITEA_TOKEN unset)")
}
@@ -490,9 +571,9 @@ func main() {
logger.Error("OAUTH_CLIENT_ID/SECRET set but MCP_RESOURCE_URL is empty; cannot derive issuer")
os.Exit(1)
}
mux.HandleFunc("GET /.well-known/oauth-authorization-server",
public["GET /.well-known/oauth-authorization-server"] = http.HandlerFunc(
oauth.MetadataHandler(issuer))
mux.HandleFunc("POST /oauth/token", oauth.TokenHandler(oauth.TokenConfig{
public["POST /oauth/token"] = http.HandlerFunc(oauth.TokenHandler(oauth.TokenConfig{
ClientID: oauthID,
ClientSecret: oauthSecret,
AccessToken: mcpToken,
@@ -505,13 +586,22 @@ func main() {
os.Exit(1)
}
// /metrics — unauthenticated Prometheus endpoint. kube-prometheus-stack
// scrapes it via the ServiceMonitor in k3s/apps/supervisor/. The metrics
// middleware below wraps every other registered handler so it observes
// real request latency. /metrics itself is excluded from its own
// observation by registering it on the outer mux (post-wrap).
reg := metrics.New()
mux.HandleFunc("GET /metrics", reg.Handler())
// /metrics — unauthenticated Prometheus endpoint, scraped via the
// ServiceMonitor in k3s/apps/supervisor/. buildMux registers it and wraps
// the whole router in the registry's middleware.
//
// The whole router, /mcp's bearer included, is built in buildMux so its
// tests are tests of production wiring (PR #152 review, hyperguild#154).
// The REST API's credential is the same as /mcp's. Untested here: the
// restAuth fields and the public map this call passes in (e.g. a nil
// validator would drop JWT auth on /mcp and REST and the suite stays
// green), and ListenAndServe serving the result unchanged (hyperguild#182).
handler, err := buildMux(restAuth{token: mcpToken, validator: jwtValidator, realm: "brain",
resourceMetadataURL: resourceMetadataURL}, h, mcpSrv, metrics.New(), public)
if err != nil {
logger.Error("build router", "err", err)
os.Exit(1)
}
logger.Info("metrics endpoint registered", "path", "/metrics")
addr := ":" + port
@@ -528,7 +618,7 @@ func main() {
"watch_interval", watchIntervalLog,
"mcp_enabled", true,
)
if err := http.ListenAndServe(addr, reg.Middleware(mux)); err != nil {
if err := http.ListenAndServe(addr, handler); err != nil {
logger.Error("server stopped", "err", err)
os.Exit(1)
}
+64
View File
@@ -6,6 +6,7 @@ import (
"log/slog"
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
@@ -36,3 +37,66 @@ func TestClaudeSink_IngestWritesToNonIndexedArchiveNotWiki(t *testing.T) {
_, err = os.Stat(filepath.Join(dir, "wiki", "claude-sessions"))
assert.True(t, os.IsNotExist(err), "raw transcripts must never land under wiki/ (ai-sessions#10 — BM25 pollution)")
}
// --- #107: an unconfigured alert must announce itself ----------------------
func TestAlertingWarnings(t *testing.T) {
tests := []struct {
name string
env map[string]string
want []string // substrings that must appear across the warnings
none bool // true = expect no warnings at all
}{
{
// The failure this exists to prevent: two alerts dark in
// production, discovered only by firing a real event and watching
// nothing arrive (#100).
name: "notifier unconfigured names what will not fire",
env: map[string]string{},
want: []string{"BRAIN_NTFY_URL", "brain_edit_root", "audit"},
},
{
name: "configured notifier is silent",
env: map[string]string{"BRAIN_NTFY_URL": "http://ntfy/brain-alerts"},
none: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := alertingWarnings(func(k string) string { return tt.env[k] })
if tt.none {
assert.Empty(t, got)
return
}
joined := strings.Join(got, " ")
for _, w := range tt.want {
assert.Contains(t, joined, w)
}
})
}
}
// hyperguild#158: the claude-session archive was written with os.MkdirAll +
// os.WriteFile by path, so a symlinked archive directory redirected it.
func TestClaudeSinkDoesNotWriteThroughASymlinkedArchiveDir(t *testing.T) {
for name, mk := range map[string]func(t *testing.T, dir string) (link, target string){
"outside": func(t *testing.T, _ string) (string, string) { o := t.TempDir(); return o, o },
"in-brain": func(t *testing.T, dir string) (string, string) {
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw"), 0o755))
return filepath.Join("..", "raw"), filepath.Join(dir, "raw")
},
} {
t.Run(name, func(t *testing.T) {
dir := t.TempDir()
link, target := mk(t, dir)
require.NoError(t, os.Symlink(link, filepath.Join(dir, "archive")))
sink := &claudeSink{brainDir: dir, logger: slog.New(slog.NewTextHandler(os.Stderr, nil))}
err := sink.Ingest(context.Background(), claudewatcher.Batch{Host: "koala", SessionID: "abc",
Turns: []claudewatcher.Turn{{Type: "assistant", Content: "x"}}})
assert.Error(t, err)
ents, _ := os.ReadDir(target)
assert.Empty(t, ents, "the archive was written at the link's target")
})
}
}
+109
View File
@@ -0,0 +1,109 @@
package main
import (
"errors"
"fmt"
"net/http"
chassisauth "git.d-ma.be/mathias/mcp-chassis/auth"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/metrics"
)
// publicRoutes is the complete set of patterns a caller may register on the
// outer mux, outside the REST API's auth (hyperguild#149). Each authenticates
// itself or is meant to be open: /capture (bearer/JWT), the webhook (HMAC),
// /oauth/token (client secret) and OAuth metadata (open). /mcp and /metrics
// are not here: buildMux builds them itself (hyperguild#154). A REST pattern
// added here would beat the authenticated catch-all by specificity and ship
// open, which is why buildMux refuses anything else.
var publicRoutes = map[string]bool{
"POST /capture": true,
"POST /webhooks/brain-sync": true,
"GET /.well-known/oauth-protected-resource": true,
"GET /.well-known/oauth-authorization-server": true,
"POST /oauth/token": true,
}
// restAuth is the credential the REST API checks: the same as /mcp's.
type restAuth struct {
token string
validator *chassisauth.JWTValidator
realm string
resourceMetadataURL string
}
// buildMux builds the handler main passes to ListenAndServe unchanged. The
// REST API is mounted at "/" and /mcp is registered, both behind bearer auth
// that buildMux constructs itself from auth (one credential), so a caller
// cannot pass a no-op. GET /metrics serves reg, and the whole router is
// wrapped in reg's middleware. Other public handlers are registered as given,
// and only for patterns on the publicRoutes allowlist. It returns an
// http.Handler, not the *ServeMux, so nothing can register a route on the
// result. What stays untested is main's call site: the restAuth fields and
// the public map main passes in, and ListenAndServe serving this handler
// unchanged (hyperguild#182). The tests build their own arguments.
func buildMux(
auth restAuth, h *api.Handler, mcpHandler http.Handler, reg *metrics.Registry, public map[string]http.Handler,
) (http.Handler, error) {
if auth.token == "" {
return nil, errors.New("buildMux: REST auth needs a token")
}
if mcpHandler == nil || reg == nil {
return nil, errors.New("buildMux: needs the MCP handler and the metrics registry")
}
mux := http.NewServeMux()
for pattern, handler := range public {
if !publicRoutes[pattern] {
return nil, fmt.Errorf("buildMux: %q is not an allowed public route", pattern)
}
mux.Handle(pattern, handler)
}
bearer := func(next http.Handler) http.Handler {
return chassisauth.BearerMiddleware(auth.token, auth.validator, auth.realm, auth.resourceMetadataURL, next)
}
mux.Handle("/mcp", bearer(mcpHandler))
// /metrics is registered on the mux and so observed by its own
// middleware, like any route.
mux.Handle("GET /metrics", reg.Handler())
mountREST(mux, h, bearer)
return reg.Middleware(mux), nil
}
// mountREST registers the REST API on its own mux and mounts it at "/" behind
// auth: anything not registered as public lands here and is authenticated,
// including routes added later.
func mountREST(mux *http.ServeMux, h *api.Handler, auth func(http.Handler) http.Handler) {
rest := http.NewServeMux()
rest.HandleFunc("POST /query", h.Query)
rest.HandleFunc("POST /write", h.Write)
rest.HandleFunc("POST /index", h.Index)
rest.HandleFunc("POST /ingest", h.Ingest)
rest.HandleFunc("POST /ingest-path", h.IngestPath)
rest.HandleFunc("POST /ingest-raw", h.IngestRaw)
rest.HandleFunc("POST /backfill-refs", h.BackfillRefs)
rest.HandleFunc("GET /pending", h.Pending)
rest.HandleFunc("POST /promote", h.Promote)
rest.HandleFunc("POST /backfill-embeddings", h.BackfillEmbeddings)
rest.HandleFunc("GET /pass-rate", h.PassRate)
mux.Handle("/", labelRESTPattern(rest, auth(rest)))
}
// labelRESTPattern sets the request's Pattern to the REST route it addresses
// before auth runs, so the metrics middleware (which reads Pattern from the
// same *http.Request) labels a refused request by its route rather than by
// the "/" mount (hyperguild#155). A refused request to a path no route
// matches keeps "/", so for refused requests the label set stays bounded by
// the registered routes. That bound does not hold for authenticated
// requests: an unregistered path reaches the REST mux, which 404s with an
// empty Pattern, and the metrics middleware falls back to the raw URL path
// (hyperguild#185).
func labelRESTPattern(rest *http.ServeMux, next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if _, pattern := rest.Handler(r); pattern != "" {
r.Pattern = pattern
}
next.ServeHTTP(w, r)
})
}
@@ -0,0 +1,88 @@
package main
import (
"bytes"
"log/slog"
"net/http"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// wrongTokSentinel is a presented credential distinctive enough that any
// occurrence in a log line is the raw token, not a coincidence.
const wrongTokSentinel = "wrong-tok-sentinel-q7zx"
// hyperguild#155: since #149 every REST route sits behind BearerMiddleware,
// and chassis v0.2.0 refused without logging, so an unknown caller got
// silent 401s. The chassis now writes an `mcp auth rejected` line, and the
// presented token appears nowhere in the log as a bare value (PR #181
// review: asserting on the "Bearer " header form could never fail, since
// the chassis logs fields, not headers).
func TestBuildMuxRejectionLogLineOmitsTheBarePresentedToken(t *testing.T) {
var buf bytes.Buffer
prev := slog.Default()
slog.SetDefault(slog.New(slog.NewTextHandler(&buf, nil)))
t.Cleanup(func() { slog.SetDefault(prev) })
code, _ := do(t, testMux(t), "POST", "/query", "Bearer "+wrongTokSentinel)
require.Equal(t, http.StatusUnauthorized, code)
assert.Contains(t, buf.String(), "mcp auth rejected")
assert.Contains(t, buf.String(), "path=/query")
assert.NotContains(t, buf.String(), wrongTokSentinel, "the rejected token is logged as a bare value")
assert.NotContains(t, buf.String(), routesTok, "the valid static token is logged")
}
// The `mcp auth accepted` line leaves out the valid static token.
func TestBuildMuxAcceptedLogLineOmitsTheBareStaticToken(t *testing.T) {
var buf bytes.Buffer
prev := slog.Default()
slog.SetDefault(slog.New(slog.NewTextHandler(&buf, nil)))
t.Cleanup(func() { slog.SetDefault(prev) })
code, _ := do(t, testMux(t), "GET", "/pending", "Bearer "+routesTok)
require.Equal(t, http.StatusOK, code)
assert.Contains(t, buf.String(), "mcp auth accepted")
assert.NotContains(t, buf.String(), routesTok, "the accepted static token is logged as a bare value")
}
// An authenticated REST request is labelled with the REST mux's pattern:
// the inner mux sets it on the same *http.Request the outer middleware reads.
func TestMetricsLabelAnAuthenticatedRESTRequestWithItsRoutePattern(t *testing.T) {
mux, reg := testMuxAndRegistry(t)
code, _ := do(t, mux, "GET", "/pending", "Bearer "+routesTok)
require.Equal(t, http.StatusOK, code)
rec := httptest.NewRecorder()
reg.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/metrics", nil))
assert.Contains(t, rec.Body.String(), `path="GET /pending"`)
assert.NotContains(t, rec.Body.String(), `path="/"`)
}
// hyperguild#155: a request refused by REST auth never reaches the REST mux,
// so it was labelled path="/" and a 4xx on /query could not be told from one
// on /write. It is now labelled with the REST pattern it was addressed to.
func TestMetricsLabelARejectedRESTRequestWithItsRoutePattern(t *testing.T) {
mux, reg := testMuxAndRegistry(t)
code, _ := do(t, mux, "POST", "/query", "")
require.Equal(t, http.StatusUnauthorized, code)
rec := httptest.NewRecorder()
reg.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/metrics", nil))
assert.Contains(t, rec.Body.String(), `path="POST /query",status="4xx"`)
}
// A path no REST route matches keeps the catch-all label, so an attacker
// probing random URLs cannot grow the series set.
func TestMetricsLabelARejectedUnknownPathWithTheCatchAll(t *testing.T) {
mux, reg := testMuxAndRegistry(t)
code, _ := do(t, mux, "POST", "/no-such-route-zq", "")
require.Equal(t, http.StatusUnauthorized, code)
rec := httptest.NewRecorder()
reg.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/metrics", nil))
assert.Contains(t, rec.Body.String(), `path="/",status="4xx"`)
assert.NotContains(t, rec.Body.String(), "no-such-route-zq")
}
+161
View File
@@ -0,0 +1,161 @@
package main
import (
"log/slog"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/metrics"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
)
const routesTok = "routes-test-token"
func okHandler(body string) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte(body)) })
}
// testMux builds the router through buildMux, the function main calls, with
// every allowlisted public pattern (stub handlers), a stub MCP server and a
// static token only: no JWT validator and no resourceMetadataURL (the tests
// in jwt_validator_test.go pass those). It does not use main's own restAuth
// or public map (hyperguild#182). hyperguild#149: the REST API is
// authenticated as a whole; the PR #152 review showed a test of a COPY of
// the router let main.go regress unseen.
func testMux(t *testing.T) http.Handler {
t.Helper()
mux, _ := testMuxAndRegistry(t)
return mux
}
func testMuxAndRegistry(t *testing.T) (http.Handler, *metrics.Registry) {
t.Helper()
public := map[string]http.Handler{}
for p := range publicRoutes {
public[p] = okHandler("public " + p)
}
reg := metrics.New()
mux, err := buildMux(restAuth{token: routesTok, realm: "brain"}, testHandler(t), okHandler("mcp"), reg, public)
require.NoError(t, err)
return mux, reg
}
func testHandler(t *testing.T) *api.Handler {
t.Helper()
return api.NewHandler(t.TempDir(), slog.New(slog.NewTextHandler(os.Stderr, nil)), pipeline.Config{})
}
func do(t *testing.T, h http.Handler, method, path, authz string) (int, string) {
t.Helper()
req := httptest.NewRequest(method, path, strings.NewReader(`{"query":"x"}`))
if authz != "" {
req.Header.Set("Authorization", authz)
}
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
return rec.Code, rec.Body.String()
}
func TestBuildMuxRefusesRESTRoutesWithoutAValidToken(t *testing.T) {
r := testMux(t)
for _, rt := range []struct{ method, path string }{
{"POST", "/query"}, {"POST", "/write"}, {"POST", "/index"}, {"POST", "/ingest"},
{"POST", "/ingest-path"}, {"POST", "/ingest-raw"}, {"POST", "/backfill-refs"},
{"GET", "/pending"}, {"POST", "/promote"}, {"POST", "/backfill-embeddings"}, {"GET", "/pass-rate"},
} {
code, _ := do(t, r, rt.method, rt.path, "")
assert.Equal(t, http.StatusUnauthorized, code, "%s %s without a token", rt.method, rt.path)
code, _ = do(t, r, rt.method, rt.path, "Bearer wrong")
assert.Equal(t, http.StatusUnauthorized, code, "%s %s with a wrong token", rt.method, rt.path)
}
}
func TestBuildMuxServesRESTRoutesToAValidToken(t *testing.T) {
r := testMux(t)
code, _ := do(t, r, "GET", "/pending", "Bearer "+routesTok)
assert.Equal(t, http.StatusOK, code)
}
func TestBuildMuxAuthenticatesAnUnregisteredPath(t *testing.T) {
code, _ := do(t, testMux(t), "POST", "/some-future-route", "")
assert.Equal(t, http.StatusUnauthorized, code)
}
// buildMux registers public handlers as given, without adding auth: each
// public route owns its own (bearer/JWT on /capture, HMAC on the webhook,
// client secret on /oauth/token) or is meant to be open (OAuth metadata).
// /mcp and /metrics are not public routes: buildMux builds them itself.
func TestBuildMuxDoesNotWrapPublicRoutes(t *testing.T) {
r := testMux(t)
for p := range publicRoutes {
method, path, ok := strings.Cut(p, " ")
if !ok {
method, path = "GET", p
}
code, body := do(t, r, method, path, "")
assert.Equal(t, http.StatusOK, code, p)
assert.Equal(t, "public "+p, body, p)
}
}
// The allowlist is the control: a REST pattern registered as public would win
// over the authenticated catch-all (more specific pattern), so it is refused.
func TestBuildMuxRefusesAPublicPatternOffTheAllowlist(t *testing.T) {
h := api.NewHandler(t.TempDir(), slog.New(slog.NewTextHandler(os.Stderr, nil)), pipeline.Config{})
// "/mcp" and "GET /metrics" are built by buildMux itself (hyperguild#154):
// passing either as public would register them unauthenticated or twice.
for _, p := range []string{"POST /query", "GET /pending", "/", "POST /metrics", "/mcp", "GET /metrics"} {
_, err := buildMux(restAuth{token: routesTok, realm: "brain"}, h, okHandler("mcp"), metrics.New(),
map[string]http.Handler{p: okHandler("x")})
assert.Error(t, err, "public pattern %q must be refused", p)
}
}
func TestBuildMuxRefusesAnEmptyToken(t *testing.T) {
h := api.NewHandler(t.TempDir(), slog.New(slog.NewTextHandler(os.Stderr, nil)), pipeline.Config{})
_, err := buildMux(restAuth{token: "", realm: "brain"}, h, okHandler("mcp"), metrics.New(), nil)
assert.Error(t, err)
}
// hyperguild#154: /mcp's bearer was built in main.go, untested; the buildMux
// tests used stubs. buildMux now wraps the MCP handler itself, with the same
// credential as the REST API.
func TestBuildMuxRefusesMCPWithoutAValidToken(t *testing.T) {
r := testMux(t)
for _, authz := range []string{"", "Bearer wrong"} {
code, body := do(t, r, "POST", "/mcp", authz)
assert.Equal(t, http.StatusUnauthorized, code, "authz %q", authz)
assert.NotEqual(t, "mcp", body)
}
code, body := do(t, r, "POST", "/mcp", "Bearer "+routesTok)
assert.Equal(t, http.StatusOK, code)
assert.Equal(t, "mcp", body)
}
func TestBuildMuxRefusesANilMCPHandlerOrRegistry(t *testing.T) {
auth := restAuth{token: routesTok, realm: "brain"}
_, err := buildMux(auth, testHandler(t), nil, metrics.New(), nil)
assert.Error(t, err)
_, err = buildMux(auth, testHandler(t), okHandler("mcp"), nil, nil)
assert.Error(t, err)
}
// hyperguild#154: main wrapped buildMux's result in the metrics middleware
// before ListenAndServe, untested. buildMux now returns the handler main
// serves as-is, metrics included: a request through it is observed and
// exposed on its own GET /metrics.
func TestBuildMuxObservesRequestsAndServesMetrics(t *testing.T) {
r := testMux(t)
code, _ := do(t, r, "GET", "/pending", "Bearer "+routesTok)
require.Equal(t, http.StatusOK, code)
code, body := do(t, r, "GET", "/metrics", "")
require.Equal(t, http.StatusOK, code)
assert.Contains(t, body, `path="GET /pending"`)
}
+39
View File
@@ -0,0 +1,39 @@
package main
import (
"errors"
"fmt"
"os"
"strconv"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/symlinkguard"
)
// startupSymlinkCheck is the hyperguild#157 tripwire at boot: a symlink in the
// brain dir would let brain readers serve a file from outside the brain, so
// the server does not start until it is removed. A directory the scan cannot
// read could hide one, so it refuses on that too. A missing brain dir (first
// boot) is the only case it lets pass.
func startupSymlinkCheck(brainDir string) error {
r, err := symlinkguard.Scan(brainDir)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return nil
}
return fmt.Errorf("refusing to start: cannot scan the brain dir %s (hyperguild#157): %w", brainDir, err)
}
if !r.Clean() {
return fmt.Errorf("refusing to start: brain dir %s has %s (hyperguild#157); fix them first", brainDir, r)
}
return nil
}
// symlinkScanInterval reads BRAIN_SYMLINK_SCAN_INTERVAL (seconds), falling
// back to the default on an unset, non-numeric or non-positive value.
func symlinkScanInterval(getenv func(string) string) time.Duration {
if s, err := strconv.Atoi(getenv("BRAIN_SYMLINK_SCAN_INTERVAL")); err == nil && s > 0 {
return time.Duration(s) * time.Second
}
return symlinkguard.DefaultInterval
}
@@ -0,0 +1,76 @@
package main
import (
"os"
"path/filepath"
"strings"
"syscall"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// hyperguild#157: the server refuses to start while a symlink exists in the
// brain dir, or while any part of it cannot be scanned (which could hide
// one); a missing brain dir is the only thing it lets pass.
func TestStartupSymlinkCheckRefusesABrainWithASymlink(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "planted")))
err := startupSymlinkCheck(brain)
require.Error(t, err)
assert.Contains(t, err.Error(), "planted")
}
func TestStartupSymlinkCheckRefusesABrainWithAnUnreadableDir(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root reads mode-000 directories")
}
brain := t.TempDir()
locked := filepath.Join(brain, "aaa")
require.NoError(t, os.MkdirAll(locked, 0o755))
require.NoError(t, os.Chmod(locked, 0o000))
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
err := startupSymlinkCheck(brain)
require.Error(t, err)
assert.Contains(t, err.Error(), "aaa")
}
func TestStartupSymlinkCheckPassesACleanOrMissingBrain(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(brain, "n.md"), []byte("x"), 0o644))
assert.NoError(t, startupSymlinkCheck(brain))
assert.NoError(t, startupSymlinkCheck(filepath.Join(brain, "missing")))
}
func TestSymlinkScanIntervalFallsBackOnNonPositive(t *testing.T) {
for _, v := range []string{"0", "-5", "", "junk"} {
got := symlinkScanInterval(func(string) string { return v })
assert.Positive(t, got, "env %q", v)
}
}
func TestAlertingWarningsNamesTheSymlinkTripwire(t *testing.T) {
w := strings.Join(alertingWarnings(func(string) string { return "" }), " ")
assert.Contains(t, w, "symlink tripwire")
}
func TestStartupSymlinkCheckRefusesAnUnlistableBrainRoot(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root lists mode-0100 directories")
}
brain := t.TempDir()
require.NoError(t, os.Chmod(brain, 0o100))
t.Cleanup(func() { _ = os.Chmod(brain, 0o755) })
assert.Error(t, startupSymlinkCheck(brain))
}
// hyperguild#179: a FIFO in the brain is refused at startup with a clear
// message, instead of hanging or exiting later in a reader.
func TestStartupSymlinkCheckRefusesABrainWithAFIFO(t *testing.T) {
brain := t.TempDir()
require.NoError(t, syscall.Mkfifo(filepath.Join(brain, "classification.yaml"), 0o644))
err := startupSymlinkCheck(brain)
require.Error(t, err)
assert.Contains(t, err.Error(), "classification.yaml")
}
+7 -7
View File
@@ -34,9 +34,9 @@ require (
github.com/rogpeppe/go-internal v1.15.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/x448/float16 v0.8.4 // indirect
golang.org/x/net v0.26.0 // indirect
golang.org/x/net v0.55.0 // indirect
golang.org/x/oauth2 v0.21.0 // indirect
golang.org/x/term v0.28.0 // indirect
golang.org/x/term v0.43.0 // indirect
golang.org/x/time v0.3.0 // indirect
google.golang.org/protobuf v1.34.2 // indirect
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
@@ -51,7 +51,7 @@ require (
)
require (
git.d-ma.be/mathias/mcp-chassis v0.2.0
git.d-ma.be/mathias/mcp-chassis v0.7.1
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 // indirect
github.com/goccy/go-json v0.10.3 // indirect
@@ -66,9 +66,9 @@ require (
github.com/lestrrat-go/option v1.0.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/segmentio/asm v1.2.0 // indirect
golang.org/x/crypto v0.32.0 // indirect
golang.org/x/sync v0.17.0 // indirect
golang.org/x/sys v0.31.0 // indirect
golang.org/x/text v0.29.0 // indirect
golang.org/x/crypto v0.51.0 // indirect
golang.org/x/sync v0.21.0 // indirect
golang.org/x/sys v0.45.0
golang.org/x/text v0.39.0 // indirect
gopkg.in/yaml.v3 v3.0.1
)
+16 -16
View File
@@ -1,5 +1,5 @@
git.d-ma.be/mathias/mcp-chassis v0.2.0 h1:6fLmb7xqRa2nNVWsHaUbbfbArgDXJw/gDhb09clBIjo=
git.d-ma.be/mathias/mcp-chassis v0.2.0/go.mod h1:Ks7EK2UnGAN0H3rJjKUxUagX8/ZBdtLrOlcUbv0RwH8=
git.d-ma.be/mathias/mcp-chassis v0.7.1 h1:4h8TOBHTA0L/no4IPcr4tLcQtmKUlhE2/cXyPdNFGmM=
git.d-ma.be/mathias/mcp-chassis v0.7.1/go.mod h1:Ks7EK2UnGAN0H3rJjKUxUagX8/ZBdtLrOlcUbv0RwH8=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
@@ -117,42 +117,42 @@ github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9dec
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.32.0 h1:euUpcYgM8WcP71gNpTqQCn6rC2t6ULUPiOzfWaXVVfc=
golang.org/x/crypto v0.32.0/go.mod h1:ZnnJkOaASj8g0AjIduWNlq2NRxL0PlBrbKVyZ6V/Ugc=
golang.org/x/crypto v0.51.0 h1:IBPXwPfKxY7cWQZ38ZCIRPI50YLeevDLlLnyC5wRGTI=
golang.org/x/crypto v0.51.0/go.mod h1:8AdwkbraGNABw2kOX6YFPs3WM22XqI4EXEd8g+x7Oc8=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.26.0 h1:soB7SVo0PWrY4vPW/+ay0jKDNScG2X9wFeYlXIvJsOQ=
golang.org/x/net v0.26.0/go.mod h1:5YKkiSynbBIh3p6iOc/vibscux0x38BZDkn8sCUPxHE=
golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
golang.org/x/oauth2 v0.21.0 h1:tsimM75w1tF/uws5rbeHzIWxEqElMehnc+iW793zsZs=
golang.org/x/oauth2 v0.21.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug=
golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/term v0.28.0 h1:/Ts8HFuMR2E6IP/jlo7QVLZHggjKQbhu/7H0LJFr3Gg=
golang.org/x/term v0.28.0/go.mod h1:Sw/lC2IAUZ92udQNf3WodGtn4k/XoLyZoh8v/8uiwek=
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4=
golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk=
golang.org/x/text v0.29.0/go.mod h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4=
golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus=
golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM=
golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4=
golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.36.0 h1:kWS0uv/zsvHEle1LbV5LE8QujrxB3wfQyxHfhOk0Qkg=
golang.org/x/tools v0.36.0/go.mod h1:WBDiHKJK8YgLHlcQPYQzNCkUxUypCaa5ZegCVutKm+s=
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+44 -12
View File
@@ -1,6 +1,12 @@
package api
import "strings"
import (
"strings"
// Aliased: this package's own frontmatter type (below) would collide
// with the imported package's name otherwise.
sharedfm "github.com/mathiasbq/hyperguild/ingestion/internal/frontmatter"
)
// frontmatter is an ordered, line-preserving view of a note's YAML
// frontmatter block. It deliberately avoids a full YAML round-trip: the
@@ -25,21 +31,37 @@ type fmLine struct {
// frontmatter block is recognised only when the file opens with a `---`
// fence and a closing `---` fence follows. Otherwise the whole input is
// the body and the returned frontmatter is empty.
//
// Fence detection itself is frontmatter.Split (hyperguild#138) — this
// package's own contribution is purely the ordered, line-preserving fmLine
// shape below, which a shared parser has no business owning since it's
// specific to how brain_update does a surgical, byte-round-trip edit.
func parseFrontmatter(src string) (frontmatter, string) {
var fm frontmatter
if !strings.HasPrefix(src, "---\n") {
blockBytes, restBytes, ok := sharedfm.Split([]byte(src))
if !ok {
return fm, src
}
rest := src[len("---\n"):]
end := strings.Index(rest, "\n---\n")
if end < 0 {
// Opening fence with no closing fence — treat as bodyless content.
return fm, src
}
block := rest[:end]
body := rest[end+len("\n---\n"):]
// sharedfm.Split's block/rest boundary sits one byte earlier than this
// package's always has: block keeps the "\n" immediately before the
// closing fence, and rest keeps the "\n" immediately after it (both by
// design, for Fix's own reconstruction). This package's block/body
// never included either — TrimSuffix/TrimPrefix restore that exact
// boundary. Split guarantees ok=true only when "\n---\n" matched, so
// both trims are always exactly one byte, never ambiguous.
block := strings.TrimSuffix(string(blockBytes), "\n")
body := strings.TrimPrefix(string(restBytes), "\n")
for _, line := range strings.Split(block, "\n") {
// An indented line is a continuation (a YAML block-list item, a
// wrapped value) never a top-level key, regardless of whether it
// happens to contain a colon — " - 'Chapter 3: The Reckoning'" is
// not a key named "- 'Chapter 3" (hyperguild#128). Checked before
// Cut so a colon inside one never reaches the key/value path.
if line != "" && (line[0] == ' ' || line[0] == '\t') {
fm.lines = append(fm.lines, fmLine{raw: line})
continue
}
key, val, ok := strings.Cut(line, ":")
key = strings.TrimSpace(key)
if !ok || key == "" || strings.HasPrefix(strings.TrimSpace(line), "#") {
@@ -83,9 +105,19 @@ func (f *frontmatter) render() string {
var b strings.Builder
b.WriteString("---\n")
for _, l := range f.lines {
if l.key == "" {
switch {
case l.key == "":
b.WriteString(l.raw)
} else {
case l.value == "":
// A key with nothing after the colon (the YAML block-list
// shape: "aliases:" followed by indented "- item" lines on
// their own raw lines) must stay bare — "key: " with a
// trailing space is a different byte sequence than the
// original, and this type's whole purpose (see the package
// doc) is a byte-preserving round-trip.
b.WriteString(l.key)
b.WriteByte(':')
default:
b.WriteString(l.key)
b.WriteString(": ")
b.WriteString(l.value)
@@ -59,3 +59,37 @@ func TestFrontmatterRenderRoundTrips(t *testing.T) {
fm, _ := parseFrontmatter(src)
assert.Equal(t, src, fm.render())
}
func TestParseFrontmatterIndentedListItemWithColonIsNotMisparsedAsKey(t *testing.T) {
// hyperguild#128: an indented YAML block-list item that happens to
// contain a colon (e.g. a title with a subtitle) is a continuation
// line, never a top-level key — strings.Cut doesn't know that just
// because TrimSpace(key) happens to be non-empty. Misparsing it as a
// key/value pair and re-rendering strips its indentation, de-indenting
// it out of the list it belongs to. Same bug class already found and
// fixed in the sibling internal/wiki package's parseFrontmatterBlock
// (hyperguild#126 review round 3) — this package was the original the
// reimplementation was modeled on and had the gap too.
src := "---\naliases:\n - 'Chapter 3: The Reckoning'\ntitle: 'X'\n---\nbody\n"
fm, _ := parseFrontmatter(src)
fm.set("updated_at", "2026-09-23T00:00:00Z")
out := fm.render()
assert.Contains(t, out, " - 'Chapter 3: The Reckoning'",
"the indented list item must survive verbatim, not get re-emitted as a de-indented fake key")
assert.NotContains(t, out, "\n- 'Chapter 3",
"must not be split into a fake key/value pair at column 0 (as opposed to surviving indented, asserted above)")
}
func TestFrontmatterRenderEmptyValueKeyStaysBareNoTrailingSpace(t *testing.T) {
// Companion to the indented-list-item fix: "aliases:" (a key with
// nothing after the colon, the YAML block-list header) must round-trip
// bare, not as "aliases: " with a trailing space — a different byte
// sequence than the original, breaking this type's whole reason for
// being a line-preserving view rather than a YAML re-marshal.
// render() returns only the frontmatter block (see
// TestFrontmatterRenderRoundTrips), no trailing body.
src := "---\naliases:\n - one\n - two\n---\n"
fm, _ := parseFrontmatter(src)
assert.Equal(t, src, fm.render())
}
+110 -83
View File
@@ -12,7 +12,8 @@ import (
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/extract"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"github.com/mathiasbq/hyperguild/ingestion/internal/claim"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
"github.com/mathiasbq/hyperguild/ingestion/internal/search"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
@@ -53,8 +54,6 @@ type queryRequest struct {
type writeRequest struct {
Content string `json:"content"`
Filename string `json:"filename,omitempty"`
Type string `json:"type,omitempty"`
Domain string `json:"domain,omitempty"`
Wing string `json:"wing,omitempty"`
Hall string `json:"hall,omitempty"`
SourceType string `json:"source_type,omitempty"` // "external" opts a hall=facts entry out of the internal default
@@ -109,20 +108,21 @@ func (h *Handler) Query(w http.ResponseWriter, r *http.Request) {
// WriteNoteOptions configures how a brain note is written.
//
// When both Wing and Hall are non-empty, the note routes into the
// structured wiki at brain/wiki/<wing>/<hall>/<slug>.md and gets
// wing/hall/created_at injected into its YAML frontmatter.
//
// When either is empty, the note falls back to brain/knowledge/<filename>
// with optional type/domain frontmatter (legacy behaviour).
// Wing and Hall are both required: the note routes into the structured
// wiki at brain/wiki/<wing>/<hall>/<slug>.md and gets wing/hall/created_at
// injected into its YAML frontmatter. There is no other valid destination
// (the legacy brain/knowledge/ fallback was removed — ADR-0012 / #79).
type WriteNoteOptions struct {
Content string
Filename string
Type string
Domain string
Wing string
Hall string
SourceType string // "internal" marks a first-party observation (e.g. claudewatcher) that needs no external citation
// Origin/WrittenBy are server-owned provenance (hyperguild#145): any
// origin/written_by in Content's own frontmatter is discarded, so a
// caller cannot claim a more trusted origin than it authenticated as.
Origin string
WrittenBy string
}
// WriteNote writes a markdown note into the brain. Returns the path
@@ -131,14 +131,21 @@ func WriteNote(brainDir string, opts WriteNoteOptions) (string, error) {
if opts.Content == "" {
return "", fmt.Errorf("content is required")
}
if opts.Wing == "" || opts.Hall == "" {
return "", fmt.Errorf("wing and hall are required")
}
if opts.Wing != "" && opts.Hall != "" {
return writeHallNote(brainDir, opts)
// In the primitive, not in a caller: POST /write reaches here directly,
// without the BrainStore, and a gate one door wide is not a gate.
// BrainStore still calls the strip itself to report Ref.Unlinked; the
// second pass is a no-op on already-clean content.
clean, _, err := brain.StripUnresolvedLinks(brainDir, opts.Content)
if err != nil {
return "", err
}
if opts.Wing != "" || opts.Hall != "" {
return "", fmt.Errorf("wing and hall must be set together")
}
return writeLegacyNote(brainDir, opts)
opts.Content = clean
return writeHallNote(brainDir, opts)
}
// writeHallNote routes a note into brain/wiki/<wing>/<hall>/ and injects
@@ -152,15 +159,21 @@ func writeHallNote(brainDir string, opts WriteNoteOptions) (string, error) {
if err != nil {
return "", err
}
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
return "", fmt.Errorf("create hall dir: %w", err)
}
existingFields, body := splitFrontmatter(opts.Content)
existingByKey := make(map[string]frontmatterField, len(existingFields))
for _, f := range existingFields {
existingByKey[f.key] = f
}
// Validate the claim-property envelope BEFORE creating the directory, so a
// rejected write leaves nothing behind — not even an empty hall dir
// (brain#21). Fail closed: a value outside the enum is worse than an absent
// one, because a lint skips what it cannot parse and the note then reads as
// classified while nothing acts on it.
if err := claim.ValidateEnvelope(scalarValues(existingFields)); err != nil {
return "", fmt.Errorf("claim envelope: %w", err)
}
emitted := make(map[string]bool, 6)
var fm strings.Builder
@@ -187,9 +200,6 @@ func writeHallNote(brainDir string, opts WriteNoteOptions) (string, error) {
fmt.Fprintf(&fm, "%s: %s\n", key, fallback)
}
}
writeField("type", opts.Type)
writeField("domain", opts.Domain)
sourceType := opts.SourceType
if sourceType == "" && opts.Hall == "facts" {
// Most hall=facts entries are first-party (an eval/benchmark the
@@ -211,11 +221,48 @@ func writeHallNote(brainDir string, opts WriteNoteOptions) (string, error) {
}
fm.WriteString("---\n")
if err := os.WriteFile(dest, []byte(fm.String()+body), 0o644); err != nil {
final, err := StampProvenance(fm.String()+body, opts.Origin, opts.WrittenBy)
if err != nil {
return "", err
}
// The directory is created only once the note is known to be writable:
// a refused write leaves nothing behind, not even an empty hall dir
// (brain#21; provenance refusal joins the envelope refusal here).
// Created through the brain root, never through a symlinked wing or
// hall directory (hyperguild#188).
rel, err := filepath.Rel(brainDir, dest)
if err != nil {
return "", fmt.Errorf("note path: %w", err)
}
rel = filepath.ToSlash(rel)
bfs, err := brainfs.Open(brainDir)
if err != nil {
return "", fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
if err := bfs.PutRegular(rel, []byte(final)); err != nil {
return "", fmt.Errorf("write: %w", err)
}
rel, _ := filepath.Rel(brainDir, dest)
return filepath.ToSlash(rel), nil
return rel, nil
}
// scalarValues flattens frontmatter fields to key -> scalar value for
// validation. Multi-line values (lists, nested blocks) yield the text after
// the colon on the first line, which is empty for a list — correct here,
// since every claim-property field is a scalar enum.
func scalarValues(fields []frontmatterField) map[string]string {
out := make(map[string]string, len(fields))
for _, f := range fields {
if len(f.lines) == 0 {
continue
}
_, value, found := strings.Cut(f.lines[0], ":")
if !found {
continue
}
out[f.key] = strings.TrimSpace(value)
}
return out
}
// frontmatterField is one top-level YAML key from a frontmatter block,
@@ -261,69 +308,32 @@ func splitFrontmatter(content string) (fields []frontmatterField, body string) {
return fields, body
}
// writeLegacyNote preserves the original brain/knowledge/ behaviour for
// callers that have not adopted the wing/hall taxonomy.
func writeLegacyNote(brainDir string, opts WriteNoteOptions) (string, error) {
filename := opts.Filename
if filename == "" {
filename = fmt.Sprintf("%s-auto.md", time.Now().UTC().Format("2006-01-02-150405"))
}
rawDir := filepath.Join(brainDir, "knowledge")
if err := os.MkdirAll(rawDir, 0o755); err != nil {
return "", fmt.Errorf("create raw dir: %w", err)
}
finalContent := opts.Content
if opts.Type != "" || opts.Domain != "" {
var fm strings.Builder
fm.WriteString("---\n")
if opts.Type != "" {
fmt.Fprintf(&fm, "type: %s\n", opts.Type)
}
if opts.Domain != "" {
fmt.Fprintf(&fm, "domain: %s\n", opts.Domain)
}
fm.WriteString("---\n")
finalContent = fm.String() + opts.Content
}
if strings.ContainsAny(filename, `/\`) {
return "", fmt.Errorf("invalid filename")
}
base := filepath.Base(filename)
if base == "." || base == ".." || base == "" {
return "", fmt.Errorf("invalid filename")
}
if !strings.HasSuffix(base, ".md") {
base += ".md"
}
dest := filepath.Join(rawDir, base)
if err := os.WriteFile(dest, []byte(finalContent), 0o644); err != nil {
return "", fmt.Errorf("write: %w", err)
}
rel, _ := filepath.Rel(brainDir, dest)
return filepath.ToSlash(rel), nil
}
// Write handles POST /write — write raw content to brain/knowledge/.
// Write handles POST /write — write a note into brain/wiki/<wing>/<hall>/.
func (h *Handler) Write(w http.ResponseWriter, r *http.Request) {
var req writeRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid JSON")
return
}
relPath, err := WriteNote(h.brainDir, WriteNoteOptions(req))
// Field by field, not a struct conversion: provenance is server-owned and
// this door carries no principal, so it stamps origin: unknown
// (hyperguild#145). A JSON body must never be able to set it.
relPath, err := WriteNote(h.brainDir, WriteNoteOptions{
Content: req.Content,
Filename: req.Filename,
Wing: req.Wing,
Hall: req.Hall,
SourceType: req.SourceType,
})
if err != nil {
h.logger.Error("write failed", "err", err)
writeError(w, http.StatusBadRequest, err.Error())
return
}
if req.Wing != "" && req.Hall != "" {
if err := brain.BuildWingIndex(h.brainDir, req.Wing); err != nil {
h.logger.Warn("auto-index failed", "wing", req.Wing, "err", err)
}
// WriteNote errors out above unless both are set, so this always runs —
// no legacy fallback path left to guard against.
if err := brain.BuildWingIndex(h.brainDir, req.Wing); err != nil {
h.logger.Warn("auto-index failed", "wing", req.Wing, "err", err)
}
writeJSON(w, map[string]string{"path": relPath})
}
@@ -445,6 +455,12 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "path is required")
return
}
resolved, err := ResolveIngestPath(h.brainDir, req.Path)
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
req.Path = resolved
info, err := os.Stat(req.Path)
if err != nil {
@@ -467,7 +483,13 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
if !isSupportedExtension(ext) {
return nil
}
content, readErr := extract.Text(path)
// Each walked file is re-resolved: a symlinked file inside raw/
// would otherwise lead extract.Text out of the inbox (#149).
if _, rerr := ResolveIngestPath(h.brainDir, path); rerr != nil {
allWarnings = append(allWarnings, fmt.Sprintf("skipped %s: %v", path, rerr))
return nil
}
content, readErr := ReadIngestFile(h.brainDir, path)
if readErr != nil {
allWarnings = append(allWarnings, fmt.Sprintf("extract %s: %v", path, readErr))
return nil
@@ -496,7 +518,7 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, fmt.Sprintf("unsupported file extension: %s", ext))
return
}
content, readErr := extract.Text(req.Path)
content, readErr := ReadIngestFile(h.brainDir, req.Path)
if readErr != nil {
writeError(w, http.StatusInternalServerError, fmt.Sprintf("extract text: %v", readErr))
return
@@ -548,7 +570,8 @@ func (h *Handler) IngestRaw(w http.ResponseWriter, r *http.Request) {
return
}
result, err := pipeline.RunRaw(h.brainDir, req.Source, req.Pages, req.DryRun)
// No principal on this door: pages are stamped origin: unknown (#145).
result, err := pipeline.RunRaw(r.Context(), h.brainDir, req.Source, req.Pages, req.DryRun)
if err != nil {
h.logger.Error("ingest-raw failed", "source", req.Source, "err", err)
writeError(w, http.StatusInternalServerError, "ingest error")
@@ -604,7 +627,11 @@ func (h *Handler) Promote(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "invalid JSON")
return
}
rel, err := PromoteNote(h.brainDir, PromoteOptions(req))
// Field by field: this door carries no principal, so the promoted note is
// stamped origin: unknown and the body cannot set provenance (#145).
rel, err := PromoteNote(h.brainDir, PromoteOptions{
Filename: req.Filename, Wing: req.Wing, Hall: req.Hall, Slug: req.Slug,
})
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
+203 -50
View File
@@ -65,11 +65,13 @@ func TestQuery_ReturnsResults(t *testing.T) {
assert.NotEmpty(t, results)
}
func TestWrite_CreatesKnowledgeFile(t *testing.T) {
func TestWrite_CreatesWikiFile(t *testing.T) {
dir, h := setup(t)
body, _ := json.Marshal(map[string]any{
"content": "# Test note\n\nSome content.",
"filename": "test-note.md",
"filename": "test-note",
"wing": "homelab",
"hall": "facts",
})
req := httptest.NewRequest(http.MethodPost, "/write", bytes.NewReader(body))
rec := httptest.NewRecorder()
@@ -81,11 +83,25 @@ func TestWrite_CreatesKnowledgeFile(t *testing.T) {
require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &resp))
assert.NotEmpty(t, resp["path"])
content, err := os.ReadFile(filepath.Join(dir, "knowledge", "test-note.md"))
content, err := os.ReadFile(filepath.Join(dir, "wiki", "homelab", "facts", "test-note.md"))
require.NoError(t, err)
assert.Contains(t, string(content), "Some content.")
}
// TestWrite_RequiresWingAndHall is the HTTP-contract counterpart of
// TestWriteNote_RequiresWingAndHall: a caller (e.g. the MCP brain_write
// handler) that omits wing/hall gets a 400, not a silent legacy write.
func TestWrite_RequiresWingAndHall(t *testing.T) {
_, h := setup(t)
body, _ := json.Marshal(map[string]any{"content": "no destination"})
req := httptest.NewRequest(http.MethodPost, "/write", bytes.NewReader(body))
rec := httptest.NewRecorder()
h.Write(rec, req)
assert.Equal(t, http.StatusBadRequest, rec.Code)
}
func TestQuery_RequiresQuery(t *testing.T) {
_, h := setup(t)
body, _ := json.Marshal(map[string]any{"limit": 5})
@@ -97,27 +113,6 @@ func TestQuery_RequiresQuery(t *testing.T) {
assert.Equal(t, http.StatusBadRequest, rec.Code)
}
func TestWrite_IncludesFrontmatterWhenTypeProvided(t *testing.T) {
dir, h := setup(t)
body, _ := json.Marshal(map[string]any{
"content": "Some learning.",
"filename": "typed-note.md",
"type": "concept",
"domain": "software",
})
req := httptest.NewRequest(http.MethodPost, "/write", bytes.NewReader(body))
rec := httptest.NewRecorder()
h.Write(rec, req)
assert.Equal(t, http.StatusOK, rec.Code)
content, err := os.ReadFile(filepath.Join(dir, "knowledge", "typed-note.md"))
require.NoError(t, err)
assert.Contains(t, string(content), "type: concept")
assert.Contains(t, string(content), "domain: software")
assert.Contains(t, string(content), "Some learning.")
}
func TestWriteNote_HallRouteIncludesSourceTypeWhenSet(t *testing.T) {
dir := t.TempDir()
@@ -126,7 +121,6 @@ func TestWriteNote_HallRouteIncludesSourceTypeWhenSet(t *testing.T) {
Filename: "session-koala-abc",
Wing: "claude-sessions",
Hall: "facts",
Type: "source",
SourceType: "internal",
})
require.NoError(t, err)
@@ -210,37 +204,95 @@ func TestWriteNote_HallRouteMergesExistingFrontmatterInsteadOfStacking(t *testin
assert.Contains(t, body, "# Body")
}
func TestWriteNote_HallRouteExistingTypeWinsOverOptsType(t *testing.T) {
dir := t.TempDir()
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "---\ntype: hypothesis\n---\n\nBody.\n",
Filename: "note",
Wing: "agentsquad",
Hall: "decisions",
Type: "decision", // should lose to content's own "type: hypothesis"
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Contains(t, string(got), "type: hypothesis")
assert.NotContains(t, string(got), "type: decision")
// TestWriteNote_RequiresWingAndHall pins the post-legacy-removal contract
// (hyperguild#79 / ADR-0012): with the knowledge/ fallback gone, wing+hall
// is the only valid destination, so omitting either is a caller error, not
// a silent fallback.
func TestWriteNote_RequiresWingAndHall(t *testing.T) {
cases := []struct {
name string
wing string
hall string
}{
{"neither set", "", ""},
{"wing only", "homelab", ""},
{"hall only", "", "facts"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
dir := t.TempDir()
_, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "body",
Wing: tc.wing,
Hall: tc.hall,
})
require.Error(t, err)
assert.Contains(t, err.Error(), "wing and hall are required")
})
}
}
func TestWrite_GeneratesFilenameIfAbsent(t *testing.T) {
dir, h := setup(t)
body, _ := json.Marshal(map[string]any{"content": "auto name"})
body, _ := json.Marshal(map[string]any{
"content": "auto name",
"wing": "homelab",
"hall": "facts",
})
req := httptest.NewRequest(http.MethodPost, "/write", bytes.NewReader(body))
rec := httptest.NewRecorder()
h.Write(rec, req)
assert.Equal(t, http.StatusOK, rec.Code)
entries, _ := os.ReadDir(filepath.Join(dir, "knowledge"))
// +1 because setup already wrote tdd.md
assert.Len(t, entries, 2)
assert.True(t, strings.HasSuffix(entries[1].Name(), ".md"))
entries, err := os.ReadDir(filepath.Join(dir, "wiki", "homelab", "facts"))
require.NoError(t, err)
require.Len(t, entries, 1)
assert.True(t, strings.HasSuffix(entries[0].Name(), ".md"))
}
// TestWrite_CLIPayloadRoundTrips exercises the real Handler.Write (not a
// hand-rolled mock) over an actual HTTP round trip, POSTing the exact JSON
// envelope cmd/hyperguild's brainClient.Write sends: {wing, hall, filename,
// content}. cmd/hyperguild lives in a separate Go module from ingestion
// (no shared go.work), so a Go-level test can't import brainClient
// directly — this test instead pins the wire contract from the server
// side and proves a real `hyperguild brain write <wing> <hall> <slug>`
// invocation writes a real file with real frontmatter under wiki/<wing>/<hall>/.
// A prior version of this contract test on the client side (in
// cmd/hyperguild/http_test.go) used only an httptest.NewServer with a
// hand-rolled mock body, so it stayed green through a regression where the
// server required wing/hall but the client never sent them (hyperguild#79
// PR review).
func TestWrite_CLIPayloadRoundTrips(t *testing.T) {
dir, h := setup(t)
srv := httptest.NewServer(http.HandlerFunc(h.Write))
defer srv.Close()
payload, err := json.Marshal(struct {
Wing string `json:"wing"`
Hall string `json:"hall"`
Filename string `json:"filename"`
Content string `json:"content"`
}{Wing: "homelab", Hall: "facts", Filename: "cli-roundtrip", Content: "# CLI roundtrip\n\nReal write.\n"})
require.NoError(t, err)
resp, err := http.Post(srv.URL+"/write", "application/json", bytes.NewReader(payload)) //nolint:noctx
require.NoError(t, err)
defer resp.Body.Close() //nolint:errcheck
assert.Equal(t, http.StatusOK, resp.StatusCode)
var out struct {
Path string `json:"path"`
}
require.NoError(t, json.NewDecoder(resp.Body).Decode(&out))
assert.Equal(t, "wiki/homelab/facts/cli-roundtrip.md", out.Path)
written, err := os.ReadFile(filepath.Join(dir, "wiki", "homelab", "facts", "cli-roundtrip.md"))
require.NoError(t, err)
assert.Contains(t, string(written), "wing: homelab")
assert.Contains(t, string(written), "hall: facts")
assert.Contains(t, string(written), "Real write.")
}
// ---------------------------------------------------------------------------
@@ -309,10 +361,11 @@ func TestIngestPath_MissingPath(t *testing.T) {
}
func TestIngestPath_File(t *testing.T) {
_, h := setup(t)
brainDir, h := setup(t)
// Create a temp file with content
dir := t.TempDir()
dir := filepath.Join(brainDir, "raw", "fixture") // ingest paths must live in brain/raw/ (#149)
require.NoError(t, os.MkdirAll(dir, 0o755))
f := filepath.Join(dir, "doc.md")
require.NoError(t, os.WriteFile(f, []byte("# Hello\nThis is markdown content."), 0o644))
@@ -416,10 +469,11 @@ func TestIngestRaw_DryRun(t *testing.T) {
}
func TestIngestPath_Directory(t *testing.T) {
_, h := setup(t)
brainDir, h := setup(t)
// Create a temp dir with one .md file
dir := t.TempDir()
dir := filepath.Join(brainDir, "raw", "fixture") // ingest paths must live in brain/raw/ (#149)
require.NoError(t, os.MkdirAll(dir, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "notes.md"), []byte("# Notes\nSome notes."), 0o644))
body, _ := json.Marshal(map[string]any{
@@ -440,3 +494,102 @@ func TestIngestPath_Directory(t *testing.T) {
require.True(t, ok, "pages must be an array")
assert.NotEmpty(t, pagesSlice)
}
// --- brain#21: claim-property envelope on the write path -------------------
func TestWriteNote_PreservesClaimEnvelopeVerbatim(t *testing.T) {
// writeHallNote already passes unknown keys through; this pins that
// behaviour for the envelope specifically, so a future refactor of the
// merge loop cannot silently drop it.
dir := t.TempDir()
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "---\nhalf_life: durable\napplies: universal\nclaim_provenance: asserted\n---\n\nA durable claim.\n",
Wing: "homelab",
Hall: "decisions",
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Contains(t, string(got), "half_life: durable")
assert.Contains(t, string(got), "applies: universal")
assert.Contains(t, string(got), "claim_provenance: asserted")
}
func TestWriteNote_RejectsInvalidClaimEnvelopeAndWritesNothing(t *testing.T) {
// Fail closed: a value outside the enum must not land. A lint skips what
// it cannot parse, so an unvalidated write produces a note that reads as
// classified while nothing acts on it.
dir := t.TempDir()
_, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "---\nhalf_life: forever\nclaim_provenance: asserted\n---\n\nBody.\n",
Filename: "bad-envelope",
Wing: "homelab",
Hall: "facts",
})
require.Error(t, err)
assert.Contains(t, err.Error(), "half_life")
_, statErr := os.Stat(filepath.Join(dir, "wiki/homelab/facts/bad-envelope.md"))
assert.True(t, os.IsNotExist(statErr), "a rejected write must leave no file behind")
}
func TestWriteNote_RejectsClaimFieldWithoutProvenance(t *testing.T) {
dir := t.TempDir()
_, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "---\napplies: homelab\n---\n\nBody.\n",
Filename: "no-provenance",
Wing: "homelab",
Hall: "facts",
})
require.Error(t, err)
assert.Contains(t, err.Error(), "claim_provenance")
_, statErr := os.Stat(filepath.Join(dir, "wiki/homelab/facts/no-provenance.md"))
assert.True(t, os.IsNotExist(statErr))
}
// The HTTP POST /write door reaches WriteNote directly, without the
// BrainStore, so the unresolved-wikilink strip has to live in the primitive
// or that door stays ungated.
func TestWriteNote_UnwrapsUnresolvedWikilinks(t *testing.T) {
dir := t.TempDir()
_, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "# Target\n\nbody\n", Filename: "target", Wing: "a", Hall: "facts",
})
require.NoError(t, err)
rel, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "See [[target]] and [[gitea-ci]].\n",
Filename: "src", Wing: "a", Hall: "facts",
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Contains(t, string(got), "[[target]]")
assert.Contains(t, string(got), "and gitea-ci.")
assert.NotContains(t, string(got), "[[gitea-ci]]")
}
func TestUpdateNote_UnwrapsUnresolvedWikilinks(t *testing.T) {
dir := t.TempDir()
_, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "old\n", Filename: "n", Wing: "a", Hall: "facts",
})
require.NoError(t, err)
rel, _, _, err := api.UpdateNote(dir, api.UpdateNoteOptions{
Wing: "a", Hall: "facts", Slug: "n",
Content: "Related: [[secret-hygiene]].\n", Reason: "r",
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Contains(t, string(got), "Related: secret-hygiene.")
assert.NotContains(t, string(got), "[[secret-hygiene]]")
}
@@ -40,9 +40,10 @@ func TestIngestPath_DocxSupportedWhenDocmarkConfigured(t *testing.T) {
t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "tok")
_, h := setup(t)
brainDir, h := setup(t)
dir := t.TempDir()
dir := filepath.Join(brainDir, "raw", "fixture") // ingest paths must live in brain/raw/ (#149)
require.NoError(t, os.MkdirAll(dir, 0o755))
f := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(f, []byte("fake docx"), 0o644))
+67
View File
@@ -0,0 +1,67 @@
package api
import (
"errors"
"os"
"github.com/mathiasbq/hyperguild/ingestion/internal/extract"
"path/filepath"
"strings"
)
// ErrOutsideIngestRoot is the one refusal for every bad ingest path. It never
// carries the underlying error: "no such file" versus "permission denied"
// would let a caller probe which server files exist (PR #152 review).
var ErrOutsideIngestRoot = errors.New("ingest path must be an existing file or directory inside brain/raw/")
// ResolveIngestPath confines a caller-supplied ingest path to brain/raw/, the
// inbox the watcher already owns (hyperguild#149). Relative paths resolve
// against that root; symlinks are evaluated before the containment check, so
// a link inside raw/ that points elsewhere is refused. The path must exist.
func ResolveIngestPath(brainDir, p string) (string, error) {
if strings.TrimSpace(p) == "" {
return "", ErrOutsideIngestRoot
}
// raw/ itself must be a real directory: if it were a symlink, containment
// inside it would mean nothing (hyperguild#153 item 4).
if fi, lerr := os.Lstat(filepath.Join(brainDir, "raw")); lerr != nil || fi.Mode()&os.ModeSymlink != 0 || !fi.IsDir() {
return "", ErrOutsideIngestRoot
}
root, err := filepath.EvalSymlinks(filepath.Join(brainDir, "raw"))
if err != nil {
return "", ErrOutsideIngestRoot
}
if !filepath.IsAbs(p) {
p = filepath.Join(root, p)
}
real, err := filepath.EvalSymlinks(filepath.Clean(p))
if err != nil {
return "", ErrOutsideIngestRoot
}
rel, err := filepath.Rel(root, real)
if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) || filepath.IsAbs(rel) {
return "", ErrOutsideIngestRoot
}
return real, nil
}
// ReadIngestFile extracts text from an ingest path already confined by
// ResolveIngestPath, reading it through the raw/ root rather than reopening
// the path by name (hyperguild#153 item 3): if the file is swapped for a
// symlink after the check, the read still cannot leave raw/.
func ReadIngestFile(brainDir, resolved string) (string, error) {
root, rawPath, err := OpenRawRoot(brainDir)
if err != nil {
return "", ErrOutsideIngestRoot
}
defer func() { _ = root.Close() }()
realRoot, err := filepath.EvalSymlinks(rawPath)
if err != nil {
return "", ErrOutsideIngestRoot
}
rel, err := filepath.Rel(realRoot, resolved)
if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) || filepath.IsAbs(rel) {
return "", ErrOutsideIngestRoot
}
return extract.TextIn(root, rel)
}
@@ -0,0 +1,88 @@
package api_test
import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// hyperguild#149: /ingest-path and MCP brain_ingest read a caller-chosen
// server path, so any file the pod could read (the ServiceAccount token,
// /etc/passwd) could be pulled into the brain and read back out. Ingest paths
// now resolve inside brain/raw/ — the inbox the watcher already owns — with
// symlinks evaluated, or are refused.
func TestResolveIngestPathConfinesToRawInbox(t *testing.T) {
brain := t.TempDir()
raw := filepath.Join(brain, "raw")
require.NoError(t, os.MkdirAll(filepath.Join(raw, "sub"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(raw, "sub", "ok.md"), []byte("x"), 0o644))
outside := filepath.Join(t.TempDir(), "secret.txt")
require.NoError(t, os.WriteFile(outside, []byte("s"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(raw, "link.md")))
require.NoError(t, os.Symlink(filepath.Dir(outside), filepath.Join(raw, "dirlink")))
for _, ok := range []string{filepath.Join(raw, "sub", "ok.md"), "sub/ok.md", raw, "sub"} {
_, err := api.ResolveIngestPath(brain, ok)
assert.NoError(t, err, ok)
}
for _, bad := range []string{
"/etc/hostname", outside, "../secret.txt", filepath.Join(raw, "..", "wiki"),
filepath.Join(raw, "link.md"), filepath.Join(raw, "dirlink", "secret.txt"),
raw + "-evil/x.md", "",
} {
_, err := api.ResolveIngestPath(brain, bad)
assert.Error(t, err, "must refuse %q", bad)
}
}
func TestIngestPathRefusesAPathOutsideTheInbox(t *testing.T) {
_, h := setup(t)
body, _ := json.Marshal(map[string]any{"path": "/etc/hostname"})
rec := httptest.NewRecorder()
h.IngestPath(rec, httptest.NewRequest(http.MethodPost, "/ingest-path", bytes.NewReader(body)))
assert.Equal(t, http.StatusBadRequest, rec.Code)
assert.Contains(t, rec.Body.String(), "raw")
}
// A directory ingest must not follow a symlinked FILE inside raw/ out of it.
func TestIngestPathDirectoryWalkSkipsSymlinksLeavingTheInbox(t *testing.T) {
dir, h := setup(t)
raw := filepath.Join(dir, "raw", "batch")
require.NoError(t, os.MkdirAll(raw, 0o755))
outside := filepath.Join(t.TempDir(), "secret.md")
require.NoError(t, os.WriteFile(outside, []byte("# secret\n\nSENTINEL-149-OUTSIDE"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(raw, "leak.md")))
body, _ := json.Marshal(map[string]any{"path": "batch", "source": "walk"})
rec := httptest.NewRecorder()
h.IngestPath(rec, httptest.NewRequest(http.MethodPost, "/ingest-path", bytes.NewReader(body)))
assert.Contains(t, rec.Body.String(), "skipped")
assert.NotContains(t, rec.Body.String(), "SENTINEL-149-OUTSIDE")
}
// PR #152 review: the raw EvalSymlinks error told an authenticated caller
// whether any server file existed ("no such file") or was unreadable
// ("permission denied"), and leaked the brain directory. Every refusal now
// reads the same.
func TestResolveIngestPathRefusalsAreIndistinguishable(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(brain, "raw"), 0o755))
var msgs []string
for _, p := range []string{"/etc/hostname", "/etc/definitely-not-here-149", "../x", "nope/missing.md", ""} {
_, err := api.ResolveIngestPath(brain, p)
require.Error(t, err, p)
msgs = append(msgs, err.Error())
assert.NotContains(t, err.Error(), brain, "refusal for %q leaks the brain dir", p)
}
for _, m := range msgs[1:] {
assert.Equal(t, msgs[0], m)
}
}
+105 -20
View File
@@ -1,7 +1,9 @@
package api
import (
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"regexp"
@@ -28,25 +30,25 @@ var datePrefix = regexp.MustCompile(`^\d{4}-\d{2}-\d{2}-`)
// (natural review order). An absent raw/ dir yields an empty slice, not an
// error. Only .md files are listed; tunnel-candidate files are skipped.
func ListPending(brainDir string) ([]PendingNote, error) {
dir := filepath.Join(brainDir, "raw")
entries, err := os.ReadDir(dir)
root, _, err := OpenRawRoot(brainDir)
if err != nil {
if os.IsNotExist(err) {
if errors.Is(err, os.ErrNotExist) {
return []PendingNote{}, nil
}
return nil, fmt.Errorf("read raw dir: %w", err)
}
defer func() { _ = root.Close() }()
entries, err := fs.ReadDir(root.FS(), ".")
if err != nil {
return nil, fmt.Errorf("read raw dir: %w", err)
}
out := make([]PendingNote, 0, len(entries))
for _, e := range entries {
if e.IsDir() || !strings.HasSuffix(e.Name(), ".md") || strings.HasPrefix(e.Name(), "tunnel-candidates-") {
continue
if !e.Type().IsRegular() || !strings.HasSuffix(e.Name(), ".md") || strings.HasPrefix(e.Name(), "tunnel-candidates-") {
continue // directories, and symlinks whose target may be anywhere (#153)
}
info, statErr := e.Info()
if statErr != nil {
continue
}
raw, readErr := os.ReadFile(filepath.Join(dir, e.Name()))
raw, info, readErr := readRawNote(root, e.Name())
if readErr != nil {
continue
}
@@ -72,6 +74,10 @@ type PromoteOptions struct {
Wing string
Hall string
Slug string // optional; defaults to Filename minus date prefix + .md
// Origin/WrittenBy re-attribute the promoted note to the promoter: the raw
// note's own frontmatter claims are not provenance (hyperguild#145).
Origin string
WrittenBy string
}
// PromoteNote moves a note from brain/raw/ into the structured wiki: it
@@ -101,12 +107,9 @@ func PromoteNote(brainDir string, opts PromoteOptions) (string, error) {
}
src := filepath.Join(brainDir, "raw", base)
raw, err := os.ReadFile(src)
raw, err := readRawSource(brainDir, base)
if err != nil {
if os.IsNotExist(err) {
return "", fmt.Errorf("pending note %q does not exist in raw/", base)
}
return "", fmt.Errorf("read source: %w", err)
return "", err
}
// Collision: never silently overwrite an existing promoted note.
@@ -124,14 +127,18 @@ func PromoteNote(brainDir string, opts PromoteOptions) (string, error) {
}
fm.set("promoted_at", now)
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
return "", fmt.Errorf("create wing dir: %w", err)
// Write-then-delete: the source survives any write failure. Both go
// through the brain root (hyperguild#188).
promoted, err := StampProvenance(fm.render()+body, opts.Origin, opts.WrittenBy)
if err != nil {
return "", err
}
// Write-then-delete: the source survives any write failure.
if err := os.WriteFile(dest, []byte(fm.render()+body), 0o644); err != nil {
// Created with O_EXCL: a note that appeared after the collision check
// above is never overwritten.
if err := createBrainFile(brainDir, dest, []byte(promoted)); err != nil {
return "", fmt.Errorf("write promoted note: %w", err)
}
if err := os.Remove(src); err != nil {
if err := removeBrainFile(brainDir, src); err != nil {
return "", fmt.Errorf("promoted note written but source removal failed: %w", err)
}
@@ -145,6 +152,63 @@ func PromoteNote(brainDir string, opts PromoteOptions) (string, error) {
return relSlash, nil
}
// RejectOptions identifies a raw note to reject and why.
type RejectOptions struct {
Filename string // basename in brain/raw/
Reason string // optional
}
// RejectNote records a human decision not to promote a brain/raw/ note,
// without silently losing it: it stamps rejected_at (+ reject_reason when
// given) into frontmatter and moves the file to brain/raw/rejected/, using
// the same write-then-delete safety PromoteNote uses — the source survives
// any write failure — and the same basename-only filename validation.
//
// A rejected note never entered the wiki, so no index rebuild or
// auto-tunnel run: unlike PromoteNote, this is the end of the note's
// journey, not the start of a new one. It stays out of ListPending's view
// (raw/rejected/ is a subdirectory, and ListPending's os.ReadDir on raw/
// does not recurse), so a rejected note is never re-surfaced as pending —
// the human decision, once made, stays made, the same principle behind
// this estate's "issue status is surfaced, not auto-closed, but once a
// human decides, the decision is recorded" rule applied one level earlier.
func RejectNote(brainDir string, opts RejectOptions) (string, error) {
base := filepath.Base(opts.Filename)
if base != opts.Filename || base == "." || base == ".." || strings.ContainsAny(opts.Filename, `/\`) {
return "", fmt.Errorf("invalid filename %q", opts.Filename)
}
src := filepath.Join(brainDir, "raw", base)
raw, err := readRawSource(brainDir, base)
if err != nil {
return "", err
}
dest := filepath.Join(brainDir, "raw", "rejected", base)
if _, statErr := os.Stat(dest); statErr == nil {
rel, _ := filepath.Rel(brainDir, dest)
return "", fmt.Errorf("a rejected note named %s already exists", filepath.ToSlash(rel))
}
fm, body := parseFrontmatter(string(raw))
fm.set("rejected_at", time.Now().UTC().Format(time.RFC3339))
if opts.Reason != "" {
fm.set("reject_reason", opts.Reason)
}
// Write-then-delete: the source survives any write failure. Both go
// through the brain root (hyperguild#188).
if err := createBrainFile(brainDir, dest, []byte(fm.render()+body)); err != nil {
return "", fmt.Errorf("write rejected note: %w", err)
}
if err := removeBrainFile(brainDir, src); err != nil {
return "", fmt.Errorf("rejected note written but source removal failed: %w", err)
}
rel, _ := filepath.Rel(brainDir, dest)
return filepath.ToSlash(rel), nil
}
// excerpt returns the first n runes of s, trimmed, single-spaced.
func excerpt(s string, n int) string {
s = strings.TrimSpace(s)
@@ -154,3 +218,24 @@ func excerpt(s string, n int) string {
}
return strings.TrimSpace(string(r))
}
// readRawSource reads a pending note for promote/reject through the raw root,
// refusing symlinks and a symlinked raw/ (hyperguild#153).
func readRawSource(brainDir, base string) ([]byte, error) {
root, _, err := OpenRawRoot(brainDir)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return nil, fmt.Errorf("pending note %q does not exist in raw/", base)
}
return nil, fmt.Errorf("read source: %w", err)
}
defer func() { _ = root.Close() }()
raw, _, err := readRawNote(root, base)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return nil, fmt.Errorf("pending note %q does not exist in raw/", base)
}
return nil, fmt.Errorf("read source: %w", err)
}
return raw, nil
}
@@ -0,0 +1,87 @@
package api_test
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// hyperguild#188: promote and reject wrote the destination with
// os.MkdirAll + os.WriteFile by path, so a symlinked wing (promote) or a
// symlinked raw/rejected (reject) redirected the note; the raw source was
// then removed.
func TestPromoteAndRejectRefuseASymlinkedDestinationDirectory(t *testing.T) {
brain := t.TempDir()
outside := t.TempDir()
raw := filepath.Join(brain, "raw")
require.NoError(t, os.MkdirAll(raw, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(raw, "2026-10-05-n.md"), []byte("---\ntitle: n\n---\nbody\n"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki"), 0o755))
require.NoError(t, os.Symlink(outside, filepath.Join(brain, "wiki", "lw")))
require.NoError(t, os.Symlink(outside, filepath.Join(raw, "rejected")))
_, err := api.PromoteNote(brain, api.PromoteOptions{Filename: "2026-10-05-n.md", Wing: "lw", Hall: "facts"})
assert.Error(t, err)
_, err = api.RejectNote(brain, api.RejectOptions{Filename: "2026-10-05-n.md"})
assert.Error(t, err)
ents, _ := os.ReadDir(outside)
assert.Empty(t, ents, "a note was written outside the brain")
_, err = os.Stat(filepath.Join(raw, "2026-10-05-n.md"))
assert.NoError(t, err, "the raw source is kept when the write is refused")
require.NoError(t, os.Remove(filepath.Join(raw, "rejected")))
_, err = api.PromoteNote(brain, api.PromoteOptions{Filename: "2026-10-05-n.md", Wing: "w", Hall: "facts"})
require.NoError(t, err, "positive control: promote into a real wing")
}
// StampProvenanceFile read and rewrote the note by path (provenance.ApplyFile).
func TestStampProvenanceFileRefusesANoteBehindASymlinkedDirectory(t *testing.T) {
brain := t.TempDir()
victim := filepath.Join(brain, "raw", "facts", "v.md")
require.NoError(t, os.MkdirAll(filepath.Dir(victim), 0o755))
require.NoError(t, os.WriteFile(victim, []byte("---\ntitle: v\n---\nuntouched\n"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki", "w"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "raw", "facts"), filepath.Join(brain, "wiki", "w", "alias")))
assert.Error(t, api.StampProvenanceFile(brain, "wiki/w/alias/v.md", "static", "x"))
got, _ := os.ReadFile(victim)
assert.Equal(t, "---\ntitle: v\n---\nuntouched\n", string(got))
real := filepath.Join(brain, "wiki", "w", "facts", "r.md")
require.NoError(t, os.MkdirAll(filepath.Dir(real), 0o755))
require.NoError(t, os.WriteFile(real, []byte("---\ntitle: r\n---\nbody\n"), 0o644))
require.NoError(t, api.StampProvenanceFile(brain, "wiki/w/facts/r.md", "static", "x"), "positive control")
got, _ = os.ReadFile(real)
assert.Contains(t, string(got), "written_by: x")
}
// The same through an in-brain symlinked wing and raw/rejected, which
// os.Root alone would follow.
func TestPromoteAndRejectRefuseAnInBrainSymlinkedDestinationDirectory(t *testing.T) {
brain := t.TempDir()
raw := filepath.Join(brain, "raw")
require.NoError(t, os.MkdirAll(raw, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(raw, "2026-10-05-n.md"), []byte("---\ntitle: n\n---\nbody\n"), 0o644))
target := filepath.Join(brain, "archive")
require.NoError(t, os.MkdirAll(target, 0o755))
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "archive"), filepath.Join(brain, "wiki", "lw")))
require.NoError(t, os.Symlink(filepath.Join("..", "archive"), filepath.Join(raw, "rejected")))
_, err := api.PromoteNote(brain, api.PromoteOptions{Filename: "2026-10-05-n.md", Wing: "lw", Hall: "facts"})
assert.Error(t, err)
_, err = api.RejectNote(brain, api.RejectOptions{Filename: "2026-10-05-n.md"})
assert.Error(t, err)
ents, _ := os.ReadDir(target)
assert.Empty(t, ents, "a note was written at the in-brain link's target")
_, err = os.Stat(filepath.Join(raw, "2026-10-05-n.md"))
assert.NoError(t, err, "the raw source is kept")
require.NoError(t, os.Remove(filepath.Join(raw, "rejected")))
_, err = api.RejectNote(brain, api.RejectOptions{Filename: "2026-10-05-n.md"})
require.NoError(t, err, "positive control: reject into a real raw/rejected")
}
+82
View File
@@ -119,3 +119,85 @@ func TestPromoteRebuildsWingIndex(t *testing.T) {
require.NoError(t, err, "wing _index regenerated")
assert.Contains(t, string(idx), "x", "promoted note appears in the index")
}
// --- RejectNote (hyperguild#81) ---------------------------------------------
func TestRejectHappyPath(t *testing.T) {
dir := t.TempDir()
writeRaw(t, dir, "2026-04-23-fed-sentiment-scrape.md",
"---\ncreated_at: 2026-04-23T13:20:07Z\n---\n# Sentiment scrape\n\nbody.\n")
rel, err := RejectNote(dir, RejectOptions{
Filename: "2026-04-23-fed-sentiment-scrape.md", Reason: "generic scrape, not homelab-relevant",
})
require.NoError(t, err)
assert.Equal(t, "raw/rejected/2026-04-23-fed-sentiment-scrape.md", rel)
// Source removed from the pending queue.
_, statErr := os.Stat(filepath.Join(dir, "raw", "2026-04-23-fed-sentiment-scrape.md"))
assert.True(t, os.IsNotExist(statErr), "source removed from raw/ after reject")
got, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
s := string(got)
assert.Contains(t, s, "created_at: 2026-04-23T13:20:07Z", "original created_at preserved")
assert.Contains(t, s, "rejected_at:")
assert.Contains(t, s, "reject_reason: generic scrape, not homelab-relevant")
assert.Contains(t, s, "# Sentiment scrape", "body preserved for later reference")
// Rejected notes never entered the wiki, and are no longer pending.
pending, err := ListPending(dir)
require.NoError(t, err)
assert.Empty(t, pending)
}
func TestRejectWithoutReason(t *testing.T) {
dir := t.TempDir()
writeRaw(t, dir, "2026-06-01-x.md", "body\n")
rel, err := RejectNote(dir, RejectOptions{Filename: "2026-06-01-x.md"})
require.NoError(t, err)
got, _ := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
assert.NotContains(t, string(got), "reject_reason:", "no reason given, none stamped")
}
func TestRejectMissingSourceErrors(t *testing.T) {
_, err := RejectNote(t.TempDir(), RejectOptions{Filename: "ghost.md"})
require.Error(t, err)
}
func TestRejectRejectsTraversalFilename(t *testing.T) {
_, err := RejectNote(t.TempDir(), RejectOptions{Filename: "../escape.md"})
require.Error(t, err)
}
func TestRejectCollisionNoOverwrite(t *testing.T) {
dir := t.TempDir()
dest := filepath.Join(dir, "raw", "rejected", "x.md")
require.NoError(t, os.MkdirAll(filepath.Dir(dest), 0o755))
require.NoError(t, os.WriteFile(dest, []byte("EXISTING\n"), 0o644))
writeRaw(t, dir, "x.md", "NEW\n")
_, err := RejectNote(dir, RejectOptions{Filename: "x.md"})
require.Error(t, err, "collision must error, not overwrite")
got, _ := os.ReadFile(dest)
assert.Equal(t, "EXISTING\n", string(got), "target not overwritten")
_, statErr := os.Stat(filepath.Join(dir, "raw", "x.md"))
assert.NoError(t, statErr, "source preserved on collision (atomic: no delete without write)")
}
func TestRejectedNotesAreNotListedAsPendingAgain(t *testing.T) {
// ListPending already skips tunnel-candidates-*.md; confirm raw/rejected/
// (a subdirectory) doesn't leak into the top-level pending listing either
// — os.ReadDir on raw/ doesn't recurse, but pin it so a future change to
// ListPending can't silently start surfacing rejected notes as pending
// again.
dir := t.TempDir()
writeRaw(t, dir, "2026-06-01-x.md", "body\n")
_, err := RejectNote(dir, RejectOptions{Filename: "2026-06-01-x.md"})
require.NoError(t, err)
pending, err := ListPending(dir)
require.NoError(t, err)
assert.Empty(t, pending, "a rejected note must not still appear as pending")
}
+75
View File
@@ -0,0 +1,75 @@
package api
import (
"fmt"
"path/filepath"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"github.com/mathiasbq/hyperguild/ingestion/internal/provenance"
)
// StampProvenance applies server-owned provenance (hyperguild#145) to note
// content. See package provenance for the rule; an error means the stamped
// note would not read back as stamped, and the write must be refused.
func StampProvenance(content, origin, writtenBy string) (string, error) {
return provenance.Apply(content, provenance.Stamp{Origin: origin, WrittenBy: writtenBy})
}
// StampProvenanceFile applies StampProvenance to a brain-relative note path,
// read and rewritten through the brain root (hyperguild#158).
func StampProvenanceFile(brainDir, relPath, origin, writtenBy string) error {
rel, _, err := resolveWithin(brainDir, relPath)
if err != nil {
return err
}
bfs, err := brainfs.Open(brainDir)
if err != nil {
return fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
b, err := bfs.ReadRegular(rel)
if err != nil {
return fmt.Errorf("read for provenance: %w", err)
}
out, err := provenance.Apply(string(b), provenance.Stamp{Origin: origin, WrittenBy: writtenBy})
if err != nil {
return err
}
if out == string(b) {
return nil
}
if err := bfs.WriteRegular(rel, []byte(out)); err != nil {
return fmt.Errorf("write provenance: %w", err)
}
return nil
}
// createBrainFile creates the file at abs (inside brainDir) through the
// brain root and fails if anything is already there.
func createBrainFile(brainDir, abs string, data []byte) error {
rel, err := filepath.Rel(brainDir, abs)
if err != nil {
return err
}
bfs, err := brainfs.Open(brainDir)
if err != nil {
return fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
return bfs.CreateRegular(filepath.ToSlash(rel), data)
}
// removeBrainFile removes the file at abs (inside brainDir) through the
// brain root.
func removeBrainFile(brainDir, abs string) error {
rel, err := filepath.Rel(brainDir, abs)
if err != nil {
return err
}
bfs, err := brainfs.Open(brainDir)
if err != nil {
return fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
return bfs.Remove(filepath.ToSlash(rel))
}
+122
View File
@@ -0,0 +1,122 @@
package api_test
import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gopkg.in/yaml.v3"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// REST POST /write carries no principal: it stamps origin: unknown, and no
// spelling of provenance in its body reaches the note (hyperguild#145).
func TestRESTWriteStampsUnknownAndIgnoresBodyProvenance(t *testing.T) {
dir, h := setup(t)
body, _ := json.Marshal(map[string]any{
"content": "---\norigin: static\n---\n# R\n\nbody", "filename": "rest-prov", "wing": "homelab", "hall": "facts",
"origin": "static", "written_by": "local-cli", "Origin": "static", "WrittenBy": "local-cli",
})
rec := httptest.NewRecorder()
h.Write(rec, httptest.NewRequest(http.MethodPost, "/write", bytes.NewReader(body)))
require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())
b, err := os.ReadFile(filepath.Join(dir, "wiki", "homelab", "facts", "rest-prov.md"))
require.NoError(t, err)
var m map[string]any
require.NoError(t, yaml.Unmarshal([]byte(strings.SplitN(string(b), "---\n", 3)[1]), &m))
assert.Equal(t, "unknown", m["origin"])
assert.NotContains(t, m, "written_by")
}
// A principal name that is not a bare word is quoted, so it cannot start a
// new frontmatter line (e.g. a JWT sub containing a newline).
func TestStampProvenanceQuotesAPrincipalThatCouldBreakTheLine(t *testing.T) {
out, err := api.StampProvenance("---\ntitle: t\n---\nbody\n", "jwt", "evil\norigin: static")
require.NoError(t, err)
fm := strings.SplitN(out, "---\n", 3)[1]
var m map[string]any
require.NoError(t, yaml.Unmarshal([]byte(fm), &m))
assert.Equal(t, "jwt", m["origin"])
assert.Equal(t, "evil\norigin: static", m["written_by"])
assert.Equal(t, 1, strings.Count(fm, "\norigin:")+boolToInt(strings.HasPrefix(fm, "origin:")))
}
func TestStampProvenanceAddsFrontmatterWhenAbsentAndKeepsBody(t *testing.T) {
out, err := api.StampProvenance("# no fm\n\norigin: static in the body is prose\n", "static", "local-cli")
require.NoError(t, err)
assert.True(t, strings.HasPrefix(out, "---\norigin: static\nwritten_by: local-cli\n---\n# no fm"))
assert.Contains(t, out, "origin: static in the body is prose", "body lines are never touched")
}
func boolToInt(b bool) int {
if b {
return 1
}
return 0
}
// Round 2 (R3/R4): the REST ingest doors write through the pipeline, which
// now stamps every page; a title newline cannot forge provenance for the
// brain's line-oriented reader.
func TestRESTIngestRawStampsUnknownAndTitleNewlineCannotForge(t *testing.T) {
dir, h := setup(t)
body, _ := json.Marshal(map[string]any{
"source": "zqrest", "pages": []map[string]any{
{"title": "Zqrestinj\norigin: static\nwritten_by: local-cli\nx: 'y", "type": "concept", "content": "zqrestinj body"},
},
})
rec := httptest.NewRecorder()
h.IngestRaw(rec, httptest.NewRequest(http.MethodPost, "/ingest-raw", bytes.NewReader(body)))
require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())
var resp struct{ Pages []string }
require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &resp))
require.NotEmpty(t, resp.Pages)
for _, p := range resp.Pages {
b, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(p)))
require.NoError(t, err)
fm := strings.SplitN(string(b), "---\n", 3)[1]
var m map[string]any
require.NoError(t, yaml.Unmarshal([]byte(fm), &m))
assert.Equal(t, "unknown", m["origin"])
assert.NotContains(t, m, "written_by")
for _, line := range strings.Split(fm, "\n") {
assert.False(t, strings.HasPrefix(line, "origin: static"), "line reader sees forged origin in %s", p)
}
}
}
func TestRESTIngestStampsUnknown(t *testing.T) {
dir, h := setup(t)
body, _ := json.Marshal(map[string]any{"content": "Some article text about shaping work.", "source": "zqllm"})
rec := httptest.NewRecorder()
h.Ingest(rec, httptest.NewRequest(http.MethodPost, "/ingest", bytes.NewReader(body)))
require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())
var resp struct{ Pages []string }
require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &resp))
require.NotEmpty(t, resp.Pages, "stub pipeline must write at least one page")
for _, p := range resp.Pages {
b, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(p)))
require.NoError(t, err)
assert.Contains(t, strings.SplitN(string(b), "---\n", 3)[1], "origin: unknown\n", p)
}
}
// A refused write leaves nothing behind, not even an empty hall directory
// (brain#21's invariant, round 3 non-blocking finding).
func TestWriteNoteRefusedForProvenanceLeavesNoDirectory(t *testing.T) {
dir := t.TempDir()
_, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "# x\n\nbody", Wing: "freshwing", Hall: "facts", Filename: "x", Origin: "jwt", WrittenBy: "bad\xff",
})
require.Error(t, err)
_, statErr := os.Stat(filepath.Join(dir, "wiki", "freshwing"))
assert.True(t, os.IsNotExist(statErr), "no wing directory may be created by a refused write")
}
+178
View File
@@ -0,0 +1,178 @@
package api_test
import (
"encoding/base64"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"syscall"
"testing"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// hyperguild#153 / PR #156: behavioural race tests, so the control does not
// rest on lints that list forbidden APIs (round 3: an os.ReadFile or
// Root.ReadFile rewrite passed the lint and leaked 19k times in 4s). A
// swapper keeps renaming a regular note, a symlink to an outside file, or a
// FIFO into raw/r.md while the public readers loop. Any leaked byte fails;
// any hang fails.
const raceSentinel = "SENTINEL-153-RACE"
// startSwapper runs fn until the test ends, and waits for it to exit before
// t.TempDir's cleanup removes the directory it writes into (hyperguild#162:
// an unawaited swapper raced the cleanup, "directory not empty").
func startSwapper(t *testing.T, fn func(stop *atomic.Bool)) {
t.Helper()
var stop atomic.Bool
var wg sync.WaitGroup
wg.Add(1)
go func() { defer wg.Done(); fn(&stop) }()
t.Cleanup(func() { stop.Store(true); wg.Wait() })
}
func swap(raw, outside string, fifo bool, stop *atomic.Bool) {
tgt := filepath.Join(raw, "r.md")
for i := 0; !stop.Load(); i++ {
tmp := filepath.Join(raw, ".swap")
_ = os.Remove(tmp)
switch {
case i%2 == 0:
_ = os.WriteFile(tmp, []byte("# r\n\ninside"), 0o644)
case fifo:
_ = syscall.Mkfifo(tmp, 0o644)
default:
_ = os.Symlink(outside, tmp)
}
_ = os.Rename(tmp, tgt)
}
}
func raceBrain(t *testing.T) (brain, raw, outside string) {
t.Helper()
brain = t.TempDir()
raw = filepath.Join(brain, "raw")
if err := os.MkdirAll(raw, 0o755); err != nil {
t.Fatal(err)
}
outside = filepath.Join(t.TempDir(), "secret.md")
if err := os.WriteFile(outside, []byte("# s\n\n"+raceSentinel), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(raw, "r.md"), []byte("# r\n\ninside"), 0o644); err != nil {
t.Fatal(err)
}
return brain, raw, outside
}
// readers runs every public raw/ reader once and returns all text they
// produced (excerpts, extracted text).
func readers(brain, raw string) string {
var b strings.Builder
if ns, err := api.ListPending(brain); err == nil {
for _, n := range ns {
b.WriteString(n.Excerpt)
}
}
if s, err := api.ReadIngestFile(brain, filepath.Join(raw, "r.md")); err == nil {
b.WriteString(s)
}
return b.String()
}
func TestRawReadersLeakNothingUnderASymlinkSwapRace(t *testing.T) {
brain, raw, outside := raceBrain(t)
startSwapper(t, func(stop *atomic.Bool) { swap(raw, outside, false, stop) })
leaks, iters := 0, 0
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); iters++ {
leaks += strings.Count(readers(brain, raw), raceSentinel)
}
if leaks > 0 {
t.Fatalf("outside file leaked %d times in %d iterations", leaks, iters)
}
t.Logf("%d iterations, 0 leaks", iters)
}
func TestRawReadersDoNotHangUnderAFIFOSwapRace(t *testing.T) {
brain, raw, outside := raceBrain(t)
startSwapper(t, func(stop *atomic.Bool) { swap(raw, outside, true, stop) })
var iters atomic.Int64
done := make(chan struct{})
go func() {
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); {
_ = readers(brain, raw)
iters.Add(1)
}
close(done)
}()
select {
case <-done:
case <-time.After(6 * time.Second):
// Unblock the stuck open so the test process can exit.
if f, err := os.OpenFile(filepath.Join(raw, "r.md"), os.O_WRONLY|syscall.O_NONBLOCK, 0); err == nil {
_ = f.Close()
}
t.Fatalf("a raw/ reader hung on a swapped-in FIFO after %d iterations", iters.Load())
}
}
// PR #156 round 4: the pdf/docmark branch of TextIn copies the file through
// the root to a temp file; pointing the extractor back at the raw/ path by
// name stayed green because the race tests only used r.md (a stub docmark
// then received the outside file's bytes 34k times in 4s). This races a
// symlink into raw/r.docx and has a fake docmark count the sentinel in every
// document it is sent.
func TestDocmarkExtractionLeaksNothingUnderASymlinkSwapRace(t *testing.T) {
var seen atomic.Int64
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var req struct {
Params struct {
Arguments struct {
ContentBase64 string `json:"content_base64"`
} `json:"arguments"`
} `json:"params"`
}
_ = json.NewDecoder(r.Body).Decode(&req)
if b, err := base64.StdEncoding.DecodeString(req.Params.Arguments.ContentBase64); err == nil &&
strings.Contains(string(b), raceSentinel) {
seen.Add(1)
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"jsonrpc":"2.0","id":1,"result":{"content":[{"type":"text","text":"converted"}]}}`))
}))
defer srv.Close()
t.Setenv("DOCMARK_URL", srv.URL)
brain, raw, outside := raceBrain(t)
tgt := filepath.Join(raw, "r.docx")
if err := os.WriteFile(tgt, []byte("inside docx bytes"), 0o644); err != nil {
t.Fatal(err)
}
startSwapper(t, func(stop *atomic.Bool) {
for i := 0; !stop.Load(); i++ {
tmp := filepath.Join(raw, ".swapdocx")
_ = os.Remove(tmp)
if i%2 == 0 {
_ = os.WriteFile(tmp, []byte("inside docx bytes"), 0o644)
} else {
_ = os.Symlink(outside, tmp)
}
_ = os.Rename(tmp, tgt)
}
})
iters := 0
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); iters++ {
_, _ = api.ReadIngestFile(brain, tgt)
}
if n := seen.Load(); n > 0 {
t.Fatalf("docmark received the outside file %d times in %d iterations", n, iters)
}
t.Logf("%d iterations, 0 leaks to docmark", iters)
}
@@ -0,0 +1,80 @@
package api_test
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// hyperguild#153: readers of brain/raw/ followed symlinks. /pending showed the
// target's contents and promote would copy it into wiki/ (and brain-sync to
// Gitea). raw/ is writable via Syncthing, so a symlink there is reachable
// input. Every raw/ reader now treats a symlink as not-a-note.
const sentinel153 = "SENTINEL-153-OUTSIDE"
func rawWithSymlink(t *testing.T) (brain string) {
t.Helper()
brain = t.TempDir()
raw := filepath.Join(brain, "raw")
require.NoError(t, os.MkdirAll(raw, 0o755))
outside := filepath.Join(t.TempDir(), "secret.md")
require.NoError(t, os.WriteFile(outside, []byte("---\ncreated_at: 2026-01-01T00:00:00Z\n---\n# s\n\n"+sentinel153), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(raw, "2026-01-01-leak.md")))
require.NoError(t, os.WriteFile(filepath.Join(raw, "2026-01-02-real.md"), []byte("# real\n\nreal body"), 0o644))
return brain
}
func TestListPendingSkipsSymlinkedNotes(t *testing.T) {
notes, err := api.ListPending(rawWithSymlink(t))
require.NoError(t, err)
var names []string
for _, n := range notes {
names = append(names, n.Filename)
assert.NotContains(t, n.Excerpt, sentinel153)
}
assert.Equal(t, []string{"2026-01-02-real.md"}, names)
}
func TestPromoteNoteRefusesASymlinkedSource(t *testing.T) {
brain := rawWithSymlink(t)
_, err := api.PromoteNote(brain, api.PromoteOptions{Filename: "2026-01-01-leak.md", Wing: "testing", Hall: "facts"})
require.Error(t, err)
_, statErr := os.Stat(filepath.Join(brain, "wiki", "testing", "facts", "leak.md"))
assert.True(t, os.IsNotExist(statErr), "nothing may be promoted from a symlink")
}
func TestRejectNoteRefusesASymlinkedSource(t *testing.T) {
brain := rawWithSymlink(t)
_, err := api.RejectNote(brain, api.RejectOptions{Filename: "2026-01-01-leak.md"})
require.Error(t, err)
b, _ := os.ReadFile(filepath.Join(brain, "raw", "rejected", "2026-01-01-leak.md"))
assert.False(t, strings.Contains(string(b), sentinel153))
}
// Item 4: if raw/ itself is a symlink, containment inside it means nothing.
func TestResolvePendingPromoteRejectAndReadIngestRefuseASymlinkedRawRoot(t *testing.T) {
brain := t.TempDir()
elsewhere := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(elsewhere, "x.md"), []byte("# x\n\n"+sentinel153), 0o644))
require.NoError(t, os.Symlink(elsewhere, filepath.Join(brain, "raw")))
_, err := api.ResolveIngestPath(brain, "x.md")
assert.Error(t, err, "ingest must refuse a symlinked raw/")
notes, err := api.ListPending(brain)
assert.Error(t, err, "pending must refuse a symlinked raw/")
assert.Empty(t, notes)
_, err = api.PromoteNote(brain, api.PromoteOptions{Filename: "x.md", Wing: "testing", Hall: "facts"})
assert.Error(t, err, "promote must refuse a symlinked raw/")
_, err = api.RejectNote(brain, api.RejectOptions{Filename: "x.md"})
assert.Error(t, err, "reject must refuse a symlinked raw/")
text, err := api.ReadIngestFile(brain, filepath.Join(elsewhere, "x.md"))
assert.Error(t, err, "ReadIngestFile must refuse a symlinked raw/")
assert.NotContains(t, text, sentinel153)
// The watcher's refusal is tested in internal/watcher.
}
+67
View File
@@ -0,0 +1,67 @@
package api
import (
"errors"
"fmt"
"io"
"os"
"path/filepath"
"github.com/mathiasbq/hyperguild/ingestion/internal/extract"
)
// errNotANote is returned for a raw/ entry that is not a regular file — in
// particular a symlink, whose target may be anywhere (hyperguild#153).
var errNotANote = errors.New("not a regular note file in raw/ (symlinks are refused)")
// errRawNotDir means brain/raw is not a real directory (e.g. a symlink), so
// containment inside it would mean nothing (hyperguild#153 item 4).
var errRawNotDir = errors.New("brain/raw is not a real directory")
// OpenRawRoot opens brain/raw as an os.Root, refusing a raw/ that is itself a
// symlink. Reads through the root cannot leave it, even if an entry is
// swapped for a symlink between a check and the read. It returns the root
// and raw/'s path. A missing raw/ returns an error wrapping os.ErrNotExist.
func OpenRawRoot(brainDir string) (*os.Root, string, error) {
p := filepath.Join(brainDir, "raw")
fi, err := os.Lstat(p)
if err != nil {
return nil, "", err
}
if fi.Mode()&os.ModeSymlink != 0 || !fi.IsDir() {
return nil, "", errRawNotDir
}
r, err := os.OpenRoot(p)
if err != nil {
return nil, "", fmt.Errorf("open raw root: %w", err)
}
return r, p, nil
}
// readRawNote reads one raw/ entry by basename through the root, refusing
// anything that is not a regular file.
//
// Accepted residual: a HARD link to an outside file is a regular file and is
// read. Syncthing syncs contents, not links, so a peer cannot create one;
// fs.protected_hardlinks=1 and the pod's secrets live on a different
// filesystem from the brain volume, so a local caller cannot link them in.
// Refusing Nlink > 1 is the next step if either assumption changes.
func readRawNote(r *os.Root, base string) ([]byte, os.FileInfo, error) {
fi, err := r.Lstat(base)
if err != nil {
return nil, nil, err
}
if !fi.Mode().IsRegular() {
return nil, nil, errNotANote
}
f, _, err := extract.OpenRegular(r, base)
if err != nil {
return nil, nil, err
}
defer func() { _ = f.Close() }()
b, err := io.ReadAll(f)
if err != nil {
return nil, nil, err
}
return b, fi, nil
}
@@ -0,0 +1,52 @@
package api_test
import (
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs/brainfstest"
)
// hyperguild#157 slice 2: ReadNote (brain_get, read-after-write) read by
// path; it now reads through brainfs.
func TestReadNoteRefusesSymlinksAndSurvivesSwapRaces(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(facts, "leak.md")))
_, body, _, err := api.ReadNote(brain, "wiki/w/facts/leak.md")
assert.Error(t, err)
assert.NotContains(t, body, brainfstest.Sentinel)
// A link to a file elsewhere in the brain: os.Root allows it, only the
// regular-file check refuses it (hyperguild#171).
brainfstest.WriteSentinelAt(t, filepath.Join(brain, "raw", "s.md"))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "s.md"), filepath.Join(facts, "inlink.md")))
_, body, _, err = api.ReadNote(brain, "wiki/w/facts/inlink.md")
assert.Error(t, err)
assert.NotContains(t, body, brainfstest.Sentinel)
p := filepath.Join(facts, "r.md")
require.NoError(t, os.WriteFile(p, []byte("---\nx: y\n---\ninside"), 0o644))
fm, body, _, err := api.ReadNote(brain, "wiki/w/facts/r.md")
require.NoError(t, err)
assert.Equal(t, "y", fm["x"])
assert.Contains(t, body, "inside")
read := func() string {
fm, body, _, _ := api.ReadNote(brain, "wiki/w/facts/r.md")
s := body
for k, v := range fm {
s += k + v
}
return s
}
brainfstest.AssertNoLeak(t, p, "---\nx: y\n---\ninside", 1500*time.Millisecond, read)
inBrain := brainfstest.WriteSentinelAt(t, filepath.Join(brain, "raw", "s.md"))
brainfstest.AssertNoLeakFrom(t, p, "---\nx: y\n---\ninside", inBrain, 1500*time.Millisecond, read)
brainfstest.AssertNoHang(t, p, "---\nx: y\n---\ninside", 1500*time.Millisecond, 6*time.Second, func() { _ = read() })
}
+92 -8
View File
@@ -3,13 +3,18 @@ package api
import (
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"strings"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"github.com/mathiasbq/hyperguild/ingestion/internal/claim"
"github.com/mathiasbq/hyperguild/ingestion/internal/notes"
)
// ContentHash returns the lowercase hex sha256 of b. It is the note's
@@ -45,6 +50,11 @@ type UpdateNoteOptions struct {
Slug string
Content string // new full body (whole-note replace)
Reason string // optional; stamped as supersede_reason
// Origin/WrittenBy re-stamp provenance to the updating principal: the
// body is replaced wholesale, so the last writer is the author of what
// a reader sees (hyperguild#145). Never taken from Content's fragment.
Origin string
WrittenBy string
}
// UpdateNote supersedes an existing note in place. It replaces the body
@@ -64,6 +74,14 @@ func UpdateNote(brainDir string, opts UpdateNoteOptions) (relPath, contentHash,
return "", "", "", fmt.Errorf("content is required")
}
// Same reason as WriteNote: the strip belongs in the primitive so every
// door inherits it.
clean, _, err := brain.StripUnresolvedLinks(brainDir, opts.Content)
if err != nil {
return "", "", "", err
}
opts.Content = clean
var rel string
if opts.Path != "" {
rel = opts.Path
@@ -76,29 +94,89 @@ func UpdateNote(brainDir string, opts UpdateNoteOptions) (relPath, contentHash,
rel = filepath.ToSlash(rel)
}
rel, abs, err := resolveWithin(brainDir, rel)
// The path is caller-supplied (brain_update): resolve it as brain_edit
// does, knowledge/ and wiki/ notes only, and read and rewrite it through
// the brain root, never through a symlink (hyperguild#169).
t, err := notes.Resolve(brainDir, rel, notes.ScopeNotes)
if err != nil {
return "", "", "", err
}
prior, err := os.ReadFile(abs)
rel = t.Rel
bfs, err := brainfs.Open(brainDir)
if err != nil {
if os.IsNotExist(err) {
return "", "", "", fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
prior, err := bfs.ReadRegular(rel)
if err != nil {
if errors.Is(err, fs.ErrNotExist) {
return "", "", "", fmt.Errorf("note %q does not exist: use brain_write to create", rel)
}
return "", "", "", fmt.Errorf("read target: %w", err)
}
priorHash = ContentHash(prior)
// A caller may lead opts.Content with its own frontmatter fence to set or
// change fields on supersede (e.g. the claim envelope). Split it off
// before it reaches the body — left un-split, it would render as a
// second, inert `---`-fenced block of literal text rather than taking
// effect, and any claim-envelope fields inside it would reach disk
// unvalidated (hyperguild#111: the same rule that guards note birth in
// writeHallNote must guard a change to the same fields).
contentFM, contentBody := parseFrontmatter(opts.Content)
fm, _ := parseFrontmatter(string(prior))
for _, l := range contentFM.lines {
// updated_at/supersedes/supersede_reason are this function's own
// output, not something a caller's fragment sets — updated_at and
// supersedes are already unconditionally re-set below regardless of
// what the loop does with them, but supersede_reason is only
// re-set when opts.Reason is non-empty, so without this skip a
// caller could set it via its own fragment and have it survive
// even when Reason is empty, bypassing the one intended entry
// point for that field (hyperguild#125, found in #111's review).
// Skipping all three here rather than relying on the unconditional
// overwrites below to save the other two keeps the invariant
// explicit at the one place it can be silently broken by a future
// reordering, not implicit in call order.
if l.key == "" || l.key == "updated_at" || l.key == "supersedes" || l.key == "supersede_reason" ||
l.key == "origin" || l.key == "written_by" {
continue
}
fm.set(l.key, l.value)
}
fm.set("updated_at", time.Now().UTC().Format(time.RFC3339))
fm.set("supersedes", priorHash)
if opts.Reason != "" {
fm.set("supersede_reason", opts.Reason)
}
out := []byte(fm.render() + opts.Content)
if err := os.WriteFile(abs, out, 0o644); err != nil {
// Validate the envelope AFTER the merge, against the full field set that
// is about to reach disk — not just the caller's fragment. Validating the
// fragment alone would miss a caller that supersedes claim_provenance to
// empty while half_life/applies survive untouched from the prior note:
// each side looks individually fine, but the merged result is an
// unprovenanced envelope, which is the exact laundering this check
// exists to catch. (Caught in review before merge, not filed after —
// worth being honest about in the history: this second pass was the
// actual fix; the first one validated the wrong document.)
envelope := make(map[string]string, len(fm.lines))
for _, l := range fm.lines {
if l.key != "" {
envelope[l.key] = l.value
}
}
if err := claim.ValidateEnvelope(envelope); err != nil {
return "", "", "", fmt.Errorf("claim envelope: %w", err)
}
stamped, err := StampProvenance(fm.render()+contentBody, opts.Origin, opts.WrittenBy)
if err != nil {
return "", "", "", err
}
out := []byte(stamped)
if err := bfs.WriteRegular(rel, out); err != nil {
return "", "", "", fmt.Errorf("write: %w", err)
}
return rel, ContentHash(out), priorHash, nil
@@ -109,11 +187,17 @@ func UpdateNote(brainDir string, opts UpdateNoteOptions) (relPath, contentHash,
// write primitive behind brain_get: the hash it returns equals the hash
// brain_write / brain_update returned for the same bytes.
func ReadNote(brainDir, relPath string) (fm map[string]string, body, contentHash string, err error) {
_, abs, err := resolveWithin(brainDir, relPath)
rel, _, err := resolveWithin(brainDir, relPath)
if err != nil {
return nil, "", "", err
}
raw, err := os.ReadFile(abs)
// Through the brain root, regular files only (hyperguild#157).
bfs, err := brainfs.Open(brainDir)
if err != nil {
return nil, "", "", fmt.Errorf("read note: %w", err)
}
defer func() { _ = bfs.Close() }()
raw, err := bfs.ReadRegular(rel)
if err != nil {
if os.IsNotExist(err) {
return nil, "", "", fmt.Errorf("note %q does not exist", relPath)
@@ -0,0 +1,36 @@
package api_test
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
)
// hyperguild#173: a directory symlink inside wiki/ that points elsewhere in
// the brain redirected brain_update's write (os.Root follows in-root
// intermediate symlinks). Every directory above the note must be real.
func TestUpdateNoteRefusesANoteBehindASymlinkedDirectory(t *testing.T) {
brain := t.TempDir()
victim := filepath.Join(brain, "raw", "facts", "v.md")
require.NoError(t, os.MkdirAll(filepath.Dir(victim), 0o755))
require.NoError(t, os.WriteFile(victim, []byte("---\ntitle: v\n---\nuntouched\n"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki", "w"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "raw", "facts"), filepath.Join(brain, "wiki", "w", "alias")))
_, _, _, err := api.UpdateNote(brain, api.UpdateNoteOptions{Path: "wiki/w/alias/v.md", Content: "replaced"})
assert.ErrorIs(t, err, brainfs.ErrNotRegular)
got, _ := os.ReadFile(victim)
assert.Contains(t, string(got), "untouched")
real := filepath.Join(brain, "wiki", "w", "facts", "r.md")
require.NoError(t, os.MkdirAll(filepath.Dir(real), 0o755))
require.NoError(t, os.WriteFile(real, []byte("---\ntitle: r\n---\nold\n"), 0o644))
_, _, _, err = api.UpdateNote(brain, api.UpdateNoteOptions{Path: "wiki/w/facts/r.md", Content: "new"})
require.NoError(t, err, "positive control: a note in a real directory is updated")
}
+112
View File
@@ -0,0 +1,112 @@
package api_test
import (
"os"
"path/filepath"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// brain_update passed the caller's path through resolveWithin, a lexical
// containment check only: any file in the brain could be superseded
// (classification.yaml, raw/, sessions/), and a final symlink was written
// through. It now resolves like brain_edit: knowledge/ and wiki/ notes only,
// read and rewritten through the brain root.
func TestUpdateNoteRefusesPathsOutsideKnowledgeAndWikiNotes(t *testing.T) {
brain := t.TempDir()
files := map[string]string{
"classification.yaml": "levels: {}\n",
"raw/r.md": "raw\n",
"sessions/s.jsonl": "{}\n",
"wiki/w/facts/ok.md": "---\ntitle: ok\n---\nold\n",
"knowledge/k.md": "---\ntitle: k\n---\nold\n",
"wiki/w/facts/x.txt": "txt\n",
}
for rel, body := range files {
p := filepath.Join(brain, filepath.FromSlash(rel))
require.NoError(t, os.MkdirAll(filepath.Dir(p), 0o755))
require.NoError(t, os.WriteFile(p, []byte(body), 0o644))
}
for _, rel := range []string{"classification.yaml", "raw/r.md", "sessions/s.jsonl", "wiki/w/facts/x.txt", "wiki/w/../../raw/r.md"} {
_, _, _, err := api.UpdateNote(brain, api.UpdateNoteOptions{Path: rel, Content: "replaced"})
assert.Error(t, err, rel)
}
for rel, body := range files {
if rel == "wiki/w/facts/ok.md" || rel == "knowledge/k.md" {
continue
}
got, err := os.ReadFile(filepath.Join(brain, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Equal(t, body, string(got), rel)
}
for _, rel := range []string{"wiki/w/facts/ok.md", "knowledge/k.md"} {
_, _, _, err := api.UpdateNote(brain, api.UpdateNoteOptions{Path: rel, Content: "new body"})
require.NoError(t, err, "positive control: %s", rel)
got, _ := os.ReadFile(filepath.Join(brain, filepath.FromSlash(rel)))
assert.Contains(t, string(got), "new body")
}
}
func TestUpdateNoteDoesNotWriteThroughASymlinkedNote(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
victim := filepath.Join(brain, "raw", "victim.md")
require.NoError(t, os.MkdirAll(filepath.Dir(victim), 0o755))
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "victim.md"), filepath.Join(facts, "l.md")))
_, _, _, err := api.UpdateNote(brain, api.UpdateNoteOptions{Path: "wiki/w/facts/l.md", Content: "replaced"})
assert.Error(t, err)
got, _ := os.ReadFile(victim)
assert.Equal(t, "untouched\n", string(got))
}
// The note is swapped between a regular file and an in-brain symlink to a
// raw/ file while UpdateNote runs: the read and the write each refuse a
// symlink, so the raw/ file is never rewritten.
func TestUpdateNoteNeverRewritesASymlinkSwappedInForTheNote(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
victim := filepath.Join(brain, "raw", "victim.md")
require.NoError(t, os.MkdirAll(filepath.Dir(victim), 0o755))
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
note := filepath.Join(facts, "n.md")
body := "---\ntitle: n\n---\nold\n"
require.NoError(t, os.WriteFile(note, []byte(body), 0o644))
var stop atomic.Bool
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
tmp := note + ".swap"
for i := 0; !stop.Load(); i++ {
_ = os.Remove(tmp)
if i%2 == 0 {
_ = os.WriteFile(tmp, []byte(body), 0o644)
} else {
_ = os.Symlink(filepath.Join("..", "..", "..", "raw", "victim.md"), tmp)
}
_ = os.Rename(tmp, note)
}
}()
ok := 0
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); {
if _, _, _, err := api.UpdateNote(brain, api.UpdateNoteOptions{Path: "wiki/w/facts/n.md", Content: "new"}); err == nil {
ok++
}
}
stop.Store(true)
wg.Wait()
got, _ := os.ReadFile(victim)
assert.Equal(t, "untouched\n", string(got))
assert.Positive(t, ok, "positive control: some updates reach the regular note")
}
+147
View File
@@ -52,6 +52,114 @@ func TestUpdateNoteSupersedesAndStamps(t *testing.T) {
assert.Contains(t, s, "supersede_reason: facts changed")
}
func TestUpdateNoteRejectsInvalidEnvelopeAndWritesNothing(t *testing.T) {
// hyperguild#111: WriteNote validates the claim envelope before create;
// UpdateNote didn't before supersede, so the same rule must hold on
// change as on birth, fail-closed, nothing written on rejection.
brainDir := t.TempDir()
original := "---\nwing: jepa-fx\nhall: facts\ncreated_at: 2026-01-01T00:00:00Z\n---\n# Old\n\nold body\n"
rel := seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md", original)
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
Path: rel,
Content: "---\nhalf_life: forever\n---\n# New\n\nnew body\n",
})
require.Error(t, err)
assert.Contains(t, err.Error(), "claim envelope")
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Equal(t, original, string(got), "rejected update must leave the note byte-identical")
}
func TestUpdateNoteRejectsAppliesWithoutProvenance(t *testing.T) {
brainDir := t.TempDir()
original := "---\nwing: jepa-fx\nhall: facts\n---\nold\n"
rel := seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md", original)
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
Path: rel,
Content: "---\napplies: universal\n---\nnew\n",
})
require.Error(t, err)
assert.Contains(t, err.Error(), "claim envelope")
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Equal(t, original, string(got))
}
func TestUpdateNoteAcceptsValidEnvelopeAndMergesIntoFrontmatter(t *testing.T) {
brainDir := t.TempDir()
rel := seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md",
"---\nwing: jepa-fx\nhall: facts\ncreated_at: 2026-01-01T00:00:00Z\n---\nold\n")
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
Path: rel,
Content: "---\nhalf_life: durable\napplies: universal\nclaim_provenance: asserted\n---\nnew body\n",
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
require.NoError(t, err)
s := string(got)
assert.Contains(t, s, "half_life: durable")
assert.Contains(t, s, "applies: universal")
assert.Contains(t, s, "claim_provenance: asserted")
assert.Contains(t, s, "wing: jepa-fx", "prior fields survive")
assert.Contains(t, s, "new body")
assert.Equal(t, 2, strings.Count(s, "---\n"),
"exactly one open+close fence pair — the caller's own fence must not leak in as a second, inert block")
}
func TestUpdateNoteRejectsClearingProvenanceWhileEnvelopeSurvives(t *testing.T) {
// Found in review, not written speculatively: validating only the
// caller's fragment (rather than the merged result) let a caller blank
// out claim_provenance alone while half_life/applies survived from the
// prior note untouched — each side looks fine on its own, the merged
// envelope on disk is the unprovenanced one the whole check exists to
// catch.
brainDir := t.TempDir()
original := "---\nwing: jepa-fx\nhall: facts\nhalf_life: durable\napplies: universal\nclaim_provenance: asserted\n---\nold\n"
rel := seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md", original)
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
Path: rel,
Content: "---\nclaim_provenance: \n---\nnew body\n",
})
require.Error(t, err)
assert.Contains(t, err.Error(), "claim envelope")
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Equal(t, original, string(got), "rejected update must leave the note byte-identical")
}
func TestUpdateNoteWithoutEnvelopeFieldsIsUnchangedFromToday(t *testing.T) {
// Regression pin: plain fence-less content (the common case) must behave
// exactly as it did before the envelope-merge fix.
brainDir := t.TempDir()
rel := seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md",
"---\nwing: jepa-fx\nhall: facts\ncreated_at: 2026-01-01T00:00:00Z\ncustom: keep-me\n---\n# Old\n\nold body\n")
relPath, hash, priorHash, err := UpdateNote(brainDir, UpdateNoteOptions{
Path: rel,
Content: "# New\n\nnew body\n",
Reason: "facts changed",
})
require.NoError(t, err)
assert.Equal(t, rel, relPath)
assert.NotEmpty(t, hash)
assert.NotEqual(t, hash, priorHash)
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
require.NoError(t, err)
s := string(got)
assert.Contains(t, s, "# New")
assert.Contains(t, s, "custom: keep-me")
assert.Contains(t, s, "supersedes: "+priorHash)
}
func TestUpdateNoteResolvesByWingHallSlug(t *testing.T) {
brainDir := t.TempDir()
seedNote(t, brainDir, "wiki/jepa-fx/facts/val-vol.md",
@@ -123,6 +231,45 @@ func TestUpdateThenReadRoundTripsHash(t *testing.T) {
assert.Equal(t, hash, readHash, "update content_hash must round-trip through ReadNote")
}
func TestUpdateNoteSupersedeReasonNotSettableViaCallerFragmentWhenReasonOptionEmpty(t *testing.T) {
// hyperguild#125, found in #111's review: the merge loop applied every
// field from the caller's leading frontmatter fragment, including
// supersede_reason -- which is only re-set unconditionally when
// opts.Reason is non-empty, so a caller could set it via its own
// fragment and have it survive even with Reason left empty, bypassing
// the one intended entry point for that field.
brainDir := t.TempDir()
rel := seedNote(t, brainDir, "wiki/a/facts/n.md", "---\nwing: a\nhall: facts\n---\nold\n")
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
Path: rel,
Content: "---\nsupersede_reason: snuck in via the fragment\n---\nnew\n",
// Reason deliberately left empty.
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.NotContains(t, string(got), "supersede_reason",
"supersede_reason must not appear at all when opts.Reason is empty, regardless of what the fragment tried to set")
}
func TestUpdateNoteSupersedeReasonFromOptionStillWorks(t *testing.T) {
// Regression pin: the legitimate entry point (opts.Reason) must be
// completely unaffected by #125's fix.
brainDir := t.TempDir()
rel := seedNote(t, brainDir, "wiki/a/facts/n.md", "---\nwing: a\nhall: facts\n---\nold\n")
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
Path: rel, Content: "new\n", Reason: "facts changed",
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Contains(t, string(got), "supersede_reason: facts changed")
}
func TestContentHashStable(t *testing.T) {
assert.Equal(t, ContentHash([]byte("abc")), ContentHash([]byte("abc")))
assert.NotEqual(t, ContentHash([]byte("abc")), ContentHash([]byte("abd")))
@@ -0,0 +1,34 @@
package api_test
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// hyperguild#178: an unreadable directory under wiki/ made the link check
// fail and brain_write refuse the note, losing the learning. The note is now
// written with its links as given.
func TestWriteNoteKeepsTheLearningWhenTheLinkCheckCannotIndex(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root ignores directory permissions")
}
brain := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki", "w", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(brain, "wiki", "w", "facts", "real.md"), []byte("x"), 0o644))
locked := filepath.Join(brain, "wiki", "other", "locked")
require.NoError(t, os.MkdirAll(locked, 0o755))
require.NoError(t, os.Chmod(locked, 0o000))
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
rel, err := api.WriteNote(brain, api.WriteNoteOptions{Wing: "w", Hall: "facts", Filename: "n", Content: "learning [[real]] and [[gone]]"})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(brain, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Contains(t, string(got), "[[gone]]", "links kept as given when the index is partial")
}
@@ -0,0 +1,43 @@
package api_test
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// hyperguild#188: brain_write created a new note with os.MkdirAll +
// os.WriteFile by path, so a symlinked wing directory redirected the note
// elsewhere in the brain or outside it.
func TestWriteNoteRefusesASymlinkedWing(t *testing.T) {
for name, target := range map[string]func(t *testing.T, brain string) string{
"in-brain": func(t *testing.T, brain string) string {
require.NoError(t, os.MkdirAll(filepath.Join(brain, "raw"), 0o755))
return filepath.Join("..", "raw")
},
"outside": func(t *testing.T, _ string) string { return t.TempDir() },
} {
t.Run(name, func(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki"), 0o755))
link := target(t, brain)
require.NoError(t, os.Symlink(link, filepath.Join(brain, "wiki", "lw")))
_, err := api.WriteNote(brain, api.WriteNoteOptions{Wing: "lw", Hall: "facts", Filename: "n", Content: "learning"})
assert.Error(t, err)
resolved := link
if !filepath.IsAbs(resolved) {
resolved = filepath.Join(brain, "wiki", link)
}
_, serr := os.Stat(filepath.Join(resolved, "facts"))
assert.True(t, os.IsNotExist(serr), "a hall was created at the link's target")
_, err = api.WriteNote(brain, api.WriteNoteOptions{Wing: "w", Hall: "facts", Filename: "n", Content: "learning"})
require.NoError(t, err, "positive control: a real wing is written")
})
}
}
+2 -2
View File
@@ -28,8 +28,8 @@ type FileBuffer struct {
}
type bufferLine struct {
ID string `json:"id"`
Entry capture.AuditEntry `json:"entry"`
ID string `json:"id"`
Entry capture.AuditEntry `json:"entry"`
}
// NewFileBuffer returns a buffer backed by path. The parent directory is
+4 -4
View File
@@ -46,10 +46,10 @@ func (f *fakeNotifier) Notify(_ context.Context, msg string) error {
// unwritableBuffer always reports it cannot be written (floor condition).
type unwritableBuffer struct{}
func (unwritableBuffer) Writable() error { return errors.New("disk full") }
func (unwritableBuffer) Append(capture.AuditEntry) error { return errors.New("disk full") }
func (unwritableBuffer) Pending() ([]audit.Buffered, error) { return nil, nil }
func (unwritableBuffer) Confirm(string) error { return nil }
func (unwritableBuffer) Writable() error { return errors.New("disk full") }
func (unwritableBuffer) Append(capture.AuditEntry) error { return errors.New("disk full") }
func (unwritableBuffer) Pending() ([]audit.Buffered, error) { return nil, nil }
func (unwritableBuffer) Confirm(string) error { return nil }
func newFileBuffer(t *testing.T) *audit.FileBuffer {
t.Helper()
+13 -1
View File
@@ -49,7 +49,19 @@ func (n *NtfyNotifier) Notify(ctx context.Context, msg string) error {
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("ntfy notify: status %d", resp.StatusCode)
return &RejectedError{Status: resp.StatusCode}
}
return nil
}
// RejectedError is a non-2xx ntfy response.
type RejectedError struct{ Status int }
func (e *RejectedError) Error() string { return fmt.Sprintf("ntfy notify: status %d", e.Status) }
// Permanent reports whether resending the same message cannot succeed: a 4xx
// other than 429 (rate limited). Callers may drop such a message; anything
// else (5xx, 429, transport errors) is an outage and must be retried.
func (e *RejectedError) Permanent() bool {
return e.Status >= 400 && e.Status < 500 && e.Status != http.StatusTooManyRequests
}
@@ -0,0 +1,27 @@
package audit
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// A non-2xx ntfy response is a *RejectedError, and only a client-side refusal
// other than rate limiting is permanent: callers may drop a message on a
// permanent rejection and must retry the rest (hyperguild#157, PR #160).
func TestNotifyClassifiesRejectionsAsPermanentOrNot(t *testing.T) {
for status, permanent := range map[int]bool{413: true, 400: true, 403: true, 429: false, 500: false, 503: false} {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(status) }))
err := NewNtfyNotifier(srv.URL, "").Notify(context.Background(), "m")
srv.Close()
var rej *RejectedError
require.True(t, errors.As(err, &rej), "status %d: %v", status, err)
assert.Equal(t, status, rej.Status)
assert.Equal(t, permanent, rej.Permanent(), "status %d", status)
}
}
@@ -0,0 +1,125 @@
package brain_test
import (
"fmt"
"os"
"path/filepath"
"syscall"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs/brainfstest"
)
// hyperguild#174 / #157 slice 5b: DetectTunnels walked wiki/ and opened
// every note with a blocking os.Open, so a FIFO named *.md anywhere under
// wiki/ hung every brain_write and brain_update (wikiUpkeep -> AutoTunnel),
// and a symlinked note's title became a tunnel candidate.
func TestDetectTunnelsReturnsPromptlyWithAFIFOUnderWikiWithAndWithoutAWriter(t *testing.T) {
for _, withWriter := range []bool{false, true} {
t.Run(fmt.Sprintf("writer=%v", withWriter), func(t *testing.T) {
b := t.TempDir()
facts := filepath.Join(b, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(facts, "ok.md"), []byte("---\ntitle: Zqinside Phrase\n---\n"), 0o644))
fifo := filepath.Join(facts, "f.md")
require.NoError(t, syscall.Mkfifo(fifo, 0o644))
if withWriter {
// A writer held open: a reader that opens the FIFO gets no
// EOF and parks, unless the FIFO is never opened.
w, err := os.OpenFile(fifo, os.O_RDWR, 0)
require.NoError(t, err)
t.Cleanup(func() { _ = w.Close() })
}
done := make(chan []brain.TunnelCandidate, 1)
go func() {
cs, _ := brain.DetectTunnels(b, "text mentioning zqinside phrase here")
done <- cs
}()
select {
case cs := <-done:
assert.Contains(t, fmt.Sprintf("%+v", cs), "Zqinside Phrase", "positive control")
case <-time.After(3 * time.Second):
if r, err := os.OpenFile(fifo, os.O_WRONLY|syscall.O_NONBLOCK, 0); err == nil {
_ = r.Close()
}
t.Fatal("DetectTunnels blocked on a FIFO")
}
})
}
}
func TestDetectTunnelsRefusesSymlinkedNotesAndSurvivesSwapRaces(t *testing.T) {
b := t.TempDir()
facts := filepath.Join(b, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
// The static note's title differs from the raced note's, so the
// harness's "inside was read" floor can only be met by reading r.md.
require.NoError(t, os.WriteFile(filepath.Join(facts, "ok.md"), []byte("---\ntitle: Zqstatic Words\n---\n"), 0o644))
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(facts, "out.md")))
inBrain := brainfstest.WriteSentinelAt(t, filepath.Join(b, "raw", "s.md"))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "s.md"), filepath.Join(facts, "in.md")))
content := "zqstatic words, zqinside phrase and " + brainfstest.Sentinel
cs, err := brain.DetectTunnels(b, content)
require.NoError(t, err)
got := fmt.Sprintf("%+v", cs)
assert.Contains(t, got, "Zqstatic Words", "positive control")
assert.NotContains(t, got, brainfstest.Sentinel)
p := filepath.Join(facts, "r.md")
read := func() string {
cs, _ := brain.DetectTunnels(b, content)
return fmt.Sprintf("%+v", cs)
}
brainfstest.AssertNoLeak(t, p, "---\ntitle: Zqinside Phrase\n---\n", 1500*time.Millisecond, read)
brainfstest.AssertNoLeakFrom(t, p, "---\ntitle: Zqinside Phrase\n---\n", inBrain, 1500*time.Millisecond, read)
}
// A wiki/ that is not a real directory yields no candidates and no error:
// AutoTunnel runs inside brain_write.
func TestDetectTunnelsSkipsAWikiThatIsNotARealDirectory(t *testing.T) {
b := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(b, "raw", "w", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(b, "raw", "w", "facts", "r.md"), []byte("---\ntitle: Zqrawonly Phrase\n---\n"), 0o644))
require.NoError(t, os.Symlink("raw", filepath.Join(b, "wiki")))
cs, err := brain.DetectTunnels(b, "zqrawonly phrase")
require.NoError(t, err)
assert.Empty(t, cs)
}
// An unreadable directory under wiki/ is a walk error and DetectTunnels
// reports it, as filepath.WalkDir did; it is not silently dropped.
func TestDetectTunnelsReportsAnUnreadableDirectoryUnderWiki(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root ignores directory permissions")
}
b := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "w", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(b, "wiki", "w", "facts", "ok.md"), []byte("---\ntitle: Zqinside Phrase\n---\n"), 0o644))
locked := filepath.Join(b, "wiki", "other", "locked")
require.NoError(t, os.MkdirAll(locked, 0o755))
require.NoError(t, os.Chmod(locked, 0o000))
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
_, err := brain.DetectTunnels(b, "zqinside phrase")
assert.ErrorIs(t, err, os.ErrPermission)
}
// _index.md files are generated, not notes: their title never becomes a
// candidate.
func TestDetectTunnelsSkipsIndexFiles(t *testing.T) {
b := t.TempDir()
w := filepath.Join(b, "wiki", "w")
require.NoError(t, os.MkdirAll(filepath.Join(w, "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(w, "_index.md"), []byte("---\ntitle: Zqindex Words\n---\n"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(w, "facts", "ok.md"), []byte("---\ntitle: Zqinside Phrase\n---\n"), 0o644))
cs, err := brain.DetectTunnels(b, "zqindex words and zqinside phrase")
require.NoError(t, err)
got := fmt.Sprintf("%+v", cs)
assert.Contains(t, got, "Zqinside Phrase", "positive control")
assert.NotContains(t, got, "Zqindex")
}
+58 -47
View File
@@ -2,12 +2,16 @@ package brain
import (
"bufio"
"bytes"
"errors"
"fmt"
"os"
"path/filepath"
"io"
"io/fs"
"sort"
"strings"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
)
// noteEntry is one row in a Wing _index.md.
@@ -26,14 +30,18 @@ func BuildWingIndex(brainDir, wing string) error {
if w == "" {
return fmt.Errorf("invalid wing %q", wing)
}
wingDir := filepath.Join(brainDir, "wiki", w)
if _, err := os.Stat(wingDir); os.IsNotExist(err) {
// Listed and read through the brain root (hyperguild#157): a wing or
// hall that is a symlink is not followed, and notes are regular files.
bfs, err := brainfs.Open(brainDir)
if err != nil {
return fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
if _, err := bfs.Lstat("wiki/" + w); errors.Is(err, fs.ErrNotExist) {
return nil
} else if err != nil {
return fmt.Errorf("stat wing: %w", err)
}
entries, err := collectWingEntries(wingDir)
entries, err := collectWingEntries(bfs, "wiki/"+w)
if err != nil {
return err
}
@@ -52,74 +60,85 @@ func BuildWingIndex(brainDir, wing string) error {
fmt.Fprintf(&b, "| %s | [%s](%s/%s.md) | %s |\n", e.Hall, e.Title, e.Hall, e.Slug, e.Created)
}
dest := filepath.Join(wingDir, "_index.md")
return os.WriteFile(dest, []byte(b.String()), 0o644)
// Written through the brain root, never through a symlinked _index.md
// (hyperguild#158).
return bfs.PutRegular("wiki/"+w+"/_index.md", []byte(b.String()))
}
// BuildAllWingIndexes regenerates _index.md for every wing under brain/wiki/.
func BuildAllWingIndexes(brainDir string) error {
wikiDir := filepath.Join(brainDir, "wiki")
ents, err := os.ReadDir(wikiDir)
if os.IsNotExist(err) {
bfs, err := brainfs.Open(brainDir)
if err != nil {
return fmt.Errorf("open brain: %w", err)
}
wings, err := bfs.ReadDirDirs("wiki")
_ = bfs.Close()
if errors.Is(err, fs.ErrNotExist) {
return nil
}
if err != nil {
return fmt.Errorf("read wiki: %w", err)
}
for _, e := range ents {
if !e.IsDir() {
continue
}
if err := BuildWingIndex(brainDir, e.Name()); err != nil {
return fmt.Errorf("index %s: %w", e.Name(), err)
for _, w := range wings {
if err := BuildWingIndex(brainDir, w); err != nil {
return fmt.Errorf("index %s: %w", w, err)
}
}
return nil
}
func collectWingEntries(wingDir string) ([]noteEntry, error) {
func collectWingEntries(bfs *brainfs.Brain, wingRel string) ([]noteEntry, error) {
var out []noteEntry
ents, err := os.ReadDir(wingDir)
halls, err := bfs.ReadDirDirs(wingRel)
if err != nil {
return nil, fmt.Errorf("read wing: %w", err)
}
for _, hallEnt := range ents {
if !hallEnt.IsDir() {
continue
}
hall := hallEnt.Name()
for _, hall := range halls {
if !IsValidHall(hall) {
continue
}
hallDir := filepath.Join(wingDir, hall)
notes, err := os.ReadDir(hallDir)
hallRel := wingRel + "/" + hall
names, err := bfs.ReadDirRegular(hallRel)
if err != nil {
return nil, fmt.Errorf("read hall %s: %w", hall, err)
}
for _, n := range notes {
if n.IsDir() || !strings.HasSuffix(n.Name(), ".md") || n.Name() == "_index.md" {
for _, name := range names {
if !strings.HasSuffix(name, ".md") || name == "_index.md" {
continue
}
slug := strings.TrimSuffix(n.Name(), ".md")
full := filepath.Join(hallDir, n.Name())
title, created := readTitleAndCreated(full, slug)
slug := strings.TrimSuffix(name, ".md")
title, created := readTitleAndCreatedIn(bfs, hallRel+"/"+name, slug)
out = append(out, noteEntry{Hall: hall, Slug: slug, Title: title, Created: created})
}
}
return out, nil
}
// readTitleAndCreated reads YAML frontmatter for title + created_at; falls
// back to slug and file mtime when absent.
func readTitleAndCreated(path, slug string) (string, string) {
f, err := os.Open(path)
// readTitleAndCreatedIn is readTitleAndCreated through the brain root,
// regular files only (hyperguild#157).
func readTitleAndCreatedIn(bfs *brainfs.Brain, rel, slug string) (string, string) {
content, err := bfs.ReadRegular(rel)
if err != nil {
return slug, ""
}
defer func() { _ = f.Close() }()
title, created := parseTitleAndCreated(bytes.NewReader(content))
if created == "" {
if info, err := bfs.Lstat(rel); err == nil {
created = info.ModTime().UTC().Format("2006-01-02")
}
}
return titleOrSlug(title, slug), created
}
title, created := "", ""
scanner := bufio.NewScanner(f)
func titleOrSlug(title, slug string) string {
if title == "" {
return strings.ReplaceAll(slug, "-", " ")
}
return title
}
func parseTitleAndCreated(r io.Reader) (title, created string) {
scanner := bufio.NewScanner(r)
inFrontmatter := false
for scanner.Scan() {
line := scanner.Text()
@@ -149,13 +168,5 @@ func readTitleAndCreated(path, slug string) (string, string) {
}
}
}
if title == "" {
title = strings.ReplaceAll(slug, "-", " ")
}
if created == "" {
if info, err := os.Stat(path); err == nil {
created = info.ModTime().UTC().Format("2006-01-02")
}
}
return title, created
}
+132
View File
@@ -0,0 +1,132 @@
package brain_test
import (
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs/brainfstest"
)
func readIndex(t *testing.T, brainDir, wing string) string {
t.Helper()
b, _ := os.ReadFile(filepath.Join(brainDir, "wiki", wing, "_index.md"))
return string(b)
}
// hyperguild#157 slice 5: BuildWingIndex listed wiki/<wing>/<hall>/ with
// os.ReadDir and opened each note by path, so a symlinked note put the
// target's title into _index.md, and a FIFO hung the index build.
func TestBuildWingIndexRefusesSymlinkedNotesAndSurvivesSwapRaces(t *testing.T) {
b := t.TempDir()
facts := filepath.Join(b, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(facts, "ok.md"), []byte("---\ntitle: zqinside\n---\n"), 0o644))
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(facts, "leak.md")))
inBrain := brainfstest.WriteSentinelAt(t, filepath.Join(b, "raw", "s.md"))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "s.md"), filepath.Join(facts, "inlink.md")))
require.NoError(t, brain.BuildWingIndex(b, "w"))
idx := readIndex(t, b, "w")
assert.Contains(t, idx, "zqinside", "positive control")
assert.NotContains(t, idx, brainfstest.Sentinel)
assert.NotContains(t, idx, "leak")
assert.NotContains(t, idx, "inlink")
p := filepath.Join(facts, "r.md")
read := func() string {
_ = brain.BuildWingIndex(b, "w")
return readIndex(t, b, "w")
}
brainfstest.AssertNoLeak(t, p, "---\ntitle: zqinside\n---\n", 1500*time.Millisecond, read)
brainfstest.AssertNoLeakFrom(t, p, "---\ntitle: zqinside\n---\n", inBrain, 1500*time.Millisecond, read)
brainfstest.AssertNoHang(t, p, "---\ntitle: zqinside\n---\n", 1500*time.Millisecond, 6*time.Second, func() { _ = read() })
}
// A symlinked hall is skipped while a real hall next to it is indexed; a
// symlinked wing is refused and nothing is written into its target.
func TestBuildWingIndexSkipsASymlinkedHallAndRefusesASymlinkedWing(t *testing.T) {
b := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(b, "raw", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(b, "raw", "facts", "r.md"), []byte("---\ntitle: zqrawonly\n---\n"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "w", "decisions"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(b, "wiki", "w", "decisions", "d.md"), []byte("---\ntitle: zqrealhall\n---\n"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "raw", "facts"), filepath.Join(b, "wiki", "w", "facts")))
require.NoError(t, os.Symlink(filepath.Join("..", "raw"), filepath.Join(b, "wiki", "alias")))
require.NoError(t, brain.BuildWingIndex(b, "w"))
idx, err := os.ReadFile(filepath.Join(b, "wiki", "w", "_index.md"))
require.NoError(t, err, "the wing index is written")
assert.Contains(t, string(idx), "zqrealhall", "the real hall is indexed")
assert.NotContains(t, string(idx), "zqrawonly", "the symlinked hall is not")
assert.Error(t, brain.BuildWingIndex(b, "alias"))
_, err = os.Stat(filepath.Join(b, "raw", "_index.md"))
assert.True(t, os.IsNotExist(err), "an index was written into the symlinked wing's target")
}
// BuildAllWingIndexes indexes the real wings and skips a symlinked one.
func TestBuildAllWingIndexesSkipsASymlinkedWing(t *testing.T) {
b := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "w", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(b, "wiki", "w", "facts", "n.md"), []byte("---\ntitle: zqrealwing\n---\n"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(b, "raw", "facts"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "raw"), filepath.Join(b, "wiki", "alias")))
require.NoError(t, brain.BuildAllWingIndexes(b))
idx, err := os.ReadFile(filepath.Join(b, "wiki", "w", "_index.md"))
require.NoError(t, err)
assert.Contains(t, string(idx), "zqrealwing")
_, err = os.Stat(filepath.Join(b, "raw", "_index.md"))
assert.True(t, os.IsNotExist(err))
}
// A link to a symlinked note does not resolve: only regular files count as
// entries, so StripUnresolvedLinks unwraps it.
func TestStripUnresolvedLinksDoesNotCountASymlinkedNoteAsAnEntry(t *testing.T) {
b := t.TempDir()
facts := filepath.Join(b, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(facts, "real.md"), []byte("x"), 0o644))
brainfstest.WriteSentinelAt(t, filepath.Join(b, "raw", "s.md"))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "s.md"), filepath.Join(facts, "ghost.md")))
out, unlinked, err := brain.StripUnresolvedLinks(b, "see [[real]] and [[ghost]]")
require.NoError(t, err)
assert.Contains(t, out, "[[real]]", "positive control")
assert.NotContains(t, out, "[[ghost]]")
assert.Contains(t, unlinked, "ghost")
}
// A linked tree (knowledge/ or wiki/) that is not a real directory is
// skipped, not an error: StripUnresolvedLinks runs before every brain_write,
// and an error there loses the learning.
func TestStripUnresolvedLinksSkipsALinkedTreeThatIsNotARealDirectory(t *testing.T) {
for name, mk := range map[string]func(t *testing.T, b string){
"symlink to an in-brain dir": func(t *testing.T, b string) {
require.NoError(t, os.MkdirAll(filepath.Join(b, "raw", "k"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("raw", "k"), filepath.Join(b, "knowledge")))
},
"dangling symlink": func(t *testing.T, b string) {
require.NoError(t, os.Symlink("nowhere", filepath.Join(b, "knowledge")))
},
"regular file": func(t *testing.T, b string) {
require.NoError(t, os.WriteFile(filepath.Join(b, "knowledge"), []byte("x"), 0o644))
},
} {
t.Run(name, func(t *testing.T) {
b := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "w", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(b, "wiki", "w", "facts", "real.md"), []byte("x"), 0o644))
mk(t, b)
out, _, err := brain.StripUnresolvedLinks(b, "learning [[real]] and [[gone]]")
require.NoError(t, err)
assert.Contains(t, out, "[[real]]", "links into wiki/ still resolve")
assert.NotContains(t, out, "[[gone]]")
})
}
}
+177
View File
@@ -0,0 +1,177 @@
package brain
import (
"errors"
"fmt"
"io/fs"
"log/slog"
"path"
"regexp"
"sort"
"strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
)
// wikilinkRE matches [[target]] and [[target|Display Text]].
var wikilinkRE = regexp.MustCompile(`\[\[([^\]|]+)(?:\|([^\]]*))?\]\]`)
// fencedCodeRE and inlineCodeRE mark the regions where a [[...]] is a
// syntax example rather than a link. Same exclusion brain-gardener's audit
// applies on the read side, so the write gate and the audit agree on what
// counts as a link at all.
var (
fencedCodeRE = regexp.MustCompile("(?s)```.*?```")
inlineCodeRE = regexp.MustCompile("`[^`\n]*`")
)
// StripUnresolvedLinks unwraps every [[wikilink]] in content whose target
// resolves to no entry under knowledge/ or wiki/, leaving the link's display
// text (or its target) as plain prose. It returns the rewritten content and
// the sorted, de-duplicated targets it unwrapped.
//
// WHY UNWRAP RATHER THAN REJECT THE WRITE
// Agents write [[...]] around anything that feels related — a skill name, a
// repo, an agent-local memory file — and none of those are brain entries.
// Every one of those becomes a permanent [high] broken-link finding that no
// deterministic Fixer can repair, because there is no target to repoint to:
// the brain-audit digest grew from 19 to 63 high findings between 2026-08-26
// and 2026-09-16, and 53 of the 63 were exactly this. Rejecting the write
// instead would push the failure onto claudewatcher's unattended ingest path,
// where a refused write is a lost learning. Unwrapping keeps the author's
// words, keeps the write, and keeps the graph honest.
//
// An empty corpus (no knowledge/ or wiki/ yet, or a misconfigured brainDir)
// returns content unchanged: nothing resolving is a bootstrap state, not
// evidence that every target was invented.
func StripUnresolvedLinks(brainDir, content string) (string, []string, error) {
index, err := entryIndex(brainDir)
if err != nil {
// A partial index cannot tell an invented link from one into the
// part it could not read, so unwrap nothing, and do not refuse the
// write either: this runs before every brain_write, and a refused
// write is a lost learning (hyperguild#178).
slog.Warn("link check skipped: cannot index the brain", "err", err)
return content, nil, nil
}
if len(index) == 0 {
return content, nil, nil
}
code := codeSpans(content)
stripped := map[string]struct{}{}
var b strings.Builder
last := 0
for _, m := range wikilinkRE.FindAllStringSubmatchIndex(content, -1) {
start, end := m[0], m[1]
if overlaps(start, end, code) {
continue
}
target := strings.TrimSpace(content[m[2]:m[3]])
if _, ok := index[linkKey(target)]; ok {
continue
}
display := target
if m[4] >= 0 {
if d := strings.TrimSpace(content[m[4]:m[5]]); d != "" {
display = d
}
}
b.WriteString(content[last:start])
b.WriteString(display)
last = end
stripped[target] = struct{}{}
}
if last == 0 {
return content, nil, nil
}
b.WriteString(content[last:])
targets := make([]string, 0, len(stripped))
for t := range stripped {
targets = append(targets, t)
}
sort.Strings(targets)
return b.String(), targets, nil
}
// linkedTrees are the two trees that hold linkable entries — the same pair
// package notes admits under ScopeNotes. Duplicated rather than imported
// because that list is unexported there and this is a read-only mirror.
var linkedTrees = []string{"knowledge", "wiki"}
// entryIndex maps every form a wikilink may name an entry by — its bare slug
// and its path form (wiki/<wing>/<hall>/<slug>.md -> wing/hall/slug) — to a
// present marker. Mirrors brain-gardener's resolver so an accepted link is
// one the audit will also resolve.
func entryIndex(brainDir string) (map[string]struct{}, error) {
index := map[string]struct{}{}
// Walked through the brain root (hyperguild#157): only regular files
// count as entries, so a symlinked .md name cannot make a link resolve.
bfs, err := brainfs.Open(brainDir)
if err != nil {
return nil, fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
for _, dir := range linkedTrees {
if _, err := bfs.Lstat(dir); errors.Is(err, fs.ErrNotExist) {
continue
}
var walkErr error
err := bfs.WalkRegular(dir, func(rel string) error {
if !strings.HasSuffix(rel, ".md") {
return nil
}
index[strings.TrimSuffix(path.Base(rel), ".md")] = struct{}{}
index[strings.TrimPrefix(strings.TrimSuffix(rel, ".md"), "wiki/")] = struct{}{}
return nil
}, func(rel string, err error) {
// A linked tree that is not a real directory (a symlink, a
// file) is skipped, as filepath.WalkDir did: this runs before
// every brain_write, and an error here loses the learning.
if rel == dir && errors.Is(err, brainfs.ErrNotRegular) {
slog.Warn("link check: skipping a linked tree that is not a real directory", "dir", dir)
return
}
if walkErr == nil {
walkErr = fmt.Errorf("%s: %w", rel, err)
}
})
if err == nil {
err = walkErr
}
if err != nil {
return nil, fmt.Errorf("walk %s: %w", dir, err)
}
}
return index, nil
}
// linkKey normalises a link target for index lookup: an exact path form is
// tried as written, and anything else falls back to its final segment.
func linkKey(target string) string {
return strings.TrimSuffix(target, ".md")
}
// codeSpans returns the byte ranges of fenced blocks and inline code spans.
func codeSpans(content string) [][2]int {
var spans [][2]int
for _, m := range fencedCodeRE.FindAllStringIndex(content, -1) {
spans = append(spans, [2]int{m[0], m[1]})
}
for _, m := range inlineCodeRE.FindAllStringIndex(content, -1) {
spans = append(spans, [2]int{m[0], m[1]})
}
return spans
}
// overlaps reports whether [start,end) intersects any span.
func overlaps(start, end int, spans [][2]int) bool {
for _, s := range spans {
if start < s[1] && end > s[0] {
return true
}
}
return false
}
@@ -0,0 +1,41 @@
package brain_test
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
)
// hyperguild#178: StripUnresolvedLinks runs before every brain_write and
// brain_update. An unreadable directory under knowledge/ or wiki/ made it
// error, the write was refused and the learning lost. When the link index
// cannot be built, the content is now returned unchanged (no link is
// unwrapped, none is invented) and no error stops the write.
func TestStripUnresolvedLinksKeepsContentWhenTheIndexCannotBeBuilt(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root ignores directory permissions")
}
b := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "w", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(b, "wiki", "w", "facts", "real.md"), []byte("x"), 0o644))
locked := filepath.Join(b, "wiki", "other", "locked")
require.NoError(t, os.MkdirAll(locked, 0o755))
require.NoError(t, os.Chmod(locked, 0o000))
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
in := "learning [[real]] and [[gone]]"
out, unlinked, err := brain.StripUnresolvedLinks(b, in)
require.NoError(t, err)
assert.Equal(t, in, out, "content unchanged: no link unwrapped on a partial index")
assert.Empty(t, unlinked)
require.NoError(t, os.Chmod(locked, 0o755))
out, _, err = brain.StripUnresolvedLinks(b, in)
require.NoError(t, err)
assert.NotContains(t, out, "[[gone]]", "positive control: with a readable tree the strip runs")
}
@@ -0,0 +1,93 @@
package brain_test
import (
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// seedCorpus writes the two entries every linkcheck case resolves against.
func seedCorpus(t *testing.T, dir string) {
t.Helper()
seedNote(t, dir, "wiki/homelab/facts/real-entry.md",
"---\nwing: homelab\nhall: facts\ntitle: 'Real entry'\n---\n\nbody.\n")
seedNote(t, dir, "knowledge/legacy-entry.md",
"---\ntitle: 'Legacy entry'\n---\n\nbody.\n")
}
func TestStripUnresolvedLinks(t *testing.T) {
tests := []struct {
name string
body string
want string
stripped []string
}{
{
name: "bare slug that resolves is kept",
body: "See [[real-entry]] for the rule.",
want: "See [[real-entry]] for the rule.",
},
{
name: "path-style target that resolves is kept",
body: "See [[homelab/facts/real-entry]].",
want: "See [[homelab/facts/real-entry]].",
},
{
name: "knowledge/ entry resolves by bare slug",
body: "See [[legacy-entry]].",
want: "See [[legacy-entry]].",
},
{
name: "unresolvable target is unwrapped to plain text",
body: "Related: [[secret-hygiene]].",
want: "Related: secret-hygiene.",
stripped: []string{"secret-hygiene"},
},
{
name: "unresolvable aliased target keeps its display text",
body: "Related: [[feedback_check_host_not_assume|check the host]].",
want: "Related: check the host.",
stripped: []string{"feedback_check_host_not_assume"},
},
{
name: "link inside a fenced code block is left alone",
body: "Syntax:\n\n```\n[[wing/hall/slug]]\n```\n",
want: "Syntax:\n\n```\n[[wing/hall/slug]]\n```\n",
},
{
name: "link inside an inline code span is left alone",
body: "Write `[[wing/hall/slug]]` to link.",
want: "Write `[[wing/hall/slug]]` to link.",
},
{
name: "every unresolvable target is reported once, sorted",
body: "[[zeta-gap]] and [[alpha-gap]] and [[zeta-gap]] again.",
want: "zeta-gap and alpha-gap and zeta-gap again.",
stripped: []string{"alpha-gap", "zeta-gap"},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
dir := t.TempDir()
seedCorpus(t, dir)
got, stripped, err := brain.StripUnresolvedLinks(dir, tc.body)
require.NoError(t, err)
assert.Equal(t, tc.want, got)
assert.Equal(t, tc.stripped, stripped)
})
}
}
func TestStripUnresolvedLinks_MissingBrainDirIsNotAGate(t *testing.T) {
// A brain dir with no knowledge//wiki/ tree yet must not turn every link
// into an unresolved one — an empty corpus is a bootstrap state, not
// evidence that the author invented the target.
got, stripped, err := brain.StripUnresolvedLinks(t.TempDir(), "See [[real-entry]].")
require.NoError(t, err)
assert.Equal(t, "See [[real-entry]].", got)
assert.Nil(t, stripped)
}
+118 -64
View File
@@ -2,11 +2,15 @@ package brain
import (
"bufio"
"errors"
"fmt"
"os"
"io/fs"
"path"
"path/filepath"
"strings"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
)
// seeAlsoHeader is the markdown heading used to group cross-wing links.
@@ -36,49 +40,80 @@ type TunnelCandidate struct {
// A note's title is read from YAML frontmatter `title:`; failing that,
// the filename slug (sans `.md`, hyphens → spaces) is used.
func DetectTunnels(brainDir, content string) ([]TunnelCandidate, error) {
wikiDir := filepath.Join(brainDir, "wiki")
if _, err := os.Stat(wikiDir); os.IsNotExist(err) {
// Walked and read through the brain root, regular files only
// (hyperguild#174, #157): this runs inside every brain_write, so a FIFO
// must not block it and a symlinked note's title must not become a
// candidate.
bfs, err := brainfs.Open(brainDir)
if err != nil {
return nil, fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
if _, err := bfs.Lstat("wiki"); errors.Is(err, fs.ErrNotExist) {
return nil, nil
} else if err != nil {
return nil, fmt.Errorf("stat wiki: %w", err)
}
lowerContent := strings.ToLower(content)
var out []TunnelCandidate
err := filepath.WalkDir(wikiDir, func(path string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() || !strings.HasSuffix(path, ".md") || d.Name() == "_index.md" {
var walkErr error
err = bfs.WalkRegular("wiki", func(rel string) error {
name := path.Base(rel)
if !strings.HasSuffix(name, ".md") || name == "_index.md" {
return nil
}
title, _ := readTitleAndCreated(path, strings.TrimSuffix(d.Name(), ".md"))
needle := strings.ToLower(strings.TrimSpace(title))
if needle == "" {
title, _ := readTitleAndCreatedIn(bfs, rel, strings.TrimSuffix(name, ".md"))
title = strings.TrimSpace(title)
if title == "" {
return nil
}
idx := strings.Index(lowerContent, needle)
// A one-word title has to match the prose case-exactly. Matched
// case-insensitively, it links every entry that happens to use the
// word: "WRONG" (wiki/telos/decisions/wrong.md) collected 185 inbound
// See-also links and "Status" 136, making the two most-linked nodes in
// the graph artifacts of the English language rather than references.
// Multi-word titles keep the case-insensitive match — "risk tier
// middleware" in prose is a real mention however it is capitalised,
// and a three-word collision is not an accident.
haystack, needle := lowerContent, strings.ToLower(title)
if isSingleWord(title) {
haystack, needle = content, title
}
idx := strings.Index(haystack, needle)
if idx == -1 {
return nil
}
rel, err := filepath.Rel(brainDir, path)
if err != nil {
return err
}
out = append(out, TunnelCandidate{
TargetPath: filepath.ToSlash(rel),
TargetPath: rel,
MatchedTerm: title,
Exact: isWholeWord(lowerContent, idx, len(needle)),
Exact: isWholeWord(haystack, idx, len(needle)),
})
return nil
}, func(rel string, err error) {
// A wiki/ that is not a real directory yields no candidates, as
// filepath.WalkDir did; other walk errors still abort.
if rel == "wiki" && errors.Is(err, brainfs.ErrNotRegular) {
return
}
if walkErr == nil {
walkErr = fmt.Errorf("%s: %w", rel, err)
}
})
if err == nil {
err = walkErr
}
if err != nil {
return nil, err
}
return out, nil
}
// isSingleWord reports whether title is one whitespace-free token.
func isSingleWord(title string) bool {
return !strings.ContainsAny(title, " \t")
}
// isWholeWord reports whether the substring at [idx, idx+n) in s is
// bounded by non-alphanumeric characters (or string edges).
func isWholeWord(s string, idx, n int) bool {
@@ -140,35 +175,23 @@ func logFuzzyCandidates(brainDir, sourcePath string, cs []TunnelCandidate) error
if len(cs) == 0 {
return nil
}
rawDir := filepath.Join(brainDir, "raw")
if err := os.MkdirAll(rawDir, 0o755); err != nil {
return err
}
stamp := time.Now().UTC().Format("2006-01-02")
path := filepath.Join(rawDir, "tunnel-candidates-"+stamp+".md")
existed := fileExists(path)
f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
// Appended through the brain root, never through a symlinked raw/ or
// log file (hyperguild#158).
bfs, err := brainfs.Open(brainDir)
if err != nil {
return err
return fmt.Errorf("open brain: %w", err)
}
defer func() { _ = f.Close() }()
if !existed {
if _, err := f.WriteString("# Tunnel candidates " + stamp + "\n\nFuzzy cross-wing matches surfaced by AutoTunnel. Review and promote to a tunnel via `brain_tunnel` if relevant.\n\n"); err != nil {
return err
}
defer func() { _ = bfs.Close() }()
stamp := time.Now().UTC().Format("2006-01-02")
rel := "raw/tunnel-candidates-" + stamp + ".md"
var buf strings.Builder
if _, err := bfs.Lstat(rel); errors.Is(err, fs.ErrNotExist) {
buf.WriteString("# Tunnel candidates " + stamp + "\n\nFuzzy cross-wing matches surfaced by AutoTunnel. Review and promote to a tunnel via `brain_tunnel` if relevant.\n\n")
}
for _, c := range cs {
line := fmt.Sprintf("- `%s` ↔ `%s` (term: %q)\n", sourcePath, c.TargetPath, c.MatchedTerm)
if _, err := f.WriteString(line); err != nil {
return err
}
fmt.Fprintf(&buf, "- `%s` ↔ `%s` (term: %q)\n", sourcePath, c.TargetPath, c.MatchedTerm)
}
return nil
}
func fileExists(p string) bool {
_, err := os.Stat(p)
return err == nil
return bfs.AppendRegular(rel, []byte(buf.String()))
}
// WriteTunnel appends a bidirectional wikilink between sourcePath and
@@ -179,6 +202,12 @@ func fileExists(p string) bool {
// section headers. Rejects same-wing pairs (a tunnel is by definition
// cross-wing) and missing notes.
func WriteTunnel(brainDir, sourcePath, targetPath string) error {
if err := checkNotePath(sourcePath); err != nil {
return fmt.Errorf("source: %w", err)
}
if err := checkNotePath(targetPath); err != nil {
return fmt.Errorf("target: %w", err)
}
srcWing, err := wingOf(sourcePath)
if err != nil {
return fmt.Errorf("source: %w", err)
@@ -191,20 +220,50 @@ func WriteTunnel(brainDir, sourcePath, targetPath string) error {
return fmt.Errorf("tunnel must cross wings; got both in %q", srcWing)
}
srcFull := filepath.Join(brainDir, filepath.FromSlash(sourcePath))
tgtFull := filepath.Join(brainDir, filepath.FromSlash(targetPath))
if _, err := os.Stat(srcFull); err != nil {
// Both notes are read and rewritten through the brain root, regular
// files only: brain_tunnel passes caller-supplied paths, and the brain is
// a Syncthing folder that can hold symlinks.
bfs, err := brainfs.Open(brainDir)
if err != nil {
return fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
srcContent, err := bfs.ReadRegular(sourcePath)
if err != nil {
return fmt.Errorf("source note: %w", err)
}
if _, err := os.Stat(tgtFull); err != nil {
tgtContent, err := bfs.ReadRegular(targetPath)
if err != nil {
return fmt.Errorf("target note: %w", err)
}
if err := appendSeeAlso(srcFull, wikilinkOf(targetPath)); err != nil {
if out, changed, err := withSeeAlso(srcContent, wikilinkOf(targetPath)); err != nil {
return fmt.Errorf("update source: %w", err)
} else if changed {
if err := bfs.WriteRegular(sourcePath, out); err != nil {
return fmt.Errorf("update source: %w", err)
}
}
if err := appendSeeAlso(tgtFull, wikilinkOf(sourcePath)); err != nil {
if out, changed, err := withSeeAlso(tgtContent, wikilinkOf(sourcePath)); err != nil {
return fmt.Errorf("update target: %w", err)
} else if changed {
if err := bfs.WriteRegular(targetPath, out); err != nil {
return fmt.Errorf("update target: %w", err)
}
}
return nil
}
// checkNotePath accepts only a clean, local wiki/<wing>/<hall>/.../<slug>.md
// path: no "." or ".." segments, no empty segments, no leading slash.
func checkNotePath(p string) error {
if p == "" || path.Clean(p) != p || !filepath.IsLocal(p) || !strings.HasSuffix(p, ".md") {
return fmt.Errorf("not a clean wiki note path: %q", p)
}
for _, part := range strings.Split(p, "/") {
if part == "" || part == "." || part == ".." {
return fmt.Errorf("not a clean wiki note path: %q", p)
}
}
return nil
}
@@ -230,16 +289,13 @@ func wingOf(relPath string) (string, error) {
return parts[1], nil
}
// appendSeeAlso inserts `- [[link]]` under the file's See also section,
// creating the section if absent. No-op when the link is already present.
func appendSeeAlso(filePath, link string) error {
content, err := os.ReadFile(filePath)
if err != nil {
return err
}
// withSeeAlso returns content with `- [[link]]` under its See also section,
// creating the section if absent. changed is false when the link is already
// present.
func withSeeAlso(content []byte, link string) (out []byte, changed bool, err error) {
wikilink := "[[" + link + "]]"
if strings.Contains(string(content), wikilink) {
return nil
return content, false, nil
}
bullet := "- " + wikilink
@@ -248,8 +304,7 @@ func appendSeeAlso(filePath, link string) error {
// No section yet — append a fresh one. Always emit a trailing
// newline so subsequent appends don't merge into the previous line.
trimmed := strings.TrimRight(string(content), "\n")
out := trimmed + "\n\n" + seeAlsoHeader + "\n\n" + bullet + "\n"
return os.WriteFile(filePath, []byte(out), 0o644)
return []byte(trimmed + "\n\n" + seeAlsoHeader + "\n\n" + bullet + "\n"), true, nil
}
// Section exists — splice the bullet in just before the next `## `
@@ -275,12 +330,11 @@ func appendSeeAlso(filePath, link string) error {
b.WriteByte('\n')
}
if err := scanner.Err(); err != nil {
return err
return nil, false, err
}
if !inserted {
// section was the last thing in the file — just append bullet
out := strings.TrimRight(b.String(), "\n") + "\n" + bullet + "\n"
return os.WriteFile(filePath, []byte(out), 0o644)
return []byte(strings.TrimRight(b.String(), "\n") + "\n" + bullet + "\n"), true, nil
}
return os.WriteFile(filePath, []byte(b.String()), 0o644)
return []byte(b.String()), true, nil
}
@@ -0,0 +1,38 @@
package brain_test
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
)
// hyperguild#173: brain_tunnel's write followed a directory symlink inside
// wiki/ that points elsewhere in the brain.
func TestWriteTunnelRefusesATargetBehindASymlinkedDirectory(t *testing.T) {
b := t.TempDir()
src := filepath.Join(b, "wiki", "a", "facts", "s.md")
require.NoError(t, os.MkdirAll(filepath.Dir(src), 0o755))
require.NoError(t, os.WriteFile(src, []byte("x\n"), 0o644))
victim := filepath.Join(b, "raw", "facts", "v.md")
require.NoError(t, os.MkdirAll(filepath.Dir(victim), 0o755))
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "b"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "raw", "facts"), filepath.Join(b, "wiki", "b", "facts")))
assert.ErrorIs(t, brain.WriteTunnel(b, "wiki/a/facts/s.md", "wiki/b/facts/v.md"), brainfs.ErrNotRegular)
got, _ := os.ReadFile(victim)
assert.Equal(t, "untouched\n", string(got))
got, _ = os.ReadFile(src)
assert.Equal(t, "x\n", string(got), "the source is not rewritten either")
real := filepath.Join(b, "wiki", "c", "facts", "t.md")
require.NoError(t, os.MkdirAll(filepath.Dir(real), 0o755))
require.NoError(t, os.WriteFile(real, []byte("t\n"), 0o644))
require.NoError(t, brain.WriteTunnel(b, "wiki/a/facts/s.md", "wiki/c/facts/t.md"), "positive control: a tunnel between real directories is written")
}
@@ -0,0 +1,141 @@
package brain_test
import (
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
)
// brain_tunnel passes caller-supplied source and target paths to WriteTunnel,
// which checked only "wiki/<wing>/..." and then joined them onto the brain
// dir: ".." segments resolved outside the brain and appendSeeAlso rewrote the
// file there.
func TestWriteTunnelRefusesATargetThatTraversesOutOfTheBrain(t *testing.T) {
brain0 := t.TempDir()
src := filepath.Join(brain0, "wiki", "a", "facts", "s.md")
require.NoError(t, os.MkdirAll(filepath.Dir(src), 0o755))
require.NoError(t, os.WriteFile(src, []byte("---\ntitle: S\n---\nbody\n"), 0o644))
victim := filepath.Join(t.TempDir(), "victim.md")
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
up := strings.Repeat("../", strings.Count(filepath.Clean(brain0), "/")+3)
target := "wiki/b/x/" + up + strings.TrimPrefix(victim, "/")
err := brain.WriteTunnel(brain0, "wiki/a/facts/s.md", target)
assert.Error(t, err)
got, rerr := os.ReadFile(victim)
require.NoError(t, rerr)
assert.Equal(t, "untouched\n", string(got), "a file outside the brain was modified")
}
func TestWriteTunnelRefusesUncleanOrNonWikiPaths(t *testing.T) {
brain0 := t.TempDir()
for _, p := range []string{"wiki/a/facts/s.md", "wiki/b/facts/t.md", "wiki/b/facts/t.txt", "raw/r.md"} {
require.NoError(t, os.MkdirAll(filepath.Dir(filepath.Join(brain0, p)), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(brain0, p), []byte("x\n"), 0o644))
}
for _, target := range []string{
"wiki/b/facts/../../../raw/r.md",
"wiki/b/./facts/t.md",
"wiki/b//facts/t.md",
"/wiki/b/facts/t.md",
"wiki/b/facts/t.txt",
"wiki/b/facts/",
} {
assert.Error(t, brain.WriteTunnel(brain0, "wiki/a/facts/s.md", target), target)
}
for _, p := range []string{"raw/r.md", "wiki/b/facts/t.txt"} {
got, err := os.ReadFile(filepath.Join(brain0, p))
require.NoError(t, err)
assert.Equal(t, "x\n", string(got), p)
}
require.NoError(t, brain.WriteTunnel(brain0, "wiki/a/facts/s.md", "wiki/b/facts/t.md"), "positive control")
got, err := os.ReadFile(filepath.Join(brain0, "wiki", "b", "facts", "t.md"))
require.NoError(t, err)
assert.Contains(t, string(got), "[[a/facts/s]]")
}
func TestWriteTunnelDoesNotWriteThroughASymlinkedNote(t *testing.T) {
brain0 := t.TempDir()
src := filepath.Join(brain0, "wiki", "a", "facts", "s.md")
require.NoError(t, os.MkdirAll(filepath.Dir(src), 0o755))
require.NoError(t, os.WriteFile(src, []byte("x\n"), 0o644))
tdir := filepath.Join(brain0, "wiki", "b", "facts")
require.NoError(t, os.MkdirAll(tdir, 0o755))
victim := filepath.Join(t.TempDir(), "victim.md")
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
require.NoError(t, os.Symlink(victim, filepath.Join(tdir, "t.md")))
assert.Error(t, brain.WriteTunnel(brain0, "wiki/a/facts/s.md", "wiki/b/facts/t.md"))
got, err := os.ReadFile(victim)
require.NoError(t, err)
assert.Equal(t, "untouched\n", string(got))
// The same with a link to a file elsewhere in the brain, which os.Root
// would follow.
inVictim := filepath.Join(brain0, "raw", "v.md")
require.NoError(t, os.MkdirAll(filepath.Dir(inVictim), 0o755))
require.NoError(t, os.WriteFile(inVictim, []byte("untouched\n"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "v.md"), filepath.Join(tdir, "u.md")))
assert.Error(t, brain.WriteTunnel(brain0, "wiki/a/facts/s.md", "wiki/b/facts/u.md"))
got, err = os.ReadFile(inVictim)
require.NoError(t, err)
assert.Equal(t, "untouched\n", string(got))
got, err = os.ReadFile(src)
require.NoError(t, err)
assert.Equal(t, "x\n", string(got), "the source is not touched when the target is refused")
}
// The target is swapped between a regular note and a symlink to an in-brain
// file outside wiki/ while WriteTunnel runs: the read and the write each
// refuse a symlink, so the other file is never rewritten.
func TestWriteTunnelNeverRewritesASymlinkSwappedInForTheTarget(t *testing.T) {
brain0 := t.TempDir()
src := filepath.Join(brain0, "wiki", "a", "facts", "s.md")
require.NoError(t, os.MkdirAll(filepath.Dir(src), 0o755))
require.NoError(t, os.WriteFile(src, []byte("x\n"), 0o644))
tdir := filepath.Join(brain0, "wiki", "b", "facts")
require.NoError(t, os.MkdirAll(tdir, 0o755))
victim := filepath.Join(brain0, "raw", "victim.md")
require.NoError(t, os.MkdirAll(filepath.Dir(victim), 0o755))
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
tgt := filepath.Join(tdir, "t.md")
require.NoError(t, os.WriteFile(tgt, []byte("t\n"), 0o644))
var stop atomic.Bool
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
tmp := tgt + ".swap"
for i := 0; !stop.Load(); i++ {
_ = os.Remove(tmp)
if i%2 == 0 {
_ = os.WriteFile(tmp, []byte("t\n"), 0o644)
} else {
_ = os.Symlink(filepath.Join("..", "..", "..", "raw", "victim.md"), tmp)
}
_ = os.Rename(tmp, tgt)
}
}()
ok := 0
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); {
if brain.WriteTunnel(brain0, "wiki/a/facts/s.md", "wiki/b/facts/t.md") == nil {
ok++
}
_ = os.WriteFile(src, []byte("x\n"), 0o644)
}
stop.Store(true)
wg.Wait()
got, err := os.ReadFile(victim)
require.NoError(t, err)
assert.Equal(t, "untouched\n", string(got), "the swapped-in symlink's target was rewritten")
assert.Positive(t, ok, "positive control: some tunnels were written to the regular target")
}
+57 -11
View File
@@ -102,16 +102,29 @@ func TestDetectTunnels_ExactTitleMatch(t *testing.T) {
func TestDetectTunnels_FuzzyMatch(t *testing.T) {
dir := t.TempDir()
seedNote(t, dir, "wiki/x/facts/routing.md",
"---\ntitle: Routing\n---\nbody.\n")
seedNote(t, dir, "wiki/x/facts/routing-table.md",
"---\ntitle: 'Routing Table'\n---\nbody.\n")
// Substring of title appears in content, but not as a whole word.
candidates, err := brain.DetectTunnels(dir, "rerouting handles failover")
candidates, err := brain.DetectTunnels(dir, "the routing tables were rebuilt")
require.NoError(t, err)
require.Len(t, candidates, 1)
assert.False(t, candidates[0].Exact, "substring-only match should be fuzzy")
}
// A one-word title whose only appearance is lowercase inside another word is
// not even a candidate: it would fill the review file with English rather than
// with references.
func TestDetectTunnels_SingleWordSubstringIsNotACandidate(t *testing.T) {
dir := t.TempDir()
seedNote(t, dir, "wiki/x/facts/routing.md",
"---\ntitle: Routing\n---\nbody.\n")
candidates, err := brain.DetectTunnels(dir, "rerouting handles failover")
require.NoError(t, err)
assert.Empty(t, candidates)
}
func TestDetectTunnels_NoFrontmatterFallsBackToSlug(t *testing.T) {
dir := t.TempDir()
seedNote(t, dir, "wiki/x/facts/widget-flags.md", "# widget flags\n\nbody.\n")
@@ -126,20 +139,21 @@ func TestDetectTunnels_NoFrontmatterFallsBackToSlug(t *testing.T) {
func TestAutoTunnel_FuzzyGoesToCandidatesFile(t *testing.T) {
dir := t.TempDir()
// Existing note in a different wing whose title is "Routing".
seedNote(t, dir, "wiki/other/facts/routing.md",
"---\nwing: other\nhall: facts\ntitle: Routing\n---\nbody.\n")
// Source note in another wing whose body mentions "rerouting" (substring match only).
// Existing note in a different wing whose title is "Routing Table".
seedNote(t, dir, "wiki/other/facts/routing-table.md",
"---\nwing: other\nhall: facts\ntitle: 'Routing Table'\n---\nbody.\n")
// Source note in another wing whose body mentions "routing tables"
// (substring match only — the title is not there as a whole word).
seedNote(t, dir, "wiki/jepa-fx/facts/new.md",
"---\nwing: jepa-fx\nhall: facts\n---\nrerouting traffic\n")
"---\nwing: jepa-fx\nhall: facts\n---\nrouting tables rebuilt\n")
require.NoError(t, brain.AutoTunnel(dir,
"wiki/jepa-fx/facts/new.md", "rerouting traffic"))
"wiki/jepa-fx/facts/new.md", "routing tables rebuilt"))
// Source must not get auto-linked (fuzzy).
got, err := os.ReadFile(filepath.Join(dir, "wiki/jepa-fx/facts/new.md"))
require.NoError(t, err)
assert.NotContains(t, string(got), "[[other/facts/routing]]")
assert.NotContains(t, string(got), "[[other/facts/routing-table]]")
// Candidates file must list the pair.
matches, err := filepath.Glob(filepath.Join(dir, "raw", "tunnel-candidates-*.md"))
@@ -148,7 +162,7 @@ func TestAutoTunnel_FuzzyGoesToCandidatesFile(t *testing.T) {
body, err := os.ReadFile(matches[0])
require.NoError(t, err)
assert.Contains(t, string(body), "wiki/jepa-fx/facts/new.md")
assert.Contains(t, string(body), "wiki/other/facts/routing.md")
assert.Contains(t, string(body), "wiki/other/facts/routing-table.md")
assert.Contains(t, string(body), "Routing")
}
@@ -175,3 +189,35 @@ func TestWriteTunnel_AppendsToExistingSeeAlso(t *testing.T) {
assert.Contains(t, s, "[[a/facts/old]]")
assert.Contains(t, s, "[[b/facts/y]]")
}
// A one-word title matched case-insensitively links every entry that happens
// to use that word. Live cost before this rule: 185 entries auto-linked to
// wiki/telos/decisions/wrong.md (title "WRONG") and 136 to .../status.md,
// making the two most-linked nodes in the graph pure artifacts.
func TestDetectTunnels_SingleWordTitleNeedsAnExactCaseMatch(t *testing.T) {
dir := t.TempDir()
seedNote(t, dir, "wiki/telos/decisions/wrong.md",
"---\nwing: telos\nhall: decisions\ntitle: WRONG\n---\n# WRONG\n\nRetired beliefs.\n")
got, err := brain.DetectTunnels(dir, "The obvious fix was to reject the write. Wrong here.")
require.NoError(t, err)
for _, c := range got {
assert.False(t, c.Exact, "a lowercase prose word must not auto-link a one-word title: %+v", c)
}
got, err = brain.DetectTunnels(dir, "The beliefs listed under WRONG were retired.")
require.NoError(t, err)
require.Len(t, got, 1)
assert.True(t, got[0].Exact, "the title's own casing appearing in prose is a real reference")
}
func TestDetectTunnels_MultiWordTitleStaysCaseInsensitive(t *testing.T) {
dir := t.TempDir()
seedNote(t, dir, "wiki/homelab/facts/risk-tier.md",
"---\nwing: homelab\nhall: facts\ntitle: 'Risk Tier Middleware'\n---\nbody\n")
got, err := brain.DetectTunnels(dir, "We rebuilt the risk tier middleware last week.")
require.NoError(t, err)
require.Len(t, got, 1)
assert.True(t, got[0].Exact)
}
@@ -0,0 +1,52 @@
package brain_test
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
)
// hyperguild#158: BuildWingIndex wrote wiki/<wing>/_index.md with
// os.WriteFile, so a symlinked _index.md was written through.
func TestBuildWingIndexDoesNotWriteThroughASymlinkedIndex(t *testing.T) {
b := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "w", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(b, "wiki", "w", "facts", "n.md"), []byte("---\ntitle: zqn\n---\n"), 0o644))
victim := filepath.Join(b, "raw", "v.md")
require.NoError(t, os.MkdirAll(filepath.Dir(victim), 0o755))
require.NoError(t, os.WriteFile(victim, []byte("untouched"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "raw", "v.md"), filepath.Join(b, "wiki", "w", "_index.md")))
assert.Error(t, brain.BuildWingIndex(b, "w"))
got, _ := os.ReadFile(victim)
assert.Equal(t, "untouched", string(got))
require.NoError(t, os.Remove(filepath.Join(b, "wiki", "w", "_index.md")))
require.NoError(t, brain.BuildWingIndex(b, "w"), "positive control")
idx, _ := os.ReadFile(filepath.Join(b, "wiki", "w", "_index.md"))
assert.Contains(t, string(idx), "zqn")
}
// AutoTunnel's fuzzy-candidate log appended to raw/tunnel-candidates-*.md
// with os.OpenFile, so a symlinked raw/ (or log file) was written through.
func TestAutoTunnelDoesNotAppendTheCandidateLogThroughASymlinkedRaw(t *testing.T) {
b := t.TempDir()
outside := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "a", "facts"), 0o755))
require.NoError(t, os.MkdirAll(filepath.Join(b, "wiki", "b", "facts"), 0o755))
// "zq widget" matches as a substring of "zq widgets", a fuzzy candidate.
require.NoError(t, os.WriteFile(filepath.Join(b, "wiki", "b", "facts", "t.md"), []byte("---\ntitle: Zq Widget\n---\n"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(b, "raw")))
_ = brain.AutoTunnel(b, "wiki/a/facts/s.md", "about zq widgets")
ents, _ := os.ReadDir(outside)
assert.Empty(t, ents, "the candidate log was written outside the brain")
require.NoError(t, os.Remove(filepath.Join(b, "raw")))
require.NoError(t, brain.AutoTunnel(b, "wiki/a/facts/s.md", "about zq widgets"))
logs, _ := filepath.Glob(filepath.Join(b, "raw", "tunnel-candidates-*.md"))
assert.Len(t, logs, 1, "positive control: the fuzzy candidate is logged in the real raw/")
}
+477
View File
@@ -0,0 +1,477 @@
// Package brainfs reads the brain directory through an os.Root (hyperguild#157).
// The brain dir is a Syncthing sendreceive folder and Syncthing syncs
// symlinks; a reader that walks and reads by path follows a planted symlink
// out of the brain, and blocks forever on a planted FIFO. Readers that go
// through a Brain only ever read regular files inside the brain: os.Root
// refuses to leave the root even if an entry is swapped mid-read, and every
// open is non-blocking with the type checked on the opened handle.
package brainfs
import (
"errors"
"fmt"
"io"
"io/fs"
"os"
"path"
"path/filepath"
"strings"
"syscall"
)
// ErrNotRegular is returned for an entry that is not a regular file.
var ErrNotRegular = errors.New("not a regular file")
// ErrTooLarge is returned by ReadRegularMax for a file over the limit.
var ErrTooLarge = errors.New("file too large")
// Brain is an open brain directory.
type Brain struct {
root *os.Root
dir string
}
// Open opens brainDir (which may itself be a mount symlink) as a root.
func Open(brainDir string) (*Brain, error) {
r, err := os.OpenRoot(brainDir)
if err != nil {
return nil, fmt.Errorf("open brain root: %w", err)
}
return &Brain{root: r, dir: brainDir}, nil
}
// Close releases the root.
func (b *Brain) Close() error { return b.root.Close() }
// Rel converts a path under the brain dir to a root-relative path.
func (b *Brain) Rel(path string) (string, error) {
rel, err := filepath.Rel(b.dir, path)
if err != nil {
return "", err
}
if rel == ".." || len(rel) > 2 && rel[:3] == ".."+string(filepath.Separator) {
return "", fmt.Errorf("%s is outside the brain dir", path)
}
return filepath.ToSlash(rel), nil
}
// Lstat stats a brain-relative path through the root without following a
// final symlink.
func (b *Brain) Lstat(rel string) (fs.FileInfo, error) {
return b.root.Lstat(filepath.ToSlash(rel))
}
// realDir returns ErrNotRegular unless every component of rel is a real
// directory. A swap after the check is not caught (accepted residual,
// hyperguild#157).
func (b *Brain) realDir(rel string) error {
if rel == "." {
return nil
}
prefix := ""
for _, part := range strings.Split(rel, "/") {
prefix = path.Join(prefix, part)
fi, err := b.root.Lstat(prefix)
if err != nil {
return err
}
if !fi.IsDir() {
return fmt.Errorf("%s: walk root component %s is not a directory: %w", rel, prefix, ErrNotRegular)
}
}
return nil
}
// ReadDirRegular lists the regular files directly in a brain-relative
// directory, by name. Every component of relDir must be a real directory
// (ErrNotRegular otherwise); symlinks, FIFOs, devices and subdirectories in
// it are left out.
func (b *Brain) ReadDirRegular(relDir string) ([]string, error) {
relDir = filepath.ToSlash(relDir)
if err := b.realDir(relDir); err != nil {
return nil, err
}
ents, err := fs.ReadDir(b.root.FS(), relDir)
if err != nil {
return nil, err
}
var names []string
for _, e := range ents {
if e.Type().IsRegular() {
names = append(names, e.Name())
}
}
return names, nil
}
// cleanRel refuses a path that is not clean and local. realDir checks the
// cleaned parent, so the I/O must use that same path: os.Root resolves ".."
// after following a symlink, and a ".." after a symlinked directory would
// otherwise pass the check and redirect the read or write.
func cleanRel(rel string) error {
if rel == "" || path.Clean(rel) != rel || !filepath.IsLocal(rel) {
return fmt.Errorf("%q: not a clean brain-relative path: %w", rel, ErrNotRegular)
}
return nil
}
// ReadDirDirs lists the real subdirectories directly in a brain-relative
// directory, by name. Every component of relDir must be a real directory
// (ErrNotRegular otherwise); symlinks to directories are left out.
func (b *Brain) ReadDirDirs(relDir string) ([]string, error) {
relDir = filepath.ToSlash(relDir)
if err := b.realDir(relDir); err != nil {
return nil, err
}
ents, err := fs.ReadDir(b.root.FS(), relDir)
if err != nil {
return nil, err
}
var names []string
for _, e := range ents {
if e.Type().IsDir() {
names = append(names, e.Name())
}
}
return names, nil
}
// PutRegular creates or replaces a regular file at a brain-relative path,
// for creating writes (hyperguild#188, #158). Missing parent directories are
// created one component at a time through the root, and every existing
// component must be a real directory, so a symlinked directory cannot
// redirect the write elsewhere in the brain or outside it. An existing file
// is replaced only if it is regular (WriteRegular's checks); a new file is
// created with O_EXCL and confirmed to be the file that was opened. A
// directory swapped in after the checks is the #167 residual.
func (b *Brain) PutRegular(rel string, data []byte) error {
if err := cleanRel(rel); err != nil {
return err
}
if err := b.mkdirReal(path.Dir(rel)); err != nil {
return err
}
if _, err := b.root.Lstat(rel); err == nil {
return b.WriteRegular(rel, data)
} else if !errors.Is(err, fs.ErrNotExist) {
return err
}
return b.createNew(rel, data)
}
// CreateRegular creates a new regular file at a brain-relative path, with
// PutRegular's parent rules, and fails with fs.ErrExist if anything is
// already there: for writers that must never overwrite (promote, reject).
func (b *Brain) CreateRegular(rel string, data []byte) error {
if err := cleanRel(rel); err != nil {
return err
}
if err := b.mkdirReal(path.Dir(rel)); err != nil {
return err
}
return b.createNew(rel, data)
}
// createNew creates rel with O_EXCL and writes data only after confirming
// the path is the regular file it created.
func (b *Brain) createNew(rel string, data []byte) error {
f, err := b.root.OpenFile(rel, os.O_WRONLY|os.O_CREATE|os.O_EXCL|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0o644)
if err != nil {
return err
}
defer func() { _ = f.Close() }()
hfi, herr := f.Stat()
lfi, lerr := b.root.Lstat(rel)
if herr != nil || lerr != nil || !lfi.Mode().IsRegular() || !os.SameFile(lfi, hfi) {
return fmt.Errorf("%s: changed while creating: %w", rel, ErrNotRegular)
}
if _, err := f.Write(data); err != nil {
return err
}
return f.Close()
}
// AppendRegular appends data to a regular file at a brain-relative path,
// creating it (and missing parent directories, as PutRegular does) if it is
// absent. It writes only after confirming the path is the regular file it
// opened, so a final symlink or special file, or one swapped in, is refused.
func (b *Brain) AppendRegular(rel string, data []byte) error {
if err := cleanRel(rel); err != nil {
return err
}
if err := b.mkdirReal(path.Dir(rel)); err != nil {
return err
}
if fi, err := b.root.Lstat(rel); err == nil && !fi.Mode().IsRegular() {
return fmt.Errorf("%s: %w", rel, ErrNotRegular)
} else if err != nil && !errors.Is(err, fs.ErrNotExist) {
return err
}
f, err := b.root.OpenFile(rel, os.O_WRONLY|os.O_APPEND|os.O_CREATE|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0o644)
if err != nil {
return err
}
defer func() { _ = f.Close() }()
hfi, herr := f.Stat()
lfi, lerr := b.root.Lstat(rel)
if herr != nil || lerr != nil || !lfi.Mode().IsRegular() || !os.SameFile(lfi, hfi) {
return fmt.Errorf("%s: changed while opening: %w", rel, ErrNotRegular)
}
if _, err := f.Write(data); err != nil {
return err
}
return f.Close()
}
// Remove removes the file at a clean brain-relative path through the root,
// refusing a path behind a symlinked directory. A final symlink is removed
// itself, never its target.
func (b *Brain) Remove(rel string) error {
if err := b.RealParent(rel); err != nil {
return err
}
return b.root.Remove(rel)
}
// RealParent returns ErrNotRegular unless rel is clean and every directory
// above it is a real directory, for writers that do their own I/O (an atomic
// rename) but must not be redirected by a symlinked directory.
func (b *Brain) RealParent(rel string) error {
if err := cleanRel(rel); err != nil {
return err
}
return b.realDir(path.Dir(rel))
}
// PutFile opens the brain dir as a root and calls PutRegular, for one-off
// writers that hold no Brain.
func PutFile(brainDir, rel string, data []byte) error {
b, err := Open(brainDir)
if err != nil {
return err
}
defer func() { _ = b.Close() }()
return b.PutRegular(rel, data)
}
// AppendFile opens the brain dir as a root and calls AppendRegular.
func AppendFile(brainDir, rel string, data []byte) error {
b, err := Open(brainDir)
if err != nil {
return err
}
defer func() { _ = b.Close() }()
return b.AppendRegular(rel, data)
}
// mkdirReal makes sure every component of rel exists as a real directory,
// creating missing ones through the root; an existing component that is
// not a real directory is ErrNotRegular.
func (b *Brain) mkdirReal(rel string) error {
if rel == "." {
return nil
}
prefix := ""
for _, part := range strings.Split(rel, "/") {
prefix = path.Join(prefix, part)
fi, err := b.root.Lstat(prefix)
if errors.Is(err, fs.ErrNotExist) {
if err := b.root.Mkdir(prefix, 0o755); err != nil && !errors.Is(err, fs.ErrExist) {
return err
}
fi, err = b.root.Lstat(prefix)
}
if err != nil {
return err
}
if !fi.IsDir() {
return fmt.Errorf("%s: component %s is not a directory: %w", rel, prefix, ErrNotRegular)
}
}
return nil
}
// ReadRegular reads a brain-relative path, refusing anything that is not a
// regular file, both before the open and on the opened handle.
func (b *Brain) ReadRegular(rel string) ([]byte, error) {
if err := cleanRel(rel); err != nil {
return nil, err
}
if err := b.realDir(path.Dir(rel)); err != nil {
return nil, err
}
fi, err := b.root.Lstat(rel)
if err != nil {
return nil, err
}
if !fi.Mode().IsRegular() {
return nil, fmt.Errorf("%s: %w", rel, ErrNotRegular)
}
return b.readOpenedMax(rel, -1, fi)
}
// readOpened opens rel non-blocking and reads it only if the OPENED handle is
// a regular file. This is the check that matters under a race: an entry
// swapped for a FIFO after the Lstat, with a writer holding it open, would
// otherwise park io.ReadAll forever (PR #161 review: removing it hung search
// after 16 iterations).
//
// Accepted residual, as in api/rawroot.go: a HARD link to an outside file is
// a regular file and is read. Syncthing syncs contents, not links, so a peer
// cannot create one; fs.protected_hardlinks=1 and the pod's secrets are on a
// different filesystem from the brain volume, so a local caller cannot link
// them in. Refusing Nlink > 1 is the next step if either changes.
func (b *Brain) readOpened(rel string) ([]byte, error) { return b.readOpenedMax(rel, -1, nil) }
// ReadRegularMax is ReadRegular with a size limit, checked on the opened
// handle before reading (max < 0 means no limit).
func (b *Brain) ReadRegularMax(rel string, max int64) ([]byte, error) {
if err := cleanRel(rel); err != nil {
return nil, err
}
if err := b.realDir(path.Dir(rel)); err != nil {
return nil, err
}
fi, err := b.root.Lstat(rel)
if err != nil {
return nil, err
}
if !fi.Mode().IsRegular() {
return nil, fmt.Errorf("%s: %w", rel, ErrNotRegular)
}
return b.readOpenedMax(rel, max, fi)
}
// readOpenedMax opens rel and reads it if the opened handle is a regular file
// and, when lfi is set, the same file the caller Lstat-ed: os.Root follows an
// in-root final symlink even with O_NOFOLLOW, so a symlink to another brain
// file swapped in between the Lstat and the open is refused only here.
func (b *Brain) readOpenedMax(rel string, max int64, lfi fs.FileInfo) ([]byte, error) {
f, err := b.root.OpenFile(rel, os.O_RDONLY|syscall.O_NONBLOCK, 0)
if err != nil {
return nil, err
}
defer func() { _ = f.Close() }()
hfi, err := f.Stat()
if err != nil {
return nil, err
}
if !hfi.Mode().IsRegular() {
return nil, fmt.Errorf("%s: %w", rel, ErrNotRegular)
}
if lfi != nil && !os.SameFile(lfi, hfi) {
return nil, fmt.Errorf("%s: changed while opening: %w", rel, ErrNotRegular)
}
if max >= 0 && hfi.Size() > max {
return nil, fmt.Errorf("%s is %d bytes (limit %d): %w", rel, hfi.Size(), max, ErrTooLarge)
}
var r io.Reader = f
if wrapAfterStat != nil {
r = wrapAfterStat(r)
}
if max < 0 {
return io.ReadAll(r)
}
// The handle's size can grow after the check; bound the read itself.
data, err := io.ReadAll(io.LimitReader(r, max+1))
if err != nil {
return nil, err
}
if int64(len(data)) > max {
return nil, fmt.Errorf("%s grew past %d bytes while read: %w", rel, max, ErrTooLarge)
}
return data, nil
}
// wrapAfterStat is a test seam: it runs between the size check and the read,
// and wraps the reader the read consumes. A package global: brainfs tests
// must not use t.Parallel.
var wrapAfterStat func(io.Reader) io.Reader
// WalkRegular walks a brain-relative directory through the root and calls fn
// for each regular file. Directories are descended; symlinks, FIFOs and
// devices found by the walk are skipped. A walk root with any component that
// is not a real directory goes to onErr as ErrNotRegular and nothing is
// yielded. An entry error is passed to onErr and the walk continues; a nil
// onErr ignores it.
//
// Accepted residual (hyperguild#157): a directory swapped for a symlink
// between the listing and the read is followed. os.Root still keeps the read
// inside the brain, but not inside relDir.
func (b *Brain) WalkRegular(relDir string, fn func(rel string) error, onErr func(rel string, err error)) error {
relDir = filepath.ToSlash(relDir)
// fs.WalkDir stats its root with a follow, and os.Root allows a symlink
// that stays inside the brain: knowledge -> . would alias the whole
// brain. Require every component of the root to be a real directory
// (Lstat checks only the last one).
err := b.realDir(relDir)
if err != nil {
if onErr != nil {
onErr(relDir, err)
}
return nil
}
return fs.WalkDir(b.root.FS(), relDir, func(p string, d fs.DirEntry, err error) error {
if err != nil {
if onErr != nil {
onErr(p, err)
}
if d != nil && d.IsDir() {
return fs.SkipDir
}
return nil
}
if d.IsDir() || !d.Type().IsRegular() {
return nil
}
return fn(p)
})
}
// WriteRegular replaces the contents of an existing regular file at a
// brain-relative path. It never creates a file and never writes through a
// final symlink: os.Root.OpenFile follows an in-root final symlink even with
// O_NOFOLLOW (verified on go1.27), so the path is Lstat-ed through the root
// first and the opened handle must be that same file (os.SameFile), which
// also catches a swap between the two. O_NONBLOCK keeps a FIFO from blocking
// the open. Every directory above the file must be a real directory
// (hyperguild#173); a directory swapped in after that check is the #167
// residual.
func (b *Brain) WriteRegular(rel string, data []byte) error {
if err := cleanRel(rel); err != nil {
return err
}
// Every directory above the file must be real: os.Root follows an
// in-brain intermediate symlink, which would redirect the write
// elsewhere in the brain (hyperguild#173). A swap after this check is
// the #167 residual.
if err := b.realDir(path.Dir(rel)); err != nil {
return err
}
lfi, err := b.root.Lstat(rel)
if err != nil {
return err
}
if !lfi.Mode().IsRegular() {
return fmt.Errorf("%s: %w", rel, ErrNotRegular)
}
f, err := b.root.OpenFile(rel, os.O_WRONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0)
if err != nil {
return err
}
defer func() { _ = f.Close() }()
fi, err := f.Stat()
if err != nil {
return err
}
if !fi.Mode().IsRegular() || !os.SameFile(lfi, fi) {
return fmt.Errorf("%s: changed while opening: %w", rel, ErrNotRegular)
}
if err := f.Truncate(0); err != nil {
return err
}
if _, err := f.Write(data); err != nil {
return err
}
return f.Close()
}
+548
View File
@@ -0,0 +1,548 @@
package brainfs
import (
"errors"
"io"
"io/fs"
"os"
"path/filepath"
"sync"
"sync/atomic"
"syscall"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// The handle check is what stops a FIFO swapped in after the Lstat. A FIFO
// with no writer reads EOF at once, so a race test could not see the check
// missing; here a writer holds the FIFO open, as any local process can, and
// readOpened must refuse it promptly (PR #161 review).
func TestReadOpenedRefusesAFIFOHeldOpenByAWriter(t *testing.T) {
dir := t.TempDir()
fifo := filepath.Join(dir, "p.md")
require.NoError(t, syscall.Mkfifo(fifo, 0o644))
w, err := os.OpenFile(fifo, os.O_RDWR, 0)
require.NoError(t, err)
defer func() { _ = w.Close() }()
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
done := make(chan error, 1)
go func() { _, err := b.readOpened("p.md"); done <- err }()
select {
case err := <-done:
assert.True(t, errors.Is(err, ErrNotRegular), "got %v", err)
case <-time.After(2 * time.Second):
t.Fatal("readOpened blocked on a FIFO with a writer")
}
}
func TestReadRegularRefusesSymlinksAndReadsRegularFiles(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "ok.md"), []byte("ok"), 0o644))
outside := filepath.Join(t.TempDir(), "s.md")
require.NoError(t, os.WriteFile(outside, []byte("secret"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(dir, "link.md")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
got, err := b.ReadRegular("ok.md")
require.NoError(t, err)
assert.Equal(t, "ok", string(got))
_, err = b.ReadRegular("link.md")
assert.True(t, errors.Is(err, ErrNotRegular), "got %v", err)
_, err = b.ReadRegular("../" + filepath.Base(outside))
assert.Error(t, err)
}
func TestRelAcceptsPathsInsideAndRefusesEscapes(t *testing.T) {
dir := t.TempDir()
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
for in, want := range map[string]string{
filepath.Join(dir, "wiki", "x.md"): "wiki/x.md",
filepath.Join(dir, "..foo", "x"): "..foo/x",
dir: ".",
} {
got, err := b.Rel(in)
require.NoError(t, err, in)
assert.Equal(t, want, got, in)
}
for _, bad := range []string{filepath.Join(dir, "..", "x"), dir + "-evil/x", "/etc/passwd", filepath.Join(dir, "wiki", "..", "..", "x")} {
_, err := b.Rel(bad)
assert.Error(t, err, bad)
}
}
func TestWalkRegularYieldsOnlyRegularFiles(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "a"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "a", "n.md"), []byte("n"), 0o644))
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(dir, "a", "dirlink")))
require.NoError(t, os.Symlink(filepath.Join(dir, "a", "n.md"), filepath.Join(dir, "a", "filelink.md")))
require.NoError(t, syscall.Mkfifo(filepath.Join(dir, "a", "p.md"), 0o644))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
var got []string
require.NoError(t, b.WalkRegular(".", func(rel string) error { got = append(got, rel); return nil }, nil))
assert.Equal(t, []string{"a/n.md"}, got)
}
func TestReadRegularMaxRefusesAFileOverTheLimit(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "big.md"), make([]byte, 100), 0o644))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
_, err = b.ReadRegularMax("big.md", 99)
assert.True(t, errors.Is(err, ErrTooLarge), "got %v", err)
got, err := b.ReadRegularMax("big.md", 100)
require.NoError(t, err)
assert.Len(t, got, 100)
}
func TestReadRegularMaxRefusesAndStopsReadingAFileThatGrowsPastTheLimit(t *testing.T) {
dir := t.TempDir()
p := filepath.Join(dir, "grow.md")
require.NoError(t, os.WriteFile(p, make([]byte, 10), 0o644))
consumed := &countingReader{}
wrapAfterStat = func(r io.Reader) io.Reader {
f, err := os.OpenFile(p, os.O_WRONLY|os.O_APPEND, 0)
require.NoError(t, err)
_, err = f.Write(make([]byte, 1000))
require.NoError(t, err)
require.NoError(t, f.Close())
consumed.r = r
return consumed
}
t.Cleanup(func() { wrapAfterStat = nil })
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
got, err := b.ReadRegularMax("grow.md", 100)
assert.True(t, errors.Is(err, ErrTooLarge), "got %v", err)
assert.Nil(t, got)
assert.LessOrEqual(t, consumed.n, 101, "read past max+1 bytes")
}
type countingReader struct {
r io.Reader
n int
}
func (c *countingReader) Read(p []byte) (int, error) {
n, err := c.r.Read(p)
c.n += n
return n, err
}
func TestLstatDoesNotFollowSymlinksAndRefusesEscapes(t *testing.T) {
dir := t.TempDir()
outside := filepath.Join(t.TempDir(), "o.md")
require.NoError(t, os.WriteFile(outside, []byte("o"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(dir, "l.md")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
fi, err := b.Lstat("l.md")
require.NoError(t, err)
assert.Equal(t, fs.ModeSymlink, fi.Mode().Type())
_, err = b.Lstat("../" + filepath.Base(filepath.Dir(outside)) + "/o.md")
assert.Error(t, err)
}
func TestWalkRegularReportsASymlinkedWalkRootAndYieldsNothing(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "real"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "real", "n.md"), []byte("n"), 0o644))
require.NoError(t, os.Symlink("real", filepath.Join(dir, "alias")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
var got []string
var errs []error
require.NoError(t, b.WalkRegular("alias", func(rel string) error { got = append(got, rel); return nil },
func(_ string, err error) { errs = append(errs, err) }))
assert.Empty(t, got)
require.Len(t, errs, 1)
assert.ErrorIs(t, errs[0], ErrNotRegular)
}
func TestWalkRegularReportsASymlinkedIntermediateComponentOfTheWalkRoot(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "real", "sub"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "real", "sub", "n.md"), []byte("n"), 0o644))
require.NoError(t, os.Symlink("real", filepath.Join(dir, "alias")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
var got []string
var errs []error
require.NoError(t, b.WalkRegular("alias/sub", func(rel string) error { got = append(got, rel); return nil },
func(_ string, err error) { errs = append(errs, err) }))
assert.Empty(t, got)
require.Len(t, errs, 1)
assert.ErrorIs(t, errs[0], ErrNotRegular)
}
func TestReadDirRegularListsOnlyRegularFilesAndRefusesSymlinkedComponents(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "a", "b", "sub"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "a", "b", "n.md"), []byte("n"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(dir, "other.md"), []byte("o"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "other.md"), filepath.Join(dir, "a", "b", "l.md")))
require.NoError(t, syscall.Mkfifo(filepath.Join(dir, "a", "b", "f.md"), 0o644))
require.NoError(t, os.Symlink("a", filepath.Join(dir, "alias")))
require.NoError(t, os.MkdirAll(filepath.Join(dir, "x"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "a"), filepath.Join(dir, "x", "mid")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
names, err := b.ReadDirRegular("a/b")
require.NoError(t, err)
assert.Equal(t, []string{"n.md"}, names)
for _, rel := range []string{"alias/b", "x/mid/b"} {
_, err := b.ReadDirRegular(rel)
assert.ErrorIs(t, err, ErrNotRegular, rel)
}
}
// A final component swapped for a symlink to another regular file in the
// brain between the Lstat and the open: os.Root follows in-root links, so
// only comparing the two (os.SameFile) refuses it.
func TestReadRegularDoesNotReturnAnInBrainFileSwappedInBehindASymlink(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "s.md"), []byte("zqsamefilesentinel"), 0o644))
p := filepath.Join(dir, "r.md")
require.NoError(t, os.WriteFile(p, []byte("inside"), 0o644))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
var stop atomic.Bool
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
tmp := p + ".swap"
for i := 0; !stop.Load(); i++ {
_ = os.Remove(tmp)
if i%2 == 0 {
_ = os.WriteFile(tmp, []byte("inside"), 0o644)
} else {
_ = os.Symlink("s.md", tmp)
}
_ = os.Rename(tmp, p)
}
}()
leaks, ok := 0, 0
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); {
got, err := b.ReadRegular("r.md")
if err == nil {
ok++
}
if string(got) == "zqsamefilesentinel" {
leaks++
}
got, _ = b.ReadRegularMax("r.md", 1<<20)
if string(got) == "zqsamefilesentinel" {
leaks++
}
}
stop.Store(true)
wg.Wait()
assert.Zero(t, leaks)
assert.Positive(t, ok, "positive control: the regular file is read")
}
func TestWriteRegularRefusesFinalSymlinksFIFOsEscapesAndMissingFilesAndRewritesRegularFiles(t *testing.T) {
dir := t.TempDir()
outside := filepath.Join(t.TempDir(), "o.md")
require.NoError(t, os.WriteFile(outside, []byte("untouched"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(dir, "l.md")))
require.NoError(t, syscall.Mkfifo(filepath.Join(dir, "f.md"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(dir, "n.md"), []byte("old content"), 0o644))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
assert.Error(t, b.WriteRegular("l.md", []byte("x")))
got, _ := os.ReadFile(outside)
assert.Equal(t, "untouched", string(got))
// A symlink to another file inside the brain: os.Root would allow it,
// only O_NOFOLLOW refuses it.
require.NoError(t, os.WriteFile(filepath.Join(dir, "in.md"), []byte("inside"), 0o644))
require.NoError(t, os.Symlink("in.md", filepath.Join(dir, "il.md")))
assert.Error(t, b.WriteRegular("il.md", []byte("x")))
got, _ = os.ReadFile(filepath.Join(dir, "in.md"))
assert.Equal(t, "inside", string(got))
// A reader held open, so a non-blocking write open would succeed. The
// Lstat type check refuses the FIFO first; the handle check and
// O_NONBLOCK matter only if it is swapped in after the Lstat.
r, err := os.OpenFile(filepath.Join(dir, "f.md"), os.O_RDONLY|syscall.O_NONBLOCK, 0)
require.NoError(t, err)
defer func() { _ = r.Close() }()
done := make(chan error, 1)
go func() { done <- b.WriteRegular("f.md", []byte("x")) }()
select {
case err := <-done:
assert.Error(t, err)
case <-time.After(3 * time.Second):
t.Fatal("WriteRegular hung on a FIFO")
}
assert.Error(t, b.WriteRegular("../"+filepath.Base(filepath.Dir(outside))+"/o.md", []byte("x")))
assert.Error(t, b.WriteRegular("missing.md", []byte("x")), "does not create files")
require.NoError(t, b.WriteRegular("n.md", []byte("new")))
got, _ = os.ReadFile(filepath.Join(dir, "n.md"))
assert.Equal(t, "new", string(got), "truncated and rewritten")
}
func TestReadDirDirsListsOnlyRealSubdirsAndRefusesSymlinkedComponents(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "a", "real"), 0o755))
require.NoError(t, os.MkdirAll(filepath.Join(dir, "other"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "other"), filepath.Join(dir, "a", "linked")))
require.NoError(t, os.WriteFile(filepath.Join(dir, "a", "f.md"), []byte("f"), 0o644))
require.NoError(t, os.Symlink("a", filepath.Join(dir, "alias")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
names, err := b.ReadDirDirs("a")
require.NoError(t, err)
assert.Equal(t, []string{"real"}, names)
_, err = b.ReadDirDirs("alias")
assert.ErrorIs(t, err, ErrNotRegular)
}
// hyperguild#173: a file reached through a symlinked directory inside the
// brain is refused for reads and writes, so a caller-supplied path cannot be
// redirected to another part of the brain.
func TestReadAndWriteRegularRefuseAFileBehindASymlinkedDirectory(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "raw", "v.md"), []byte("untouched"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "raw"), filepath.Join(dir, "wiki", "alias")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
_, err = b.ReadRegular("wiki/alias/v.md")
assert.ErrorIs(t, err, ErrNotRegular)
_, err = b.ReadRegularMax("wiki/alias/v.md", 100)
assert.ErrorIs(t, err, ErrNotRegular)
assert.ErrorIs(t, b.WriteRegular("wiki/alias/v.md", []byte("x")), ErrNotRegular)
got, _ := os.ReadFile(filepath.Join(dir, "raw", "v.md"))
assert.Equal(t, "untouched", string(got))
got, err = b.ReadRegular("raw/v.md")
require.NoError(t, err, "positive control: the real path still reads")
assert.Equal(t, "untouched", string(got))
}
// realDir checks the cleaned parent, so the I/O must use that same clean
// path: os.Root resolves ".." after following a symlink, and a ".." placed
// after a symlinked directory would otherwise pass the check and redirect.
func TestReadAndWriteRegularRefuseAPathThatIsNotClean(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "raw", "v.md"), []byte("untouched"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "raw", "facts"), filepath.Join(dir, "wiki", "alias")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
for _, rel := range []string{"wiki/alias/../v.md", "./raw/v.md", "raw//v.md", "/raw/v.md"} {
got, err := b.ReadRegular(rel)
assert.ErrorIs(t, err, ErrNotRegular, rel)
assert.NotContains(t, string(got), "untouched", rel)
_, err = b.ReadRegularMax(rel, 100)
assert.ErrorIs(t, err, ErrNotRegular, rel)
assert.ErrorIs(t, b.WriteRegular(rel, []byte("OVERWRITTEN")), ErrNotRegular, rel)
}
got, _ := os.ReadFile(filepath.Join(dir, "raw", "v.md"))
assert.Equal(t, "untouched", string(got))
}
// hyperguild#188: PutRegular creates or replaces a regular file through the
// root. Missing parent directories are created; an existing component that
// is not a real directory, a final symlink or special file, and an unclean
// path are refused, and nothing is created at a link's target.
func TestPutRegularCreatesOrReplacesOnlyThroughRealDirectories(t *testing.T) {
dir := t.TempDir()
outside := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw"), 0o755))
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "raw"), filepath.Join(dir, "wiki", "inwing")))
require.NoError(t, os.Symlink(outside, filepath.Join(dir, "wiki", "outwing")))
require.NoError(t, os.WriteFile(filepath.Join(dir, "raw", "v.md"), []byte("untouched"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki", "w"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "raw", "v.md"), filepath.Join(dir, "wiki", "w", "link.md")))
require.NoError(t, syscall.Mkfifo(filepath.Join(dir, "wiki", "w", "f.md"), 0o644))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
for _, rel := range []string{"wiki/inwing/facts/n.md", "wiki/outwing/facts/n.md", "wiki/w/link.md", "wiki/w/f.md", "wiki/w/../../raw/v.md", "/abs.md"} {
assert.ErrorIs(t, b.PutRegular(rel, []byte("x")), ErrNotRegular, rel)
}
ents, _ := os.ReadDir(outside)
assert.Empty(t, ents, "nothing created outside the brain")
_, err = os.Stat(filepath.Join(dir, "raw", "facts"))
assert.True(t, os.IsNotExist(err), "nothing created through the in-brain link")
got, _ := os.ReadFile(filepath.Join(dir, "raw", "v.md"))
assert.Equal(t, "untouched", string(got))
require.NoError(t, b.PutRegular("wiki/new/facts/n.md", []byte("one")), "positive control: create with missing parents")
got, _ = os.ReadFile(filepath.Join(dir, "wiki", "new", "facts", "n.md"))
assert.Equal(t, "one", string(got))
require.NoError(t, b.PutRegular("wiki/new/facts/n.md", []byte("two")), "positive control: replace")
got, _ = os.ReadFile(filepath.Join(dir, "wiki", "new", "facts", "n.md"))
assert.Equal(t, "two", string(got))
}
func TestAppendRegularCreatesOrAppendsOnlyThroughRealDirectories(t *testing.T) {
dir := t.TempDir()
outside := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "raw", "v.md"), []byte("untouched"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(dir, "outdir")))
require.NoError(t, os.Symlink("raw", filepath.Join(dir, "indir")))
require.NoError(t, os.Symlink(filepath.Join("raw", "v.md"), filepath.Join(dir, "log.md")))
require.NoError(t, syscall.Mkfifo(filepath.Join(dir, "f.md"), 0o644))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
for _, rel := range []string{"outdir/l.md", "indir/v.md", "log.md", "f.md", "raw/../log.md"} {
assert.ErrorIs(t, b.AppendRegular(rel, []byte("x")), ErrNotRegular, rel)
}
ents, _ := os.ReadDir(outside)
assert.Empty(t, ents)
got, _ := os.ReadFile(filepath.Join(dir, "raw", "v.md"))
assert.Equal(t, "untouched", string(got))
require.NoError(t, b.AppendRegular("logs/a.md", []byte("one\n")), "positive control: create")
require.NoError(t, b.AppendRegular("logs/a.md", []byte("two\n")), "positive control: append")
got, _ = os.ReadFile(filepath.Join(dir, "logs", "a.md"))
assert.Equal(t, "one\ntwo\n", string(got))
}
func TestRealParentRefusesASymlinkedDirectoryOrUncleanPath(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw"), 0o755))
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "raw"), filepath.Join(dir, "wiki", "alias")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
assert.ErrorIs(t, b.RealParent("wiki/alias/x.md"), ErrNotRegular)
assert.ErrorIs(t, b.RealParent("wiki/../raw/x.md"), ErrNotRegular)
assert.NoError(t, b.RealParent("wiki/x.md"), "positive control")
}
// cleanRel must refuse an unclean path on its own: here the ".." resolves,
// after a symlinked directory, to a real regular file, so only the clean
// check stands between the call and that file.
func TestAppendRegularRefusesAnUncleanPathThatResolvesToARealFile(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw", "keep"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "raw", "v.md"), []byte("untouched"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("raw", "keep"), filepath.Join(dir, "indir")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
assert.ErrorIs(t, b.AppendRegular("indir/../v.md", []byte("x")), ErrNotRegular)
assert.ErrorIs(t, b.PutRegular("indir/../v.md", []byte("x")), ErrNotRegular)
got, _ := os.ReadFile(filepath.Join(dir, "raw", "v.md"))
assert.Equal(t, "untouched", string(got))
}
func TestRemoveRefusesAPathBehindASymlinkedDirectoryAndRemovesOnlyTheLink(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "raw", "v.md"), []byte("keep"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "raw"), filepath.Join(dir, "wiki", "alias")))
require.NoError(t, os.Symlink(filepath.Join("..", "raw", "v.md"), filepath.Join(dir, "wiki", "link.md")))
require.NoError(t, os.WriteFile(filepath.Join(dir, "wiki", "n.md"), []byte("n"), 0o644))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
assert.ErrorIs(t, b.Remove("wiki/alias/v.md"), ErrNotRegular)
_, err = os.Stat(filepath.Join(dir, "raw", "v.md"))
assert.NoError(t, err, "the file behind the symlinked directory is kept")
require.NoError(t, b.Remove("wiki/link.md"))
_, err = os.Stat(filepath.Join(dir, "raw", "v.md"))
assert.NoError(t, err, "removing a final symlink never removes its target")
require.NoError(t, b.Remove("wiki/n.md"), "positive control")
_, err = os.Stat(filepath.Join(dir, "wiki", "n.md"))
assert.True(t, os.IsNotExist(err))
}
// CreateRegular is the only write path for promote and reject: it creates
// missing parents through the root, refuses an in-brain symlinked parent and
// an unclean path, and never replaces anything already there (fs.ErrExist,
// content unchanged), including a dangling link.
func TestCreateRegularCreatesOnlyNewFilesThroughRealDirectories(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw", "keep"), 0o755))
require.NoError(t, os.MkdirAll(filepath.Join(dir, "wiki"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "raw"), filepath.Join(dir, "wiki", "inwing")))
require.NoError(t, os.WriteFile(filepath.Join(dir, "raw", "v.md"), []byte("untouched"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("raw", "keep"), filepath.Join(dir, "indir")))
require.NoError(t, os.WriteFile(filepath.Join(dir, "wiki", "exists.md"), []byte("original"), 0o644))
require.NoError(t, os.Symlink("nowhere.md", filepath.Join(dir, "wiki", "dangling.md")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
assert.ErrorIs(t, b.CreateRegular("wiki/inwing/facts/n.md", []byte("x")), ErrNotRegular)
_, err = os.Stat(filepath.Join(dir, "raw", "facts"))
assert.True(t, os.IsNotExist(err), "nothing created through the in-brain link")
assert.ErrorIs(t, b.CreateRegular("indir/../v.md", []byte("x")), ErrNotRegular)
got, _ := os.ReadFile(filepath.Join(dir, "raw", "v.md"))
assert.Equal(t, "untouched", string(got))
assert.ErrorIs(t, b.CreateRegular("wiki/exists.md", []byte("x")), fs.ErrExist)
got, _ = os.ReadFile(filepath.Join(dir, "wiki", "exists.md"))
assert.Equal(t, "original", string(got))
assert.ErrorIs(t, b.CreateRegular("wiki/dangling.md", []byte("x")), fs.ErrExist)
_, err = os.Stat(filepath.Join(dir, "wiki", "nowhere.md"))
assert.True(t, os.IsNotExist(err), "nothing created at the dangling link's target")
require.NoError(t, b.CreateRegular("wiki/new/facts/n.md", []byte("one")), "positive control: create with missing parents")
got, _ = os.ReadFile(filepath.Join(dir, "wiki", "new", "facts", "n.md"))
assert.Equal(t, "one", string(got))
}
// On Linux a backslash is an ordinary filename byte, not a separator, so a
// name containing one stays a single in-root component (hyperguild#158: the
// watcher retried such a note forever while cleanRel refused it).
func TestReadAndPutRegularTreatABackslashAsAFilenameByte(t *testing.T) {
dir := t.TempDir()
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
require.NoError(t, b.PutRegular(`a\..\b.md`, []byte("x")))
got, err := os.ReadFile(filepath.Join(dir, `a\..\b.md`))
require.NoError(t, err, "the name was not kept as one component")
assert.Equal(t, "x", string(got))
got, err = b.ReadRegular(`a\..\b.md`)
require.NoError(t, err)
assert.Equal(t, "x", string(got))
}
@@ -0,0 +1,173 @@
// Package brainfstest is the shared harness for hyperguild#157's swap-race
// tests: it keeps swapping a regular note, a symlink to a sentinel target file
// (outside the brain, or inside it for hyperguild#171), or a FIFO (with a
// writer held open) into a path while a reader under test loops.
// It observes two things, over a sampled window only: no sentinel byte from
// the target reaches the reader's output, and the reader returns within a
// timeout. It does not cover
// hard links, other paths into the reader, or races it does not generate.
package brainfstest
import (
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"syscall"
"testing"
"time"
)
// Sentinel marks the target file's contents.
const Sentinel = "zqbrainfssentinelonefiftyseven"
// outsideContent carries Sentinel in every field a reader might extract: the
// frontmatter title and aliases, a heading, a wikilink and the body.
const outsideContent = "---\ntitle: " + Sentinel + "\naliases:\n - " + Sentinel + "\n---\n# " + Sentinel +
"\n\n" + Sentinel + " [[" + Sentinel + "]]\n"
// Outside writes a file outside any brain dir containing Sentinel.
func Outside(t testing.TB) string {
t.Helper()
p := filepath.Join(t.TempDir(), "secret.md")
if err := os.WriteFile(p, []byte(outsideContent), 0o644); err != nil {
t.Fatal(err)
}
return p
}
var swapSeq atomic.Int64
// StartSwap renames, in a loop, a regular note (content inside) and either
// a symlink to outside or a FIFO into path, until the test ends. FIFOs get a
// writer held open until the swapper exits, so a reader that skips the
// handle check parks forever instead of reading EOF. The swapper is stopped
// and awaited before t.TempDir cleanup. It returns the number of symlinks
// or FIFOs actually renamed into path so far, so a caller can tell a race
// that ran from one whose every swap failed (hyperguild#177).
func StartSwap(t testing.TB, path, inside, outside string, fifo bool) *atomic.Int64 {
t.Helper()
var swaps atomic.Int64
var stop atomic.Bool
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
// A name of its own: swappers on one path share a test, and one
// opening another's FIFO for write would block forever.
tmp := fmt.Sprintf("%s.swap%d", path, swapSeq.Add(1))
// A relative link: os.Root refuses every absolute link on its own, so
// an absolute in-brain target would test nothing.
link := outside
if rel, err := filepath.Rel(filepath.Dir(path), outside); err == nil {
link = rel
}
var writers []*os.File
defer func() {
for _, w := range writers {
_ = w.Close()
}
}()
for i := 0; !stop.Load(); i++ {
_ = os.Remove(tmp)
var made error
switch {
case i%2 == 0:
_ = os.WriteFile(tmp, []byte(inside), 0o644)
case fifo:
if made = syscall.Mkfifo(tmp, 0o644); made == nil {
if w, err := os.OpenFile(tmp, os.O_RDWR, 0); err == nil {
writers = append(writers, w)
}
}
default:
made = os.Symlink(link, tmp)
}
if err := os.Rename(tmp, path); err == nil && made == nil && i%2 == 1 {
swaps.Add(1)
}
}
}()
t.Cleanup(func() { stop.Store(true); wg.Wait() })
return &swaps
}
// WriteSentinelAt writes a file carrying Sentinel at p (creating its
// directory), for an in-brain symlink target.
func WriteSentinelAt(t *testing.T, p string) string {
t.Helper()
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(outsideContent), 0o644); err != nil {
t.Fatal(err)
}
return p
}
// AssertNoLeak loops read for d while a symlink swapper runs on path, and
// fails if any output contains Sentinel. The symlink points outside the
// brain; os.Root refuses that on its own, so pair it with AssertNoLeakFrom
// and an in-brain target.
func AssertNoLeak(t *testing.T, path, inside string, d time.Duration, read func() string) {
t.Helper()
AssertNoLeakFrom(t, path, inside, Outside(t), d, read)
}
// InsideMarker must appear in the inside note's content and in read's
// output whenever read saw that note: the floor below counts such reads.
const InsideMarker = "inside"
// AssertNoLeakFrom is AssertNoLeak with the symlink pointing at target, e.g.
// a file elsewhere in the brain written with WriteSentinelAt.
//
// Beyond "no Sentinel in any output" it sets two floors, because without
// them the assertion passes when nothing was tested (hyperguild#177): at
// least one symlink must have been swapped into path, and at least one read
// must have returned InsideMarker (a reader that always errors returns
// nothing, and so never leaks).
func AssertNoLeakFrom(t testing.TB, path, inside, target string, d time.Duration, read func() string) {
t.Helper()
if !strings.Contains(inside, InsideMarker) {
t.Fatalf("inside content must contain %q", InsideMarker)
}
swaps := StartSwap(t, path, inside, target, false)
leaks, iters, insideReads := 0, 0, 0
for deadline := time.Now().Add(d); time.Now().Before(deadline); iters++ {
out := read()
leaks += strings.Count(out, Sentinel)
if strings.Contains(out, InsideMarker) {
insideReads++
}
}
if leaks > 0 {
t.Fatalf("symlink target %s leaked %d times in %d iterations", target, leaks, iters)
}
if n := swaps.Load(); n == 0 {
t.Fatalf("no symlink was swapped into %s in %d iterations: the race did not run", path, iters)
}
if insideReads == 0 {
t.Fatalf("no read in %d iterations returned the inside note (%q): the reader read nothing", iters, InsideMarker)
}
}
// AssertNoHang loops read for d while a FIFO swapper (held writers) runs on
// path, and fails if a read does not return within timeout.
func AssertNoHang(t *testing.T, path, inside string, d, timeout time.Duration, read func()) {
t.Helper()
StartSwap(t, path, inside, Outside(t), true)
done := make(chan struct{})
go func() {
for deadline := time.Now().Add(d); time.Now().Before(deadline); {
read()
}
close(done)
}()
select {
case <-done:
case <-time.After(timeout):
t.Fatal("reader hung on a swapped-in FIFO with a writer held open")
}
}
@@ -0,0 +1,79 @@
package brainfstest
import (
"fmt"
"os"
"path/filepath"
"runtime"
"syscall"
"testing"
"time"
)
// recordingTB stands in for the test under AssertNoLeakFrom so a fatal
// there can be observed instead of failing this test. Fatalf ends the
// calling goroutine the way testing.T does.
type recordingTB struct {
testing.TB
failed bool
msg string
}
func (r *recordingTB) Fatalf(format string, args ...any) {
r.failed, r.msg = true, fmt.Sprintf(format, args...)
runtime.Goexit()
}
func runAssert(t *testing.T, path, target string, read func() string) *recordingTB {
t.Helper()
r := &recordingTB{TB: t}
done := make(chan struct{})
go func() {
defer close(done)
AssertNoLeakFrom(r, path, "inside", target, 200*time.Millisecond, read)
}()
<-done
return r
}
// hyperguild#177: with every swap failing, no leak can occur, so the leak
// assertion passed without testing anything.
func TestAssertNoLeakFromFailsWhenNoSymlinkSwapSucceeds(t *testing.T) {
path := filepath.Join(t.TempDir(), "missing-dir", "r.md")
r := runAssert(t, path, Outside(t), func() string { return "inside" })
if !r.failed {
t.Fatal("AssertNoLeakFrom passed with zero successful symlink swaps")
}
}
// hyperguild#177: a reader that always errors returns nothing, so it never
// leaks either.
func TestAssertNoLeakFromFailsWhenNoReadReturnsTheInsideNote(t *testing.T) {
path := filepath.Join(t.TempDir(), "r.md")
r := runAssert(t, path, Outside(t), func() string { return "" })
if !r.failed {
t.Fatal("AssertNoLeakFrom passed with zero reads returning the inside note")
}
}
// Positive control: a reader that refuses symlinks and reads regular files
// passes, so the floors do not fail a working reader.
func TestAssertNoLeakFromPassesAReaderThatRefusesSymlinks(t *testing.T) {
path := filepath.Join(t.TempDir(), "r.md")
read := func() string {
f, err := os.OpenFile(path, os.O_RDONLY|noFollow, 0)
if err != nil {
return ""
}
defer func() { _ = f.Close() }()
b := make([]byte, 64)
n, _ := f.Read(b)
return string(b[:n])
}
r := runAssert(t, path, Outside(t), read)
if r.failed {
t.Fatalf("a symlink-refusing reader failed: %s", r.msg)
}
}
const noFollow = syscall.O_NOFOLLOW
+21 -9
View File
@@ -42,13 +42,19 @@ func (s *Store) WithGraph(g graphsync.Store) *Store {
// Write creates a brain note and returns its read-after-write handle.
func (s *Store) Write(ctx context.Context, n capture.Note) (capture.Ref, error) {
content, unlinked, err := brain.StripUnresolvedLinks(s.brainDir, n.Content)
if err != nil {
return capture.Ref{}, err
}
n.Content = content
relPath, err := api.WriteNote(s.brainDir, api.WriteNoteOptions{
Content: n.Content,
Filename: n.Filename,
Type: n.Type,
Domain: n.Domain,
Wing: n.Wing,
Hall: n.Hall,
Content: n.Content,
Filename: n.Filename,
Wing: n.Wing,
Hall: n.Hall,
Origin: n.Origin,
WrittenBy: n.WrittenBy,
})
if err != nil {
return capture.Ref{}, err
@@ -57,14 +63,20 @@ func (s *Store) Write(ctx context.Context, n capture.Note) (capture.Ref, error)
s.indexInGraph(ctx, "brain_write", relPath)
_, _, hash, _ := api.ReadNote(s.brainDir, relPath)
return capture.Ref{ID: relPath, Path: relPath, ContentHash: hash}, nil
return capture.Ref{ID: relPath, Path: relPath, ContentHash: hash, Unlinked: unlinked}, nil
}
// Update supersedes an existing note in place. slug may be a bare slug
// (resolved against n.Wing/n.Hall) or a full brain-relative path (when it
// contains a slash). It never creates — a missing target is an error.
func (s *Store) Update(ctx context.Context, slug string, n capture.Note) (capture.Ref, error) {
opts := api.UpdateNoteOptions{Content: n.Content, Reason: n.Reason}
content, unlinked, err := brain.StripUnresolvedLinks(s.brainDir, n.Content)
if err != nil {
return capture.Ref{}, err
}
n.Content = content
opts := api.UpdateNoteOptions{Content: n.Content, Reason: n.Reason, Origin: n.Origin, WrittenBy: n.WrittenBy}
if strings.Contains(slug, "/") {
opts.Path = slug
} else {
@@ -80,7 +92,7 @@ func (s *Store) Update(ctx context.Context, slug string, n capture.Note) (captur
}
s.indexInGraph(ctx, "brain_update", relPath)
return capture.Ref{ID: relPath, Path: relPath, ContentHash: hash, Superseded: true}, nil
return capture.Ref{ID: relPath, Path: relPath, ContentHash: hash, Superseded: true, Unlinked: unlinked}, nil
}
// Get fetches a note by id/path — the read-after-write confirmation
@@ -83,3 +83,45 @@ func TestStoreGetRoundTripsHash(t *testing.T) {
assert.Equal(t, "a", note.Frontmatter["wing"])
assert.Contains(t, note.Body, "# Body")
}
func TestStoreWriteUnwrapsUnresolvedWikilinks(t *testing.T) {
dir := t.TempDir()
s := brainstore.New(dir)
_, err := s.Write(context.Background(), capture.Note{
Content: "# Target\n\nbody\n", Filename: "target", Wing: "a", Hall: "facts",
})
require.NoError(t, err)
ref, err := s.Write(context.Background(), capture.Note{
Content: "See [[target]] and [[secret-hygiene]].\n",
Filename: "src", Wing: "a", Hall: "facts",
})
require.NoError(t, err)
assert.Equal(t, []string{"secret-hygiene"}, ref.Unlinked)
got, err := os.ReadFile(filepath.Join(dir, "wiki/a/facts/src.md"))
require.NoError(t, err)
assert.Contains(t, string(got), "[[target]]")
assert.Contains(t, string(got), "and secret-hygiene.")
assert.NotContains(t, string(got), "[[secret-hygiene]]")
}
func TestStoreUpdateUnwrapsUnresolvedWikilinks(t *testing.T) {
dir := t.TempDir()
s := brainstore.New(dir)
_, err := s.Write(context.Background(), capture.Note{
Content: "old\n", Filename: "n", Wing: "a", Hall: "facts",
})
require.NoError(t, err)
ref, err := s.Update(context.Background(), "n", capture.Note{
Content: "Related: [[cad-babysit]].\n", Wing: "a", Hall: "facts", Reason: "r",
})
require.NoError(t, err)
assert.Equal(t, []string{"cad-babysit"}, ref.Unlinked)
got, err := os.ReadFile(filepath.Join(dir, "wiki/a/facts/n.md"))
require.NoError(t, err)
assert.Contains(t, string(got), "Related: cad-babysit.")
assert.NotContains(t, string(got), "[[cad-babysit]]")
}
+3 -18
View File
@@ -1,8 +1,7 @@
// Package capture is the Clean-Architecture use-case for the uniform
// capture capability (issue #49/#51): persist a finished session's
// valuable output — insights → brain, action items → Gitea tickets,
// optional summary → ai-sessions — with one invocation, identical core
// behaviour across every harness.
// valuable output — insights → brain, action items → Gitea tickets —
// with one invocation, identical core behaviour across every harness.
//
// This package is pure orchestration. It depends only on ports
// (interfaces) and plain entities — no HTTP, no live Gitea, no embedding
@@ -61,6 +60,7 @@ type CaptureContext struct {
Classification string // caller-declared level token ("" = unspecified)
Principal string // server-derived (auth); audit identity
Origin Zone // server-derived trust zone; the I1 gate input
AuthPath string // server-derived: "static" or "jwt"; stamped as note provenance (#145)
}
// Insight is one piece of session knowledge bound for the brain. A
@@ -83,19 +83,11 @@ type Ticket struct {
Body string
}
// Summary is an optional session summary bound for ai-sessions.
type Summary struct {
Title string
Body string
ReposTouched []string
}
// CaptureInput is the whole capture request.
type CaptureInput struct {
Context CaptureContext
Insights []Insight
Tickets []Ticket
Summary *Summary
DryRun bool
}
@@ -117,12 +109,6 @@ type TicketResult struct {
OK bool `json:"ok"`
}
// SummaryResult is the summary outcome in the receipt.
type SummaryResult struct {
Path string `json:"path,omitempty"`
OK bool `json:"ok"`
}
// ItemError pins a failure to a specific request item for the partial
// receipt. Item is a stable locator like "insight[1]" or "ticket[0]".
type ItemError struct {
@@ -136,7 +122,6 @@ type ItemError struct {
type CaptureReceipt struct {
Insights []InsightResult `json:"insights"`
Tickets []TicketResult `json:"tickets"`
Summary *SummaryResult `json:"summary,omitempty"`
Errors []ItemError `json:"errors"`
EffectiveClassification string `json:"effective_classification,omitempty"`
DryRun bool `json:"dry_run"`
+11 -10
View File
@@ -15,6 +15,10 @@ type Ref struct {
Path string
ContentHash string
Superseded bool
// Unlinked names the [[wikilink]] targets the store unwrapped to plain
// text because they resolve to no entry. Reported rather than rejected:
// see brain.StripUnresolvedLinks for why the write still lands.
Unlinked []string
}
// StoredNote is a brain note fetched by Get: the read-after-write
@@ -27,18 +31,20 @@ type StoredNote struct {
Body string
}
// Note is the brain-write payload. It carries both the wing/hall taxonomy
// and the legacy type/domain fields so a single BrainStore serves both
// capture insights and the existing MCP brain_write surface. Reason is
// Note is the brain-write payload: wing+hall are the only routing fields
// (the legacy type/domain params were removed — ADR-0012 / #79). Reason is
// the supersede rationale, used only by Update.
type Note struct {
Content string
Filename string
Wing string
Hall string
Type string
Domain string
Reason string
// Provenance (hyperguild#145) is stamped by the server from the
// authenticated principal, never taken from Content's frontmatter.
// Origin is "static", "jwt" or "unknown"; empty means "unknown".
Origin string
WrittenBy string
}
// BrainStore is the brain persistence port — the shared implementation of
@@ -69,11 +75,6 @@ type IssueTracker interface {
CommentIssue(ctx context.Context, repo string, number int, body string) (IssueRef, error)
}
// SummaryWriter is the ai-sessions summary port.
type SummaryWriter interface {
WriteFile(ctx context.Context, repo, path, content string) error
}
// ClassificationPolicy derives a target's sensitivity (model C). The
// "stricter wins" combination of declared vs derived is use-case policy
// and lives in the service, so the port stays minimal. Satisfied by
+55 -100
View File
@@ -2,8 +2,6 @@ package capture
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"strings"
"time"
@@ -14,22 +12,19 @@ import (
// Service is the CaptureSession use-case. It depends only on ports.
type Service struct {
brain BrainStore
issues IssueTracker
summaries SummaryWriter
policy ClassificationPolicy
audit AuditSink
brain BrainStore
issues IssueTracker
policy ClassificationPolicy
audit AuditSink
// now is the clock, injectable for deterministic summary paths and
// audit timestamps in tests.
// now is the clock, injectable for deterministic audit timestamps in
// tests.
now func() time.Time
}
// NewService constructs a Service from its ports. summaries may be nil
// when no summary persistence is wired; a CaptureInput with a Summary
// then fails that item rather than panicking.
func NewService(b BrainStore, tr IssueTracker, sw SummaryWriter, p ClassificationPolicy, a AuditSink) *Service {
return &Service{brain: b, issues: tr, summaries: sw, policy: p, audit: a, now: time.Now}
// NewService constructs a Service from its ports.
func NewService(b BrainStore, tr IssueTracker, p ClassificationPolicy, a AuditSink) *Service {
return &Service{brain: b, issues: tr, policy: p, audit: a, now: time.Now}
}
var validActions = map[string]bool{"create": true, "close": true, "comment": true}
@@ -88,7 +83,7 @@ func (s *Service) Capture(ctx context.Context, in CaptureInput) (CaptureReceipt,
declared, _ = classification.ParseLevel(in.Context.Classification)
}
effective, securityEvents := s.resolveClassification(declared, in)
effective, securityEvents, escalators := s.resolveClassification(declared, in)
// Server-derived origin governs the I1 gate; a caller-asserted harness
// label that names a different zone is descriptive-only and logged as a
@@ -112,8 +107,8 @@ func (s *Service) Capture(ctx context.Context, in CaptureInput) (CaptureReceipt,
Items: nil, // refused before any write
SecurityEvents: append(securityEvents, "I1 refusal: confidential capture via us-nexus origin"),
}, AuditCentral)
return CaptureReceipt{}, fmt.Errorf("%w: effective classification confidential through %s origin",
ErrSovereigntyRefused, in.Context.Origin)
return CaptureReceipt{}, fmt.Errorf("%w: effective classification confidential through %s origin (%s)",
ErrSovereigntyRefused, in.Context.Origin, escalationReason(escalators))
}
receipt := CaptureReceipt{
@@ -131,9 +126,6 @@ func (s *Service) Capture(ctx context.Context, in CaptureInput) (CaptureReceipt,
for _, tk := range in.Tickets {
receipt.Tickets = append(receipt.Tickets, TicketResult{Repo: tk.Repo, Action: tk.Action, Number: tk.Number, OK: true})
}
if in.Summary != nil {
receipt.Summary = &SummaryResult{Path: s.summaryPath(in.Context, in.Summary), OK: true}
}
return receipt, nil
}
@@ -150,7 +142,7 @@ func (s *Service) Capture(ctx context.Context, in CaptureInput) (CaptureReceipt,
var landed []string
for i, ins := range in.Insights {
res, item, err := s.persistInsight(ctx, ins)
res, item, err := s.persistInsight(ctx, ins, in.Context)
receipt.Insights = append(receipt.Insights, res)
if err != nil {
receipt.Errors = append(receipt.Errors, ItemError{Item: fmt.Sprintf("insight[%d]", i), Error: err.Error()})
@@ -169,16 +161,6 @@ func (s *Service) Capture(ctx context.Context, in CaptureInput) (CaptureReceipt,
landed = append(landed, fmt.Sprintf("ticket:%s#%d", tk.Repo, res.Number))
}
if in.Summary != nil {
res, err := s.persistSummary(ctx, in.Context, in.Summary)
receipt.Summary = &res
if err != nil {
receipt.Errors = append(receipt.Errors, ItemError{Item: "summary", Error: err.Error()})
} else {
landed = append(landed, "summary:"+res.Path)
}
}
// I5: persist the request-level audit record of exactly what landed,
// using the outcome reserved before the writes. AuditBuffered surfaces
// the degraded (locally-buffered) state on the receipt.
@@ -239,15 +221,28 @@ func (s *Service) validate(in CaptureInput) error {
return nil
}
// resolveClassification computes the effective level (stricter of
// declared and every target's derived level) and collects a security
// event whenever the caller under-declared relative to a target floor.
func (s *Service) resolveClassification(declared classification.Level, in CaptureInput) (classification.Level, []string) {
// resolveClassification computes the effective level (stricter of declared
// and every target's derived level), collects a security event whenever the
// caller under-declared relative to a target floor, and names the targets
// that carry the strictest level.
//
// That third return is what makes an I1 refusal actionable. A caller sending
// twenty tickets gets one refusal; without the offending target it cannot
// tell which repo tripped the gate, and infra#265 shows what happens next —
// the caller concluded the tickets ARRAY was broken, built a fallback of
// twenty individual calls, and filed a bug against the wrong thing.
func (s *Service) resolveClassification(declared classification.Level, in CaptureInput) (classification.Level, []string, []string) {
effective := declared
var events []string
derivedFor := map[string]classification.Level{}
var order []string
consider := func(kind classification.TargetKind, name string) {
derived := s.policy.Derive(classification.Target{Kind: kind, Name: name})
effective = classification.Stricter(effective, derived)
if _, seen := derivedFor[name]; !seen {
derivedFor[name] = derived
order = append(order, name)
}
if declared < derived {
events = append(events, fmt.Sprintf("classification under-declared: declared=%s target=%s(%s) derived=%s",
declared, name, kindString(kind), derived))
@@ -259,16 +254,35 @@ func (s *Service) resolveClassification(declared classification.Level, in Captur
for _, tk := range in.Tickets {
consider(classification.RepoTarget, tk.Repo)
}
if in.Summary != nil {
for _, repo := range in.Summary.ReposTouched {
consider(classification.RepoTarget, repo)
// Only targets that actually reach the effective level are named: at
// twenty tickets, listing the innocent ones is noise that hides the one
// that matters.
var escalators []string
for _, name := range order {
if derivedFor[name] == effective && effective > declared {
escalators = append(escalators, name)
}
}
return effective, events
return effective, events, escalators
}
func (s *Service) persistInsight(ctx context.Context, ins Insight) (InsightResult, string, error) {
note := Note{Content: ins.Text, Wing: ins.Wing, Hall: ins.Hall, Filename: brain.Sanitise(firstLine(ins.Text))}
// escalationReason renders why the effective classification landed where it
// did, for an operator reading a refusal. Naming the caller's own declaration
// matters as much as naming a target: "you asked for this" and "a repo you
// touched required it" call for different fixes.
func escalationReason(escalators []string) string {
if len(escalators) == 0 {
return "declared by the caller; no target required it"
}
return "required by: " + strings.Join(escalators, ", ")
}
func (s *Service) persistInsight(ctx context.Context, ins Insight, cc CaptureContext) (InsightResult, string, error) {
note := Note{
Content: ins.Text, Wing: ins.Wing, Hall: ins.Hall, Filename: brain.Sanitise(firstLine(ins.Text)),
Origin: cc.AuthPath, WrittenBy: cc.Principal,
}
var ref Ref
var err error
if ins.SupersedeSlug != "" {
@@ -309,60 +323,6 @@ func (s *Service) persistTicket(ctx context.Context, tk Ticket) (TicketResult, e
return res, nil
}
func (s *Service) persistSummary(ctx context.Context, c CaptureContext, sum *Summary) (SummaryResult, error) {
if s.summaries == nil {
return SummaryResult{OK: false}, fmt.Errorf("no summary writer configured")
}
path := s.summaryPath(c, sum)
content := s.renderSummary(c, sum)
repo := "ai-sessions"
if err := s.summaries.WriteFile(ctx, repo, path, content); err != nil {
return SummaryResult{Path: path, OK: false}, err
}
return SummaryResult{Path: path, OK: true}, nil
}
// summaryPath builds summaries/<harness>/<YYYY-MM>/<date>-<slug>-<ref8>.md.
// The ref8 disambiguator is derived from the session_ref (or the title
// when no ref is present) so distinct sessions never collide.
func (s *Service) summaryPath(c CaptureContext, sum *Summary) string {
t := s.now().UTC()
slug := brain.Sanitise(sum.Title)
if slug == "" {
slug = "summary"
}
seed := c.SessionRef
if seed == "" {
seed = sum.Title + sum.Body
}
sum8 := shortHash(seed)
return fmt.Sprintf("summaries/%s/%s/%s-%s-%s.md",
brain.Sanitise(c.Harness), t.Format("2006-01"), t.Format("2006-01-02"), slug, sum8)
}
// renderSummary stamps fidelity + session metadata into frontmatter so the
// richer-fidelity-supersedes-thinner collision rule has the data it needs.
func (s *Service) renderSummary(c CaptureContext, sum *Summary) string {
var b strings.Builder
b.WriteString("---\n")
fmt.Fprintf(&b, "title: %s\n", sum.Title)
fmt.Fprintf(&b, "harness: %s\n", c.Harness)
if c.SessionRef != "" {
fmt.Fprintf(&b, "session_ref: %s\n", c.SessionRef)
}
fmt.Fprintf(&b, "fidelity: %s\n", c.Fidelity)
fmt.Fprintf(&b, "captured_at: %s\n", s.now().UTC().Format(time.RFC3339))
if len(sum.ReposTouched) > 0 {
fmt.Fprintf(&b, "repos_touched: [%s]\n", strings.Join(sum.ReposTouched, ", "))
}
b.WriteString("---\n\n")
b.WriteString(sum.Body)
if !strings.HasSuffix(sum.Body, "\n") {
b.WriteByte('\n')
}
return b.String()
}
func kindString(k classification.TargetKind) string {
if k == classification.RepoTarget {
return "repo"
@@ -381,8 +341,3 @@ func firstLine(s string) string {
}
return s
}
func shortHash(s string) string {
sum := sha256.Sum256([]byte(s))
return hex.EncodeToString(sum[:])[:8]
}
+122 -53
View File
@@ -3,6 +3,7 @@ package capture
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
@@ -59,12 +60,18 @@ type fakeTracker struct {
closed []int
comments []int
err error
// failOnTitle makes CreateIssue fail for one specific title, so a
// partial-failure receipt can be exercised without failing every item.
failOnTitle string
}
func (f *fakeTracker) CreateIssue(_ context.Context, repo, title, _ string) (IssueRef, error) {
if f.err != nil {
return IssueRef{}, f.err
}
if f.failOnTitle != "" && title == f.failOnTitle {
return IssueRef{}, errors.New("tracker refused this one")
}
f.created = append(f.created, repo+":"+title)
return IssueRef{Repo: repo, Number: 100 + len(f.created), URL: "https://git/" + repo + "/issues/x"}, nil
}
@@ -85,24 +92,11 @@ func (f *fakeTracker) CommentIssue(_ context.Context, repo string, number int, _
return IssueRef{Repo: repo, Number: number}, nil
}
type fakeSummary struct {
paths []string
content []string
err error
}
func (f *fakeSummary) WriteFile(_ context.Context, _, path, content string) error {
if f.err != nil {
return f.err
}
f.paths = append(f.paths, path)
f.content = append(f.content, content)
return nil
}
// fakePolicy derives from an explicit map; default Internal so tests pin
// behaviour without depending on the real defaulting.
type fakePolicy struct{ tags map[string]classification.Level }
type fakePolicy struct {
tags map[string]classification.Level
}
func (p fakePolicy) Derive(t classification.Target) classification.Level {
if lvl, ok := p.tags[t.Name]; ok {
@@ -135,8 +129,8 @@ func (f *fakeAudit) Record(_ context.Context, e AuditEntry, _ AuditOutcome) erro
// --- helpers ---
func newSvc(b BrainStore, tr IssueTracker, sw SummaryWriter, p ClassificationPolicy, a AuditSink) *Service {
s := NewService(b, tr, sw, p, a)
func newSvc(b BrainStore, tr IssueTracker, p ClassificationPolicy, a AuditSink) *Service {
s := NewService(b, tr, p, a)
s.now = func() time.Time { return time.Date(2026, 6, 22, 12, 0, 0, 0, time.UTC) }
return s
}
@@ -151,7 +145,7 @@ func TestCaptureHappyPath(t *testing.T) {
b := &fakeBrain{}
tr := &fakeTracker{}
au := &fakeAudit{}
svc := newSvc(b, tr, nil, fakePolicy{}, au)
svc := newSvc(b, tr, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
@@ -180,7 +174,7 @@ func TestCaptureHappyPath(t *testing.T) {
func TestCaptureSupersedeNotDuplicate(t *testing.T) {
b := &fakeBrain{}
svc := newSvc(b, &fakeTracker{}, nil, fakePolicy{}, &fakeAudit{})
svc := newSvc(b, &fakeTracker{}, fakePolicy{}, &fakeAudit{})
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
@@ -197,7 +191,7 @@ func TestCaptureValidationFailClosed(t *testing.T) {
b := &fakeBrain{}
tr := &fakeTracker{}
au := &fakeAudit{}
svc := newSvc(b, tr, nil, fakePolicy{}, au)
svc := newSvc(b, tr, fakePolicy{}, au)
_, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
@@ -216,7 +210,7 @@ func TestCaptureValidationFailClosed(t *testing.T) {
}
func TestCaptureValidationRejectsBadTicket(t *testing.T) {
svc := newSvc(&fakeBrain{}, &fakeTracker{}, nil, fakePolicy{}, &fakeAudit{})
svc := newSvc(&fakeBrain{}, &fakeTracker{}, fakePolicy{}, &fakeAudit{})
_, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
Tickets: []Ticket{{Repo: "hyperguild", Action: "frobnicate"}}, // bad action
@@ -239,7 +233,7 @@ func TestCapturePartialFailureBestEffort(t *testing.T) {
}}
tr := &fakeTracker{}
au := &fakeAudit{}
svc := newSvc(b, tr, nil, fakePolicy{}, au)
svc := newSvc(b, tr, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
@@ -264,7 +258,7 @@ func TestCaptureDryRunWritesNothing(t *testing.T) {
b := &fakeBrain{}
tr := &fakeTracker{}
au := &fakeAudit{}
svc := newSvc(b, tr, nil, fakePolicy{}, au)
svc := newSvc(b, tr, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
@@ -287,7 +281,7 @@ func TestCaptureStricterClassificationWins(t *testing.T) {
b := &fakeBrain{}
au := &fakeAudit{}
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
svc := newSvc(b, &fakeTracker{}, nil, pol, au)
svc := newSvc(b, &fakeTracker{}, pol, au)
ctx := baseCtx()
ctx.Classification = "internal"
@@ -306,7 +300,7 @@ func TestCaptureCallerRaisingSensitivityHonoured(t *testing.T) {
// Caller declares confidential; target internal → effective confidential, NOT a security event.
au := &fakeAudit{}
pol := fakePolicy{tags: map[string]classification.Level{"hyperguild": classification.Internal}}
svc := newSvc(&fakeBrain{}, &fakeTracker{}, nil, pol, au)
svc := newSvc(&fakeBrain{}, &fakeTracker{}, pol, au)
ctx := baseCtx()
ctx.Classification = "confidential"
@@ -319,26 +313,6 @@ func TestCaptureCallerRaisingSensitivityHonoured(t *testing.T) {
assert.Empty(t, au.entries[0].SecurityEvents, "raising sensitivity is honoured, not flagged")
}
func TestCaptureSummaryPathAndFidelity(t *testing.T) {
sw := &fakeSummary{}
svc := newSvc(&fakeBrain{}, &fakeTracker{}, sw, fakePolicy{}, &fakeAudit{})
ctx := baseCtx()
ctx.Fidelity = "transcript-parse"
ctx.SessionRef = "abc123def456"
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: ctx,
Summary: &Summary{Title: "Session Wrap", Body: "did stuff", ReposTouched: []string{"hyperguild"}},
})
require.NoError(t, err)
require.NotNil(t, rec.Summary)
assert.True(t, rec.Summary.OK)
require.Len(t, sw.paths, 1)
assert.True(t, strings.HasPrefix(sw.paths[0], "summaries/claude-code/2026-06/"), "path: %s", sw.paths[0])
assert.Contains(t, sw.paths[0], "session-wrap")
assert.Contains(t, sw.content[0], "fidelity: transcript-parse", "fidelity stamped in frontmatter")
}
// --- I1 sovereignty gate (#53) ---
func TestCaptureRefusesConfidentialViaUSNexus(t *testing.T) {
@@ -346,7 +320,7 @@ func TestCaptureRefusesConfidentialViaUSNexus(t *testing.T) {
tr := &fakeTracker{}
au := &fakeAudit{}
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
svc := newSvc(b, tr, nil, pol, au)
svc := newSvc(b, tr, pol, au)
ctx := baseCtx()
ctx.Classification = "confidential"
@@ -368,7 +342,7 @@ func TestCaptureRefusesConfidentialViaUSNexus(t *testing.T) {
func TestCaptureAllowsConfidentialViaSovereign(t *testing.T) {
b := &fakeBrain{}
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
svc := newSvc(b, &fakeTracker{}, nil, pol, &fakeAudit{})
svc := newSvc(b, &fakeTracker{}, pol, &fakeAudit{})
ctx := baseCtx()
ctx.Classification = "confidential"
@@ -387,7 +361,7 @@ func TestCaptureAssertedLabelIgnoredAndLogged(t *testing.T) {
// us-nexus; confidential ⇒ refused, and the discrepancy is a security event.
au := &fakeAudit{}
pol := fakePolicy{tags: map[string]classification.Level{"client-seb": classification.Confidential}}
svc := newSvc(&fakeBrain{}, &fakeTracker{}, nil, pol, au)
svc := newSvc(&fakeBrain{}, &fakeTracker{}, pol, au)
ctx := baseCtx()
ctx.Harness = "sovereign-soil" // asserted
@@ -407,7 +381,7 @@ func TestCaptureAssertedLabelIgnoredAndLogged(t *testing.T) {
func TestCaptureInternalViaUSNexusAllowed(t *testing.T) {
// us-nexus origin is fine for non-confidential data.
b := &fakeBrain{}
svc := newSvc(b, &fakeTracker{}, nil, fakePolicy{}, &fakeAudit{})
svc := newSvc(b, &fakeTracker{}, fakePolicy{}, &fakeAudit{})
ctx := baseCtx()
ctx.Origin = ZoneUSNexus // internal classification, so gate doesn't fire
rec, err := svc.Capture(context.Background(), CaptureInput{
@@ -426,7 +400,7 @@ func TestCaptureRefusesWhenAuditReserveFails(t *testing.T) {
b := &fakeBrain{}
tr := &fakeTracker{}
au := &fakeAudit{reserveErr: errors.New("central sink unreachable")}
svc := newSvc(b, tr, nil, fakePolicy{}, au)
svc := newSvc(b, tr, fakePolicy{}, au)
_, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
@@ -443,7 +417,7 @@ func TestCaptureFlagsLocallyBufferedAudit(t *testing.T) {
// proceeds and the receipt flags the degraded audit state.
b := &fakeBrain{}
au := &fakeAudit{reserveMode: AuditBuffered}
svc := newSvc(b, &fakeTracker{}, nil, fakePolicy{}, au)
svc := newSvc(b, &fakeTracker{}, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
@@ -458,7 +432,7 @@ func TestCaptureFlagsLocallyBufferedAudit(t *testing.T) {
func TestCaptureDryRunSkipsAuditGate(t *testing.T) {
// dry_run must not even probe the audit sink (writes nothing anywhere).
au := &fakeAudit{reserveErr: errors.New("would refuse")}
svc := newSvc(&fakeBrain{}, &fakeTracker{}, nil, fakePolicy{}, au)
svc := newSvc(&fakeBrain{}, &fakeTracker{}, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
@@ -469,3 +443,98 @@ func TestCaptureDryRunSkipsAuditGate(t *testing.T) {
assert.True(t, rec.DryRun)
assert.Empty(t, au.entries)
}
// ---- hyperguild#91 part 1 -------------------------------------------------
//
// The reported symptom was "capture with a tickets array fails" (infra#265:
// 20 tickets errored, 5 errored, caller fell back to 20 individual
// issue_create calls). The array was never the failing dimension — every
// ticket targeted a client repo, and the I1 gate refuses that through a
// us-nexus origin at ANY length. The refusal did not say which target
// escalated the classification, so it read as a malfunction.
func TestCapture_TicketsArrayWorksAtLengthOne(t *testing.T) {
b, tr, au := &fakeBrain{}, &fakeTracker{}, &fakeAudit{}
svc := newSvc(b, tr, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
Tickets: []Ticket{{Repo: "hyperguild", Action: "create", Title: "one", Body: "b"}},
})
require.NoError(t, err)
require.Len(t, rec.Tickets, 1)
assert.True(t, rec.Tickets[0].OK)
assert.Len(t, tr.created, 1)
assert.Empty(t, rec.Errors)
}
func TestCapture_TicketsArrayWorksAtLengthMany(t *testing.T) {
b, tr, au := &fakeBrain{}, &fakeTracker{}, &fakeAudit{}
svc := newSvc(b, tr, fakePolicy{}, au)
in := CaptureInput{Context: baseCtx()}
for i := range 20 {
in.Tickets = append(in.Tickets, Ticket{
Repo: "hyperguild", Action: "create", Title: fmt.Sprintf("t%02d", i), Body: "b",
})
}
rec, err := svc.Capture(context.Background(), in)
require.NoError(t, err)
require.Len(t, rec.Tickets, 20, "one receipt entry per ticket")
for i, r := range rec.Tickets {
assert.True(t, r.OK, "ticket %d failed: %+v", i, rec.Errors)
}
assert.Len(t, tr.created, 20)
assert.Empty(t, rec.Errors)
}
func TestCapture_OneFailingTicketDoesNotSinkTheRest(t *testing.T) {
// infra#265 asked explicitly: "if ticket 3 of 5 fails, what happened to
// 1 and 2?" They land, and the receipt says so per item.
b, au := &fakeBrain{}, &fakeAudit{}
tr := &fakeTracker{failOnTitle: "boom"}
svc := newSvc(b, tr, fakePolicy{}, au)
rec, err := svc.Capture(context.Background(), CaptureInput{
Context: baseCtx(),
Tickets: []Ticket{
{Repo: "hyperguild", Action: "create", Title: "ok-1", Body: "b"},
{Repo: "hyperguild", Action: "create", Title: "boom", Body: "b"},
{Repo: "hyperguild", Action: "create", Title: "ok-3", Body: "b"},
},
})
require.NoError(t, err, "a per-item failure is a partial receipt, not a call error")
require.Len(t, rec.Tickets, 3)
assert.True(t, rec.Tickets[0].OK)
assert.False(t, rec.Tickets[1].OK)
assert.True(t, rec.Tickets[2].OK)
require.Len(t, rec.Errors, 1)
assert.Contains(t, rec.Errors[0].Item, "ticket[1]")
}
func TestCapture_I1RefusalNamesWhatEscalatedTheClassification(t *testing.T) {
// The actual defect behind infra#265. Without the offending target in
// the message, a caller with 20 tickets cannot tell which repo tripped
// the gate — and reasonably concludes the array itself is broken.
b, tr, au := &fakeBrain{}, &fakeTracker{}, &fakeAudit{}
pol := fakePolicy{tags: map[string]classification.Level{"trucker-ecommerce": classification.Confidential}}
svc := newSvc(b, tr, pol, au)
ctx := baseCtx()
ctx.Origin = ZoneUSNexus
_, err := svc.Capture(context.Background(), CaptureInput{
Context: ctx,
Tickets: []Ticket{
{Repo: "hyperguild", Action: "create", Title: "fine", Body: "b"},
{Repo: "trucker-ecommerce", Action: "create", Title: "client work", Body: "b"},
},
})
require.ErrorIs(t, err, ErrSovereigntyRefused)
assert.Contains(t, err.Error(), "trucker-ecommerce",
"the refusal must name the target that escalated the classification")
assert.NotContains(t, err.Error(), "hyperguild",
"only the offending target, or the message is noise at 20 tickets")
assert.Empty(t, tr.created, "refused before any write")
}
+4 -15
View File
@@ -55,7 +55,6 @@ type request struct {
Context contextBody `json:"context"`
Insights []insightBody `json:"insights"`
Tickets []ticketBody `json:"tickets"`
Summary *summaryBody `json:"summary,omitempty"`
DryRun bool `json:"dry_run"`
}
@@ -82,12 +81,6 @@ type ticketBody struct {
Body string `json:"body,omitempty"`
}
type summaryBody struct {
Title string `json:"title"`
Body string `json:"body"`
ReposTouched []string `json:"repos_touched,omitempty"`
}
// ServeHTTP authenticates, derives origin, runs the use-case, and maps the
// result to an HTTP status.
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
@@ -111,6 +104,10 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// caller may have tried to put in the body.
in.Context.Principal = principal
in.Context.Origin = h.resolver.Resolve(principal, viaStatic)
in.Context.AuthPath = "jwt"
if viaStatic {
in.Context.AuthPath = "static"
}
rec, err := h.svc.Capture(r.Context(), in)
switch {
@@ -183,11 +180,6 @@ func (b request) toInput() capture.CaptureInput {
Repo: t.Repo, Action: t.Action, Number: t.Number, Title: t.Title, Body: t.Body,
})
}
if b.Summary != nil {
in.Summary = &capture.Summary{
Title: b.Summary.Title, Body: b.Summary.Body, ReposTouched: b.Summary.ReposTouched,
}
}
return in
}
@@ -204,9 +196,6 @@ func statusFor(rec capture.CaptureReceipt) int {
for _, t := range rec.Tickets {
count(&ok, &fail, t.OK)
}
if rec.Summary != nil {
count(&ok, &fail, rec.Summary.OK)
}
switch {
case fail == 0:
return http.StatusOK
@@ -38,9 +38,11 @@ func (f fakeTracker) CreateIssue(context.Context, string, string, string) (captu
}
return capture.IssueRef{Repo: "hyperguild", Number: 1, URL: "https://git/1"}, nil
}
func (fakeTracker) CloseIssue(context.Context, string, int, string) (capture.IssueRef, error) {
return capture.IssueRef{}, nil
}
func (fakeTracker) CommentIssue(context.Context, string, int, string) (capture.IssueRef, error) {
return capture.IssueRef{}, nil
}
@@ -49,7 +51,7 @@ func newHandler(t *testing.T, v capturehttp.Validator, tr capture.IssueTracker,
t.Helper()
cfg, err := classification.Load(t.TempDir())
require.NoError(t, err)
svc := capture.NewService(brainstore.New(t.TempDir()), tr, nil, cfg, audit.NewSlogSink(nil))
svc := capture.NewService(brainstore.New(t.TempDir()), tr, cfg, audit.NewSlogSink(nil))
return capturehttp.New(svc, v, staticTok, "local-cli", capturehttp.NewOriginResolver(sovereign))
}
@@ -183,6 +185,7 @@ type refusingAudit struct{}
func (refusingAudit) Reserve(context.Context, classification.Level) (capture.AuditOutcome, error) {
return 0, errors.New("central audit sink unreachable")
}
func (refusingAudit) Record(context.Context, capture.AuditEntry, capture.AuditOutcome) error {
return nil
}
@@ -190,7 +193,7 @@ func (refusingAudit) Record(context.Context, capture.AuditEntry, capture.AuditOu
func TestAuditUnavailableIs503(t *testing.T) {
cfg, err := classification.Load(t.TempDir())
require.NoError(t, err)
svc := capture.NewService(brainstore.New(t.TempDir()), fakeTracker{}, nil, cfg, refusingAudit{})
svc := capture.NewService(brainstore.New(t.TempDir()), fakeTracker{}, cfg, refusingAudit{})
h := capturehttp.New(svc, nil, staticTok, "local-cli", capturehttp.NewOriginResolver(nil))
rr := do(t, h, "Bearer "+staticTok, internalReq())
@@ -0,0 +1,51 @@
package capturehttp_test
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/audit"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/capturehttp"
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
)
// The REST capture door derives AuthPath itself (hyperguild#145); a body
// field cannot set it, and static vs JWT are told apart.
func TestRESTCaptureStampsAuthPathFromAuthentication(t *testing.T) {
for _, tc := range []struct {
name, authz, wantOrigin, wantBy string
v capturehttp.Validator
sovereign []string
}{
{"static", "Bearer " + staticTok, "origin: static\n", "written_by: local-cli\n", nil, nil},
{"jwt", "Bearer jwt", "origin: jwt\n", "written_by: koala-cli\n", fakeValidator{subject: "koala-cli"}, []string{"koala-cli"}},
} {
t.Run(tc.name, func(t *testing.T) {
brainDir := t.TempDir()
cfg, err := classification.Load(t.TempDir())
require.NoError(t, err)
svc := capture.NewService(brainstore.New(brainDir), fakeTracker{}, cfg, audit.NewSlogSink(nil))
h := capturehttp.New(svc, tc.v, staticTok, "local-cli", capturehttp.NewOriginResolver(tc.sovereign))
rr := do(t, h, tc.authz, map[string]any{
"context": map[string]any{
"harness": "claude-code", "actor": "mathias", "classification": "internal",
"AuthPath": "static", "auth_path": "static", "principal": "local-cli",
},
"insights": []map[string]any{{"text": "rest provenance probe", "wing": "hyperguild", "hall": "facts"}},
})
require.Equal(t, 200, rr.Code, rr.Body.String())
matches, _ := filepath.Glob(filepath.Join(brainDir, "wiki/hyperguild/facts/*.md"))
require.Len(t, matches, 1)
b, err := os.ReadFile(matches[0])
require.NoError(t, err)
assert.Contains(t, string(b), tc.wantOrigin)
assert.Contains(t, string(b), tc.wantBy)
})
}
}
+98
View File
@@ -0,0 +1,98 @@
// Package claim validates the claim-property frontmatter that says when and
// where a note's claim holds, and who decided that (brain#21).
//
// Two families were designed together and land separately. This is the
// applicability envelope:
//
// half_life: volatile | seasonal | durable — how fast it decays
// applies: local | homelab | universal — how widely it holds
// claim_provenance: asserted | inferred — who set the above
//
// The strength family (confidence, contested) follows once the ingest-time
// contradiction detector exists to populate it.
//
// Why the envelope is validated rather than merely accepted: a value outside
// the enum is worse than an absent one. A lint skips what it cannot parse, so
// the note reads as classified while nothing acts on it — the field becomes
// decoration that suppresses the very review it was added to trigger.
//
// Why `applies` and not `scope`: `scope:` has been in use since brain#5 for
// glossary terms (`homelab-core`, `project:<repo>`) — a disjoint value space
// on the same key. The shipped schema keeps the name; this one takes another.
package claim
import (
"fmt"
"strings"
)
// Field names, exported so writers and lints agree on the spelling.
const (
FieldHalfLife = "half_life"
FieldApplies = "applies"
FieldProvenance = "claim_provenance"
)
var (
halfLives = []string{"volatile", "seasonal", "durable"}
appliesTo = []string{"local", "homelab", "universal"}
provenances = []string{"asserted", "inferred"}
)
// ValidateEnvelope checks the envelope fields of one note's frontmatter.
//
// Absent fields are valid: the envelope is opt-in for new writes, and a
// corpus-wide backfill is deliberately out of scope (brain#21) — inferring
// 1113 notes' properties in bulk has the same failure mode as the extraction
// pass that produced 28 wrong wikilinks out of 67.
func ValidateEnvelope(fm map[string]string) error {
halfLife, hasHalfLife := present(fm, FieldHalfLife)
applies, hasApplies := present(fm, FieldApplies)
prov, hasProv := present(fm, FieldProvenance)
if hasHalfLife {
if err := oneOf(FieldHalfLife, halfLife, halfLives); err != nil {
return err
}
}
if hasApplies {
if err := oneOf(FieldApplies, applies, appliesTo); err != nil {
return err
}
}
if hasProv {
if err := oneOf(FieldProvenance, prov, provenances); err != nil {
return err
}
}
// The field and its provenance travel together or not at all. A value
// whose author is unknown cannot be trusted differently from one Mathias
// asserted, which is how an inferred guess hardens into accepted fact.
switch {
case (hasHalfLife || hasApplies) && !hasProv:
return fmt.Errorf("%s is required when %s or %s is set",
FieldProvenance, FieldHalfLife, FieldApplies)
case hasProv && !hasHalfLife && !hasApplies:
return fmt.Errorf("%s is set but neither %s nor %s is",
FieldProvenance, FieldHalfLife, FieldApplies)
}
return nil
}
// present reports a trimmed field value and whether it was set to anything
// non-empty. A key written with an empty value is treated as absent rather
// than as an error: that is a caller emitting a placeholder, not asserting.
func present(fm map[string]string, key string) (string, bool) {
v := strings.TrimSpace(fm[key])
return v, v != ""
}
func oneOf(field, value string, allowed []string) error {
for _, a := range allowed {
if a == value {
return nil
}
}
return fmt.Errorf("%s: %q is not one of %s", field, value, strings.Join(allowed, ", "))
}
+89
View File
@@ -0,0 +1,89 @@
// Package claim_test covers the claim-property envelope (brain#21).
package claim_test
import (
"testing"
"github.com/mathiasbq/hyperguild/ingestion/internal/claim"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestValidateEnvelope(t *testing.T) {
tests := []struct {
name string
fm map[string]string
wantErr string
}{
{
name: "absent is fine — the fields are opt-in on new writes",
fm: map[string]string{"wing": "homelab", "hall": "facts"},
},
{
name: "complete envelope with provenance",
fm: map[string]string{
"half_life": "durable", "applies": "universal", "claim_provenance": "asserted",
},
},
{
name: "inferred provenance is equally valid — it just must be stated",
fm: map[string]string{
"half_life": "volatile", "applies": "local", "claim_provenance": "inferred",
},
},
{
// A value nobody can act on is worse than no value: a lint
// silently skips it, so the note looks classified and isn't.
name: "unknown half_life value",
fm: map[string]string{"half_life": "forever", "claim_provenance": "asserted"},
wantErr: `half_life: "forever" is not one of volatile, seasonal, durable`,
},
{
name: "unknown applies value",
fm: map[string]string{"applies": "galaxy", "claim_provenance": "asserted"},
wantErr: `applies: "galaxy" is not one of local, homelab, universal`,
},
{
// brain#21: a value a model inferred must never be
// indistinguishable from one Mathias asserted. Writing the field
// without saying who set it is exactly that laundering.
name: "envelope field without provenance",
fm: map[string]string{"half_life": "durable"},
wantErr: "claim_provenance is required when half_life or applies is set",
},
{
name: "applies without provenance",
fm: map[string]string{"applies": "homelab"},
wantErr: "claim_provenance is required when half_life or applies is set",
},
{
name: "unknown provenance value",
fm: map[string]string{"half_life": "durable", "claim_provenance": "probably"},
wantErr: `claim_provenance: "probably" is not one of asserted, inferred`,
},
{
// Provenance alone says who set fields that aren't there.
name: "provenance without any envelope field",
fm: map[string]string{"claim_provenance": "asserted"},
wantErr: "claim_provenance is set but neither half_life nor applies is",
},
{
// The glossary wing has used scope: homelab-core since #5. The
// envelope deliberately does NOT claim that key (brain#21).
name: "glossary scope is untouched",
fm: map[string]string{"scope": "homelab-core", "term": "koala"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := claim.ValidateEnvelope(tt.fm)
if tt.wantErr == "" {
assert.NoError(t, err)
return
}
require.Error(t, err)
assert.Contains(t, err.Error(), tt.wantErr)
})
}
}
@@ -16,9 +16,10 @@
package classification
import (
"errors"
"fmt"
"os"
"path/filepath"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"io/fs"
"strings"
"gopkg.in/yaml.v3"
@@ -110,9 +111,20 @@ type rawConfig struct {
func Load(brainDir string) (*Config, error) {
cfg := &Config{wings: map[string]Level{}, repos: map[string]Level{}}
data, err := os.ReadFile(filepath.Join(brainDir, "classification.yaml"))
// Read through the brain root, regular files only (hyperguild#157): a
// symlinked or special file is not trusted as the config, which is a hard
// error like a malformed one.
bfs, err := brainfs.Open(brainDir)
if errors.Is(err, fs.ErrNotExist) {
return cfg, nil
}
if err != nil {
if os.IsNotExist(err) {
return nil, fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
data, err := bfs.ReadRegular("classification.yaml")
if err != nil {
if errors.Is(err, fs.ErrNotExist) {
return cfg, nil
}
return nil, fmt.Errorf("read classification.yaml: %w", err)
@@ -0,0 +1,46 @@
package classification
import (
"os"
"path/filepath"
"syscall"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// hyperguild#157 slice 6: Load read classification.yaml with os.ReadFile. It
// sets the I1 sovereignty levels; a symlinked file must not be trusted as the
// config (a hard error, as for a malformed file), and a FIFO must not hang
// startup.
func TestLoadRefusesASymlinkedOrFIFOClassificationFile(t *testing.T) {
ok := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(ok, "classification.yaml"), []byte("wings:\n w: confidential\n"), 0o644))
_, err := Load(ok)
require.NoError(t, err, "positive control")
linked := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(linked, "real.yaml"), []byte("wings:\n w: public\n"), 0o644))
require.NoError(t, os.Symlink("real.yaml", filepath.Join(linked, "classification.yaml")))
_, err = Load(linked)
assert.Error(t, err, "an in-brain symlink is not trusted as the config")
fifo := t.TempDir()
require.NoError(t, syscall.Mkfifo(filepath.Join(fifo, "classification.yaml"), 0o644))
w, err := os.OpenFile(filepath.Join(fifo, "classification.yaml"), os.O_RDWR, 0)
require.NoError(t, err)
defer func() { _ = w.Close() }()
done := make(chan error, 1)
go func() { _, err := Load(fifo); done <- err }()
select {
case err := <-done:
assert.Error(t, err)
case <-time.After(3 * time.Second):
t.Fatal("Load hung on a FIFO")
}
_, err = Load(t.TempDir())
assert.NoError(t, err, "an absent file is still not an error")
}
@@ -43,6 +43,26 @@ var DefaultRules = []Rule{
// JSON, so both segments begin with "eyJ".
{Name: "jwt", RE: regexp.MustCompile(`eyJ[A-Za-z0-9_\-]{8,}\.eyJ[A-Za-z0-9_\-]{8,}\.[A-Za-z0-9_\-]{8,}`)},
{Name: "postgres-uri-with-password", RE: regexp.MustCompile(`postgres(?:ql)?://[^:\s/]+:[^@\s/]+@`)},
// Any credential embedded in a URL's userinfo, which is what a git credential
// store holds: `https://<user>:<token>@host`. The postgres rule above is the
// same shape narrowed to one scheme, and is kept ahead of this one only so a
// DSN reports the more specific name -- same ordering convention as
// authorization-header before bearer-token.
//
// Added after this shape passed ALL rules and was therefore ingested rather
// than dropped (hyperguild#142). It reached ~/.claude/projects -- this
// watcher's own input -- because a test suite in mathias/infra executed a
// script that configures git while inheriting the operator's real $HOME, read
// koala's live credential store, and printed it. Two credentials were rotated
// (infra#466). The scrubber is the control that decides whether that class of
// mistake is survivable, and for this shape it was failing open.
//
// The {8,} floor on the secret half is what keeps `https://host:443/path` and
// `https://user:@host` out; the `[^:\s/@]+` user half stops a host:port pair
// being read as userinfo. Both directions are covered by
// TestScrub_URLUserinfoCredential, because a rule that dropped every URL with
// a colon in it would discard most of the signal this watcher exists to keep.
{Name: "url-userinfo-credential", RE: regexp.MustCompile(`[a-zA-Z][a-zA-Z0-9+.\-]*://[^:\s/@]+:[^@\s/]{8,}@`)},
{Name: "private-key", RE: regexp.MustCompile(`-----BEGIN[^-]*PRIVATE KEY-----`)},
{Name: "ssh-key", RE: regexp.MustCompile(`ssh-(?:rsa|ed25519|ecdsa)\s+[A-Za-z0-9+/=]{40,}`)},
{Name: "github-pat", RE: regexp.MustCompile(`\b(?:ghp|gho|ghu|ghr|gha)_[A-Za-z0-9]{30,}\b`)},
@@ -46,6 +46,64 @@ func TestScrub_PoisonedFixtures(t *testing.T) {
}
}
// A credential embedded in a URL's userinfo is the shape a git credential store
// holds -- `https://<user>:<token>@host` -- and it walked past every rule until
// hyperguild#142. There was a rule for `postgres://user:pass@` and nothing for the
// general form, so `~/.git-credentials` content was ingested rather than dropped.
//
// Found the hard way: a test suite in mathias/infra executed a script that
// configures git while inheriting the operator's real $HOME, read koala's live
// credential store, and printed two live credentials into a session transcript.
// ~/.claude/projects is this watcher's own input, so the scrubber was the last
// line of defence for that mistake and it failed open. Two credentials were
// rotated (infra#466).
//
// The lesson is not "do not read the credential file" -- that mistake will recur.
// It is that this shape must fail closed, because the scrubber is what decides
// whether such a mistake is survivable.
func TestScrub_URLUserinfoCredential(t *testing.T) {
// Every secret below is INVENTED. None is or was a real credential.
mustScrub := []struct {
name string
content string
}{
{"git-credential-store-user", "https://mathias:0123456789abcdef0123456789abcdef01234567@git.d-ma.be"},
{"git-credential-store-oauth2", "https://oauth2:0123456789abcdef0123456789abcdef01234567@git.d-ma.be"},
{"in-context-payload-line", `payload line: https://mathias:0123456789abcdef0123456789abcdef01234567@git.d-ma.be`},
{"clone-url-in-argv", "git clone https://oauth2:abcdefghijklmnopqrstuvwx@git.d-ma.be/mathias/infra.git"},
{"http-not-https", "http://admin:supersecretvalue123@10.0.1.20/api"},
{"git-plus-ssh-scheme", "git+ssh://deploy:abcdefghijklmnop@example.com/repo"},
{"redis-scheme", "redis://default:abcdefghijklmnopqrst@10.0.1.20:6379/0"},
}
for _, tc := range mustScrub {
t.Run(tc.name, func(t *testing.T) {
assert.NotEmpty(t, Scrub(tc.content),
"a credential in URL userinfo must be dropped, not ingested")
})
}
// The other direction matters as much: a rule that scrubs every URL with a
// colon in it would drop most of the signal this watcher exists to keep.
mustNotScrub := []struct {
name string
content string
}{
{"host-with-port", "cloning https://git.d-ma.be:443/mathias/infra.git"},
{"postgres-no-password", "postgres://localhost:5432/brain"},
{"url-with-colon-in-query", "GET https://example.com/path?filter=a:b&x=1"},
{"plain-url", "see https://git.d-ma.be/mathias/infra/issues/466"},
{"userinfo-no-secret", "https://user:@example.com"},
{"format-string-placeholder", `printf 'https://oauth2:%s@%s\n' "$TOKEN" "$HOST"`},
{"scheme-relative-time", "duration https://example.com took 00:03:22"},
}
for _, tc := range mustNotScrub {
t.Run(tc.name, func(t *testing.T) {
assert.Empty(t, Scrub(tc.content),
"this is not a credential and must pass through")
})
}
}
func TestScrub_CleanContentPassesThrough(t *testing.T) {
cases := []string{
"",
@@ -0,0 +1,231 @@
package claudewatcher
import (
"context"
"log/slog"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// recordingHandler captures log records so a test can assert that a file was
// never even attempted — the absence of a read error is the evidence.
type recordingHandler struct {
mu sync.Mutex
records []string
}
func (h *recordingHandler) Enabled(context.Context, slog.Level) bool { return true }
func (h *recordingHandler) Handle(_ context.Context, r slog.Record) error {
h.mu.Lock()
defer h.mu.Unlock()
var sb strings.Builder
sb.WriteString(r.Message)
r.Attrs(func(a slog.Attr) bool {
sb.WriteString(" " + a.Key + "=" + a.Value.String())
return true
})
h.records = append(h.records, sb.String())
return nil
}
func (h *recordingHandler) WithAttrs([]slog.Attr) slog.Handler { return h }
func (h *recordingHandler) WithGroup(string) slog.Handler { return h }
func (h *recordingHandler) contains(substr string) bool {
h.mu.Lock()
defer h.mu.Unlock()
for _, r := range h.records {
if strings.Contains(r, substr) {
return true
}
}
return false
}
func session(t *testing.T, root, project, id string) string {
t.Helper()
dir := filepath.Join(root, project)
require.NoError(t, os.MkdirAll(dir, 0o755))
return writeSession(t, dir, id, []string{
`{"type":"user","sessionId":"` + id + `","message":"a prompt"}`,
`{"type":"assistant","sessionId":"` + id + `","message":{"content":[{"type":"text","text":"an answer"}]}}`,
})
}
// A named project is not ingested, and its neighbours still are.
func TestSkipProjects_skipsTheNamedProjectOnly(t *testing.T) {
tmp := t.TempDir()
session(t, tmp, "-home-mathias-dev-AGENTS-cobalt-dingo", "skipped")
session(t, tmp, "-home-mathias-dev", "kept")
sink := &memSink{}
require.NoError(t, TickOnce(context.Background(), Config{
SessionsDir: tmp,
Host: "koala",
Sink: sink,
SkipProjects: []string{"-home-mathias-dev-AGENTS-cobalt-dingo"},
}))
require.Len(t, sink.batches, 1)
assert.Equal(t, "kept", sink.batches[0].SessionID)
assert.Equal(t, "-home-mathias-dev", sink.batches[0].ProjectID)
}
// Globs are necessary, not a convenience. A project directory is derived from
// the session's WORKING DIRECTORY, so a git worktree produces an ephemeral
// name like "-home-mathias-dev--claude-worktrees-bridge-cse-017xoC...". An
// exact-match list could never keep up with those.
func TestSkipProjects_globMatchesEphemeralWorktreeNames(t *testing.T) {
tmp := t.TempDir()
session(t, tmp, "-home-mathias-dev--claude-worktrees-bridge-cse-017xoCEcrc35", "worktree")
session(t, tmp, "-home-mathias-dev", "kept")
sink := &memSink{}
require.NoError(t, TickOnce(context.Background(), Config{
SessionsDir: tmp,
Host: "koala",
Sink: sink,
SkipProjects: []string{"*claude-worktrees-bridge-cse*"},
}))
require.Len(t, sink.batches, 1)
assert.Equal(t, "kept", sink.batches[0].SessionID)
}
// The point of skipping at the directory walk: the file is never OPENED, so
// its content never enters this process. Proven by making it unreadable — a
// skipped file produces no read error because nothing tried to read it, while
// the same file in a non-skipped project does.
func TestSkipProjects_theFileIsNeverOpened(t *testing.T) {
tmp := t.TempDir()
path := session(t, tmp, "-home-mathias-dev-AGENTS-cobalt-dingo", "unreadable")
require.NoError(t, os.Chmod(path, 0o000))
t.Cleanup(func() { _ = os.Chmod(path, 0o644) })
rec := &recordingHandler{}
sink := &memSink{}
require.NoError(t, TickOnce(context.Background(), Config{
SessionsDir: tmp,
Host: "koala",
Sink: sink,
Logger: slog.New(rec),
SkipProjects: []string{"*cobalt-dingo*"},
}))
assert.Empty(t, sink.batches)
assert.False(t, rec.contains("file failed"),
"a skipped file must not even be attempted")
}
// The control test for the one above: without the skip, the unreadable file IS
// attempted and fails. Otherwise the assertion above could pass for the wrong
// reason.
func TestSkipProjects_withoutTheSkipTheSameFileIsAttempted(t *testing.T) {
tmp := t.TempDir()
path := session(t, tmp, "-home-mathias-dev-AGENTS-cobalt-dingo", "unreadable")
require.NoError(t, os.Chmod(path, 0o000))
t.Cleanup(func() { _ = os.Chmod(path, 0o644) })
rec := &recordingHandler{}
require.NoError(t, TickOnce(context.Background(), Config{
SessionsDir: tmp,
Host: "koala",
Sink: &memSink{},
Logger: slog.New(rec),
}))
assert.True(t, rec.contains("file failed"),
"the unreadable file must fail when not skipped — otherwise the skip test proves nothing")
}
// Patterns match the project directory NAME, not the whole path. A temp
// directory or a home directory component must not accidentally match and
// silently disable ingestion.
func TestSkipProjects_matchesTheProjectNameNotTheFullPath(t *testing.T) {
tmp := t.TempDir()
session(t, tmp, "-home-mathias-dev", "kept")
sink := &memSink{}
require.NoError(t, TickOnce(context.Background(), Config{
SessionsDir: tmp,
Host: "koala",
Sink: sink,
// tmp is somewhere under /tmp; this must not match the project name.
SkipProjects: []string{"*tmp*"},
}))
require.Len(t, sink.batches, 1, "a path component outside the project name must not match")
}
// An unparseable pattern must stop the watcher rather than be ignored. A
// silently dropped skip rule means ingesting what was meant to be excluded,
// which is the failure that cannot be noticed after the fact.
func TestSkipProjects_invalidPatternRefusesToRun(t *testing.T) {
tmp := t.TempDir()
session(t, tmp, "-home-mathias-dev", "kept")
sink := &memSink{}
err := TickOnce(context.Background(), Config{
SessionsDir: tmp,
Host: "koala",
Sink: sink,
SkipProjects: []string{"[unclosed"},
})
require.Error(t, err)
assert.Contains(t, err.Error(), "[unclosed")
assert.Empty(t, sink.batches, "nothing may be ingested while a skip rule is broken")
}
func TestSkipProjects_emptyListChangesNothing(t *testing.T) {
tmp := t.TempDir()
session(t, tmp, "-home-mathias-dev", "kept")
sink := &memSink{}
require.NoError(t, TickOnce(context.Background(), Config{
SessionsDir: tmp, Host: "koala", Sink: sink, SkipProjects: nil,
}))
require.Len(t, sink.batches, 1)
}
// The difference between SkipDir and filtering after the walk, made
// observable: an unreadable skipped directory must not disturb anything.
//
// Without SkipDir, WalkDir reads the directory's entries, gets a permission
// error, and that error aborts the ENTIRE walk — so one skipped project with
// awkward permissions would silently stop ingesting every other project.
// Returning SkipDir means its contents are never listed at all.
//
// Added after a mutation test showed the earlier assertions could not tell the
// two implementations apart.
func TestSkipProjects_anUnreadableSkippedDirDoesNotBreakTheWalk(t *testing.T) {
tmp := t.TempDir()
session(t, tmp, "-home-mathias-dev", "kept")
blocked := filepath.Join(tmp, "-home-mathias-dev-AGENTS-cobalt-dingo")
require.NoError(t, os.MkdirAll(blocked, 0o755))
_ = writeSession(t, blocked, "secret", []string{
`{"type":"user","sessionId":"secret","message":"should never be read"}`,
})
require.NoError(t, os.Chmod(blocked, 0o000))
t.Cleanup(func() { _ = os.Chmod(blocked, 0o755) })
sink := &memSink{}
require.NoError(t, TickOnce(context.Background(), Config{
SessionsDir: tmp,
Host: "koala",
Sink: sink,
SkipProjects: []string{"*cobalt-dingo*"},
}))
require.Len(t, sink.batches, 1, "an unreadable skipped directory must not abort the walk")
assert.Equal(t, "kept", sink.batches[0].SessionID)
}
+59 -2
View File
@@ -45,6 +45,25 @@ type Config struct {
Sink Sink
Cursors *CursorStore
Logger *slog.Logger
// SkipProjects excludes whole project directories from ingestion, as
// filepath.Match globs tested against the project directory NAME (the
// basename, e.g. "-home-mathias-dev"), not the full path.
//
// Globs rather than exact names because a project directory is derived
// from the session's working directory, so a git worktree yields an
// ephemeral name like
// "-home-mathias-dev--claude-worktrees-bridge-cse-017xoC..." that no
// exact-match list could keep up with.
//
// Matching directories are skipped during the directory WALK, so their
// files are never opened and their content never enters this process —
// which is a stronger property than filtering it out afterwards.
//
// An unparseable pattern stops the watcher. A silently dropped skip rule
// means ingesting what was meant to be excluded, and that is not
// noticeable after the fact.
SkipProjects []string
}
// Watch runs the polling loop until ctx is cancelled. Returns ctx.Err()
@@ -68,6 +87,9 @@ func Watch(ctx context.Context, cfg Config) error {
if cfg.Logger == nil {
cfg.Logger = slog.Default()
}
if err := validateSkipPatterns(cfg.SkipProjects); err != nil {
return err
}
cfg.Logger.Info("claudewatcher: started",
"sessions_dir", cfg.SessionsDir,
"host", cfg.Host,
@@ -91,7 +113,7 @@ func Watch(ctx context.Context, cfg Config) error {
// runTick is one polling pass. Exposed (lowercase) for tests via
// TickOnce.
func runTick(ctx context.Context, cfg Config) {
files, err := listSessionFiles(cfg.SessionsDir)
files, err := listSessionFiles(cfg.SessionsDir, cfg.SkipProjects, cfg.Logger)
if err != nil {
cfg.Logger.Warn("claudewatcher: list session files", "err", err)
return
@@ -119,17 +141,52 @@ func TickOnce(ctx context.Context, cfg Config) error {
if cfg.Logger == nil {
cfg.Logger = slog.Default()
}
if err := validateSkipPatterns(cfg.SkipProjects); err != nil {
return err
}
runTick(ctx, cfg)
return nil
}
func listSessionFiles(root string) ([]string, error) {
// validateSkipPatterns rejects a pattern filepath.Match cannot parse.
//
// Checked up front rather than per file: a bad pattern that only failed at
// match time would be reported once per session file per tick, and — worse —
// filepath.Match returns an error alongside a false match, so the directory
// would be ingested while the log complained.
func validateSkipPatterns(patterns []string) error {
for _, p := range patterns {
if _, err := filepath.Match(p, "probe"); err != nil {
return fmt.Errorf("invalid skip-project pattern %q: %w", p, err)
}
}
return nil
}
// skipProject reports whether a project directory name matches any pattern.
func skipProject(name string, patterns []string) bool {
for _, p := range patterns {
// The error case is already excluded by validateSkipPatterns.
if ok, _ := filepath.Match(p, name); ok {
return true
}
}
return false
}
func listSessionFiles(root string, skip []string, log *slog.Logger) ([]string, error) {
var out []string
err := filepath.WalkDir(root, func(path string, d os.DirEntry, walkErr error) error {
if walkErr != nil {
return walkErr
}
if d.IsDir() {
// SkipDir, so nothing inside is even listed. The files are never
// opened and their content never reaches this process.
if path != root && skipProject(d.Name(), skip) {
log.Info("claudewatcher: project skipped by configuration", "project", d.Name())
return filepath.SkipDir
}
return nil
}
if !strings.HasSuffix(path, ".jsonl") {
@@ -0,0 +1,129 @@
package extract
import (
"go/ast"
"go/parser"
"go/token"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
)
const extractPkg = "github.com/mathiasbq/hyperguild/ingestion/internal/extract"
// parseNonTest parses every non-test Go file under the module.
func parseNonTest(t *testing.T) map[string]*ast.File {
t.Helper()
fset := token.NewFileSet()
out := map[string]*ast.File{}
err := filepath.WalkDir(filepath.Join("..", ".."), func(p string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() && (d.Name() == "vendor" || d.Name() == ".git") {
return filepath.SkipDir
}
if d.IsDir() || !strings.HasSuffix(p, ".go") || strings.HasSuffix(p, "_test.go") {
return nil
}
f, perr := parser.ParseFile(fset, p, nil, 0)
if perr != nil {
return perr
}
out[filepath.ToSlash(p)] = f
return nil
})
if err != nil {
t.Fatal(err)
}
return out
}
// PR #156 review: reading a caller-chosen path by name reopens it after the
// containment check (hyperguild#153 item 3). Outside this package, nothing
// may reference extract.Text — called OR taken as a value — under whatever
// name the package is imported (round 2: an alias and a func value slipped
// past a plain text search).
func TestNoPackageReferencesExtractTextOutsideExtract(t *testing.T) {
for path, f := range parseNonTest(t) {
if strings.Contains(path, "internal/extract/") {
continue
}
names := map[string]bool{}
for _, imp := range f.Imports {
if p, _ := strconv.Unquote(imp.Path.Value); p == extractPkg {
name := "extract"
if imp.Name != nil {
name = imp.Name.Name
}
if name == "." {
t.Errorf("%s dot-imports extract; import it by name so its use stays checkable", path)
}
names[name] = true
}
}
if len(names) == 0 {
continue
}
ast.Inspect(f, func(n ast.Node) bool {
if sel, ok := n.(*ast.SelectorExpr); ok {
if id, ok := sel.X.(*ast.Ident); ok && names[id.Name] && sel.Sel.Name == "Text" {
t.Errorf("%s references %s.Text; use extract.TextIn through an os.Root", path, id.Name)
}
}
return true
})
}
}
// PR #156 review round 2: the FIFO fix (open non-blocking, check the handle)
// lives in OpenRegular, and reverting any caller to a plain Root.Open stayed
// green. In the three raw/ reader FILES parsed below (and only those: a
// wrapper in another file is not seen), no Open/OpenFile/ReadFile selector
// may appear outside OpenRegular, called or taken as a method value. This is
// a second layer; the behavioural race tests in internal/api and
// internal/watcher are what catch a rewrite through any API (round 3).
// osOpenAllowed names the functions in guarded files that may call os.Open*
// on a path that is not a raw/ read.
var osOpenAllowed = map[string]bool{
// Text reads by path; outside this package only TextIn's temp copy reaches
// it (TestNoPackageReferencesExtractTextOutsideExtract enforces that).
"internal/extract/extract.go:Text": true,
}
func TestRawReaderFilesUseNoOpenOrReadFileOutsideOpenRegular(t *testing.T) {
guarded := []string{"internal/api/rawroot.go", "internal/extract/extract.go", "internal/watcher/watcher.go"}
files := parseNonTest(t)
for _, g := range guarded {
var f *ast.File
for p, af := range files {
if strings.HasSuffix(p, g) {
f = af
}
}
if f == nil {
t.Fatalf("guarded file %s not found", g)
}
for _, decl := range f.Decls {
fn, ok := decl.(*ast.FuncDecl)
if !ok || fn.Name.Name == "OpenRegular" {
continue
}
ast.Inspect(fn, func(n ast.Node) bool {
sel, ok := n.(*ast.SelectorExpr)
if !ok {
return true
}
if sel.Sel.Name == "Open" || sel.Sel.Name == "OpenFile" || sel.Sel.Name == "ReadFile" {
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "os" && osOpenAllowed[g+":"+fn.Name.Name] {
return true // a named non-raw write (the ingest log append)
}
t.Errorf("%s: %s calls .%s; raw/ reads must go through OpenRegular", g, fn.Name.Name, sel.Sel.Name)
}
return true
})
}
}
}
+69
View File
@@ -3,8 +3,10 @@ package extract
import (
"fmt"
"io"
"os"
"strings"
"syscall"
)
// Text reads the file at path and returns its plain-text content.
@@ -41,3 +43,70 @@ func fileExt(path string) string {
}
return ""
}
// TextIn is Text for a file read through an os.Root (hyperguild#153): the
// read cannot leave the root, so a path checked earlier cannot be swapped
// for a symlink pointing elsewhere before it is read. Formats that need an
// external tool (pdf, docmark) are copied to a temp file from the handle
// opened through the root, then extracted from the copy.
func TextIn(r *os.Root, rel string) (string, error) {
fi, err := r.Lstat(rel)
if err != nil {
return "", fmt.Errorf("stat %s: %w", rel, err)
}
if !fi.Mode().IsRegular() {
return "", fmt.Errorf("%s: not a regular file", rel)
}
f, _, err := OpenRegular(r, rel)
if err != nil {
return "", fmt.Errorf("open %s: %w", rel, err)
}
defer func() { _ = f.Close() }()
ext := strings.ToLower(fileExt(rel))
switch ext {
case ".md", ".txt":
b, err := io.ReadAll(f)
if err != nil {
return "", fmt.Errorf("read %s: %w", rel, err)
}
return string(b), nil
case ".pdf", ".docx", ".xlsx", ".pptx", ".png", ".jpg", ".jpeg":
tmp, err := os.CreateTemp("", "ingest-*"+ext)
if err != nil {
return "", fmt.Errorf("temp copy: %w", err)
}
defer func() { _ = os.Remove(tmp.Name()) }()
if _, err := io.Copy(tmp, f); err != nil {
_ = tmp.Close()
return "", fmt.Errorf("copy %s: %w", rel, err)
}
if err := tmp.Close(); err != nil {
return "", fmt.Errorf("copy %s: %w", rel, err)
}
return Text(tmp.Name())
default:
return "", fmt.Errorf("unsupported file extension: %s", ext)
}
}
// OpenRegular opens rel inside r and returns it only if the OPENED handle is
// a regular file. The open is non-blocking, so a FIFO swapped in after an
// Lstat (PR #156 review: it hung open(2) forever) is refused promptly rather
// than stalling the reader. Callers keep their Lstat check, which refuses
// symlinks outright; this closes the race between that check and the open.
func OpenRegular(r *os.Root, rel string) (*os.File, os.FileInfo, error) {
f, err := r.OpenFile(rel, os.O_RDONLY|syscall.O_NONBLOCK, 0)
if err != nil {
return nil, nil, err
}
fi, err := f.Stat()
if err != nil {
_ = f.Close()
return nil, nil, err
}
if !fi.Mode().IsRegular() {
_ = f.Close()
return nil, nil, fmt.Errorf("%s: not a regular file", rel)
}
return f, fi, nil
}
+64
View File
@@ -0,0 +1,64 @@
package extract
import (
"os"
"path/filepath"
"syscall"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// hyperguild#153 item 3: ingest checked a path, then extract.Text reopened it
// by name, so a file swapped for a symlink in between was followed out of
// raw/. TextIn reads through an os.Root, which refuses to leave the root.
func TestTextInReadsInsideTheRootAndRefusesEscapes(t *testing.T) {
root := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(root, "ok.md"), []byte("inside"), 0o644))
outside := filepath.Join(t.TempDir(), "secret.md")
require.NoError(t, os.WriteFile(outside, []byte("SENTINEL-153-TEXTIN"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(root, "leak.md")))
r, err := os.OpenRoot(root)
require.NoError(t, err)
defer func() { _ = r.Close() }()
got, err := TextIn(r, "ok.md")
require.NoError(t, err)
assert.Equal(t, "inside", got)
got, err = TextIn(r, "leak.md")
assert.Error(t, err)
assert.NotContains(t, got, "SENTINEL-153-TEXTIN")
_, err = TextIn(r, "../"+filepath.Base(outside))
assert.Error(t, err)
}
// PR #156 review: a regular file swapped for a FIFO between Lstat and Open
// blocked open(2) forever. OpenRegular opens non-blocking and checks the type
// on the opened handle, so a FIFO is refused promptly instead of hanging.
func TestOpenRegularRefusesAFIFOWithoutBlocking(t *testing.T) {
dir := t.TempDir()
require.NoError(t, syscall.Mkfifo(filepath.Join(dir, "pipe.md"), 0o644))
r, err := os.OpenRoot(dir)
require.NoError(t, err)
defer func() { _ = r.Close() }()
done := make(chan error, 1)
go func() {
f, _, err := OpenRegular(r, "pipe.md")
if f != nil {
_ = f.Close()
}
done <- err
}()
select {
case err := <-done:
assert.Error(t, err)
case <-time.After(2 * time.Second):
t.Fatal("OpenRegular blocked on a FIFO")
}
}
@@ -0,0 +1,167 @@
// Package frontmatter lints and repairs the YAML frontmatter block of a
// markdown note.
//
// Notes written directly — hand-authored, or promoted from brain/raw/ — do not
// pass through pipeline.BuildPages, so nothing quotes their scalars. A title
// containing an unescaped ": " then produces a block that yaml.Unmarshal
// rejects with "mapping values are not allowed here", and every consumer that
// reads frontmatter silently skips the note.
//
// Fix is deliberately conservative: it only adds quoting, never reorders or
// rewrites, and it refuses to return a document that still does not parse.
package frontmatter
import (
"bytes"
"fmt"
"regexp"
"strings"
"gopkg.in/yaml.v3"
)
const fence = "---"
// Lint reports whether doc's frontmatter block parses as YAML.
//
// A document with no frontmatter block, or with an unterminated one, is not an
// error: there is nothing claiming to be YAML.
func Lint(doc []byte) error {
b, ok := block(doc)
if !ok {
return nil
}
var v any
if err := yaml.Unmarshal(b, &v); err != nil {
return fmt.Errorf("frontmatter: %w", err)
}
return nil
}
// Fix quotes unquoted scalars whose value breaks YAML parsing and returns the
// repaired document.
//
// It returns (doc, false, nil) when the document already parses, and
// (doc, false, err) when quoting cannot rescue it — an unfixable document is
// always returned byte-identical, never half-repaired.
func Fix(doc []byte) ([]byte, bool, error) {
if err := Lint(doc); err == nil {
return doc, false, nil
}
b, ok := block(doc)
if !ok {
// Lint only fails when there is a block, so this is unreachable in
// practice; return the original rather than guess.
return doc, false, nil
}
fixed := quoteScalars(b)
if bytes.Equal(fixed, b) {
return doc, false, fmt.Errorf("frontmatter: not repairable by quoting: %w", Lint(doc))
}
var v any
if err := yaml.Unmarshal(fixed, &v); err != nil {
return doc, false, fmt.Errorf("frontmatter: quoting did not repair the block: %w", err)
}
_, rest, _ := Split(doc)
out := []byte(fence + "\n")
out = append(out, fixed...)
out = append(out, []byte(fence)...)
out = append(out, rest...)
return out, true, nil
}
// mappingLine matches "key: value" at any indentation, capturing the prefix so
// it can be re-emitted untouched.
//
// Sequence items ("- value") are deliberately not handled: "- foo: bar" is
// valid YAML, so a document containing one does not reach Fix in the first
// place. See the Lint table for why that gap is left open.
var mappingLine = regexp.MustCompile(`^(\s*[A-Za-z_][\w.-]*:[ \t]+)(\S.*)$`)
// quoteScalars single-quotes every unquoted scalar value in the block that
// contains a colon. Values that are already quoted, or that carry YAML syntax
// this package has no business rewriting, are left alone.
func quoteScalars(block []byte) []byte {
lines := strings.Split(string(block), "\n")
for i, line := range lines {
m := mappingLine.FindStringSubmatch(line)
if m == nil {
continue
}
prefix, value := m[1], m[2]
if !needsQuoting(value) {
continue
}
lines[i] = prefix + "'" + strings.ReplaceAll(value, "'", "''") + "'"
}
return []byte(strings.Join(lines, "\n"))
}
// needsQuoting reports whether a scalar value has to be quoted to load as the
// string it plainly is.
//
// The test is YAML's own parser rather than a pattern: ask whether the value
// stands alone as the right-hand side of a mapping. Heuristics get this wrong
// in both directions — `title: "Foo" — bar` opens with a quote but is not
// quoted, and a value may contain a colon inside quotes and be perfectly fine.
//
// Anything that already parses is left alone, whatever it parses into — a flow
// collection, a number, a bool. It is not broken, and quoting would change its
// type.
func needsQuoting(value string) bool {
if strings.HasPrefix(value, "|") || strings.HasPrefix(value, ">") {
return false // block scalar; its content is not parsed as YAML
}
var probe map[string]any
return yaml.Unmarshal([]byte("v: "+value+"\n"), &probe) != nil
}
// block returns the raw YAML between the opening and closing fences.
func block(doc []byte) ([]byte, bool) {
b, _, ok := Split(doc)
return b, ok
}
// Split returns doc's frontmatter block and everything from the closing
// fence onward (the closing fence itself excluded, its trailing newline
// included), or ok=false if doc has no properly-fenced frontmatter.
//
// The canonical fence detector for this codebase (hyperguild#138): three
// independent reimplementations of this same logic each grew their own gap
// through separate bug-fixing before being consolidated here —
// internal/mcp's needed a CRLF/BOM guard added after both silently defeated
// its own HasPrefix check, and internal/api's closing-fence match had no
// trailing-newline requirement, so a document ending in "\n---extra" with no
// real newline after the fence was treated as validly closed. Both classes
// are closed here, once, for every caller.
func Split(doc []byte) (block, rest []byte, ok bool) {
s := strings.TrimPrefix(string(doc), "\xef\xbb\xbf")
s = strings.ReplaceAll(s, "\r\n", "\n")
if !strings.HasPrefix(s, fence+"\n") {
return nil, nil, false
}
body := s[len(fence)+1:]
idx := strings.Index(body, "\n"+fence)
if idx < 0 {
return nil, nil, false
}
// What follows the closing fence must be either a newline (a real body
// follows) or nothing at all (the fence is the literal end of the
// document — no body, not a bypass attempt). Anything else, e.g.
// "\n---extra", is not a valid close: an earlier version of this check
// accepted it, which is exactly the looser match hyperguild#138 closes.
after := idx + 1 + len(fence)
switch {
case after == len(body):
return []byte(body[:idx+1]), nil, true
case body[after] == '\n':
return []byte(body[:idx+1]), []byte(body[after:]), true
default:
return nil, nil, false
}
}
@@ -0,0 +1,240 @@
// ingestion/internal/frontmatter/frontmatter_test.go
package frontmatter
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gopkg.in/yaml.v3"
)
func TestLint(t *testing.T) {
tests := []struct {
name string
doc string
wantErr bool
}{
{
name: "no frontmatter block is not an error",
doc: "# Just a heading\n\nSome body text.\n",
},
{
name: "quoted title parses",
doc: "---\ntitle: 'Headless dispatch runs: Bash is denied'\n---\n\nbody\n",
},
{
name: "plain title parses",
doc: "---\ntitle: A simple title\nwing: homelab\n---\n\nbody\n",
},
{
name: "unquoted colon in title does not parse",
doc: "---\ntitle: Headless dispatch runs: Bash is denied — use gitea MCP\nwing: homelab\n---\n\nbody\n",
wantErr: true,
},
{
// Known gap, deliberately not fixed here: this parses, but the
// item becomes map[string]any{"Headless dispatch": "denied"}
// rather than a string. It is a silent type corruption, not a
// parse failure, so repairing it would rewrite documents that
// currently load — out of scope for a lint about broken YAML.
name: "unquoted colon in a list item parses as a mapping, not an error",
doc: "---\ntitle: Fine\naliases:\n - Headless dispatch: denied\n---\n\nbody\n",
},
{
name: "unterminated frontmatter is not treated as frontmatter",
doc: "---\ntitle: no closing fence\n\nbody\n",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := Lint([]byte(tt.doc))
if tt.wantErr {
require.Error(t, err)
return
}
require.NoError(t, err)
})
}
}
func TestFix(t *testing.T) {
tests := []struct {
name string
doc string
wantChanged bool
wantKey string
wantValue string
}{
{
name: "quotes an unquoted title containing a colon",
doc: "---\ntitle: Headless dispatch runs: Bash is denied — use gitea MCP\nwing: homelab\n---\n\nbody\n",
wantChanged: true,
wantKey: "title",
wantValue: "Headless dispatch runs: Bash is denied — use gitea MCP",
},
{
// Parses already (see the Lint table), so Fix must leave it alone
// rather than rewrite a document that loads.
name: "leaves a colon-bearing list item alone because it already parses",
doc: "---\ntitle: Fine\naliases:\n - Headless dispatch: denied\n---\n\nbody\n",
wantChanged: false,
wantKey: "title",
wantValue: "Fine",
},
{
name: "escapes an embedded single quote when quoting",
doc: "---\ntitle: Don't do this: it breaks\n---\n\nbody\n",
wantChanged: true,
wantKey: "title",
wantValue: "Don't do this: it breaks",
},
{
// The value opens with a double quote and then keeps going after
// the closing one, so YAML reads it as a quoted scalar followed by
// junk. Treating "starts with a quote" as "already quoted" leaves
// this broken forever.
name: "quotes a value that only looks quoted",
doc: "---\ntitle: \"Evaluation Driven Generation\" — a TDD-style scaffold\ntags: [a, b]\n---\n\nbody\n",
wantChanged: true,
wantKey: "title",
wantValue: "\"Evaluation Driven Generation\" — a TDD-style scaffold",
},
{
name: "leaves an already-valid document untouched",
doc: "---\ntitle: 'Already quoted: fine'\nwing: homelab\n---\n\nbody\n",
wantChanged: false,
wantKey: "title",
wantValue: "Already quoted: fine",
},
{
name: "leaves a document with no frontmatter untouched",
doc: "# Heading\n\nbody: with a colon\n",
wantChanged: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, changed, err := Fix([]byte(tt.doc))
require.NoError(t, err)
assert.Equal(t, tt.wantChanged, changed)
if !tt.wantChanged {
assert.Equal(t, tt.doc, string(got), "unchanged docs must be returned byte-identical")
}
// Whatever came back must parse — that is the whole point.
require.NoError(t, Lint(got), "Fix must not return a document that still fails to parse")
if tt.wantKey == "" {
return
}
block, ok := block(got)
require.True(t, ok)
var m map[string]any
require.NoError(t, yaml.Unmarshal(block, &m))
assert.Equal(t, tt.wantValue, m[tt.wantKey], "the value must survive quoting unchanged")
})
}
}
func TestFix_LeavesStructuredValuesStructured(t *testing.T) {
// Only the broken scalar may be touched. A flow sequence on a neighbouring
// line still has to load as a sequence afterwards, not as a string.
doc := "---\ntitle: Broken: here\ntags: [prompting, llm]\ncount: 3\n---\n\nbody\n"
got, changed, err := Fix([]byte(doc))
require.NoError(t, err)
require.True(t, changed)
b, ok := block(got)
require.True(t, ok)
var m map[string]any
require.NoError(t, yaml.Unmarshal(b, &m))
assert.Equal(t, "Broken: here", m["title"])
assert.Equal(t, []any{"prompting", "llm"}, m["tags"])
assert.Equal(t, 3, m["count"])
}
func TestFix_PreservesBodyExactly(t *testing.T) {
body := "\nThe body has: colons, 'quotes', and\n---\na horizontal rule.\n"
doc := "---\ntitle: Broken: title\n---" + body
got, changed, err := Fix([]byte(doc))
require.NoError(t, err)
require.True(t, changed)
assert.True(t, len(got) > 0)
_, rest, ok := Split([]byte(doc))
require.True(t, ok)
_, gotRest, ok := Split(got)
require.True(t, ok)
assert.Equal(t, string(rest), string(gotRest), "the body must not be touched")
}
func TestFix_RefusesToReturnAnUnparseableResult(t *testing.T) {
// A frontmatter block that quoting cannot rescue: the structure itself is
// broken, not a scalar. Fix must return the original and an error rather
// than a half-repaired document.
doc := "---\ntitle: fine\n\tbad: tab-indented mapping\n - stray list item\n---\n\nbody\n"
got, changed, err := Fix([]byte(doc))
require.Error(t, err)
assert.False(t, changed)
assert.Equal(t, doc, string(got), "an unfixable document must come back byte-identical")
}
// TestSplit_NormalizesCRLFBeforeFenceDetection is hyperguild#138: a note
// whose fence lines use CRLF ("---\r\n") failed strings.HasPrefix(doc,
// "---\n") on the precondition read in internal/mcp's independent
// reimplementation -- encoding alone, no adversarial input, silently
// defeated fence detection there before this consolidation. Split must
// never repeat that.
func TestSplit_NormalizesCRLFBeforeFenceDetection(t *testing.T) {
doc := "---\r\ntitle: 'n'\r\nhalf_life: durable\r\n---\r\n\r\nbody\r\n"
block, rest, ok := Split([]byte(doc))
require.True(t, ok)
assert.Contains(t, string(block), "half_life: durable")
assert.Contains(t, string(rest), "body")
}
// TestSplit_StripsALeadingUTF8BOM is the same mechanism one layer earlier:
// a leading BOM (Notepad's default "UTF-8" save, PowerShell's pre-v6
// Out-File/Set-Content) defeats the same HasPrefix check before CRLF
// normalization even applies.
func TestSplit_StripsALeadingUTF8BOM(t *testing.T) {
doc := "\xef\xbb\xbf---\ntitle: 'n'\n---\n\nbody\n"
block, rest, ok := Split([]byte(doc))
require.True(t, ok)
assert.Contains(t, string(block), "title: 'n'")
assert.Contains(t, string(rest), "body")
}
// TestSplit_RequiresANewlineAfterTheClosingFence closes the looser match
// internal/api's independent reimplementation carried (strings.Index(body,
// "\n"+fence) with no trailing-newline requirement): a document ending in
// "\n---extra" with no newline separating the fence from what follows it is
// not a valid closing fence and must not be treated as one.
func TestSplit_RequiresANewlineAfterTheClosingFence(t *testing.T) {
doc := "---\ntitle: 'n'\n---extra-not-a-real-close"
_, _, ok := Split([]byte(doc))
assert.False(t, ok, "a fence line not followed by a newline is not a valid closing fence")
}
// TestSplit_AcceptsAClosingFenceThatIsTheLastBytesOfTheDocument is a
// regression pin found in review of hyperguild#138: the fix for the
// "\n---extra" bypass above must not also reject the legitimate case of a
// closing fence with nothing after it at all (no trailing newline because
// there's no body, not because a bypass is being attempted) -- this
// package's own private split() accepted that shape before the
// consolidation, and cmd/frontmatterlint's Lint/Fix relied on it.
func TestSplit_AcceptsAClosingFenceThatIsTheLastBytesOfTheDocument(t *testing.T) {
doc := "---\ntitle: 'n'\n---"
block, rest, ok := Split([]byte(doc))
require.True(t, ok, "a closing fence at literal end-of-file, with nothing after it, is valid -- there is no body to require a newline before")
assert.Contains(t, string(block), "title: 'n'")
assert.Equal(t, "", string(rest))
}

Some files were not shown because too many files have changed in this diff Show More