Compare commits

..
Author SHA1 Message Date
mathiasandClaude Sonnet 5 442557ca31 docs(mcp): document checkClaimEnvelope's accepted v1 gap
CI / Lint / Test / Vet (pull_request) Successful in 33s
CI / Mirror to GitHub (pull_request) Skipped
Round 7's review found an 8th bypass in the same "value hidden from the
extractor" family as round 6, inside the fold logic round 7's own fix had
just added (a blank line interrupting a folded continuation) — the second
distinct gap found on the first alternate shape tried against that fix.

Decision (Mathias, via the running sweep session): ship this PR's rounds
1-7 as v1 rather than continue incremental patches past the point of
diminishing returns. Every bypass found across all 7 rounds requires the
caller to already hold brain_edit write access and to deliberately
construct a specific malformed-YAML shape with knowledge of this exact
gap — narrow and low-likelihood next to another patch cycle. The real fix
(one shared, tested, encoding-normalized frontmatter parser, replacing the
three independent hand-rolled implementations this bug class keeps
surfacing in) is tracked separately as hyperguild#138.

Document the boundary directly on checkClaimEnvelope so the next person
reading this function knows what's covered and what's a deliberate,
recorded trade-off rather than an unnoticed gap.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 12:25:30 +02:00
mathiasandClaude Sonnet 5 42578c21b9 fix(mcp): fold YAML block-scalar continuations before validating the envelope
CI / Lint / Test / Vet (pull_request) Successful in 33s
CI / Mirror to GitHub (pull_request) Skipped
Round-6 review found a bypass in a different mechanism than rounds 1-6's
fence-detection family: frontmatterScalars's indentation guard skips any
indented continuation line outright, so old_str="half_life: durable",
new_str="half_life:\n  garbage-value" turns the field into a YAML
block-scalar shape whose real value lives on the continuation line.
present()'s own "empty value = placeholder, not an assertion" contract then
treats the field as absent, and claim.ValidateEnvelope never sees the
invalid value at all — no encoding quirk needed, one ordinary brain_edit
call smuggles an out-of-enum value straight past the check.

Fold indented continuation lines into the value when a top-level key's
inline value is empty, so oneOf() still sees what was actually written. A
genuinely empty value (no indented line follows) still folds to "",
preserving the existing placeholder-not-assertion behavior for legitimately
unset fields.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 10:51:09 +02:00
mathiasandClaude Sonnet 5 1d67e16c0d fix(mcp): strip a leading UTF-8 BOM before frontmatter fence detection
CI / Lint / Test / Vet (pull_request) Successful in 42s
CI / Mirror to GitHub (pull_request) Skipped
Round-5 review found a sixth bypass, same mechanism as round 4's CRLF gap,
one layer earlier: a leading UTF-8 BOM (EF BB BF) defeats
strings.HasPrefix(doc, "---\n") before CRLF normalization even applies.
Judged live-plausible rather than exotic — Notepad's default "UTF-8" save
and PowerShell's pre-v6 Out-File/Set-Content both emit a BOM — so fixed
inline at the same cost as the CRLF fix rather than deferred wholesale to
hyperguild#138.

Strip the BOM before the CRLF normalization in frontmatterBlock, so every
caller (fence-presence detection and frontmatterScalars) agrees regardless
of encoding.

Drawing the line here: further encoding variants in this family (NEL line
separators, UTF-16 BOM, lone-CR classic-Mac endings) are judged not
live-plausible in a git-tracked UTF-8 markdown corpus and are tracked
wholesale under hyperguild#138's parser consolidation rather than chased as
individual rounds on this PR.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 10:44:54 +02:00
mathiasandClaude Sonnet 5 4f75279ceb fix(mcp): normalize CRLF before frontmatter fence detection
CI / Lint / Test / Vet (pull_request) Successful in 33s
CI / Mirror to GitHub (pull_request) Skipped
Round-4 review found a fifth bypass, and this one needs no adversarial
old_str/new_str crafting at all: frontmatterBlock's fence detection matches
the literal byte sequence "---\n". A note whose fence lines use CRLF
("---\r\n") fails strings.HasPrefix(doc, "---\n") on the precondition read
purely from line-ending encoding — hadFence is false, so the round-3/round-4
checks can never fire, frontmatterScalars falls back to an empty map, and
claim.ValidateEnvelope({}) is nil by design. An invalid claim_provenance
value lands with zero error.

Zero CRLF-fenced notes exist in the live corpus today, but nothing in the
code prevented one, and this class of encoding-dependent false negative
could resurface in any of the other fence-based checks. Normalize once
inside frontmatterBlock so every caller (fence-presence detection and
frontmatterScalars) agrees regardless of line ending.

The broader structural concern — three independent, subtly inconsistent
fence-parsing implementations across internal/mcp, internal/api, and
internal/frontmatter, of which this CRLF gap is a direct symptom — is out
of scope for this PR and filed separately as hyperguild#138.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 10:40:00 +02:00
mathiasandClaude Sonnet 5 110e8b8f75 fix(mcp): reject edits that strip a note's frontmatter fence
CI / Lint / Test / Vet (pull_request) Successful in 34s
CI / Mirror to GitHub (pull_request) Skipped
Round-3 review found a fourth bypass: checkClaimEnvelope always validates
frontmatterScalars(after), but an edit whose new_str drops the closing
"---" degrades `after` to "unfenced" in the same call. frontmatterBlock(after)
then returns ok=false, frontmatterScalars returns an empty map, and
claim.ValidateEnvelope({}) is nil by design (absent fields are valid) — so
an invalid enum value lands on disk with zero error, and the note is left
permanently unfenced.

A note that had a valid fence before the edit must still have one after, or
the edit is rejected outright — closes this off structurally rather than
special-casing the fence-removal shape. Also guards oldStr == "" as a no-op
(strings.Replace treats "" as matching position 0, which would otherwise
look like an edit that inserts text before the fence): that degenerate case
belongs to notes.Edit's own "old_str is empty" refusal (#115), not to the
envelope check — TestBrainEdit_EmptyOldStrErrorNamesRawArgumentBytes caught
the regression when this guard was still missing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 10:32:46 +02:00
mathiasandClaude Sonnet 5 0f932d8f77 fix(mcp): drop the containment gate, always validate the post-edit envelope
CI / Lint / Test / Vet (pull_request) Successful in 41s
CI / Mirror to GitHub (pull_request) Skipped
Round-2 review found a third bypass: checkClaimEnvelope required old_str to
be fully contained within the extracted frontmatter block (which excludes
both "---" fence lines). An old_str disambiguated across the closing fence
into the body — a natural choice, since notes.Edit requires old_str to be
unique — failed that containment test, so the gate skipped validation
entirely while notes.Edit applied the replace verbatim, reaching the same
invalid-enum outcome as the two bypasses already closed.

Remove the containment gate structurally instead of patching another edge:
checkClaimEnvelope now always computes the post-edit doc and validates its
resulting frontmatter, regardless of old_str's textual relationship to the
fence. A body-only edit leaves the frontmatter unchanged (still valid, since
the note started valid), so this is a no-op for the common case and
fail-closed for every case that touches the envelope, however old_str is
phrased.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 10:27:30 +02:00
mathiasandClaude Sonnet 5 062574dad8 fix(mcp): close claim-envelope bypass on value-only brain_edit diffs
CI / Lint / Test / Vet (pull_request) Successful in 33s
CI / Mirror to GitHub (pull_request) Skipped
Review of PR #136 found the gate trivially bypassable: touchesClaimEnvelope
only fired when old_str/new_str contained the claim-envelope FIELD NAME as a
substring, but a natural value-only diff (old_str: "asserted", new_str:
"guessed" against a claim_provenance line) never mentions the field name and
walked straight past it — notes.Edit wrote the invalid enum value to disk
with zero error.

Redesign the gate to be position-based instead of name-based: checkClaimEnvelope
now checks whether old_str is physically contained within the note's
frontmatter block (via the new frontmatterBlock helper), previews the exact
strings.Replace the real edit would perform, and validates the resulting
envelope before the edit is allowed to run. frontmatterScalars now builds on
the shared frontmatterBlock helper.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 10:21:53 +02:00
mathiasandClaude Sonnet 5 f7fe00cecf fix(mcp): validate the claim envelope before brain_edit writes too
CI / Lint / Test / Vet (pull_request) Successful in 36s
CI / Mirror to GitHub (pull_request) Skipped
hyperguild#123, the audit follow-up from #111: writeHallNote and
UpdateNote both validate the claim envelope (half_life/applies/
claim_provenance) before writing; brain_edit didn't -- a byte-level
string replace has no frontmatter awareness at all, so a caller could
rewrite an existing envelope line (e.g. flip claim_provenance from
asserted to guessed, or set half_life to a value outside the enum)
straight past the validator. This was the one live, model-facing gap
#123 identified; the ingestion pipeline write paths audited alongside it
turned out not to touch these fields at all, so this closes the issue.

Fixed by previewing what old_str->new_str would produce -- the exact
strings.Replace(before, old_str, new_str, 1) formula notes.Edit itself
uses internally -- and validating the envelope on that preview before
the real edit is allowed to run. Fail-closed: an invalid result blocks
the edit entirely, matching create/supersede's existing posture. Scoped
to notes only (scope == notes.ScopeNotes) since governance files have no
claim-envelope concept, and gated on a cheap substring check so an
ordinary body edit that never mentions an envelope field name pays
nothing extra.

The preview doesn't replicate notes.Edit's uniqueness/CRLF/conflict-
marker checks -- if old_str is absent or ambiguous, the preview may not
match what Edit ultimately does, but Edit independently and correctly
rejects those cases at write time regardless, so nothing about write
safety rides on this preview's accuracy, only the envelope check itself.

frontmatterScalars is a third, minimal implementation of the same
indented-line-safe scalar extraction as the sibling packages
(hyperguild#126/#128) -- deliberately not shared/exported across
packages (each package's frontmatter handling has slightly different
needs and none currently import each other), but built the same way
from the start this time rather than needing a follow-up fix.

Flip-tested: disabling the pre-check gate lets the invalid-envelope test
fail (the edit silently succeeds); restoring it passes. Full module
suite green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 10:13:59 +02:00
88 changed files with 283 additions and 5992 deletions
+1 -5
View File
@@ -64,11 +64,7 @@ the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`, The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300 `/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients, **with the same bearer token as `/mcp`** for shell scripts and non-MCP clients.
(`BRAIN_MCP_TOKEN`; hyperguild#149). It is not served on the public host
(infra#517): use the tailnet (`koala:8443`) or in-cluster
`ingestion.supervisor.svc:3300`. `/ingest-path` and `brain_ingest` `path`
accept only paths inside `brain/raw/`.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai `brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title, gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
+5 -85
View File
@@ -78,27 +78,6 @@ These rules apply to every task across every project, regardless of harness.
when a human reviewer outside the project is required. Document the reason in when a human reviewer outside the project is required. Document the reason in
PROJECT.md. PROJECT.md.
**Exception — a GitOps repo where a branch cannot be reconciled.** In a repo whose
deployment watches one branch (`mathias/infra`: Flux watches `main`), a branch is
invisible to the thing that applies it. A `risk/medium` change to `alerts.yaml` or
`litellm-config.yaml` can be linted on a branch but never *exercised* there, so a
PR would buy a review of a state that was never deployed.
There, `risk/medium`+ work stays on `main` and instead requires a **recorded
adversarial review pass before the push** — `skills:pr-gatekeeper` or equivalent,
reviewing the staged diff. **The reviewer's verdict goes in the issue, not a PR
thread.** No verdict recorded means the lane was not used, and saying "the gate was
green" is not the same claim.
Be honest about what this is: a recorded *independent* pass, not four-eyes in the
sense `assessor-loop` §4.1 means for a regulated organisation, because the reviewer
is an agent. It is more than the repo had and less than the model it mirrors; do not
describe it as the stronger property. (infra#470, decided 2026-09-29.)
This lane depends on the `risk/` label being right. A label that triggers a heavier
lane is load-bearing, not cosmetic — if work that belongs in it is unlabelled, the
lane silently does not apply.
6. **Close the loop — every substantive task ends with the same ritual.** Shipping 6. **Close the loop — every substantive task ends with the same ritual.** Shipping
the code is not the end of the task; capturing it is. Run this unprompted: the code is not the end of the task; capturing it is. Run this unprompted:
- **Tag + bump SemVer** on the change (annotated tag; minor for a feature or - **Tag + bump SemVer** on the change (annotated tag; minor for a feature or
@@ -127,43 +106,6 @@ These rules apply to every task across every project, regardless of harness.
revisit Y". Those are exactly the items that need a locator. If it is worth revisit Y". Those are exactly the items that need a locator. If it is worth
saying in a summary, it is worth writing where it can be found again without you. saying in a summary, it is worth writing where it can be found again without you.
7. **Finish what's closable — only involve me for a genuine call.** If a
tracker issue, PR, or checklist item is demonstrably resolved — the fix
landed, the state changed, you verified it live — close/merge/check it off
in the same turn. Don't leave it open "for the record" or park it waiting
on me to confirm something that already happened; that's bookkeeping, not
a decision. Reserve asking me (`AskUserQuestion`, a `DECISION NEEDED`
comment, or otherwise pausing) for when it's genuinely mine to call:
irreversible or prod/host-impacting actions, a real judgment call with no
clearly-better option, ambiguous intent, or anything secret-writes/
destructive-git already gate. Before claiming something is closable,
verify it live (re-check the issue, re-run the command) rather than
trusting your own earlier summary (brain: `filed-issue-not-same-as-fixed`).
8. **Control code needs an adversary, not a checklist.** Code whose job is to
refuse, deny, gate or contain is not adequately verified by a green gate.
Rule 5 makes CI the quality gate and that holds for ordinary work — but a
passing suite only asserts what its author already thought of, and for a
control the author is precisely the person whose imagination just failed.
Measured on `dispatch#75`: five review rounds, five real escapes verified by
execution, and **CI was green on every one of them** while `git rm
././.dispatch-allow` deleted the file the control existed to protect. So
route this work through a review with authority to block, instructed to
verify by execution rather than by reading the implementation. Expect to be
blocked more than once; that is the control working, not the reviewer being
difficult. (brain: `a-green-gate-is-not-evidence-for-control-code`; the
practices live in the `pr-gatekeeper` agent.)
9. **A test's name must not claim more than its assertions prove.** Name a test
after the mechanism it observes, not the goal it serves.
`DoesNotInheritEnv` is honest; `IsContained` is a claim the test cannot
support. This is not pedantry about naming — when name and assertion
diverge, the name wins in the reader's head and the gap becomes invisible,
which is what lets the defect survive review. Found three times in one PR,
twice in tests written to fix the previous instance of it; each passed while
the thing it was named for was exploitable. If you cannot name the mechanism,
you do not yet know what the test observes.
## CAD sprint patterns ## CAD sprint patterns
Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`): Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`):
@@ -304,16 +246,10 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
|---------|-------|-------| |---------|-------|-------|
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool | | **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same | | **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | Tailnet only: `POST https://koala.tail30472e.ts.net:8443/query` with `{"query":"..."}`, sending `BRAIN_MCP_TOKEN` (not yet enforced on REST, hyperguild#149). The public `brain-mcp.d-ma.be` serves `/mcp` only (infra#517). | `POST .../write` with `{"content":"...","wing":"...","hall":"..."}` | | **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` |
| **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files | | **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files |
- **Scoping — the brain holds no confidential client content.** Nothing on the read path - **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`.
filters by client: every token holder can read every wing, and `wing` on `brain_query`
is an optional argument, not an access control. Client confidentiality is enforced only
at write time (`capture`'s I1 gate refuses internal/confidential). So never `brain_write`
client material; generalise it first or keep it in the client's own repo. (hyperguild#144,
decided 2026-10-01. This line previously claimed client projects were filtered; they
were not.)
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as - **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
fallback. Both are configurable in the `supervisor/ingestion-deployment.yaml` fallback. Both are configurable in the `supervisor/ingestion-deployment.yaml`
on the koala k3s cluster; don't hardcode local-only model names into the on the koala k3s cluster; don't hardcode local-only model names into the
@@ -383,20 +319,8 @@ Workflow:
and missing-untracked adapters). A drift fails the check with a and missing-untracked adapters). A drift fails the check with a
message telling you to stage the regenerated files. message telling you to stage the regenerated files.
Behavior rules in this file and per-project rules in `PROJECT.md` apply on every Behavior rules in this file and per-project rules in `PROJECT.md` apply
host and every harness. unconditionally on every host, every harness.
**Where a project rule contradicts a rule here, the project rule governs — and it
must state its reason.** Both halves matter: the project file wins because it knows
something this file cannot (infra's Flux constraint is real and not visible from
here), and it must say why because a bare override is indistinguishable from drift.
This replaces the previous wording, which said rules here and in `PROJECT.md` both
apply *"unconditionally"*. That was not a tiebreak but a contradiction, and it went
unnoticed until rule 5 and infra's "never create feature branches" had to be obeyed
at the same time (infra#470). A standard can hold two individually sensible clauses
that are jointly unsatisfiable, and survive for as long as everyone reads it
charitably — the same failure the Inception Sprint Oath's G6 had.
## Engineering Skills ## Engineering Skills
@@ -493,11 +417,7 @@ the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`, The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300 `/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients, **with the same bearer token as `/mcp`** for shell scripts and non-MCP clients.
(`BRAIN_MCP_TOKEN`; hyperguild#149). It is not served on the public host
(infra#517): use the tailnet (`koala:8443`) or in-cluster
`ingestion.supervisor.svc:3300`. `/ingest-path` and `brain_ingest` `path`
accept only paths inside `brain/raw/`.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai `brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title, gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
+5 -85
View File
@@ -73,27 +73,6 @@ These rules apply to every task across every project, regardless of harness.
when a human reviewer outside the project is required. Document the reason in when a human reviewer outside the project is required. Document the reason in
PROJECT.md. PROJECT.md.
**Exception — a GitOps repo where a branch cannot be reconciled.** In a repo whose
deployment watches one branch (`mathias/infra`: Flux watches `main`), a branch is
invisible to the thing that applies it. A `risk/medium` change to `alerts.yaml` or
`litellm-config.yaml` can be linted on a branch but never *exercised* there, so a
PR would buy a review of a state that was never deployed.
There, `risk/medium`+ work stays on `main` and instead requires a **recorded
adversarial review pass before the push** — `skills:pr-gatekeeper` or equivalent,
reviewing the staged diff. **The reviewer's verdict goes in the issue, not a PR
thread.** No verdict recorded means the lane was not used, and saying "the gate was
green" is not the same claim.
Be honest about what this is: a recorded *independent* pass, not four-eyes in the
sense `assessor-loop` §4.1 means for a regulated organisation, because the reviewer
is an agent. It is more than the repo had and less than the model it mirrors; do not
describe it as the stronger property. (infra#470, decided 2026-09-29.)
This lane depends on the `risk/` label being right. A label that triggers a heavier
lane is load-bearing, not cosmetic — if work that belongs in it is unlabelled, the
lane silently does not apply.
6. **Close the loop — every substantive task ends with the same ritual.** Shipping 6. **Close the loop — every substantive task ends with the same ritual.** Shipping
the code is not the end of the task; capturing it is. Run this unprompted: the code is not the end of the task; capturing it is. Run this unprompted:
- **Tag + bump SemVer** on the change (annotated tag; minor for a feature or - **Tag + bump SemVer** on the change (annotated tag; minor for a feature or
@@ -122,43 +101,6 @@ These rules apply to every task across every project, regardless of harness.
revisit Y". Those are exactly the items that need a locator. If it is worth revisit Y". Those are exactly the items that need a locator. If it is worth
saying in a summary, it is worth writing where it can be found again without you. saying in a summary, it is worth writing where it can be found again without you.
7. **Finish what's closable — only involve me for a genuine call.** If a
tracker issue, PR, or checklist item is demonstrably resolved — the fix
landed, the state changed, you verified it live — close/merge/check it off
in the same turn. Don't leave it open "for the record" or park it waiting
on me to confirm something that already happened; that's bookkeeping, not
a decision. Reserve asking me (`AskUserQuestion`, a `DECISION NEEDED`
comment, or otherwise pausing) for when it's genuinely mine to call:
irreversible or prod/host-impacting actions, a real judgment call with no
clearly-better option, ambiguous intent, or anything secret-writes/
destructive-git already gate. Before claiming something is closable,
verify it live (re-check the issue, re-run the command) rather than
trusting your own earlier summary (brain: `filed-issue-not-same-as-fixed`).
8. **Control code needs an adversary, not a checklist.** Code whose job is to
refuse, deny, gate or contain is not adequately verified by a green gate.
Rule 5 makes CI the quality gate and that holds for ordinary work — but a
passing suite only asserts what its author already thought of, and for a
control the author is precisely the person whose imagination just failed.
Measured on `dispatch#75`: five review rounds, five real escapes verified by
execution, and **CI was green on every one of them** while `git rm
././.dispatch-allow` deleted the file the control existed to protect. So
route this work through a review with authority to block, instructed to
verify by execution rather than by reading the implementation. Expect to be
blocked more than once; that is the control working, not the reviewer being
difficult. (brain: `a-green-gate-is-not-evidence-for-control-code`; the
practices live in the `pr-gatekeeper` agent.)
9. **A test's name must not claim more than its assertions prove.** Name a test
after the mechanism it observes, not the goal it serves.
`DoesNotInheritEnv` is honest; `IsContained` is a claim the test cannot
support. This is not pedantry about naming — when name and assertion
diverge, the name wins in the reader's head and the gap becomes invisible,
which is what lets the defect survive review. Found three times in one PR,
twice in tests written to fix the previous instance of it; each passed while
the thing it was named for was exploitable. If you cannot name the mechanism,
you do not yet know what the test observes.
## CAD sprint patterns ## CAD sprint patterns
Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`): Two CAD sprint archetypes (defs: brain `wiki/homelab/decisions/continuous-agentic-development-cad-concept-2026-06-16`):
@@ -299,16 +241,10 @@ entries that age well are about *why*, *how to avoid*, and *what to do when*.
|---------|-------|-------| |---------|-------|-------|
| **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool | | **Claude Code, Claude Desktop** | `brain_query` (BM25), `brain_answer` (LLM-synth + sources) MCP tools | `brain_write` MCP tool |
| **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same | | **Crush, Pi, Antigravity, other MCP-capable** | same MCP server: `ingestion-brain` (via the `mcp__*_brain__*` namespace once authenticated) | same |
| **Anything HTTP-only (curl, scripts)** | Tailnet only: `POST https://koala.tail30472e.ts.net:8443/query` with `{"query":"..."}`, sending `BRAIN_MCP_TOKEN` (not yet enforced on REST, hyperguild#149). The public `brain-mcp.d-ma.be` serves `/mcp` only (infra#517). | `POST .../write` with `{"content":"...","wing":"...","hall":"..."}` | | **Anything HTTP-only (curl, scripts)** | `POST https://brain-mcp.d-ma.be/query` with `{"query":"..."}` (auth via `BRAIN_MCP_TOKEN`) | `POST .../write` with `{"content":"...","filename":"..."}` |
| **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files | | **Browser / human inspection** | `https://git.d-ma.be/mathias/hyperguild` → `knowledge/` and `wiki/` markdown files |
- **Scoping — the brain holds no confidential client content.** Nothing on the read path - **Scoping**: defaults to `public` collection; client projects filter to `{client}` + `public`.
filters by client: every token holder can read every wing, and `wing` on `brain_query`
is an optional argument, not an access control. Client confidentiality is enforced only
at write time (`capture`'s I1 gate refuses internal/confidential). So never `brain_write`
client material; generalise it first or keep it in the client's own repo. (hyperguild#144,
decided 2026-10-01. This line previously claimed client projects were filtered; they
were not.)
- **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as - **Routing**: brain_answer's LLM uses berget.ai as primary, iguana ollama as
fallback. Both are configurable in the `supervisor/ingestion-deployment.yaml` fallback. Both are configurable in the `supervisor/ingestion-deployment.yaml`
on the koala k3s cluster; don't hardcode local-only model names into the on the koala k3s cluster; don't hardcode local-only model names into the
@@ -378,20 +314,8 @@ Workflow:
and missing-untracked adapters). A drift fails the check with a and missing-untracked adapters). A drift fails the check with a
message telling you to stage the regenerated files. message telling you to stage the regenerated files.
Behavior rules in this file and per-project rules in `PROJECT.md` apply on every Behavior rules in this file and per-project rules in `PROJECT.md` apply
host and every harness. unconditionally on every host, every harness.
**Where a project rule contradicts a rule here, the project rule governs — and it
must state its reason.** Both halves matter: the project file wins because it knows
something this file cannot (infra's Flux constraint is real and not visible from
here), and it must say why because a bare override is indistinguishable from drift.
This replaces the previous wording, which said rules here and in `PROJECT.md` both
apply *"unconditionally"*. That was not a tiebreak but a contradiction, and it went
unnoticed until rule 5 and infra's "never create feature branches" had to be obeyed
at the same time (infra#470). A standard can hold two individually sensible clauses
that are jointly unsatisfiable, and survive for as long as everyone reads it
charitably — the same failure the Inception Sprint Oath's G6 had.
## Engineering Skills ## Engineering Skills
@@ -488,11 +412,7 @@ the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`, The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300 `/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients, **with the same bearer token as `/mcp`** for shell scripts and non-MCP clients.
(`BRAIN_MCP_TOKEN`; hyperguild#149). It is not served on the public host
(infra#517): use the tailnet (`koala:8443`) or in-cluster
`ingestion.supervisor.svc:3300`. `/ingest-path` and `brain_ingest` `path`
accept only paths inside `brain/raw/`.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai `brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title, gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
+1 -5
View File
@@ -64,11 +64,7 @@ the routing pod; brain tools moved to the brain MCP.
The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`, The brain HTTP REST API (`/query`, `/write`, `/ingest`, `/ingest-raw`,
`/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300 `/ingest-path`, `/backfill-refs`, `/pass-rate`) remains available on port 3300
for shell scripts and non-MCP clients, **with the same bearer token as `/mcp`** for shell scripts and non-MCP clients.
(`BRAIN_MCP_TOKEN`; hyperguild#149). It is not served on the public host
(infra#517): use the tailnet (`koala:8443`) or in-cluster
`ingestion.supervisor.svc:3300`. `/ingest-path` and `brain_ingest` `path`
accept only paths inside `brain/raw/`.
`brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai `brain_answer(query)` performs BM25 retrieval + LLM synthesis (berget.ai
gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title, gemma4:31b → iguana fallback). `brain_classify(text)` infers doc type, title,
+5 -15
View File
@@ -20,7 +20,6 @@ const defaultBrainURL = "http://koala:30330"
// shell-friendly client; MCP framing is unnecessary. // shell-friendly client; MCP framing is unnecessary.
type brainClient struct { type brainClient struct {
baseURL string baseURL string
token string // BRAIN_MCP_TOKEN; the REST API requires it since hyperguild#149
http *http.Client http *http.Client
} }
@@ -31,7 +30,6 @@ func newBrainClient() *brainClient {
} }
return &brainClient{ return &brainClient{
baseURL: u, baseURL: u,
token: os.Getenv("BRAIN_MCP_TOKEN"),
http: &http.Client{Timeout: 5 * time.Second}, http: &http.Client{Timeout: 5 * time.Second},
} }
} }
@@ -64,7 +62,8 @@ func (c *brainClient) Query(ctx context.Context, topic string, limit int) (*Quer
return nil, fmt.Errorf("build request: %w", err) return nil, fmt.Errorf("build request: %w", err)
} }
req.Header.Set("Content-Type", "application/json") req.Header.Set("Content-Type", "application/json")
resp, err := c.do(req)
resp, err := c.http.Do(req)
if err != nil { if err != nil {
return nil, fmt.Errorf("brain POST /query: %w", err) return nil, fmt.Errorf("brain POST /query: %w", err)
} }
@@ -106,7 +105,8 @@ func (c *brainClient) Write(ctx context.Context, wing, hall, filename string, co
return nil, fmt.Errorf("build request: %w", err) return nil, fmt.Errorf("build request: %w", err)
} }
req.Header.Set("Content-Type", "application/json") req.Header.Set("Content-Type", "application/json")
resp, err := c.do(req)
resp, err := c.http.Do(req)
if err != nil { if err != nil {
return nil, fmt.Errorf("brain POST /write: %w", err) return nil, fmt.Errorf("brain POST /write: %w", err)
} }
@@ -143,7 +143,7 @@ func (c *brainClient) PassRate(ctx context.Context, skill, window string) (*Pass
if err != nil { if err != nil {
return nil, fmt.Errorf("build request: %w", err) return nil, fmt.Errorf("build request: %w", err)
} }
resp, err := c.do(req) resp, err := c.http.Do(req)
if err != nil { if err != nil {
return nil, fmt.Errorf("brain GET /pass-rate: %w", err) return nil, fmt.Errorf("brain GET /pass-rate: %w", err)
} }
@@ -158,13 +158,3 @@ func (c *brainClient) PassRate(ctx context.Context, skill, window string) (*Pass
} }
return &out, nil return &out, nil
} }
// do sends every brain request, adding the bearer token when one is
// configured. All methods go through here so none can forget it (PassRate
// did, PR #152 review).
func (c *brainClient) do(req *http.Request) (*http.Response, error) {
if c.token != "" {
req.Header.Set("Authorization", "Bearer "+c.token)
}
return c.http.Do(req)
}
-57
View File
@@ -1,57 +0,0 @@
package main
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// hyperguild#149: the ingestion REST API now requires a bearer token; the CLI
// sends BRAIN_MCP_TOKEN, or it gets 401 on every brain call.
func TestBrainClientSendsBrainMCPTokenOnQueryWriteAndPassRate(t *testing.T) {
var got []string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
got = append(got, r.URL.Path+" "+r.Header.Get("Authorization"))
switch r.URL.Path {
case "/write":
_, _ = w.Write([]byte(`{"path":"wiki/x/facts/y.md"}`))
return
case "/pass-rate":
_, _ = w.Write([]byte(`{}`))
return
}
_, _ = w.Write([]byte(`{"results":[]}`))
}))
defer srv.Close()
t.Setenv("BRAIN_URL", srv.URL)
t.Setenv("BRAIN_MCP_TOKEN", "cli-test-token")
c := newBrainClient()
_, err := c.Query(context.Background(), "x", 1)
require.NoError(t, err)
_, err = c.Write(context.Background(), "x", "facts", "y", strings.NewReader("# y"))
require.NoError(t, err)
// PR #152 review: PassRate sent no token and would 401 in production.
_, _ = c.PassRate(context.Background(), "tdd", "7d")
assert.Equal(t, []string{"/query Bearer cli-test-token", "/write Bearer cli-test-token",
"/pass-rate Bearer cli-test-token"}, got)
}
func TestBrainClientSendsNoAuthorizationWhenTokenUnset(t *testing.T) {
var auth string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
auth = r.Header.Get("Authorization")
_, _ = w.Write([]byte(`{"results":[]}`))
}))
defer srv.Close()
t.Setenv("BRAIN_URL", srv.URL)
t.Setenv("BRAIN_MCP_TOKEN", "")
_, err := newBrainClient().Query(context.Background(), "x", 1)
require.NoError(t, err)
assert.Empty(t, auth)
}
@@ -24,7 +24,7 @@ Only a `tools/call` to the routing pod records a pass/fail.
|---------|-----|------| |---------|-----|------|
| Routing MCP (local, Tailscale) | `http://koala:30310/mcp` | Bearer `ROUTING_MCP_TOKEN` | | Routing MCP (local, Tailscale) | `http://koala:30310/mcp` | Bearer `ROUTING_MCP_TOKEN` |
| Routing MCP (remote) | `https://routing-mcp.d-ma.be/mcp` | OAuth via `auth.d-ma.be` (audience `claude-ai`) | | Routing MCP (remote) | `https://routing-mcp.d-ma.be/mcp` | OAuth via `auth.d-ma.be` (audience `claude-ai`) |
| Pass-rate readout | `http://koala:30330/pass-rate?skill=<name>` | Bearer `BRAIN_MCP_TOKEN` (since hyperguild#149) | | Pass-rate readout | `http://koala:30330/pass-rate?skill=<name>` | none (read-only) |
Tools advertised: **`review`**, **`debug`** (the two the #35 gate measures), Tools advertised: **`review`**, **`debug`** (the two the #35 gate measures),
plus `session_log`, `retrospective`, `trainer`. plus `session_log`, `retrospective`, `trainer`.
+19 -62
View File
@@ -33,7 +33,6 @@ import (
"github.com/mathiasbq/hyperguild/ingestion/internal/reranker" "github.com/mathiasbq/hyperguild/ingestion/internal/reranker"
"github.com/mathiasbq/hyperguild/ingestion/internal/search" "github.com/mathiasbq/hyperguild/ingestion/internal/search"
"github.com/mathiasbq/hyperguild/ingestion/internal/startupretry" "github.com/mathiasbq/hyperguild/ingestion/internal/startupretry"
"github.com/mathiasbq/hyperguild/ingestion/internal/symlinkguard"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore" "github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/watcher" "github.com/mathiasbq/hyperguild/ingestion/internal/watcher"
"github.com/mathiasbq/hyperguild/ingestion/internal/webhook" "github.com/mathiasbq/hyperguild/ingestion/internal/webhook"
@@ -148,38 +147,6 @@ func envInt(key string, fallback int) int {
// buildNotifier returns the ntfy notifier, or a nil interface when // buildNotifier returns the ntfy notifier, or a nil interface when
// BRAIN_NTFY_URL is unset. A nil *interface* (not a typed-nil) is required so // BRAIN_NTFY_URL is unset. A nil *interface* (not a typed-nil) is required so
// DegradingSink/Reconcile and the governance observer skip it cleanly. // DegradingSink/Reconcile and the governance observer skip it cleanly.
// alertingWarnings names every alert that will not fire under the current
// configuration (#107).
//
// buildNotifier returns a nil interface when BRAIN_NTFY_URL is unset, and every
// consumer skips a nil notifier silently — right for an unconfigured homelab,
// wrong for an alert whose entire value is that someone finds out. The cost of
// the silence is that "configured and working" and "never configured" are
// byte-identical in behaviour: no error, no warning, only a message that never
// arrives, which nobody notices because nobody is waiting for an alert they
// have never seen. Two alerts were dark in production for exactly that reason
// (#100), found by firing a real event rather than by reading anything.
//
// WARN rather than fatal, deliberately. The brain sibling rule
// (`empty-credential-must-refuse-to-start-not-switch-modes`) says an empty
// credential must be a startup error, never a silent mode switch — but that is
// about a control being removed. Here observability is degraded, not a gate
// bypassed, and refusing to boot would break the unconfigured-homelab case the
// nil notifier exists to support. What the rule does demand, and what this
// provides, is that the mode switch stops being silent.
//
// Pure and env-injected so the message is testable without a running server:
// the reason this gap survived is that nothing about it was observable.
func alertingWarnings(getenv func(string) string) []string {
var out []string
if getenv("BRAIN_NTFY_URL") == "" {
out = append(out, "alerting DISABLED: BRAIN_NTFY_URL unset — "+
"brain_edit_root governance rewrites, capture audit degradation and the "+
"brain symlink tripwire (hyperguild#157) will not notify")
}
return out
}
func buildNotifier() audit.Notifier { func buildNotifier() audit.Notifier {
if n := audit.NewNtfyNotifier(os.Getenv("BRAIN_NTFY_URL"), os.Getenv("BRAIN_NTFY_TOKEN")); n != nil { if n := audit.NewNtfyNotifier(os.Getenv("BRAIN_NTFY_URL"), os.Getenv("BRAIN_NTFY_TOKEN")); n != nil {
return n return n
@@ -236,15 +203,7 @@ func systemHostname() string {
func main() { func main() {
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil)) logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
for _, w := range alertingWarnings(os.Getenv) {
logger.Warn(w)
}
brainDir := envOr("INGEST_BRAIN_DIR", "../brain") brainDir := envOr("INGEST_BRAIN_DIR", "../brain")
if err := startupSymlinkCheck(brainDir); err != nil {
logger.Error("brain symlink tripwire", "err", err)
os.Exit(1)
}
port := envOr("INGEST_PORT", "3300") port := envOr("INGEST_PORT", "3300")
llmURL := envOr("INGEST_LLM_URL", "http://iguana:4000/v1") llmURL := envOr("INGEST_LLM_URL", "http://iguana:4000/v1")
@@ -366,8 +325,6 @@ func main() {
} }
ctx := context.Background() ctx := context.Background()
// hyperguild#157: report a symlink that appears in the brain while running.
go symlinkguard.Watch(ctx, brainDir, symlinkScanInterval(os.Getenv), buildNotifier())
// Built before the watcher, the graph backfill and session ingestion, and // Built before the watcher, the graph backfill and session ingestion, and
// retried rather than fatal (#83). Two separate reasons: // retried rather than fatal (#83). Two separate reasons:
@@ -511,10 +468,20 @@ func main() {
logger.Info("embed sync started", "interval_s", embedSyncInterval) logger.Info("embed sync started", "interval_s", embedSyncInterval)
} }
// Public routes only; buildMux mounts the authenticated REST API (#149). mux := http.NewServeMux()
public := map[string]http.Handler{} mux.HandleFunc("POST /query", h.Query)
mux.HandleFunc("POST /write", h.Write)
mux.HandleFunc("POST /index", h.Index)
mux.HandleFunc("POST /ingest", h.Ingest)
mux.HandleFunc("POST /ingest-path", h.IngestPath)
mux.HandleFunc("POST /ingest-raw", h.IngestRaw)
mux.HandleFunc("POST /backfill-refs", h.BackfillRefs)
mux.HandleFunc("GET /pending", h.Pending)
mux.HandleFunc("POST /promote", h.Promote)
mux.HandleFunc("POST /backfill-embeddings", h.BackfillEmbeddings)
mux.HandleFunc("GET /pass-rate", h.PassRate)
if webhookHandler != nil { if webhookHandler != nil {
public["POST /webhooks/brain-sync"] = webhookHandler mux.Handle("POST /webhooks/brain-sync", webhookHandler)
} }
// Resource-metadata URL is only emitted on 401 when Dex OAuth is // Resource-metadata URL is only emitted on 401 when Dex OAuth is
// configured. Static-Bearer-only deployments leave this empty so // configured. Static-Bearer-only deployments leave this empty so
@@ -522,14 +489,14 @@ func main() {
var resourceMetadataURL string var resourceMetadataURL string
if dexURL := os.Getenv("DEX_ISSUER_URL"); dexURL != "" { if dexURL := os.Getenv("DEX_ISSUER_URL"); dexURL != "" {
resourceURL := os.Getenv("MCP_RESOURCE_URL") resourceURL := os.Getenv("MCP_RESOURCE_URL")
public["GET /.well-known/oauth-protected-resource"] = http.HandlerFunc( mux.HandleFunc("GET /.well-known/oauth-protected-resource",
chassisauth.ProtectedResourceHandler(resourceURL, dexURL)) chassisauth.ProtectedResourceHandler(resourceURL, dexURL))
if resourceURL != "" { if resourceURL != "" {
resourceMetadataURL = strings.TrimRight(resourceURL, "/") + "/.well-known/oauth-protected-resource" resourceMetadataURL = strings.TrimRight(resourceURL, "/") + "/.well-known/oauth-protected-resource"
} }
} }
public["/mcp"] = chassisauth.BearerMiddleware(mcpToken, jwtValidator, "brain", resourceMetadataURL, mcpSrv) mux.Handle("/mcp", chassisauth.BearerMiddleware(mcpToken, jwtValidator, "brain", resourceMetadataURL, mcpSrv))
// POST /capture (#53/#54): the uniform capture REST door. Needs a ticket // POST /capture (#53/#54): the uniform capture REST door. Needs a ticket
// tracker to file action items, so it only mounts when Gitea is // tracker to file action items, so it only mounts when Gitea is
@@ -551,7 +518,7 @@ func main() {
sovereign := splitList(os.Getenv("BRAIN_CAPTURE_SOVEREIGN_PRINCIPALS")) sovereign := splitList(os.Getenv("BRAIN_CAPTURE_SOVEREIGN_PRINCIPALS"))
resolver := capturehttp.NewOriginResolver(sovereign) resolver := capturehttp.NewOriginResolver(sovereign)
captureH := capturehttp.New(captureSvc, jwtValidator, mcpToken, "local-cli", resolver) captureH := capturehttp.New(captureSvc, jwtValidator, mcpToken, "local-cli", resolver)
public["POST /capture"] = captureH mux.Handle("POST /capture", captureH)
// Same use-case behind the MCP `capture` tool (#55 relay) so MCP-native // Same use-case behind the MCP `capture` tool (#55 relay) so MCP-native
// harnesses (claude.ai, Crush, Pi, LLM Council) reach capture through // harnesses (claude.ai, Crush, Pi, LLM Council) reach capture through
// the existing /mcp OAuth connector. mcpSrv is already wrapped above; // the existing /mcp OAuth connector. mcpSrv is already wrapped above;
@@ -581,9 +548,9 @@ func main() {
logger.Error("OAUTH_CLIENT_ID/SECRET set but MCP_RESOURCE_URL is empty; cannot derive issuer") logger.Error("OAUTH_CLIENT_ID/SECRET set but MCP_RESOURCE_URL is empty; cannot derive issuer")
os.Exit(1) os.Exit(1)
} }
public["GET /.well-known/oauth-authorization-server"] = http.HandlerFunc( mux.HandleFunc("GET /.well-known/oauth-authorization-server",
oauth.MetadataHandler(issuer)) oauth.MetadataHandler(issuer))
public["POST /oauth/token"] = http.HandlerFunc(oauth.TokenHandler(oauth.TokenConfig{ mux.HandleFunc("POST /oauth/token", oauth.TokenHandler(oauth.TokenConfig{
ClientID: oauthID, ClientID: oauthID,
ClientSecret: oauthSecret, ClientSecret: oauthSecret,
AccessToken: mcpToken, AccessToken: mcpToken,
@@ -602,17 +569,7 @@ func main() {
// real request latency. /metrics itself is excluded from its own // real request latency. /metrics itself is excluded from its own
// observation by registering it on the outer mux (post-wrap). // observation by registering it on the outer mux (post-wrap).
reg := metrics.New() reg := metrics.New()
public["GET /metrics"] = http.HandlerFunc(reg.Handler()) mux.HandleFunc("GET /metrics", reg.Handler())
// The whole router, built in one place so the test of buildMux is a test
// of production wiring (PR #152 review). The REST API's credential is the
// same as /mcp's.
mux, err := buildMux(restAuth{token: mcpToken, validator: jwtValidator, realm: "brain",
resourceMetadataURL: resourceMetadataURL}, h, public)
if err != nil {
logger.Error("build router", "err", err)
os.Exit(1)
}
logger.Info("metrics endpoint registered", "path", "/metrics") logger.Info("metrics endpoint registered", "path", "/metrics")
addr := ":" + port addr := ":" + port
-40
View File
@@ -6,7 +6,6 @@ import (
"log/slog" "log/slog"
"os" "os"
"path/filepath" "path/filepath"
"strings"
"testing" "testing"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
@@ -37,42 +36,3 @@ func TestClaudeSink_IngestWritesToNonIndexedArchiveNotWiki(t *testing.T) {
_, err = os.Stat(filepath.Join(dir, "wiki", "claude-sessions")) _, err = os.Stat(filepath.Join(dir, "wiki", "claude-sessions"))
assert.True(t, os.IsNotExist(err), "raw transcripts must never land under wiki/ (ai-sessions#10 — BM25 pollution)") assert.True(t, os.IsNotExist(err), "raw transcripts must never land under wiki/ (ai-sessions#10 — BM25 pollution)")
} }
// --- #107: an unconfigured alert must announce itself ----------------------
func TestAlertingWarnings(t *testing.T) {
tests := []struct {
name string
env map[string]string
want []string // substrings that must appear across the warnings
none bool // true = expect no warnings at all
}{
{
// The failure this exists to prevent: two alerts dark in
// production, discovered only by firing a real event and watching
// nothing arrive (#100).
name: "notifier unconfigured names what will not fire",
env: map[string]string{},
want: []string{"BRAIN_NTFY_URL", "brain_edit_root", "audit"},
},
{
name: "configured notifier is silent",
env: map[string]string{"BRAIN_NTFY_URL": "http://ntfy/brain-alerts"},
none: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := alertingWarnings(func(k string) string { return tt.env[k] })
if tt.none {
assert.Empty(t, got)
return
}
joined := strings.Join(got, " ")
for _, w := range tt.want {
assert.Contains(t, joined, w)
}
})
}
}
-77
View File
@@ -1,77 +0,0 @@
package main
import (
"errors"
"fmt"
"net/http"
chassisauth "git.d-ma.be/mathias/mcp-chassis/auth"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// publicRoutes is the complete set of patterns allowed on the outer mux,
// outside the REST API's auth (hyperguild#149). Each authenticates itself or is
// meant to be open: /mcp (bearer/JWT), /capture (bearer/JWT), the webhook
// (HMAC), /oauth/token (client secret), OAuth metadata and /metrics (open).
// A REST pattern added here would beat the authenticated catch-all by
// specificity and ship open, which is why buildMux refuses anything else.
var publicRoutes = map[string]bool{
"/mcp": true,
"POST /capture": true,
"POST /webhooks/brain-sync": true,
"GET /.well-known/oauth-protected-resource": true,
"GET /.well-known/oauth-authorization-server": true,
"POST /oauth/token": true,
"GET /metrics": true,
}
// restAuth is the credential the REST API checks: the same as /mcp's.
type restAuth struct {
token string
validator *chassisauth.JWTValidator
realm string
resourceMetadataURL string
}
// buildMux is the server's whole router. The REST API is mounted at "/"
// behind bearer auth that buildMux constructs itself, so a caller cannot pass
// a no-op; public handlers are registered as given, and only for patterns on
// the publicRoutes allowlist. It returns an http.Handler, not the
// *ServeMux, so nothing can register a route on the result afterwards and
// bypass the allowlist.
func buildMux(auth restAuth, h *api.Handler, public map[string]http.Handler) (http.Handler, error) {
if auth.token == "" {
return nil, errors.New("buildMux: REST auth needs a token")
}
mux := http.NewServeMux()
for pattern, handler := range public {
if !publicRoutes[pattern] {
return nil, fmt.Errorf("buildMux: %q is not an allowed public route", pattern)
}
mux.Handle(pattern, handler)
}
mountREST(mux, h, func(next http.Handler) http.Handler {
return chassisauth.BearerMiddleware(auth.token, auth.validator, auth.realm, auth.resourceMetadataURL, next)
})
return mux, nil
}
// mountREST registers the REST API on its own mux and mounts it at "/" behind
// auth: anything not registered as public lands here and is authenticated,
// including routes added later.
func mountREST(mux *http.ServeMux, h *api.Handler, auth func(http.Handler) http.Handler) {
rest := http.NewServeMux()
rest.HandleFunc("POST /query", h.Query)
rest.HandleFunc("POST /write", h.Write)
rest.HandleFunc("POST /index", h.Index)
rest.HandleFunc("POST /ingest", h.Ingest)
rest.HandleFunc("POST /ingest-path", h.IngestPath)
rest.HandleFunc("POST /ingest-raw", h.IngestRaw)
rest.HandleFunc("POST /backfill-refs", h.BackfillRefs)
rest.HandleFunc("GET /pending", h.Pending)
rest.HandleFunc("POST /promote", h.Promote)
rest.HandleFunc("POST /backfill-embeddings", h.BackfillEmbeddings)
rest.HandleFunc("GET /pass-rate", h.PassRate)
mux.Handle("/", auth(rest))
}
-106
View File
@@ -1,106 +0,0 @@
package main
import (
"log/slog"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
)
const routesTok = "routes-test-token"
func okHandler(body string) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte(body)) })
}
// testMux builds the router exactly as main does: buildMux with the full
// public set main registers (stub handlers) and real bearer config.
// hyperguild#149: the REST API is authenticated as a whole; the PR #152
// review showed a test of a COPY of the router let main.go regress unseen.
func testMux(t *testing.T) http.Handler {
t.Helper()
h := api.NewHandler(t.TempDir(), slog.New(slog.NewTextHandler(os.Stderr, nil)), pipeline.Config{})
public := map[string]http.Handler{}
for p := range publicRoutes {
public[p] = okHandler("public " + p)
}
mux, err := buildMux(restAuth{token: routesTok, realm: "brain"}, h, public)
require.NoError(t, err)
return mux
}
func do(t *testing.T, h http.Handler, method, path, authz string) (int, string) {
t.Helper()
req := httptest.NewRequest(method, path, strings.NewReader(`{"query":"x"}`))
if authz != "" {
req.Header.Set("Authorization", authz)
}
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
return rec.Code, rec.Body.String()
}
func TestBuildMuxRefusesRESTRoutesWithoutAValidToken(t *testing.T) {
r := testMux(t)
for _, rt := range []struct{ method, path string }{
{"POST", "/query"}, {"POST", "/write"}, {"POST", "/index"}, {"POST", "/ingest"},
{"POST", "/ingest-path"}, {"POST", "/ingest-raw"}, {"POST", "/backfill-refs"},
{"GET", "/pending"}, {"POST", "/promote"}, {"POST", "/backfill-embeddings"}, {"GET", "/pass-rate"},
} {
code, _ := do(t, r, rt.method, rt.path, "")
assert.Equal(t, http.StatusUnauthorized, code, "%s %s without a token", rt.method, rt.path)
code, _ = do(t, r, rt.method, rt.path, "Bearer wrong")
assert.Equal(t, http.StatusUnauthorized, code, "%s %s with a wrong token", rt.method, rt.path)
}
}
func TestBuildMuxServesRESTRoutesToAValidToken(t *testing.T) {
r := testMux(t)
code, _ := do(t, r, "GET", "/pending", "Bearer "+routesTok)
assert.Equal(t, http.StatusOK, code)
}
func TestBuildMuxAuthenticatesAnUnregisteredPath(t *testing.T) {
code, _ := do(t, testMux(t), "POST", "/some-future-route", "")
assert.Equal(t, http.StatusUnauthorized, code)
}
// buildMux registers public handlers as given, without adding auth: each
// public route owns its own (bearer on /mcp, HMAC on the webhook, client
// secret on /oauth/token) or is meant to be open (/metrics, OAuth metadata).
func TestBuildMuxDoesNotWrapPublicRoutes(t *testing.T) {
r := testMux(t)
for p := range publicRoutes {
method, path, ok := strings.Cut(p, " ")
if !ok {
method, path = "GET", p
}
code, body := do(t, r, method, path, "")
assert.Equal(t, http.StatusOK, code, p)
assert.Equal(t, "public "+p, body, p)
}
}
// The allowlist is the control: a REST pattern registered as public would win
// over the authenticated catch-all (more specific pattern), so it is refused.
func TestBuildMuxRefusesAPublicPatternOffTheAllowlist(t *testing.T) {
h := api.NewHandler(t.TempDir(), slog.New(slog.NewTextHandler(os.Stderr, nil)), pipeline.Config{})
for _, p := range []string{"POST /query", "GET /pending", "/", "POST /metrics"} {
_, err := buildMux(restAuth{token: routesTok, realm: "brain"}, h, map[string]http.Handler{p: okHandler("x")})
assert.Error(t, err, "public pattern %q must be refused", p)
}
}
func TestBuildMuxRefusesAnEmptyToken(t *testing.T) {
h := api.NewHandler(t.TempDir(), slog.New(slog.NewTextHandler(os.Stderr, nil)), pipeline.Config{})
_, err := buildMux(restAuth{token: "", realm: "brain"}, h, nil)
assert.Error(t, err)
}
-39
View File
@@ -1,39 +0,0 @@
package main
import (
"errors"
"fmt"
"os"
"strconv"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/symlinkguard"
)
// startupSymlinkCheck is the hyperguild#157 tripwire at boot: a symlink in the
// brain dir would let brain readers serve a file from outside the brain, so
// the server does not start until it is removed. A directory the scan cannot
// read could hide one, so it refuses on that too. A missing brain dir (first
// boot) is the only case it lets pass.
func startupSymlinkCheck(brainDir string) error {
r, err := symlinkguard.Scan(brainDir)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return nil
}
return fmt.Errorf("refusing to start: cannot scan the brain dir %s (hyperguild#157): %w", brainDir, err)
}
if !r.Clean() {
return fmt.Errorf("refusing to start: brain dir %s has %s (hyperguild#157); fix them first", brainDir, r)
}
return nil
}
// symlinkScanInterval reads BRAIN_SYMLINK_SCAN_INTERVAL (seconds), falling
// back to the default on an unset, non-numeric or non-positive value.
func symlinkScanInterval(getenv func(string) string) time.Duration {
if s, err := strconv.Atoi(getenv("BRAIN_SYMLINK_SCAN_INTERVAL")); err == nil && s > 0 {
return time.Duration(s) * time.Second
}
return symlinkguard.DefaultInterval
}
@@ -1,65 +0,0 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// hyperguild#157: the server refuses to start while a symlink exists in the
// brain dir, or while any part of it cannot be scanned (which could hide
// one); a missing brain dir is the only thing it lets pass.
func TestStartupSymlinkCheckRefusesABrainWithASymlink(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "planted")))
err := startupSymlinkCheck(brain)
require.Error(t, err)
assert.Contains(t, err.Error(), "planted")
}
func TestStartupSymlinkCheckRefusesABrainWithAnUnreadableDir(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root reads mode-000 directories")
}
brain := t.TempDir()
locked := filepath.Join(brain, "aaa")
require.NoError(t, os.MkdirAll(locked, 0o755))
require.NoError(t, os.Chmod(locked, 0o000))
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
err := startupSymlinkCheck(brain)
require.Error(t, err)
assert.Contains(t, err.Error(), "aaa")
}
func TestStartupSymlinkCheckPassesACleanOrMissingBrain(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(brain, "n.md"), []byte("x"), 0o644))
assert.NoError(t, startupSymlinkCheck(brain))
assert.NoError(t, startupSymlinkCheck(filepath.Join(brain, "missing")))
}
func TestSymlinkScanIntervalFallsBackOnNonPositive(t *testing.T) {
for _, v := range []string{"0", "-5", "", "junk"} {
got := symlinkScanInterval(func(string) string { return v })
assert.Positive(t, got, "env %q", v)
}
}
func TestAlertingWarningsNamesTheSymlinkTripwire(t *testing.T) {
w := strings.Join(alertingWarnings(func(string) string { return "" }), " ")
assert.Contains(t, w, "symlink tripwire")
}
func TestStartupSymlinkCheckRefusesAnUnlistableBrainRoot(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root lists mode-0100 directories")
}
brain := t.TempDir()
require.NoError(t, os.Chmod(brain, 0o100))
t.Cleanup(func() { _ = os.Chmod(brain, 0o755) })
assert.Error(t, startupSymlinkCheck(brain))
}
+10 -23
View File
@@ -1,12 +1,6 @@
package api package api
import ( import "strings"
"strings"
// Aliased: this package's own frontmatter type (below) would collide
// with the imported package's name otherwise.
sharedfm "github.com/mathiasbq/hyperguild/ingestion/internal/frontmatter"
)
// frontmatter is an ordered, line-preserving view of a note's YAML // frontmatter is an ordered, line-preserving view of a note's YAML
// frontmatter block. It deliberately avoids a full YAML round-trip: the // frontmatter block. It deliberately avoids a full YAML round-trip: the
@@ -31,26 +25,19 @@ type fmLine struct {
// frontmatter block is recognised only when the file opens with a `---` // frontmatter block is recognised only when the file opens with a `---`
// fence and a closing `---` fence follows. Otherwise the whole input is // fence and a closing `---` fence follows. Otherwise the whole input is
// the body and the returned frontmatter is empty. // the body and the returned frontmatter is empty.
//
// Fence detection itself is frontmatter.Split (hyperguild#138) — this
// package's own contribution is purely the ordered, line-preserving fmLine
// shape below, which a shared parser has no business owning since it's
// specific to how brain_update does a surgical, byte-round-trip edit.
func parseFrontmatter(src string) (frontmatter, string) { func parseFrontmatter(src string) (frontmatter, string) {
var fm frontmatter var fm frontmatter
blockBytes, restBytes, ok := sharedfm.Split([]byte(src)) if !strings.HasPrefix(src, "---\n") {
if !ok {
return fm, src return fm, src
} }
// sharedfm.Split's block/rest boundary sits one byte earlier than this rest := src[len("---\n"):]
// package's always has: block keeps the "\n" immediately before the end := strings.Index(rest, "\n---\n")
// closing fence, and rest keeps the "\n" immediately after it (both by if end < 0 {
// design, for Fix's own reconstruction). This package's block/body // Opening fence with no closing fence — treat as bodyless content.
// never included either — TrimSuffix/TrimPrefix restore that exact return fm, src
// boundary. Split guarantees ok=true only when "\n---\n" matched, so }
// both trims are always exactly one byte, never ambiguous. block := rest[:end]
block := strings.TrimSuffix(string(blockBytes), "\n") body := rest[end+len("\n---\n"):]
body := strings.TrimPrefix(string(restBytes), "\n")
for _, line := range strings.Split(block, "\n") { for _, line := range strings.Split(block, "\n") {
// An indented line is a continuation (a YAML block-list item, a // An indented line is a continuation (a YAML block-list item, a
+10 -47
View File
@@ -13,6 +13,7 @@ import (
"github.com/mathiasbq/hyperguild/ingestion/internal/brain" "github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/claim" "github.com/mathiasbq/hyperguild/ingestion/internal/claim"
"github.com/mathiasbq/hyperguild/ingestion/internal/extract"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline" "github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
"github.com/mathiasbq/hyperguild/ingestion/internal/search" "github.com/mathiasbq/hyperguild/ingestion/internal/search"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore" "github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
@@ -117,11 +118,6 @@ type WriteNoteOptions struct {
Wing string Wing string
Hall string Hall string
SourceType string // "internal" marks a first-party observation (e.g. claudewatcher) that needs no external citation SourceType string // "internal" marks a first-party observation (e.g. claudewatcher) that needs no external citation
// Origin/WrittenBy are server-owned provenance (hyperguild#145): any
// origin/written_by in Content's own frontmatter is discarded, so a
// caller cannot claim a more trusted origin than it authenticated as.
Origin string
WrittenBy string
} }
// WriteNote writes a markdown note into the brain. Returns the path // WriteNote writes a markdown note into the brain. Returns the path
@@ -173,6 +169,9 @@ func writeHallNote(brainDir string, opts WriteNoteOptions) (string, error) {
return "", fmt.Errorf("claim envelope: %w", err) return "", fmt.Errorf("claim envelope: %w", err)
} }
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
return "", fmt.Errorf("create hall dir: %w", err)
}
emitted := make(map[string]bool, 6) emitted := make(map[string]bool, 6)
var fm strings.Builder var fm strings.Builder
@@ -220,17 +219,7 @@ func writeHallNote(brainDir string, opts WriteNoteOptions) (string, error) {
} }
fm.WriteString("---\n") fm.WriteString("---\n")
final, err := StampProvenance(fm.String()+body, opts.Origin, opts.WrittenBy) if err := os.WriteFile(dest, []byte(fm.String()+body), 0o644); err != nil {
if err != nil {
return "", err
}
// The directory is created only once the note is known to be writable:
// a refused write leaves nothing behind, not even an empty hall dir
// (brain#21; provenance refusal joins the envelope refusal here).
if err := os.MkdirAll(filepath.Dir(dest), 0o755); err != nil {
return "", fmt.Errorf("create hall dir: %w", err)
}
if err := os.WriteFile(dest, []byte(final), 0o644); err != nil {
return "", fmt.Errorf("write: %w", err) return "", fmt.Errorf("write: %w", err)
} }
rel, _ := filepath.Rel(brainDir, dest) rel, _ := filepath.Rel(brainDir, dest)
@@ -306,16 +295,7 @@ func (h *Handler) Write(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "invalid JSON") writeError(w, http.StatusBadRequest, "invalid JSON")
return return
} }
// Field by field, not a struct conversion: provenance is server-owned and relPath, err := WriteNote(h.brainDir, WriteNoteOptions(req))
// this door carries no principal, so it stamps origin: unknown
// (hyperguild#145). A JSON body must never be able to set it.
relPath, err := WriteNote(h.brainDir, WriteNoteOptions{
Content: req.Content,
Filename: req.Filename,
Wing: req.Wing,
Hall: req.Hall,
SourceType: req.SourceType,
})
if err != nil { if err != nil {
h.logger.Error("write failed", "err", err) h.logger.Error("write failed", "err", err)
writeError(w, http.StatusBadRequest, err.Error()) writeError(w, http.StatusBadRequest, err.Error())
@@ -446,12 +426,6 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "path is required") writeError(w, http.StatusBadRequest, "path is required")
return return
} }
resolved, err := ResolveIngestPath(h.brainDir, req.Path)
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
req.Path = resolved
info, err := os.Stat(req.Path) info, err := os.Stat(req.Path)
if err != nil { if err != nil {
@@ -474,13 +448,7 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
if !isSupportedExtension(ext) { if !isSupportedExtension(ext) {
return nil return nil
} }
// Each walked file is re-resolved: a symlinked file inside raw/ content, readErr := extract.Text(path)
// would otherwise lead extract.Text out of the inbox (#149).
if _, rerr := ResolveIngestPath(h.brainDir, path); rerr != nil {
allWarnings = append(allWarnings, fmt.Sprintf("skipped %s: %v", path, rerr))
return nil
}
content, readErr := ReadIngestFile(h.brainDir, path)
if readErr != nil { if readErr != nil {
allWarnings = append(allWarnings, fmt.Sprintf("extract %s: %v", path, readErr)) allWarnings = append(allWarnings, fmt.Sprintf("extract %s: %v", path, readErr))
return nil return nil
@@ -509,7 +477,7 @@ func (h *Handler) IngestPath(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, fmt.Sprintf("unsupported file extension: %s", ext)) writeError(w, http.StatusBadRequest, fmt.Sprintf("unsupported file extension: %s", ext))
return return
} }
content, readErr := ReadIngestFile(h.brainDir, req.Path) content, readErr := extract.Text(req.Path)
if readErr != nil { if readErr != nil {
writeError(w, http.StatusInternalServerError, fmt.Sprintf("extract text: %v", readErr)) writeError(w, http.StatusInternalServerError, fmt.Sprintf("extract text: %v", readErr))
return return
@@ -561,8 +529,7 @@ func (h *Handler) IngestRaw(w http.ResponseWriter, r *http.Request) {
return return
} }
// No principal on this door: pages are stamped origin: unknown (#145). result, err := pipeline.RunRaw(h.brainDir, req.Source, req.Pages, req.DryRun)
result, err := pipeline.RunRaw(r.Context(), h.brainDir, req.Source, req.Pages, req.DryRun)
if err != nil { if err != nil {
h.logger.Error("ingest-raw failed", "source", req.Source, "err", err) h.logger.Error("ingest-raw failed", "source", req.Source, "err", err)
writeError(w, http.StatusInternalServerError, "ingest error") writeError(w, http.StatusInternalServerError, "ingest error")
@@ -618,11 +585,7 @@ func (h *Handler) Promote(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "invalid JSON") writeError(w, http.StatusBadRequest, "invalid JSON")
return return
} }
// Field by field: this door carries no principal, so the promoted note is rel, err := PromoteNote(h.brainDir, PromoteOptions(req))
// stamped origin: unknown and the body cannot set provenance (#145).
rel, err := PromoteNote(h.brainDir, PromoteOptions{
Filename: req.Filename, Wing: req.Wing, Hall: req.Hall, Slug: req.Slug,
})
if err != nil { if err != nil {
writeError(w, http.StatusBadRequest, err.Error()) writeError(w, http.StatusBadRequest, err.Error())
return return
+4 -6
View File
@@ -361,11 +361,10 @@ func TestIngestPath_MissingPath(t *testing.T) {
} }
func TestIngestPath_File(t *testing.T) { func TestIngestPath_File(t *testing.T) {
brainDir, h := setup(t) _, h := setup(t)
// Create a temp file with content // Create a temp file with content
dir := filepath.Join(brainDir, "raw", "fixture") // ingest paths must live in brain/raw/ (#149) dir := t.TempDir()
require.NoError(t, os.MkdirAll(dir, 0o755))
f := filepath.Join(dir, "doc.md") f := filepath.Join(dir, "doc.md")
require.NoError(t, os.WriteFile(f, []byte("# Hello\nThis is markdown content."), 0o644)) require.NoError(t, os.WriteFile(f, []byte("# Hello\nThis is markdown content."), 0o644))
@@ -469,11 +468,10 @@ func TestIngestRaw_DryRun(t *testing.T) {
} }
func TestIngestPath_Directory(t *testing.T) { func TestIngestPath_Directory(t *testing.T) {
brainDir, h := setup(t) _, h := setup(t)
// Create a temp dir with one .md file // Create a temp dir with one .md file
dir := filepath.Join(brainDir, "raw", "fixture") // ingest paths must live in brain/raw/ (#149) dir := t.TempDir()
require.NoError(t, os.MkdirAll(dir, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "notes.md"), []byte("# Notes\nSome notes."), 0o644)) require.NoError(t, os.WriteFile(filepath.Join(dir, "notes.md"), []byte("# Notes\nSome notes."), 0o644))
body, _ := json.Marshal(map[string]any{ body, _ := json.Marshal(map[string]any{
@@ -40,10 +40,9 @@ func TestIngestPath_DocxSupportedWhenDocmarkConfigured(t *testing.T) {
t.Setenv("DOCMARK_URL", srv.URL+"/mcp") t.Setenv("DOCMARK_URL", srv.URL+"/mcp")
t.Setenv("DOCMARK_BEARER_TOKEN", "tok") t.Setenv("DOCMARK_BEARER_TOKEN", "tok")
brainDir, h := setup(t) _, h := setup(t)
dir := filepath.Join(brainDir, "raw", "fixture") // ingest paths must live in brain/raw/ (#149) dir := t.TempDir()
require.NoError(t, os.MkdirAll(dir, 0o755))
f := filepath.Join(dir, "doc.docx") f := filepath.Join(dir, "doc.docx")
require.NoError(t, os.WriteFile(f, []byte("fake docx"), 0o644)) require.NoError(t, os.WriteFile(f, []byte("fake docx"), 0o644))
-67
View File
@@ -1,67 +0,0 @@
package api
import (
"errors"
"os"
"github.com/mathiasbq/hyperguild/ingestion/internal/extract"
"path/filepath"
"strings"
)
// ErrOutsideIngestRoot is the one refusal for every bad ingest path. It never
// carries the underlying error: "no such file" versus "permission denied"
// would let a caller probe which server files exist (PR #152 review).
var ErrOutsideIngestRoot = errors.New("ingest path must be an existing file or directory inside brain/raw/")
// ResolveIngestPath confines a caller-supplied ingest path to brain/raw/, the
// inbox the watcher already owns (hyperguild#149). Relative paths resolve
// against that root; symlinks are evaluated before the containment check, so
// a link inside raw/ that points elsewhere is refused. The path must exist.
func ResolveIngestPath(brainDir, p string) (string, error) {
if strings.TrimSpace(p) == "" {
return "", ErrOutsideIngestRoot
}
// raw/ itself must be a real directory: if it were a symlink, containment
// inside it would mean nothing (hyperguild#153 item 4).
if fi, lerr := os.Lstat(filepath.Join(brainDir, "raw")); lerr != nil || fi.Mode()&os.ModeSymlink != 0 || !fi.IsDir() {
return "", ErrOutsideIngestRoot
}
root, err := filepath.EvalSymlinks(filepath.Join(brainDir, "raw"))
if err != nil {
return "", ErrOutsideIngestRoot
}
if !filepath.IsAbs(p) {
p = filepath.Join(root, p)
}
real, err := filepath.EvalSymlinks(filepath.Clean(p))
if err != nil {
return "", ErrOutsideIngestRoot
}
rel, err := filepath.Rel(root, real)
if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) || filepath.IsAbs(rel) {
return "", ErrOutsideIngestRoot
}
return real, nil
}
// ReadIngestFile extracts text from an ingest path already confined by
// ResolveIngestPath, reading it through the raw/ root rather than reopening
// the path by name (hyperguild#153 item 3): if the file is swapped for a
// symlink after the check, the read still cannot leave raw/.
func ReadIngestFile(brainDir, resolved string) (string, error) {
root, rawPath, err := OpenRawRoot(brainDir)
if err != nil {
return "", ErrOutsideIngestRoot
}
defer func() { _ = root.Close() }()
realRoot, err := filepath.EvalSymlinks(rawPath)
if err != nil {
return "", ErrOutsideIngestRoot
}
rel, err := filepath.Rel(realRoot, resolved)
if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) || filepath.IsAbs(rel) {
return "", ErrOutsideIngestRoot
}
return extract.TextIn(root, rel)
}
@@ -1,88 +0,0 @@
package api_test
import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// hyperguild#149: /ingest-path and MCP brain_ingest read a caller-chosen
// server path, so any file the pod could read (the ServiceAccount token,
// /etc/passwd) could be pulled into the brain and read back out. Ingest paths
// now resolve inside brain/raw/ — the inbox the watcher already owns — with
// symlinks evaluated, or are refused.
func TestResolveIngestPathConfinesToRawInbox(t *testing.T) {
brain := t.TempDir()
raw := filepath.Join(brain, "raw")
require.NoError(t, os.MkdirAll(filepath.Join(raw, "sub"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(raw, "sub", "ok.md"), []byte("x"), 0o644))
outside := filepath.Join(t.TempDir(), "secret.txt")
require.NoError(t, os.WriteFile(outside, []byte("s"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(raw, "link.md")))
require.NoError(t, os.Symlink(filepath.Dir(outside), filepath.Join(raw, "dirlink")))
for _, ok := range []string{filepath.Join(raw, "sub", "ok.md"), "sub/ok.md", raw, "sub"} {
_, err := api.ResolveIngestPath(brain, ok)
assert.NoError(t, err, ok)
}
for _, bad := range []string{
"/etc/hostname", outside, "../secret.txt", filepath.Join(raw, "..", "wiki"),
filepath.Join(raw, "link.md"), filepath.Join(raw, "dirlink", "secret.txt"),
raw + "-evil/x.md", "",
} {
_, err := api.ResolveIngestPath(brain, bad)
assert.Error(t, err, "must refuse %q", bad)
}
}
func TestIngestPathRefusesAPathOutsideTheInbox(t *testing.T) {
_, h := setup(t)
body, _ := json.Marshal(map[string]any{"path": "/etc/hostname"})
rec := httptest.NewRecorder()
h.IngestPath(rec, httptest.NewRequest(http.MethodPost, "/ingest-path", bytes.NewReader(body)))
assert.Equal(t, http.StatusBadRequest, rec.Code)
assert.Contains(t, rec.Body.String(), "raw")
}
// A directory ingest must not follow a symlinked FILE inside raw/ out of it.
func TestIngestPathDirectoryWalkSkipsSymlinksLeavingTheInbox(t *testing.T) {
dir, h := setup(t)
raw := filepath.Join(dir, "raw", "batch")
require.NoError(t, os.MkdirAll(raw, 0o755))
outside := filepath.Join(t.TempDir(), "secret.md")
require.NoError(t, os.WriteFile(outside, []byte("# secret\n\nSENTINEL-149-OUTSIDE"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(raw, "leak.md")))
body, _ := json.Marshal(map[string]any{"path": "batch", "source": "walk"})
rec := httptest.NewRecorder()
h.IngestPath(rec, httptest.NewRequest(http.MethodPost, "/ingest-path", bytes.NewReader(body)))
assert.Contains(t, rec.Body.String(), "skipped")
assert.NotContains(t, rec.Body.String(), "SENTINEL-149-OUTSIDE")
}
// PR #152 review: the raw EvalSymlinks error told an authenticated caller
// whether any server file existed ("no such file") or was unreadable
// ("permission denied"), and leaked the brain directory. Every refusal now
// reads the same.
func TestResolveIngestPathRefusalsAreIndistinguishable(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(brain, "raw"), 0o755))
var msgs []string
for _, p := range []string{"/etc/hostname", "/etc/definitely-not-here-149", "../x", "nope/missing.md", ""} {
_, err := api.ResolveIngestPath(brain, p)
require.Error(t, err, p)
msgs = append(msgs, err.Error())
assert.NotContains(t, err.Error(), brain, "refusal for %q leaks the brain dir", p)
}
for _, m := range msgs[1:] {
assert.Equal(t, msgs[0], m)
}
}
+21 -46
View File
@@ -1,9 +1,7 @@
package api package api
import ( import (
"errors"
"fmt" "fmt"
"io/fs"
"os" "os"
"path/filepath" "path/filepath"
"regexp" "regexp"
@@ -30,25 +28,25 @@ var datePrefix = regexp.MustCompile(`^\d{4}-\d{2}-\d{2}-`)
// (natural review order). An absent raw/ dir yields an empty slice, not an // (natural review order). An absent raw/ dir yields an empty slice, not an
// error. Only .md files are listed; tunnel-candidate files are skipped. // error. Only .md files are listed; tunnel-candidate files are skipped.
func ListPending(brainDir string) ([]PendingNote, error) { func ListPending(brainDir string) ([]PendingNote, error) {
root, _, err := OpenRawRoot(brainDir) dir := filepath.Join(brainDir, "raw")
entries, err := os.ReadDir(dir)
if err != nil { if err != nil {
if errors.Is(err, os.ErrNotExist) { if os.IsNotExist(err) {
return []PendingNote{}, nil return []PendingNote{}, nil
} }
return nil, fmt.Errorf("read raw dir: %w", err) return nil, fmt.Errorf("read raw dir: %w", err)
} }
defer func() { _ = root.Close() }()
entries, err := fs.ReadDir(root.FS(), ".")
if err != nil {
return nil, fmt.Errorf("read raw dir: %w", err)
}
out := make([]PendingNote, 0, len(entries)) out := make([]PendingNote, 0, len(entries))
for _, e := range entries { for _, e := range entries {
if !e.Type().IsRegular() || !strings.HasSuffix(e.Name(), ".md") || strings.HasPrefix(e.Name(), "tunnel-candidates-") { if e.IsDir() || !strings.HasSuffix(e.Name(), ".md") || strings.HasPrefix(e.Name(), "tunnel-candidates-") {
continue // directories, and symlinks whose target may be anywhere (#153) continue
} }
raw, info, readErr := readRawNote(root, e.Name()) info, statErr := e.Info()
if statErr != nil {
continue
}
raw, readErr := os.ReadFile(filepath.Join(dir, e.Name()))
if readErr != nil { if readErr != nil {
continue continue
} }
@@ -74,10 +72,6 @@ type PromoteOptions struct {
Wing string Wing string
Hall string Hall string
Slug string // optional; defaults to Filename minus date prefix + .md Slug string // optional; defaults to Filename minus date prefix + .md
// Origin/WrittenBy re-attribute the promoted note to the promoter: the raw
// note's own frontmatter claims are not provenance (hyperguild#145).
Origin string
WrittenBy string
} }
// PromoteNote moves a note from brain/raw/ into the structured wiki: it // PromoteNote moves a note from brain/raw/ into the structured wiki: it
@@ -107,9 +101,12 @@ func PromoteNote(brainDir string, opts PromoteOptions) (string, error) {
} }
src := filepath.Join(brainDir, "raw", base) src := filepath.Join(brainDir, "raw", base)
raw, err := readRawSource(brainDir, base) raw, err := os.ReadFile(src)
if err != nil { if err != nil {
return "", err if os.IsNotExist(err) {
return "", fmt.Errorf("pending note %q does not exist in raw/", base)
}
return "", fmt.Errorf("read source: %w", err)
} }
// Collision: never silently overwrite an existing promoted note. // Collision: never silently overwrite an existing promoted note.
@@ -131,11 +128,7 @@ func PromoteNote(brainDir string, opts PromoteOptions) (string, error) {
return "", fmt.Errorf("create wing dir: %w", err) return "", fmt.Errorf("create wing dir: %w", err)
} }
// Write-then-delete: the source survives any write failure. // Write-then-delete: the source survives any write failure.
promoted, err := StampProvenance(fm.render()+body, opts.Origin, opts.WrittenBy) if err := os.WriteFile(dest, []byte(fm.render()+body), 0o644); err != nil {
if err != nil {
return "", err
}
if err := os.WriteFile(dest, []byte(promoted), 0o644); err != nil {
return "", fmt.Errorf("write promoted note: %w", err) return "", fmt.Errorf("write promoted note: %w", err)
} }
if err := os.Remove(src); err != nil { if err := os.Remove(src); err != nil {
@@ -179,9 +172,12 @@ func RejectNote(brainDir string, opts RejectOptions) (string, error) {
} }
src := filepath.Join(brainDir, "raw", base) src := filepath.Join(brainDir, "raw", base)
raw, err := readRawSource(brainDir, base) raw, err := os.ReadFile(src)
if err != nil { if err != nil {
return "", err if os.IsNotExist(err) {
return "", fmt.Errorf("pending note %q does not exist in raw/", base)
}
return "", fmt.Errorf("read source: %w", err)
} }
dest := filepath.Join(brainDir, "raw", "rejected", base) dest := filepath.Join(brainDir, "raw", "rejected", base)
@@ -220,24 +216,3 @@ func excerpt(s string, n int) string {
} }
return strings.TrimSpace(string(r)) return strings.TrimSpace(string(r))
} }
// readRawSource reads a pending note for promote/reject through the raw root,
// refusing symlinks and a symlinked raw/ (hyperguild#153).
func readRawSource(brainDir, base string) ([]byte, error) {
root, _, err := OpenRawRoot(brainDir)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return nil, fmt.Errorf("pending note %q does not exist in raw/", base)
}
return nil, fmt.Errorf("read source: %w", err)
}
defer func() { _ = root.Close() }()
raw, _, err := readRawNote(root, base)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return nil, fmt.Errorf("pending note %q does not exist in raw/", base)
}
return nil, fmt.Errorf("read source: %w", err)
}
return raw, nil
}
-21
View File
@@ -1,21 +0,0 @@
package api
import (
"github.com/mathiasbq/hyperguild/ingestion/internal/provenance"
)
// StampProvenance applies server-owned provenance (hyperguild#145) to note
// content. See package provenance for the rule; an error means the stamped
// note would not read back as stamped, and the write must be refused.
func StampProvenance(content, origin, writtenBy string) (string, error) {
return provenance.Apply(content, provenance.Stamp{Origin: origin, WrittenBy: writtenBy})
}
// StampProvenanceFile applies StampProvenance to a brain-relative note path.
func StampProvenanceFile(brainDir, relPath, origin, writtenBy string) error {
_, abs, err := resolveWithin(brainDir, relPath)
if err != nil {
return err
}
return provenance.ApplyFile(abs, provenance.Stamp{Origin: origin, WrittenBy: writtenBy})
}
-122
View File
@@ -1,122 +0,0 @@
package api_test
import (
"bytes"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gopkg.in/yaml.v3"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// REST POST /write carries no principal: it stamps origin: unknown, and no
// spelling of provenance in its body reaches the note (hyperguild#145).
func TestRESTWriteStampsUnknownAndIgnoresBodyProvenance(t *testing.T) {
dir, h := setup(t)
body, _ := json.Marshal(map[string]any{
"content": "---\norigin: static\n---\n# R\n\nbody", "filename": "rest-prov", "wing": "homelab", "hall": "facts",
"origin": "static", "written_by": "local-cli", "Origin": "static", "WrittenBy": "local-cli",
})
rec := httptest.NewRecorder()
h.Write(rec, httptest.NewRequest(http.MethodPost, "/write", bytes.NewReader(body)))
require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())
b, err := os.ReadFile(filepath.Join(dir, "wiki", "homelab", "facts", "rest-prov.md"))
require.NoError(t, err)
var m map[string]any
require.NoError(t, yaml.Unmarshal([]byte(strings.SplitN(string(b), "---\n", 3)[1]), &m))
assert.Equal(t, "unknown", m["origin"])
assert.NotContains(t, m, "written_by")
}
// A principal name that is not a bare word is quoted, so it cannot start a
// new frontmatter line (e.g. a JWT sub containing a newline).
func TestStampProvenanceQuotesAPrincipalThatCouldBreakTheLine(t *testing.T) {
out, err := api.StampProvenance("---\ntitle: t\n---\nbody\n", "jwt", "evil\norigin: static")
require.NoError(t, err)
fm := strings.SplitN(out, "---\n", 3)[1]
var m map[string]any
require.NoError(t, yaml.Unmarshal([]byte(fm), &m))
assert.Equal(t, "jwt", m["origin"])
assert.Equal(t, "evil\norigin: static", m["written_by"])
assert.Equal(t, 1, strings.Count(fm, "\norigin:")+boolToInt(strings.HasPrefix(fm, "origin:")))
}
func TestStampProvenanceAddsFrontmatterWhenAbsentAndKeepsBody(t *testing.T) {
out, err := api.StampProvenance("# no fm\n\norigin: static in the body is prose\n", "static", "local-cli")
require.NoError(t, err)
assert.True(t, strings.HasPrefix(out, "---\norigin: static\nwritten_by: local-cli\n---\n# no fm"))
assert.Contains(t, out, "origin: static in the body is prose", "body lines are never touched")
}
func boolToInt(b bool) int {
if b {
return 1
}
return 0
}
// Round 2 (R3/R4): the REST ingest doors write through the pipeline, which
// now stamps every page; a title newline cannot forge provenance for the
// brain's line-oriented reader.
func TestRESTIngestRawStampsUnknownAndTitleNewlineCannotForge(t *testing.T) {
dir, h := setup(t)
body, _ := json.Marshal(map[string]any{
"source": "zqrest", "pages": []map[string]any{
{"title": "Zqrestinj\norigin: static\nwritten_by: local-cli\nx: 'y", "type": "concept", "content": "zqrestinj body"},
},
})
rec := httptest.NewRecorder()
h.IngestRaw(rec, httptest.NewRequest(http.MethodPost, "/ingest-raw", bytes.NewReader(body)))
require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())
var resp struct{ Pages []string }
require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &resp))
require.NotEmpty(t, resp.Pages)
for _, p := range resp.Pages {
b, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(p)))
require.NoError(t, err)
fm := strings.SplitN(string(b), "---\n", 3)[1]
var m map[string]any
require.NoError(t, yaml.Unmarshal([]byte(fm), &m))
assert.Equal(t, "unknown", m["origin"])
assert.NotContains(t, m, "written_by")
for _, line := range strings.Split(fm, "\n") {
assert.False(t, strings.HasPrefix(line, "origin: static"), "line reader sees forged origin in %s", p)
}
}
}
func TestRESTIngestStampsUnknown(t *testing.T) {
dir, h := setup(t)
body, _ := json.Marshal(map[string]any{"content": "Some article text about shaping work.", "source": "zqllm"})
rec := httptest.NewRecorder()
h.Ingest(rec, httptest.NewRequest(http.MethodPost, "/ingest", bytes.NewReader(body)))
require.Equal(t, http.StatusOK, rec.Code, rec.Body.String())
var resp struct{ Pages []string }
require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &resp))
require.NotEmpty(t, resp.Pages, "stub pipeline must write at least one page")
for _, p := range resp.Pages {
b, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(p)))
require.NoError(t, err)
assert.Contains(t, strings.SplitN(string(b), "---\n", 3)[1], "origin: unknown\n", p)
}
}
// A refused write leaves nothing behind, not even an empty hall directory
// (brain#21's invariant, round 3 non-blocking finding).
func TestWriteNoteRefusedForProvenanceLeavesNoDirectory(t *testing.T) {
dir := t.TempDir()
_, err := api.WriteNote(dir, api.WriteNoteOptions{
Content: "# x\n\nbody", Wing: "freshwing", Hall: "facts", Filename: "x", Origin: "jwt", WrittenBy: "bad\xff",
})
require.Error(t, err)
_, statErr := os.Stat(filepath.Join(dir, "wiki", "freshwing"))
assert.True(t, os.IsNotExist(statErr), "no wing directory may be created by a refused write")
}
-178
View File
@@ -1,178 +0,0 @@
package api_test
import (
"encoding/base64"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"syscall"
"testing"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// hyperguild#153 / PR #156: behavioural race tests, so the control does not
// rest on lints that list forbidden APIs (round 3: an os.ReadFile or
// Root.ReadFile rewrite passed the lint and leaked 19k times in 4s). A
// swapper keeps renaming a regular note, a symlink to an outside file, or a
// FIFO into raw/r.md while the public readers loop. Any leaked byte fails;
// any hang fails.
const raceSentinel = "SENTINEL-153-RACE"
// startSwapper runs fn until the test ends, and waits for it to exit before
// t.TempDir's cleanup removes the directory it writes into (hyperguild#162:
// an unawaited swapper raced the cleanup, "directory not empty").
func startSwapper(t *testing.T, fn func(stop *atomic.Bool)) {
t.Helper()
var stop atomic.Bool
var wg sync.WaitGroup
wg.Add(1)
go func() { defer wg.Done(); fn(&stop) }()
t.Cleanup(func() { stop.Store(true); wg.Wait() })
}
func swap(raw, outside string, fifo bool, stop *atomic.Bool) {
tgt := filepath.Join(raw, "r.md")
for i := 0; !stop.Load(); i++ {
tmp := filepath.Join(raw, ".swap")
_ = os.Remove(tmp)
switch {
case i%2 == 0:
_ = os.WriteFile(tmp, []byte("# r\n\ninside"), 0o644)
case fifo:
_ = syscall.Mkfifo(tmp, 0o644)
default:
_ = os.Symlink(outside, tmp)
}
_ = os.Rename(tmp, tgt)
}
}
func raceBrain(t *testing.T) (brain, raw, outside string) {
t.Helper()
brain = t.TempDir()
raw = filepath.Join(brain, "raw")
if err := os.MkdirAll(raw, 0o755); err != nil {
t.Fatal(err)
}
outside = filepath.Join(t.TempDir(), "secret.md")
if err := os.WriteFile(outside, []byte("# s\n\n"+raceSentinel), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(raw, "r.md"), []byte("# r\n\ninside"), 0o644); err != nil {
t.Fatal(err)
}
return brain, raw, outside
}
// readers runs every public raw/ reader once and returns all text they
// produced (excerpts, extracted text).
func readers(brain, raw string) string {
var b strings.Builder
if ns, err := api.ListPending(brain); err == nil {
for _, n := range ns {
b.WriteString(n.Excerpt)
}
}
if s, err := api.ReadIngestFile(brain, filepath.Join(raw, "r.md")); err == nil {
b.WriteString(s)
}
return b.String()
}
func TestRawReadersLeakNothingUnderASymlinkSwapRace(t *testing.T) {
brain, raw, outside := raceBrain(t)
startSwapper(t, func(stop *atomic.Bool) { swap(raw, outside, false, stop) })
leaks, iters := 0, 0
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); iters++ {
leaks += strings.Count(readers(brain, raw), raceSentinel)
}
if leaks > 0 {
t.Fatalf("outside file leaked %d times in %d iterations", leaks, iters)
}
t.Logf("%d iterations, 0 leaks", iters)
}
func TestRawReadersDoNotHangUnderAFIFOSwapRace(t *testing.T) {
brain, raw, outside := raceBrain(t)
startSwapper(t, func(stop *atomic.Bool) { swap(raw, outside, true, stop) })
var iters atomic.Int64
done := make(chan struct{})
go func() {
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); {
_ = readers(brain, raw)
iters.Add(1)
}
close(done)
}()
select {
case <-done:
case <-time.After(6 * time.Second):
// Unblock the stuck open so the test process can exit.
if f, err := os.OpenFile(filepath.Join(raw, "r.md"), os.O_WRONLY|syscall.O_NONBLOCK, 0); err == nil {
_ = f.Close()
}
t.Fatalf("a raw/ reader hung on a swapped-in FIFO after %d iterations", iters.Load())
}
}
// PR #156 round 4: the pdf/docmark branch of TextIn copies the file through
// the root to a temp file; pointing the extractor back at the raw/ path by
// name stayed green because the race tests only used r.md (a stub docmark
// then received the outside file's bytes 34k times in 4s). This races a
// symlink into raw/r.docx and has a fake docmark count the sentinel in every
// document it is sent.
func TestDocmarkExtractionLeaksNothingUnderASymlinkSwapRace(t *testing.T) {
var seen atomic.Int64
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var req struct {
Params struct {
Arguments struct {
ContentBase64 string `json:"content_base64"`
} `json:"arguments"`
} `json:"params"`
}
_ = json.NewDecoder(r.Body).Decode(&req)
if b, err := base64.StdEncoding.DecodeString(req.Params.Arguments.ContentBase64); err == nil &&
strings.Contains(string(b), raceSentinel) {
seen.Add(1)
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"jsonrpc":"2.0","id":1,"result":{"content":[{"type":"text","text":"converted"}]}}`))
}))
defer srv.Close()
t.Setenv("DOCMARK_URL", srv.URL)
brain, raw, outside := raceBrain(t)
tgt := filepath.Join(raw, "r.docx")
if err := os.WriteFile(tgt, []byte("inside docx bytes"), 0o644); err != nil {
t.Fatal(err)
}
startSwapper(t, func(stop *atomic.Bool) {
for i := 0; !stop.Load(); i++ {
tmp := filepath.Join(raw, ".swapdocx")
_ = os.Remove(tmp)
if i%2 == 0 {
_ = os.WriteFile(tmp, []byte("inside docx bytes"), 0o644)
} else {
_ = os.Symlink(outside, tmp)
}
_ = os.Rename(tmp, tgt)
}
})
iters := 0
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); iters++ {
_, _ = api.ReadIngestFile(brain, tgt)
}
if n := seen.Load(); n > 0 {
t.Fatalf("docmark received the outside file %d times in %d iterations", n, iters)
}
t.Logf("%d iterations, 0 leaks to docmark", iters)
}
@@ -1,80 +0,0 @@
package api_test
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// hyperguild#153: readers of brain/raw/ followed symlinks. /pending showed the
// target's contents and promote would copy it into wiki/ (and brain-sync to
// Gitea). raw/ is writable via Syncthing, so a symlink there is reachable
// input. Every raw/ reader now treats a symlink as not-a-note.
const sentinel153 = "SENTINEL-153-OUTSIDE"
func rawWithSymlink(t *testing.T) (brain string) {
t.Helper()
brain = t.TempDir()
raw := filepath.Join(brain, "raw")
require.NoError(t, os.MkdirAll(raw, 0o755))
outside := filepath.Join(t.TempDir(), "secret.md")
require.NoError(t, os.WriteFile(outside, []byte("---\ncreated_at: 2026-01-01T00:00:00Z\n---\n# s\n\n"+sentinel153), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(raw, "2026-01-01-leak.md")))
require.NoError(t, os.WriteFile(filepath.Join(raw, "2026-01-02-real.md"), []byte("# real\n\nreal body"), 0o644))
return brain
}
func TestListPendingSkipsSymlinkedNotes(t *testing.T) {
notes, err := api.ListPending(rawWithSymlink(t))
require.NoError(t, err)
var names []string
for _, n := range notes {
names = append(names, n.Filename)
assert.NotContains(t, n.Excerpt, sentinel153)
}
assert.Equal(t, []string{"2026-01-02-real.md"}, names)
}
func TestPromoteNoteRefusesASymlinkedSource(t *testing.T) {
brain := rawWithSymlink(t)
_, err := api.PromoteNote(brain, api.PromoteOptions{Filename: "2026-01-01-leak.md", Wing: "testing", Hall: "facts"})
require.Error(t, err)
_, statErr := os.Stat(filepath.Join(brain, "wiki", "testing", "facts", "leak.md"))
assert.True(t, os.IsNotExist(statErr), "nothing may be promoted from a symlink")
}
func TestRejectNoteRefusesASymlinkedSource(t *testing.T) {
brain := rawWithSymlink(t)
_, err := api.RejectNote(brain, api.RejectOptions{Filename: "2026-01-01-leak.md"})
require.Error(t, err)
b, _ := os.ReadFile(filepath.Join(brain, "raw", "rejected", "2026-01-01-leak.md"))
assert.False(t, strings.Contains(string(b), sentinel153))
}
// Item 4: if raw/ itself is a symlink, containment inside it means nothing.
func TestResolvePendingPromoteRejectAndReadIngestRefuseASymlinkedRawRoot(t *testing.T) {
brain := t.TempDir()
elsewhere := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(elsewhere, "x.md"), []byte("# x\n\n"+sentinel153), 0o644))
require.NoError(t, os.Symlink(elsewhere, filepath.Join(brain, "raw")))
_, err := api.ResolveIngestPath(brain, "x.md")
assert.Error(t, err, "ingest must refuse a symlinked raw/")
notes, err := api.ListPending(brain)
assert.Error(t, err, "pending must refuse a symlinked raw/")
assert.Empty(t, notes)
_, err = api.PromoteNote(brain, api.PromoteOptions{Filename: "x.md", Wing: "testing", Hall: "facts"})
assert.Error(t, err, "promote must refuse a symlinked raw/")
_, err = api.RejectNote(brain, api.RejectOptions{Filename: "x.md"})
assert.Error(t, err, "reject must refuse a symlinked raw/")
text, err := api.ReadIngestFile(brain, filepath.Join(elsewhere, "x.md"))
assert.Error(t, err, "ReadIngestFile must refuse a symlinked raw/")
assert.NotContains(t, text, sentinel153)
// The watcher's refusal is tested in internal/watcher.
}
-67
View File
@@ -1,67 +0,0 @@
package api
import (
"errors"
"fmt"
"io"
"os"
"path/filepath"
"github.com/mathiasbq/hyperguild/ingestion/internal/extract"
)
// errNotANote is returned for a raw/ entry that is not a regular file — in
// particular a symlink, whose target may be anywhere (hyperguild#153).
var errNotANote = errors.New("not a regular note file in raw/ (symlinks are refused)")
// errRawNotDir means brain/raw is not a real directory (e.g. a symlink), so
// containment inside it would mean nothing (hyperguild#153 item 4).
var errRawNotDir = errors.New("brain/raw is not a real directory")
// OpenRawRoot opens brain/raw as an os.Root, refusing a raw/ that is itself a
// symlink. Reads through the root cannot leave it, even if an entry is
// swapped for a symlink between a check and the read. It returns the root
// and raw/'s path. A missing raw/ returns an error wrapping os.ErrNotExist.
func OpenRawRoot(brainDir string) (*os.Root, string, error) {
p := filepath.Join(brainDir, "raw")
fi, err := os.Lstat(p)
if err != nil {
return nil, "", err
}
if fi.Mode()&os.ModeSymlink != 0 || !fi.IsDir() {
return nil, "", errRawNotDir
}
r, err := os.OpenRoot(p)
if err != nil {
return nil, "", fmt.Errorf("open raw root: %w", err)
}
return r, p, nil
}
// readRawNote reads one raw/ entry by basename through the root, refusing
// anything that is not a regular file.
//
// Accepted residual: a HARD link to an outside file is a regular file and is
// read. Syncthing syncs contents, not links, so a peer cannot create one;
// fs.protected_hardlinks=1 and the pod's secrets live on a different
// filesystem from the brain volume, so a local caller cannot link them in.
// Refusing Nlink > 1 is the next step if either assumption changes.
func readRawNote(r *os.Root, base string) ([]byte, os.FileInfo, error) {
fi, err := r.Lstat(base)
if err != nil {
return nil, nil, err
}
if !fi.Mode().IsRegular() {
return nil, nil, errNotANote
}
f, _, err := extract.OpenRegular(r, base)
if err != nil {
return nil, nil, err
}
defer func() { _ = f.Close() }()
b, err := io.ReadAll(f)
if err != nil {
return nil, nil, err
}
return b, fi, nil
}
@@ -1,43 +0,0 @@
package api_test
import (
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs/brainfstest"
)
// hyperguild#157 slice 2: ReadNote (brain_get, read-after-write) read by
// path; it now reads through brainfs.
func TestReadNoteRefusesSymlinksAndSurvivesSwapRaces(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(facts, "leak.md")))
_, body, _, err := api.ReadNote(brain, "wiki/w/facts/leak.md")
assert.Error(t, err)
assert.NotContains(t, body, brainfstest.Sentinel)
p := filepath.Join(facts, "r.md")
require.NoError(t, os.WriteFile(p, []byte("---\nx: y\n---\ninside"), 0o644))
fm, body, _, err := api.ReadNote(brain, "wiki/w/facts/r.md")
require.NoError(t, err)
assert.Equal(t, "y", fm["x"])
assert.Contains(t, body, "inside")
read := func() string {
fm, body, _, _ := api.ReadNote(brain, "wiki/w/facts/r.md")
s := body
for k, v := range fm {
s += k + v
}
return s
}
brainfstest.AssertNoLeak(t, p, "---\nx: y\n---\ninside", 1500*time.Millisecond, read)
brainfstest.AssertNoHang(t, p, "---\nx: y\n---\ninside", 1500*time.Millisecond, 6*time.Second, func() { _ = read() })
}
+9 -51
View File
@@ -3,18 +3,14 @@ package api
import ( import (
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
"errors"
"fmt" "fmt"
"io/fs"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
"time" "time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain" "github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"github.com/mathiasbq/hyperguild/ingestion/internal/claim" "github.com/mathiasbq/hyperguild/ingestion/internal/claim"
"github.com/mathiasbq/hyperguild/ingestion/internal/notes"
) )
// ContentHash returns the lowercase hex sha256 of b. It is the note's // ContentHash returns the lowercase hex sha256 of b. It is the note's
@@ -50,11 +46,6 @@ type UpdateNoteOptions struct {
Slug string Slug string
Content string // new full body (whole-note replace) Content string // new full body (whole-note replace)
Reason string // optional; stamped as supersede_reason Reason string // optional; stamped as supersede_reason
// Origin/WrittenBy re-stamp provenance to the updating principal: the
// body is replaced wholesale, so the last writer is the author of what
// a reader sees (hyperguild#145). Never taken from Content's fragment.
Origin string
WrittenBy string
} }
// UpdateNote supersedes an existing note in place. It replaces the body // UpdateNote supersedes an existing note in place. It replaces the body
@@ -94,23 +85,14 @@ func UpdateNote(brainDir string, opts UpdateNoteOptions) (relPath, contentHash,
rel = filepath.ToSlash(rel) rel = filepath.ToSlash(rel)
} }
// The path is caller-supplied (brain_update): resolve it as brain_edit rel, abs, err := resolveWithin(brainDir, rel)
// does, knowledge/ and wiki/ notes only, and read and rewrite it through
// the brain root, never through a symlink (hyperguild#169).
t, err := notes.Resolve(brainDir, rel, notes.ScopeNotes)
if err != nil { if err != nil {
return "", "", "", err return "", "", "", err
} }
rel = t.Rel
bfs, err := brainfs.Open(brainDir)
if err != nil {
return "", "", "", fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
prior, err := bfs.ReadRegular(rel) prior, err := os.ReadFile(abs)
if err != nil { if err != nil {
if errors.Is(err, fs.ErrNotExist) { if os.IsNotExist(err) {
return "", "", "", fmt.Errorf("note %q does not exist: use brain_write to create", rel) return "", "", "", fmt.Errorf("note %q does not exist: use brain_write to create", rel)
} }
return "", "", "", fmt.Errorf("read target: %w", err) return "", "", "", fmt.Errorf("read target: %w", err)
@@ -128,23 +110,9 @@ func UpdateNote(brainDir string, opts UpdateNoteOptions) (relPath, contentHash,
fm, _ := parseFrontmatter(string(prior)) fm, _ := parseFrontmatter(string(prior))
for _, l := range contentFM.lines { for _, l := range contentFM.lines {
// updated_at/supersedes/supersede_reason are this function's own if l.key != "" {
// output, not something a caller's fragment sets — updated_at and fm.set(l.key, l.value)
// supersedes are already unconditionally re-set below regardless of
// what the loop does with them, but supersede_reason is only
// re-set when opts.Reason is non-empty, so without this skip a
// caller could set it via its own fragment and have it survive
// even when Reason is empty, bypassing the one intended entry
// point for that field (hyperguild#125, found in #111's review).
// Skipping all three here rather than relying on the unconditional
// overwrites below to save the other two keeps the invariant
// explicit at the one place it can be silently broken by a future
// reordering, not implicit in call order.
if l.key == "" || l.key == "updated_at" || l.key == "supersedes" || l.key == "supersede_reason" ||
l.key == "origin" || l.key == "written_by" {
continue
} }
fm.set(l.key, l.value)
} }
fm.set("updated_at", time.Now().UTC().Format(time.RFC3339)) fm.set("updated_at", time.Now().UTC().Format(time.RFC3339))
fm.set("supersedes", priorHash) fm.set("supersedes", priorHash)
@@ -171,12 +139,8 @@ func UpdateNote(brainDir string, opts UpdateNoteOptions) (relPath, contentHash,
return "", "", "", fmt.Errorf("claim envelope: %w", err) return "", "", "", fmt.Errorf("claim envelope: %w", err)
} }
stamped, err := StampProvenance(fm.render()+contentBody, opts.Origin, opts.WrittenBy) out := []byte(fm.render() + contentBody)
if err != nil { if err := os.WriteFile(abs, out, 0o644); err != nil {
return "", "", "", err
}
out := []byte(stamped)
if err := bfs.WriteRegular(rel, out); err != nil {
return "", "", "", fmt.Errorf("write: %w", err) return "", "", "", fmt.Errorf("write: %w", err)
} }
return rel, ContentHash(out), priorHash, nil return rel, ContentHash(out), priorHash, nil
@@ -187,17 +151,11 @@ func UpdateNote(brainDir string, opts UpdateNoteOptions) (relPath, contentHash,
// write primitive behind brain_get: the hash it returns equals the hash // write primitive behind brain_get: the hash it returns equals the hash
// brain_write / brain_update returned for the same bytes. // brain_write / brain_update returned for the same bytes.
func ReadNote(brainDir, relPath string) (fm map[string]string, body, contentHash string, err error) { func ReadNote(brainDir, relPath string) (fm map[string]string, body, contentHash string, err error) {
rel, _, err := resolveWithin(brainDir, relPath) _, abs, err := resolveWithin(brainDir, relPath)
if err != nil { if err != nil {
return nil, "", "", err return nil, "", "", err
} }
// Through the brain root, regular files only (hyperguild#157). raw, err := os.ReadFile(abs)
bfs, err := brainfs.Open(brainDir)
if err != nil {
return nil, "", "", fmt.Errorf("read note: %w", err)
}
defer func() { _ = bfs.Close() }()
raw, err := bfs.ReadRegular(rel)
if err != nil { if err != nil {
if os.IsNotExist(err) { if os.IsNotExist(err) {
return nil, "", "", fmt.Errorf("note %q does not exist", relPath) return nil, "", "", fmt.Errorf("note %q does not exist", relPath)
-112
View File
@@ -1,112 +0,0 @@
package api_test
import (
"os"
"path/filepath"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
)
// brain_update passed the caller's path through resolveWithin, a lexical
// containment check only: any file in the brain could be superseded
// (classification.yaml, raw/, sessions/), and a final symlink was written
// through. It now resolves like brain_edit: knowledge/ and wiki/ notes only,
// read and rewritten through the brain root.
func TestUpdateNoteRefusesPathsOutsideKnowledgeAndWikiNotes(t *testing.T) {
brain := t.TempDir()
files := map[string]string{
"classification.yaml": "levels: {}\n",
"raw/r.md": "raw\n",
"sessions/s.jsonl": "{}\n",
"wiki/w/facts/ok.md": "---\ntitle: ok\n---\nold\n",
"knowledge/k.md": "---\ntitle: k\n---\nold\n",
"wiki/w/facts/x.txt": "txt\n",
}
for rel, body := range files {
p := filepath.Join(brain, filepath.FromSlash(rel))
require.NoError(t, os.MkdirAll(filepath.Dir(p), 0o755))
require.NoError(t, os.WriteFile(p, []byte(body), 0o644))
}
for _, rel := range []string{"classification.yaml", "raw/r.md", "sessions/s.jsonl", "wiki/w/facts/x.txt", "wiki/w/../../raw/r.md"} {
_, _, _, err := api.UpdateNote(brain, api.UpdateNoteOptions{Path: rel, Content: "replaced"})
assert.Error(t, err, rel)
}
for rel, body := range files {
if rel == "wiki/w/facts/ok.md" || rel == "knowledge/k.md" {
continue
}
got, err := os.ReadFile(filepath.Join(brain, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Equal(t, body, string(got), rel)
}
for _, rel := range []string{"wiki/w/facts/ok.md", "knowledge/k.md"} {
_, _, _, err := api.UpdateNote(brain, api.UpdateNoteOptions{Path: rel, Content: "new body"})
require.NoError(t, err, "positive control: %s", rel)
got, _ := os.ReadFile(filepath.Join(brain, filepath.FromSlash(rel)))
assert.Contains(t, string(got), "new body")
}
}
func TestUpdateNoteDoesNotWriteThroughASymlinkedNote(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
victim := filepath.Join(brain, "raw", "victim.md")
require.NoError(t, os.MkdirAll(filepath.Dir(victim), 0o755))
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "victim.md"), filepath.Join(facts, "l.md")))
_, _, _, err := api.UpdateNote(brain, api.UpdateNoteOptions{Path: "wiki/w/facts/l.md", Content: "replaced"})
assert.Error(t, err)
got, _ := os.ReadFile(victim)
assert.Equal(t, "untouched\n", string(got))
}
// The note is swapped between a regular file and an in-brain symlink to a
// raw/ file while UpdateNote runs: the read and the write each refuse a
// symlink, so the raw/ file is never rewritten.
func TestUpdateNoteNeverRewritesASymlinkSwappedInForTheNote(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
victim := filepath.Join(brain, "raw", "victim.md")
require.NoError(t, os.MkdirAll(filepath.Dir(victim), 0o755))
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
note := filepath.Join(facts, "n.md")
body := "---\ntitle: n\n---\nold\n"
require.NoError(t, os.WriteFile(note, []byte(body), 0o644))
var stop atomic.Bool
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
tmp := note + ".swap"
for i := 0; !stop.Load(); i++ {
_ = os.Remove(tmp)
if i%2 == 0 {
_ = os.WriteFile(tmp, []byte(body), 0o644)
} else {
_ = os.Symlink(filepath.Join("..", "..", "..", "raw", "victim.md"), tmp)
}
_ = os.Rename(tmp, note)
}
}()
ok := 0
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); {
if _, _, _, err := api.UpdateNote(brain, api.UpdateNoteOptions{Path: "wiki/w/facts/n.md", Content: "new"}); err == nil {
ok++
}
}
stop.Store(true)
wg.Wait()
got, _ := os.ReadFile(victim)
assert.Equal(t, "untouched\n", string(got))
assert.Positive(t, ok, "positive control: some updates reach the regular note")
}
-39
View File
@@ -231,45 +231,6 @@ func TestUpdateThenReadRoundTripsHash(t *testing.T) {
assert.Equal(t, hash, readHash, "update content_hash must round-trip through ReadNote") assert.Equal(t, hash, readHash, "update content_hash must round-trip through ReadNote")
} }
func TestUpdateNoteSupersedeReasonNotSettableViaCallerFragmentWhenReasonOptionEmpty(t *testing.T) {
// hyperguild#125, found in #111's review: the merge loop applied every
// field from the caller's leading frontmatter fragment, including
// supersede_reason -- which is only re-set unconditionally when
// opts.Reason is non-empty, so a caller could set it via its own
// fragment and have it survive even with Reason left empty, bypassing
// the one intended entry point for that field.
brainDir := t.TempDir()
rel := seedNote(t, brainDir, "wiki/a/facts/n.md", "---\nwing: a\nhall: facts\n---\nold\n")
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
Path: rel,
Content: "---\nsupersede_reason: snuck in via the fragment\n---\nnew\n",
// Reason deliberately left empty.
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.NotContains(t, string(got), "supersede_reason",
"supersede_reason must not appear at all when opts.Reason is empty, regardless of what the fragment tried to set")
}
func TestUpdateNoteSupersedeReasonFromOptionStillWorks(t *testing.T) {
// Regression pin: the legitimate entry point (opts.Reason) must be
// completely unaffected by #125's fix.
brainDir := t.TempDir()
rel := seedNote(t, brainDir, "wiki/a/facts/n.md", "---\nwing: a\nhall: facts\n---\nold\n")
_, _, _, err := UpdateNote(brainDir, UpdateNoteOptions{
Path: rel, Content: "new\n", Reason: "facts changed",
})
require.NoError(t, err)
got, err := os.ReadFile(filepath.Join(brainDir, filepath.FromSlash(rel)))
require.NoError(t, err)
assert.Contains(t, string(got), "supersede_reason: facts changed")
}
func TestContentHashStable(t *testing.T) { func TestContentHashStable(t *testing.T) {
assert.Equal(t, ContentHash([]byte("abc")), ContentHash([]byte("abc"))) assert.Equal(t, ContentHash([]byte("abc")), ContentHash([]byte("abc")))
assert.NotEqual(t, ContentHash([]byte("abc")), ContentHash([]byte("abd"))) assert.NotEqual(t, ContentHash([]byte("abc")), ContentHash([]byte("abd")))
+1 -13
View File
@@ -49,19 +49,7 @@ func (n *NtfyNotifier) Notify(ctx context.Context, msg string) error {
} }
defer func() { _ = resp.Body.Close() }() defer func() { _ = resp.Body.Close() }()
if resp.StatusCode < 200 || resp.StatusCode >= 300 { if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return &RejectedError{Status: resp.StatusCode} return fmt.Errorf("ntfy notify: status %d", resp.StatusCode)
} }
return nil return nil
} }
// RejectedError is a non-2xx ntfy response.
type RejectedError struct{ Status int }
func (e *RejectedError) Error() string { return fmt.Sprintf("ntfy notify: status %d", e.Status) }
// Permanent reports whether resending the same message cannot succeed: a 4xx
// other than 429 (rate limited). Callers may drop such a message; anything
// else (5xx, 429, transport errors) is an outage and must be retried.
func (e *RejectedError) Permanent() bool {
return e.Status >= 400 && e.Status < 500 && e.Status != http.StatusTooManyRequests
}
@@ -1,27 +0,0 @@
package audit
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// A non-2xx ntfy response is a *RejectedError, and only a client-side refusal
// other than rate limiting is permanent: callers may drop a message on a
// permanent rejection and must retry the rest (hyperguild#157, PR #160).
func TestNotifyClassifiesRejectionsAsPermanentOrNot(t *testing.T) {
for status, permanent := range map[int]bool{413: true, 400: true, 403: true, 429: false, 500: false, 503: false} {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(status) }))
err := NewNtfyNotifier(srv.URL, "").Notify(context.Background(), "m")
srv.Close()
var rej *RejectedError
require.True(t, errors.As(err, &rej), "status %d: %v", status, err)
assert.Equal(t, status, rej.Status)
assert.Equal(t, permanent, rej.Permanent(), "status %d", status)
}
}
+20 -54
View File
@@ -4,12 +4,9 @@ import (
"bufio" "bufio"
"fmt" "fmt"
"os" "os"
"path"
"path/filepath" "path/filepath"
"strings" "strings"
"time" "time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
) )
// seeAlsoHeader is the markdown heading used to group cross-wing links. // seeAlsoHeader is the markdown heading used to group cross-wing links.
@@ -200,12 +197,6 @@ func fileExists(p string) bool {
// section headers. Rejects same-wing pairs (a tunnel is by definition // section headers. Rejects same-wing pairs (a tunnel is by definition
// cross-wing) and missing notes. // cross-wing) and missing notes.
func WriteTunnel(brainDir, sourcePath, targetPath string) error { func WriteTunnel(brainDir, sourcePath, targetPath string) error {
if err := checkNotePath(sourcePath); err != nil {
return fmt.Errorf("source: %w", err)
}
if err := checkNotePath(targetPath); err != nil {
return fmt.Errorf("target: %w", err)
}
srcWing, err := wingOf(sourcePath) srcWing, err := wingOf(sourcePath)
if err != nil { if err != nil {
return fmt.Errorf("source: %w", err) return fmt.Errorf("source: %w", err)
@@ -218,50 +209,20 @@ func WriteTunnel(brainDir, sourcePath, targetPath string) error {
return fmt.Errorf("tunnel must cross wings; got both in %q", srcWing) return fmt.Errorf("tunnel must cross wings; got both in %q", srcWing)
} }
// Both notes are read and rewritten through the brain root, regular srcFull := filepath.Join(brainDir, filepath.FromSlash(sourcePath))
// files only: brain_tunnel passes caller-supplied paths, and the brain is tgtFull := filepath.Join(brainDir, filepath.FromSlash(targetPath))
// a Syncthing folder that can hold symlinks. if _, err := os.Stat(srcFull); err != nil {
bfs, err := brainfs.Open(brainDir)
if err != nil {
return fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
srcContent, err := bfs.ReadRegular(sourcePath)
if err != nil {
return fmt.Errorf("source note: %w", err) return fmt.Errorf("source note: %w", err)
} }
tgtContent, err := bfs.ReadRegular(targetPath) if _, err := os.Stat(tgtFull); err != nil {
if err != nil {
return fmt.Errorf("target note: %w", err) return fmt.Errorf("target note: %w", err)
} }
if out, changed, err := withSeeAlso(srcContent, wikilinkOf(targetPath)); err != nil { if err := appendSeeAlso(srcFull, wikilinkOf(targetPath)); err != nil {
return fmt.Errorf("update source: %w", err) return fmt.Errorf("update source: %w", err)
} else if changed {
if err := bfs.WriteRegular(sourcePath, out); err != nil {
return fmt.Errorf("update source: %w", err)
}
} }
if out, changed, err := withSeeAlso(tgtContent, wikilinkOf(sourcePath)); err != nil { if err := appendSeeAlso(tgtFull, wikilinkOf(sourcePath)); err != nil {
return fmt.Errorf("update target: %w", err) return fmt.Errorf("update target: %w", err)
} else if changed {
if err := bfs.WriteRegular(targetPath, out); err != nil {
return fmt.Errorf("update target: %w", err)
}
}
return nil
}
// checkNotePath accepts only a clean, local wiki/<wing>/<hall>/.../<slug>.md
// path: no "." or ".." segments, no empty segments, no leading slash.
func checkNotePath(p string) error {
if p == "" || path.Clean(p) != p || !filepath.IsLocal(p) || !strings.HasSuffix(p, ".md") {
return fmt.Errorf("not a clean wiki note path: %q", p)
}
for _, part := range strings.Split(p, "/") {
if part == "" || part == "." || part == ".." {
return fmt.Errorf("not a clean wiki note path: %q", p)
}
} }
return nil return nil
} }
@@ -287,13 +248,16 @@ func wingOf(relPath string) (string, error) {
return parts[1], nil return parts[1], nil
} }
// withSeeAlso returns content with `- [[link]]` under its See also section, // appendSeeAlso inserts `- [[link]]` under the file's See also section,
// creating the section if absent. changed is false when the link is already // creating the section if absent. No-op when the link is already present.
// present. func appendSeeAlso(filePath, link string) error {
func withSeeAlso(content []byte, link string) (out []byte, changed bool, err error) { content, err := os.ReadFile(filePath)
if err != nil {
return err
}
wikilink := "[[" + link + "]]" wikilink := "[[" + link + "]]"
if strings.Contains(string(content), wikilink) { if strings.Contains(string(content), wikilink) {
return content, false, nil return nil
} }
bullet := "- " + wikilink bullet := "- " + wikilink
@@ -302,7 +266,8 @@ func withSeeAlso(content []byte, link string) (out []byte, changed bool, err err
// No section yet — append a fresh one. Always emit a trailing // No section yet — append a fresh one. Always emit a trailing
// newline so subsequent appends don't merge into the previous line. // newline so subsequent appends don't merge into the previous line.
trimmed := strings.TrimRight(string(content), "\n") trimmed := strings.TrimRight(string(content), "\n")
return []byte(trimmed + "\n\n" + seeAlsoHeader + "\n\n" + bullet + "\n"), true, nil out := trimmed + "\n\n" + seeAlsoHeader + "\n\n" + bullet + "\n"
return os.WriteFile(filePath, []byte(out), 0o644)
} }
// Section exists — splice the bullet in just before the next `## ` // Section exists — splice the bullet in just before the next `## `
@@ -328,11 +293,12 @@ func withSeeAlso(content []byte, link string) (out []byte, changed bool, err err
b.WriteByte('\n') b.WriteByte('\n')
} }
if err := scanner.Err(); err != nil { if err := scanner.Err(); err != nil {
return nil, false, err return err
} }
if !inserted { if !inserted {
// section was the last thing in the file — just append bullet // section was the last thing in the file — just append bullet
return []byte(strings.TrimRight(b.String(), "\n") + "\n" + bullet + "\n"), true, nil out := strings.TrimRight(b.String(), "\n") + "\n" + bullet + "\n"
return os.WriteFile(filePath, []byte(out), 0o644)
} }
return []byte(b.String()), true, nil return os.WriteFile(filePath, []byte(b.String()), 0o644)
} }
@@ -1,141 +0,0 @@
package brain_test
import (
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain"
)
// brain_tunnel passes caller-supplied source and target paths to WriteTunnel,
// which checked only "wiki/<wing>/..." and then joined them onto the brain
// dir: ".." segments resolved outside the brain and appendSeeAlso rewrote the
// file there.
func TestWriteTunnelRefusesATargetThatTraversesOutOfTheBrain(t *testing.T) {
brain0 := t.TempDir()
src := filepath.Join(brain0, "wiki", "a", "facts", "s.md")
require.NoError(t, os.MkdirAll(filepath.Dir(src), 0o755))
require.NoError(t, os.WriteFile(src, []byte("---\ntitle: S\n---\nbody\n"), 0o644))
victim := filepath.Join(t.TempDir(), "victim.md")
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
up := strings.Repeat("../", strings.Count(filepath.Clean(brain0), "/")+3)
target := "wiki/b/x/" + up + strings.TrimPrefix(victim, "/")
err := brain.WriteTunnel(brain0, "wiki/a/facts/s.md", target)
assert.Error(t, err)
got, rerr := os.ReadFile(victim)
require.NoError(t, rerr)
assert.Equal(t, "untouched\n", string(got), "a file outside the brain was modified")
}
func TestWriteTunnelRefusesUncleanOrNonWikiPaths(t *testing.T) {
brain0 := t.TempDir()
for _, p := range []string{"wiki/a/facts/s.md", "wiki/b/facts/t.md", "wiki/b/facts/t.txt", "raw/r.md"} {
require.NoError(t, os.MkdirAll(filepath.Dir(filepath.Join(brain0, p)), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(brain0, p), []byte("x\n"), 0o644))
}
for _, target := range []string{
"wiki/b/facts/../../../raw/r.md",
"wiki/b/./facts/t.md",
"wiki/b//facts/t.md",
"/wiki/b/facts/t.md",
"wiki/b/facts/t.txt",
"wiki/b/facts/",
} {
assert.Error(t, brain.WriteTunnel(brain0, "wiki/a/facts/s.md", target), target)
}
for _, p := range []string{"raw/r.md", "wiki/b/facts/t.txt"} {
got, err := os.ReadFile(filepath.Join(brain0, p))
require.NoError(t, err)
assert.Equal(t, "x\n", string(got), p)
}
require.NoError(t, brain.WriteTunnel(brain0, "wiki/a/facts/s.md", "wiki/b/facts/t.md"), "positive control")
got, err := os.ReadFile(filepath.Join(brain0, "wiki", "b", "facts", "t.md"))
require.NoError(t, err)
assert.Contains(t, string(got), "[[a/facts/s]]")
}
func TestWriteTunnelDoesNotWriteThroughASymlinkedNote(t *testing.T) {
brain0 := t.TempDir()
src := filepath.Join(brain0, "wiki", "a", "facts", "s.md")
require.NoError(t, os.MkdirAll(filepath.Dir(src), 0o755))
require.NoError(t, os.WriteFile(src, []byte("x\n"), 0o644))
tdir := filepath.Join(brain0, "wiki", "b", "facts")
require.NoError(t, os.MkdirAll(tdir, 0o755))
victim := filepath.Join(t.TempDir(), "victim.md")
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
require.NoError(t, os.Symlink(victim, filepath.Join(tdir, "t.md")))
assert.Error(t, brain.WriteTunnel(brain0, "wiki/a/facts/s.md", "wiki/b/facts/t.md"))
got, err := os.ReadFile(victim)
require.NoError(t, err)
assert.Equal(t, "untouched\n", string(got))
// The same with a link to a file elsewhere in the brain, which os.Root
// would follow.
inVictim := filepath.Join(brain0, "raw", "v.md")
require.NoError(t, os.MkdirAll(filepath.Dir(inVictim), 0o755))
require.NoError(t, os.WriteFile(inVictim, []byte("untouched\n"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "v.md"), filepath.Join(tdir, "u.md")))
assert.Error(t, brain.WriteTunnel(brain0, "wiki/a/facts/s.md", "wiki/b/facts/u.md"))
got, err = os.ReadFile(inVictim)
require.NoError(t, err)
assert.Equal(t, "untouched\n", string(got))
got, err = os.ReadFile(src)
require.NoError(t, err)
assert.Equal(t, "x\n", string(got), "the source is not touched when the target is refused")
}
// The target is swapped between a regular note and a symlink to an in-brain
// file outside wiki/ while WriteTunnel runs: the read and the write each
// refuse a symlink, so the other file is never rewritten.
func TestWriteTunnelNeverRewritesASymlinkSwappedInForTheTarget(t *testing.T) {
brain0 := t.TempDir()
src := filepath.Join(brain0, "wiki", "a", "facts", "s.md")
require.NoError(t, os.MkdirAll(filepath.Dir(src), 0o755))
require.NoError(t, os.WriteFile(src, []byte("x\n"), 0o644))
tdir := filepath.Join(brain0, "wiki", "b", "facts")
require.NoError(t, os.MkdirAll(tdir, 0o755))
victim := filepath.Join(brain0, "raw", "victim.md")
require.NoError(t, os.MkdirAll(filepath.Dir(victim), 0o755))
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
tgt := filepath.Join(tdir, "t.md")
require.NoError(t, os.WriteFile(tgt, []byte("t\n"), 0o644))
var stop atomic.Bool
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
tmp := tgt + ".swap"
for i := 0; !stop.Load(); i++ {
_ = os.Remove(tmp)
if i%2 == 0 {
_ = os.WriteFile(tmp, []byte("t\n"), 0o644)
} else {
_ = os.Symlink(filepath.Join("..", "..", "..", "raw", "victim.md"), tmp)
}
_ = os.Rename(tmp, tgt)
}
}()
ok := 0
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); {
if brain.WriteTunnel(brain0, "wiki/a/facts/s.md", "wiki/b/facts/t.md") == nil {
ok++
}
_ = os.WriteFile(src, []byte("x\n"), 0o644)
}
stop.Store(true)
wg.Wait()
got, err := os.ReadFile(victim)
require.NoError(t, err)
assert.Equal(t, "untouched\n", string(got), "the swapped-in symlink's target was rewritten")
assert.Positive(t, ok, "positive control: some tunnels were written to the regular target")
}
-268
View File
@@ -1,268 +0,0 @@
// Package brainfs reads the brain directory through an os.Root (hyperguild#157).
// The brain dir is a Syncthing sendreceive folder and Syncthing syncs
// symlinks; a reader that walks and reads by path follows a planted symlink
// out of the brain, and blocks forever on a planted FIFO. Readers that go
// through a Brain only ever read regular files inside the brain: os.Root
// refuses to leave the root even if an entry is swapped mid-read, and every
// open is non-blocking with the type checked on the opened handle.
package brainfs
import (
"errors"
"fmt"
"io"
"io/fs"
"os"
"path"
"path/filepath"
"strings"
"syscall"
)
// ErrNotRegular is returned for an entry that is not a regular file.
var ErrNotRegular = errors.New("not a regular file")
// ErrTooLarge is returned by ReadRegularMax for a file over the limit.
var ErrTooLarge = errors.New("file too large")
// Brain is an open brain directory.
type Brain struct {
root *os.Root
dir string
}
// Open opens brainDir (which may itself be a mount symlink) as a root.
func Open(brainDir string) (*Brain, error) {
r, err := os.OpenRoot(brainDir)
if err != nil {
return nil, fmt.Errorf("open brain root: %w", err)
}
return &Brain{root: r, dir: brainDir}, nil
}
// Close releases the root.
func (b *Brain) Close() error { return b.root.Close() }
// Rel converts a path under the brain dir to a root-relative path.
func (b *Brain) Rel(path string) (string, error) {
rel, err := filepath.Rel(b.dir, path)
if err != nil {
return "", err
}
if rel == ".." || len(rel) > 2 && rel[:3] == ".."+string(filepath.Separator) {
return "", fmt.Errorf("%s is outside the brain dir", path)
}
return filepath.ToSlash(rel), nil
}
// Lstat stats a brain-relative path through the root without following a
// final symlink.
func (b *Brain) Lstat(rel string) (fs.FileInfo, error) {
return b.root.Lstat(filepath.ToSlash(rel))
}
// realDir returns ErrNotRegular unless every component of rel is a real
// directory. A swap after the check is not caught (accepted residual,
// hyperguild#157).
func (b *Brain) realDir(rel string) error {
if rel == "." {
return nil
}
prefix := ""
for _, part := range strings.Split(rel, "/") {
prefix = path.Join(prefix, part)
fi, err := b.root.Lstat(prefix)
if err != nil {
return err
}
if !fi.IsDir() {
return fmt.Errorf("%s: walk root component %s is not a directory: %w", rel, prefix, ErrNotRegular)
}
}
return nil
}
// ReadDirRegular lists the regular files directly in a brain-relative
// directory, by name. Every component of relDir must be a real directory
// (ErrNotRegular otherwise); symlinks, FIFOs, devices and subdirectories in
// it are left out.
func (b *Brain) ReadDirRegular(relDir string) ([]string, error) {
relDir = filepath.ToSlash(relDir)
if err := b.realDir(relDir); err != nil {
return nil, err
}
ents, err := fs.ReadDir(b.root.FS(), relDir)
if err != nil {
return nil, err
}
var names []string
for _, e := range ents {
if e.Type().IsRegular() {
names = append(names, e.Name())
}
}
return names, nil
}
// ReadRegular reads a brain-relative path, refusing anything that is not a
// regular file, both before the open and on the opened handle.
func (b *Brain) ReadRegular(rel string) ([]byte, error) {
fi, err := b.root.Lstat(rel)
if err != nil {
return nil, err
}
if !fi.Mode().IsRegular() {
return nil, fmt.Errorf("%s: %w", rel, ErrNotRegular)
}
return b.readOpenedMax(rel, -1, fi)
}
// readOpened opens rel non-blocking and reads it only if the OPENED handle is
// a regular file. This is the check that matters under a race: an entry
// swapped for a FIFO after the Lstat, with a writer holding it open, would
// otherwise park io.ReadAll forever (PR #161 review: removing it hung search
// after 16 iterations).
//
// Accepted residual, as in api/rawroot.go: a HARD link to an outside file is
// a regular file and is read. Syncthing syncs contents, not links, so a peer
// cannot create one; fs.protected_hardlinks=1 and the pod's secrets are on a
// different filesystem from the brain volume, so a local caller cannot link
// them in. Refusing Nlink > 1 is the next step if either changes.
func (b *Brain) readOpened(rel string) ([]byte, error) { return b.readOpenedMax(rel, -1, nil) }
// ReadRegularMax is ReadRegular with a size limit, checked on the opened
// handle before reading (max < 0 means no limit).
func (b *Brain) ReadRegularMax(rel string, max int64) ([]byte, error) {
fi, err := b.root.Lstat(rel)
if err != nil {
return nil, err
}
if !fi.Mode().IsRegular() {
return nil, fmt.Errorf("%s: %w", rel, ErrNotRegular)
}
return b.readOpenedMax(rel, max, fi)
}
// readOpenedMax opens rel and reads it if the opened handle is a regular file
// and, when lfi is set, the same file the caller Lstat-ed: os.Root follows an
// in-root final symlink even with O_NOFOLLOW, so a symlink to another brain
// file swapped in between the Lstat and the open is refused only here.
func (b *Brain) readOpenedMax(rel string, max int64, lfi fs.FileInfo) ([]byte, error) {
f, err := b.root.OpenFile(rel, os.O_RDONLY|syscall.O_NONBLOCK, 0)
if err != nil {
return nil, err
}
defer func() { _ = f.Close() }()
hfi, err := f.Stat()
if err != nil {
return nil, err
}
if !hfi.Mode().IsRegular() {
return nil, fmt.Errorf("%s: %w", rel, ErrNotRegular)
}
if lfi != nil && !os.SameFile(lfi, hfi) {
return nil, fmt.Errorf("%s: changed while opening: %w", rel, ErrNotRegular)
}
if max >= 0 && hfi.Size() > max {
return nil, fmt.Errorf("%s is %d bytes (limit %d): %w", rel, hfi.Size(), max, ErrTooLarge)
}
var r io.Reader = f
if wrapAfterStat != nil {
r = wrapAfterStat(r)
}
if max < 0 {
return io.ReadAll(r)
}
// The handle's size can grow after the check; bound the read itself.
data, err := io.ReadAll(io.LimitReader(r, max+1))
if err != nil {
return nil, err
}
if int64(len(data)) > max {
return nil, fmt.Errorf("%s grew past %d bytes while read: %w", rel, max, ErrTooLarge)
}
return data, nil
}
// wrapAfterStat is a test seam: it runs between the size check and the read,
// and wraps the reader the read consumes. A package global: brainfs tests
// must not use t.Parallel.
var wrapAfterStat func(io.Reader) io.Reader
// WalkRegular walks a brain-relative directory through the root and calls fn
// for each regular file. Directories are descended; symlinks, FIFOs and
// devices found by the walk are skipped. A walk root with any component that
// is not a real directory goes to onErr as ErrNotRegular and nothing is
// yielded. An entry error is passed to onErr and the walk continues; a nil
// onErr ignores it.
//
// Accepted residual (hyperguild#157): a directory swapped for a symlink
// between the listing and the read is followed. os.Root still keeps the read
// inside the brain, but not inside relDir.
func (b *Brain) WalkRegular(relDir string, fn func(rel string) error, onErr func(rel string, err error)) error {
relDir = filepath.ToSlash(relDir)
// fs.WalkDir stats its root with a follow, and os.Root allows a symlink
// that stays inside the brain: knowledge -> . would alias the whole
// brain. Require every component of the root to be a real directory
// (Lstat checks only the last one).
err := b.realDir(relDir)
if err != nil {
if onErr != nil {
onErr(relDir, err)
}
return nil
}
return fs.WalkDir(b.root.FS(), relDir, func(p string, d fs.DirEntry, err error) error {
if err != nil {
if onErr != nil {
onErr(p, err)
}
if d != nil && d.IsDir() {
return fs.SkipDir
}
return nil
}
if d.IsDir() || !d.Type().IsRegular() {
return nil
}
return fn(p)
})
}
// WriteRegular replaces the contents of an existing regular file at a
// brain-relative path. It never creates a file and never writes through a
// final symlink: os.Root.OpenFile follows an in-root final symlink even with
// O_NOFOLLOW (verified on go1.27), so the path is Lstat-ed through the root
// first and the opened handle must be that same file (os.SameFile), which
// also catches a swap between the two. O_NONBLOCK keeps a FIFO from blocking
// the open. Intermediate in-brain symlinks are still followed (os.Root keeps
// the write inside the brain; hyperguild#167).
func (b *Brain) WriteRegular(rel string, data []byte) error {
rel = filepath.ToSlash(rel)
lfi, err := b.root.Lstat(rel)
if err != nil {
return err
}
if !lfi.Mode().IsRegular() {
return fmt.Errorf("%s: %w", rel, ErrNotRegular)
}
f, err := b.root.OpenFile(rel, os.O_WRONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0)
if err != nil {
return err
}
defer func() { _ = f.Close() }()
fi, err := f.Stat()
if err != nil {
return err
}
if !fi.Mode().IsRegular() || !os.SameFile(lfi, fi) {
return fmt.Errorf("%s: changed while opening: %w", rel, ErrNotRegular)
}
if err := f.Truncate(0); err != nil {
return err
}
if _, err := f.Write(data); err != nil {
return err
}
return f.Close()
}
-310
View File
@@ -1,310 +0,0 @@
package brainfs
import (
"errors"
"io"
"io/fs"
"os"
"path/filepath"
"sync"
"sync/atomic"
"syscall"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// The handle check is what stops a FIFO swapped in after the Lstat. A FIFO
// with no writer reads EOF at once, so a race test could not see the check
// missing; here a writer holds the FIFO open, as any local process can, and
// readOpened must refuse it promptly (PR #161 review).
func TestReadOpenedRefusesAFIFOHeldOpenByAWriter(t *testing.T) {
dir := t.TempDir()
fifo := filepath.Join(dir, "p.md")
require.NoError(t, syscall.Mkfifo(fifo, 0o644))
w, err := os.OpenFile(fifo, os.O_RDWR, 0)
require.NoError(t, err)
defer func() { _ = w.Close() }()
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
done := make(chan error, 1)
go func() { _, err := b.readOpened("p.md"); done <- err }()
select {
case err := <-done:
assert.True(t, errors.Is(err, ErrNotRegular), "got %v", err)
case <-time.After(2 * time.Second):
t.Fatal("readOpened blocked on a FIFO with a writer")
}
}
func TestReadRegularRefusesSymlinksAndReadsRegularFiles(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "ok.md"), []byte("ok"), 0o644))
outside := filepath.Join(t.TempDir(), "s.md")
require.NoError(t, os.WriteFile(outside, []byte("secret"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(dir, "link.md")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
got, err := b.ReadRegular("ok.md")
require.NoError(t, err)
assert.Equal(t, "ok", string(got))
_, err = b.ReadRegular("link.md")
assert.True(t, errors.Is(err, ErrNotRegular), "got %v", err)
_, err = b.ReadRegular("../" + filepath.Base(outside))
assert.Error(t, err)
}
func TestRelAcceptsPathsInsideAndRefusesEscapes(t *testing.T) {
dir := t.TempDir()
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
for in, want := range map[string]string{
filepath.Join(dir, "wiki", "x.md"): "wiki/x.md",
filepath.Join(dir, "..foo", "x"): "..foo/x",
dir: ".",
} {
got, err := b.Rel(in)
require.NoError(t, err, in)
assert.Equal(t, want, got, in)
}
for _, bad := range []string{filepath.Join(dir, "..", "x"), dir + "-evil/x", "/etc/passwd", filepath.Join(dir, "wiki", "..", "..", "x")} {
_, err := b.Rel(bad)
assert.Error(t, err, bad)
}
}
func TestWalkRegularYieldsOnlyRegularFiles(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "a"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "a", "n.md"), []byte("n"), 0o644))
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(dir, "a", "dirlink")))
require.NoError(t, os.Symlink(filepath.Join(dir, "a", "n.md"), filepath.Join(dir, "a", "filelink.md")))
require.NoError(t, syscall.Mkfifo(filepath.Join(dir, "a", "p.md"), 0o644))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
var got []string
require.NoError(t, b.WalkRegular(".", func(rel string) error { got = append(got, rel); return nil }, nil))
assert.Equal(t, []string{"a/n.md"}, got)
}
func TestReadRegularMaxRefusesAFileOverTheLimit(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "big.md"), make([]byte, 100), 0o644))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
_, err = b.ReadRegularMax("big.md", 99)
assert.True(t, errors.Is(err, ErrTooLarge), "got %v", err)
got, err := b.ReadRegularMax("big.md", 100)
require.NoError(t, err)
assert.Len(t, got, 100)
}
func TestReadRegularMaxRefusesAndStopsReadingAFileThatGrowsPastTheLimit(t *testing.T) {
dir := t.TempDir()
p := filepath.Join(dir, "grow.md")
require.NoError(t, os.WriteFile(p, make([]byte, 10), 0o644))
consumed := &countingReader{}
wrapAfterStat = func(r io.Reader) io.Reader {
f, err := os.OpenFile(p, os.O_WRONLY|os.O_APPEND, 0)
require.NoError(t, err)
_, err = f.Write(make([]byte, 1000))
require.NoError(t, err)
require.NoError(t, f.Close())
consumed.r = r
return consumed
}
t.Cleanup(func() { wrapAfterStat = nil })
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
got, err := b.ReadRegularMax("grow.md", 100)
assert.True(t, errors.Is(err, ErrTooLarge), "got %v", err)
assert.Nil(t, got)
assert.LessOrEqual(t, consumed.n, 101, "read past max+1 bytes")
}
type countingReader struct {
r io.Reader
n int
}
func (c *countingReader) Read(p []byte) (int, error) {
n, err := c.r.Read(p)
c.n += n
return n, err
}
func TestLstatDoesNotFollowSymlinksAndRefusesEscapes(t *testing.T) {
dir := t.TempDir()
outside := filepath.Join(t.TempDir(), "o.md")
require.NoError(t, os.WriteFile(outside, []byte("o"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(dir, "l.md")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
fi, err := b.Lstat("l.md")
require.NoError(t, err)
assert.Equal(t, fs.ModeSymlink, fi.Mode().Type())
_, err = b.Lstat("../" + filepath.Base(filepath.Dir(outside)) + "/o.md")
assert.Error(t, err)
}
func TestWalkRegularReportsASymlinkedWalkRootAndYieldsNothing(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "real"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "real", "n.md"), []byte("n"), 0o644))
require.NoError(t, os.Symlink("real", filepath.Join(dir, "alias")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
var got []string
var errs []error
require.NoError(t, b.WalkRegular("alias", func(rel string) error { got = append(got, rel); return nil },
func(_ string, err error) { errs = append(errs, err) }))
assert.Empty(t, got)
require.Len(t, errs, 1)
assert.ErrorIs(t, errs[0], ErrNotRegular)
}
func TestWalkRegularReportsASymlinkedIntermediateComponentOfTheWalkRoot(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "real", "sub"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "real", "sub", "n.md"), []byte("n"), 0o644))
require.NoError(t, os.Symlink("real", filepath.Join(dir, "alias")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
var got []string
var errs []error
require.NoError(t, b.WalkRegular("alias/sub", func(rel string) error { got = append(got, rel); return nil },
func(_ string, err error) { errs = append(errs, err) }))
assert.Empty(t, got)
require.Len(t, errs, 1)
assert.ErrorIs(t, errs[0], ErrNotRegular)
}
func TestReadDirRegularListsOnlyRegularFilesAndRefusesSymlinkedComponents(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "a", "b", "sub"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "a", "b", "n.md"), []byte("n"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(dir, "other.md"), []byte("o"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "other.md"), filepath.Join(dir, "a", "b", "l.md")))
require.NoError(t, syscall.Mkfifo(filepath.Join(dir, "a", "b", "f.md"), 0o644))
require.NoError(t, os.Symlink("a", filepath.Join(dir, "alias")))
require.NoError(t, os.MkdirAll(filepath.Join(dir, "x"), 0o755))
require.NoError(t, os.Symlink(filepath.Join("..", "a"), filepath.Join(dir, "x", "mid")))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
names, err := b.ReadDirRegular("a/b")
require.NoError(t, err)
assert.Equal(t, []string{"n.md"}, names)
for _, rel := range []string{"alias/b", "x/mid/b"} {
_, err := b.ReadDirRegular(rel)
assert.ErrorIs(t, err, ErrNotRegular, rel)
}
}
// A final component swapped for a symlink to another regular file in the
// brain between the Lstat and the open: os.Root follows in-root links, so
// only comparing the two (os.SameFile) refuses it.
func TestReadRegularNeverReturnsAnInBrainFileSwappedInBehindASymlink(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "s.md"), []byte("zqsamefilesentinel"), 0o644))
p := filepath.Join(dir, "r.md")
require.NoError(t, os.WriteFile(p, []byte("inside"), 0o644))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
var stop atomic.Bool
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
tmp := p + ".swap"
for i := 0; !stop.Load(); i++ {
_ = os.Remove(tmp)
if i%2 == 0 {
_ = os.WriteFile(tmp, []byte("inside"), 0o644)
} else {
_ = os.Symlink("s.md", tmp)
}
_ = os.Rename(tmp, p)
}
}()
leaks, ok := 0, 0
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); {
got, err := b.ReadRegular("r.md")
if err == nil {
ok++
}
if string(got) == "zqsamefilesentinel" {
leaks++
}
got, _ = b.ReadRegularMax("r.md", 1<<20)
if string(got) == "zqsamefilesentinel" {
leaks++
}
}
stop.Store(true)
wg.Wait()
assert.Zero(t, leaks)
assert.Positive(t, ok, "positive control: the regular file is read")
}
func TestWriteRegularRefusesFinalSymlinksFIFOsEscapesAndMissingFilesAndRewritesRegularFiles(t *testing.T) {
dir := t.TempDir()
outside := filepath.Join(t.TempDir(), "o.md")
require.NoError(t, os.WriteFile(outside, []byte("untouched"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(dir, "l.md")))
require.NoError(t, syscall.Mkfifo(filepath.Join(dir, "f.md"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(dir, "n.md"), []byte("old content"), 0o644))
b, err := Open(dir)
require.NoError(t, err)
defer func() { _ = b.Close() }()
assert.Error(t, b.WriteRegular("l.md", []byte("x")))
got, _ := os.ReadFile(outside)
assert.Equal(t, "untouched", string(got))
// A symlink to another file inside the brain: os.Root would allow it,
// only O_NOFOLLOW refuses it.
require.NoError(t, os.WriteFile(filepath.Join(dir, "in.md"), []byte("inside"), 0o644))
require.NoError(t, os.Symlink("in.md", filepath.Join(dir, "il.md")))
assert.Error(t, b.WriteRegular("il.md", []byte("x")))
got, _ = os.ReadFile(filepath.Join(dir, "in.md"))
assert.Equal(t, "inside", string(got))
// A reader held open, so a non-blocking write open would succeed. The
// Lstat type check refuses the FIFO first; the handle check and
// O_NONBLOCK matter only if it is swapped in after the Lstat.
r, err := os.OpenFile(filepath.Join(dir, "f.md"), os.O_RDONLY|syscall.O_NONBLOCK, 0)
require.NoError(t, err)
defer func() { _ = r.Close() }()
done := make(chan error, 1)
go func() { done <- b.WriteRegular("f.md", []byte("x")) }()
select {
case err := <-done:
assert.Error(t, err)
case <-time.After(3 * time.Second):
t.Fatal("WriteRegular hung on a FIFO")
}
assert.Error(t, b.WriteRegular("../"+filepath.Base(filepath.Dir(outside))+"/o.md", []byte("x")))
assert.Error(t, b.WriteRegular("missing.md", []byte("x")), "does not create files")
require.NoError(t, b.WriteRegular("n.md", []byte("new")))
got, _ = os.ReadFile(filepath.Join(dir, "n.md"))
assert.Equal(t, "new", string(got), "truncated and rewritten")
}
@@ -1,141 +0,0 @@
// Package brainfstest is the shared harness for hyperguild#157's swap-race
// tests: it keeps swapping a regular note, a symlink to an outside file, or a
// FIFO (with a writer held open) into a path while a reader under test loops.
// It observes two things: no sentinel byte from the outside file reaches the
// reader's output, and the reader returns within a timeout. It does not cover
// hard links, other paths into the reader, or races it does not generate.
package brainfstest
import (
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"syscall"
"testing"
"time"
)
// Sentinel marks the outside file's contents.
const Sentinel = "zqbrainfssentinelonefiftyseven"
// outsideContent carries Sentinel in every field a reader might extract: the
// frontmatter title and aliases, a heading, a wikilink and the body.
const outsideContent = "---\ntitle: " + Sentinel + "\naliases:\n - " + Sentinel + "\n---\n# " + Sentinel +
"\n\n" + Sentinel + " [[" + Sentinel + "]]\n"
// Outside writes a file outside any brain dir containing Sentinel.
func Outside(t *testing.T) string {
t.Helper()
p := filepath.Join(t.TempDir(), "secret.md")
if err := os.WriteFile(p, []byte(outsideContent), 0o644); err != nil {
t.Fatal(err)
}
return p
}
var swapSeq atomic.Int64
// StartSwap renames, in a loop, a regular note (content inside) and either
// a symlink to outside or a FIFO into path, until the test ends. FIFOs get a
// writer held open until the swapper exits, so a reader that skips the
// handle check parks forever instead of reading EOF. The swapper is stopped
// and awaited before t.TempDir cleanup.
func StartSwap(t *testing.T, path, inside, outside string, fifo bool) {
t.Helper()
var stop atomic.Bool
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
// A name of its own: swappers on one path share a test, and one
// opening another's FIFO for write would block forever.
tmp := fmt.Sprintf("%s.swap%d", path, swapSeq.Add(1))
// A relative link: os.Root refuses every absolute link on its own, so
// an absolute in-brain target would test nothing.
link := outside
if rel, err := filepath.Rel(filepath.Dir(path), outside); err == nil {
link = rel
}
var writers []*os.File
defer func() {
for _, w := range writers {
_ = w.Close()
}
}()
for i := 0; !stop.Load(); i++ {
_ = os.Remove(tmp)
switch {
case i%2 == 0:
_ = os.WriteFile(tmp, []byte(inside), 0o644)
case fifo:
if err := syscall.Mkfifo(tmp, 0o644); err == nil {
if w, err := os.OpenFile(tmp, os.O_RDWR, 0); err == nil {
writers = append(writers, w)
}
}
default:
_ = os.Symlink(link, tmp)
}
_ = os.Rename(tmp, path)
}
}()
t.Cleanup(func() { stop.Store(true); wg.Wait() })
}
// WriteSentinelAt writes a file carrying Sentinel at p (creating its
// directory), for an in-brain symlink target.
func WriteSentinelAt(t *testing.T, p string) string {
t.Helper()
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(outsideContent), 0o644); err != nil {
t.Fatal(err)
}
return p
}
// AssertNoLeak loops read for d while a symlink swapper runs on path, and
// fails if any output contains Sentinel. The symlink points outside the
// brain; os.Root refuses that on its own, so pair it with AssertNoLeakFrom
// and an in-brain target.
func AssertNoLeak(t *testing.T, path, inside string, d time.Duration, read func() string) {
t.Helper()
AssertNoLeakFrom(t, path, inside, Outside(t), d, read)
}
// AssertNoLeakFrom is AssertNoLeak with the symlink pointing at target, e.g.
// a file elsewhere in the brain written with WriteSentinelAt.
func AssertNoLeakFrom(t *testing.T, path, inside, target string, d time.Duration, read func() string) {
t.Helper()
StartSwap(t, path, inside, target, false)
leaks, iters := 0, 0
for deadline := time.Now().Add(d); time.Now().Before(deadline); iters++ {
leaks += strings.Count(read(), Sentinel)
}
if leaks > 0 {
t.Fatalf("outside file leaked %d times in %d iterations", leaks, iters)
}
}
// AssertNoHang loops read for d while a FIFO swapper (held writers) runs on
// path, and fails if a read does not return within timeout.
func AssertNoHang(t *testing.T, path, inside string, d, timeout time.Duration, read func()) {
t.Helper()
StartSwap(t, path, inside, Outside(t), true)
done := make(chan struct{})
go func() {
for deadline := time.Now().Add(d); time.Now().Before(deadline); {
read()
}
close(done)
}()
select {
case <-done:
case <-time.After(timeout):
t.Fatal("reader hung on a swapped-in FIFO with a writer held open")
}
}
+5 -7
View File
@@ -49,12 +49,10 @@ func (s *Store) Write(ctx context.Context, n capture.Note) (capture.Ref, error)
n.Content = content n.Content = content
relPath, err := api.WriteNote(s.brainDir, api.WriteNoteOptions{ relPath, err := api.WriteNote(s.brainDir, api.WriteNoteOptions{
Content: n.Content, Content: n.Content,
Filename: n.Filename, Filename: n.Filename,
Wing: n.Wing, Wing: n.Wing,
Hall: n.Hall, Hall: n.Hall,
Origin: n.Origin,
WrittenBy: n.WrittenBy,
}) })
if err != nil { if err != nil {
return capture.Ref{}, err return capture.Ref{}, err
@@ -76,7 +74,7 @@ func (s *Store) Update(ctx context.Context, slug string, n capture.Note) (captur
} }
n.Content = content n.Content = content
opts := api.UpdateNoteOptions{Content: n.Content, Reason: n.Reason, Origin: n.Origin, WrittenBy: n.WrittenBy} opts := api.UpdateNoteOptions{Content: n.Content, Reason: n.Reason}
if strings.Contains(slug, "/") { if strings.Contains(slug, "/") {
opts.Path = slug opts.Path = slug
} else { } else {
-1
View File
@@ -60,7 +60,6 @@ type CaptureContext struct {
Classification string // caller-declared level token ("" = unspecified) Classification string // caller-declared level token ("" = unspecified)
Principal string // server-derived (auth); audit identity Principal string // server-derived (auth); audit identity
Origin Zone // server-derived trust zone; the I1 gate input Origin Zone // server-derived trust zone; the I1 gate input
AuthPath string // server-derived: "static" or "jwt"; stamped as note provenance (#145)
} }
// Insight is one piece of session knowledge bound for the brain. A // Insight is one piece of session knowledge bound for the brain. A
-5
View File
@@ -40,11 +40,6 @@ type Note struct {
Wing string Wing string
Hall string Hall string
Reason string Reason string
// Provenance (hyperguild#145) is stamped by the server from the
// authenticated principal, never taken from Content's frontmatter.
// Origin is "static", "jwt" or "unknown"; empty means "unknown".
Origin string
WrittenBy string
} }
// BrainStore is the brain persistence port — the shared implementation of // BrainStore is the brain persistence port — the shared implementation of
+3 -6
View File
@@ -142,7 +142,7 @@ func (s *Service) Capture(ctx context.Context, in CaptureInput) (CaptureReceipt,
var landed []string var landed []string
for i, ins := range in.Insights { for i, ins := range in.Insights {
res, item, err := s.persistInsight(ctx, ins, in.Context) res, item, err := s.persistInsight(ctx, ins)
receipt.Insights = append(receipt.Insights, res) receipt.Insights = append(receipt.Insights, res)
if err != nil { if err != nil {
receipt.Errors = append(receipt.Errors, ItemError{Item: fmt.Sprintf("insight[%d]", i), Error: err.Error()}) receipt.Errors = append(receipt.Errors, ItemError{Item: fmt.Sprintf("insight[%d]", i), Error: err.Error()})
@@ -278,11 +278,8 @@ func escalationReason(escalators []string) string {
return "required by: " + strings.Join(escalators, ", ") return "required by: " + strings.Join(escalators, ", ")
} }
func (s *Service) persistInsight(ctx context.Context, ins Insight, cc CaptureContext) (InsightResult, string, error) { func (s *Service) persistInsight(ctx context.Context, ins Insight) (InsightResult, string, error) {
note := Note{ note := Note{Content: ins.Text, Wing: ins.Wing, Hall: ins.Hall, Filename: brain.Sanitise(firstLine(ins.Text))}
Content: ins.Text, Wing: ins.Wing, Hall: ins.Hall, Filename: brain.Sanitise(firstLine(ins.Text)),
Origin: cc.AuthPath, WrittenBy: cc.Principal,
}
var ref Ref var ref Ref
var err error var err error
if ins.SupersedeSlug != "" { if ins.SupersedeSlug != "" {
@@ -104,10 +104,6 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// caller may have tried to put in the body. // caller may have tried to put in the body.
in.Context.Principal = principal in.Context.Principal = principal
in.Context.Origin = h.resolver.Resolve(principal, viaStatic) in.Context.Origin = h.resolver.Resolve(principal, viaStatic)
in.Context.AuthPath = "jwt"
if viaStatic {
in.Context.AuthPath = "static"
}
rec, err := h.svc.Capture(r.Context(), in) rec, err := h.svc.Capture(r.Context(), in)
switch { switch {
@@ -1,51 +0,0 @@
package capturehttp_test
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/audit"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/capturehttp"
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
)
// The REST capture door derives AuthPath itself (hyperguild#145); a body
// field cannot set it, and static vs JWT are told apart.
func TestRESTCaptureStampsAuthPathFromAuthentication(t *testing.T) {
for _, tc := range []struct {
name, authz, wantOrigin, wantBy string
v capturehttp.Validator
sovereign []string
}{
{"static", "Bearer " + staticTok, "origin: static\n", "written_by: local-cli\n", nil, nil},
{"jwt", "Bearer jwt", "origin: jwt\n", "written_by: koala-cli\n", fakeValidator{subject: "koala-cli"}, []string{"koala-cli"}},
} {
t.Run(tc.name, func(t *testing.T) {
brainDir := t.TempDir()
cfg, err := classification.Load(t.TempDir())
require.NoError(t, err)
svc := capture.NewService(brainstore.New(brainDir), fakeTracker{}, cfg, audit.NewSlogSink(nil))
h := capturehttp.New(svc, tc.v, staticTok, "local-cli", capturehttp.NewOriginResolver(tc.sovereign))
rr := do(t, h, tc.authz, map[string]any{
"context": map[string]any{
"harness": "claude-code", "actor": "mathias", "classification": "internal",
"AuthPath": "static", "auth_path": "static", "principal": "local-cli",
},
"insights": []map[string]any{{"text": "rest provenance probe", "wing": "hyperguild", "hall": "facts"}},
})
require.Equal(t, 200, rr.Code, rr.Body.String())
matches, _ := filepath.Glob(filepath.Join(brainDir, "wiki/hyperguild/facts/*.md"))
require.Len(t, matches, 1)
b, err := os.ReadFile(matches[0])
require.NoError(t, err)
assert.Contains(t, string(b), tc.wantOrigin)
assert.Contains(t, string(b), tc.wantBy)
})
}
}
@@ -43,26 +43,6 @@ var DefaultRules = []Rule{
// JSON, so both segments begin with "eyJ". // JSON, so both segments begin with "eyJ".
{Name: "jwt", RE: regexp.MustCompile(`eyJ[A-Za-z0-9_\-]{8,}\.eyJ[A-Za-z0-9_\-]{8,}\.[A-Za-z0-9_\-]{8,}`)}, {Name: "jwt", RE: regexp.MustCompile(`eyJ[A-Za-z0-9_\-]{8,}\.eyJ[A-Za-z0-9_\-]{8,}\.[A-Za-z0-9_\-]{8,}`)},
{Name: "postgres-uri-with-password", RE: regexp.MustCompile(`postgres(?:ql)?://[^:\s/]+:[^@\s/]+@`)}, {Name: "postgres-uri-with-password", RE: regexp.MustCompile(`postgres(?:ql)?://[^:\s/]+:[^@\s/]+@`)},
// Any credential embedded in a URL's userinfo, which is what a git credential
// store holds: `https://<user>:<token>@host`. The postgres rule above is the
// same shape narrowed to one scheme, and is kept ahead of this one only so a
// DSN reports the more specific name -- same ordering convention as
// authorization-header before bearer-token.
//
// Added after this shape passed ALL rules and was therefore ingested rather
// than dropped (hyperguild#142). It reached ~/.claude/projects -- this
// watcher's own input -- because a test suite in mathias/infra executed a
// script that configures git while inheriting the operator's real $HOME, read
// koala's live credential store, and printed it. Two credentials were rotated
// (infra#466). The scrubber is the control that decides whether that class of
// mistake is survivable, and for this shape it was failing open.
//
// The {8,} floor on the secret half is what keeps `https://host:443/path` and
// `https://user:@host` out; the `[^:\s/@]+` user half stops a host:port pair
// being read as userinfo. Both directions are covered by
// TestScrub_URLUserinfoCredential, because a rule that dropped every URL with
// a colon in it would discard most of the signal this watcher exists to keep.
{Name: "url-userinfo-credential", RE: regexp.MustCompile(`[a-zA-Z][a-zA-Z0-9+.\-]*://[^:\s/@]+:[^@\s/]{8,}@`)},
{Name: "private-key", RE: regexp.MustCompile(`-----BEGIN[^-]*PRIVATE KEY-----`)}, {Name: "private-key", RE: regexp.MustCompile(`-----BEGIN[^-]*PRIVATE KEY-----`)},
{Name: "ssh-key", RE: regexp.MustCompile(`ssh-(?:rsa|ed25519|ecdsa)\s+[A-Za-z0-9+/=]{40,}`)}, {Name: "ssh-key", RE: regexp.MustCompile(`ssh-(?:rsa|ed25519|ecdsa)\s+[A-Za-z0-9+/=]{40,}`)},
{Name: "github-pat", RE: regexp.MustCompile(`\b(?:ghp|gho|ghu|ghr|gha)_[A-Za-z0-9]{30,}\b`)}, {Name: "github-pat", RE: regexp.MustCompile(`\b(?:ghp|gho|ghu|ghr|gha)_[A-Za-z0-9]{30,}\b`)},
@@ -46,64 +46,6 @@ func TestScrub_PoisonedFixtures(t *testing.T) {
} }
} }
// A credential embedded in a URL's userinfo is the shape a git credential store
// holds -- `https://<user>:<token>@host` -- and it walked past every rule until
// hyperguild#142. There was a rule for `postgres://user:pass@` and nothing for the
// general form, so `~/.git-credentials` content was ingested rather than dropped.
//
// Found the hard way: a test suite in mathias/infra executed a script that
// configures git while inheriting the operator's real $HOME, read koala's live
// credential store, and printed two live credentials into a session transcript.
// ~/.claude/projects is this watcher's own input, so the scrubber was the last
// line of defence for that mistake and it failed open. Two credentials were
// rotated (infra#466).
//
// The lesson is not "do not read the credential file" -- that mistake will recur.
// It is that this shape must fail closed, because the scrubber is what decides
// whether such a mistake is survivable.
func TestScrub_URLUserinfoCredential(t *testing.T) {
// Every secret below is INVENTED. None is or was a real credential.
mustScrub := []struct {
name string
content string
}{
{"git-credential-store-user", "https://mathias:0123456789abcdef0123456789abcdef01234567@git.d-ma.be"},
{"git-credential-store-oauth2", "https://oauth2:0123456789abcdef0123456789abcdef01234567@git.d-ma.be"},
{"in-context-payload-line", `payload line: https://mathias:0123456789abcdef0123456789abcdef01234567@git.d-ma.be`},
{"clone-url-in-argv", "git clone https://oauth2:abcdefghijklmnopqrstuvwx@git.d-ma.be/mathias/infra.git"},
{"http-not-https", "http://admin:supersecretvalue123@10.0.1.20/api"},
{"git-plus-ssh-scheme", "git+ssh://deploy:abcdefghijklmnop@example.com/repo"},
{"redis-scheme", "redis://default:abcdefghijklmnopqrst@10.0.1.20:6379/0"},
}
for _, tc := range mustScrub {
t.Run(tc.name, func(t *testing.T) {
assert.NotEmpty(t, Scrub(tc.content),
"a credential in URL userinfo must be dropped, not ingested")
})
}
// The other direction matters as much: a rule that scrubs every URL with a
// colon in it would drop most of the signal this watcher exists to keep.
mustNotScrub := []struct {
name string
content string
}{
{"host-with-port", "cloning https://git.d-ma.be:443/mathias/infra.git"},
{"postgres-no-password", "postgres://localhost:5432/brain"},
{"url-with-colon-in-query", "GET https://example.com/path?filter=a:b&x=1"},
{"plain-url", "see https://git.d-ma.be/mathias/infra/issues/466"},
{"userinfo-no-secret", "https://user:@example.com"},
{"format-string-placeholder", `printf 'https://oauth2:%s@%s\n' "$TOKEN" "$HOST"`},
{"scheme-relative-time", "duration https://example.com took 00:03:22"},
}
for _, tc := range mustNotScrub {
t.Run(tc.name, func(t *testing.T) {
assert.Empty(t, Scrub(tc.content),
"this is not a credential and must pass through")
})
}
}
func TestScrub_CleanContentPassesThrough(t *testing.T) { func TestScrub_CleanContentPassesThrough(t *testing.T) {
cases := []string{ cases := []string{
"", "",
@@ -1,130 +0,0 @@
package extract
import (
"go/ast"
"go/parser"
"go/token"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
)
const extractPkg = "github.com/mathiasbq/hyperguild/ingestion/internal/extract"
// parseNonTest parses every non-test Go file under the module.
func parseNonTest(t *testing.T) map[string]*ast.File {
t.Helper()
fset := token.NewFileSet()
out := map[string]*ast.File{}
err := filepath.WalkDir(filepath.Join("..", ".."), func(p string, d os.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() && (d.Name() == "vendor" || d.Name() == ".git") {
return filepath.SkipDir
}
if d.IsDir() || !strings.HasSuffix(p, ".go") || strings.HasSuffix(p, "_test.go") {
return nil
}
f, perr := parser.ParseFile(fset, p, nil, 0)
if perr != nil {
return perr
}
out[filepath.ToSlash(p)] = f
return nil
})
if err != nil {
t.Fatal(err)
}
return out
}
// PR #156 review: reading a caller-chosen path by name reopens it after the
// containment check (hyperguild#153 item 3). Outside this package, nothing
// may reference extract.Text — called OR taken as a value — under whatever
// name the package is imported (round 2: an alias and a func value slipped
// past a plain text search).
func TestNoPackageReferencesExtractTextOutsideExtract(t *testing.T) {
for path, f := range parseNonTest(t) {
if strings.Contains(path, "internal/extract/") {
continue
}
names := map[string]bool{}
for _, imp := range f.Imports {
if p, _ := strconv.Unquote(imp.Path.Value); p == extractPkg {
name := "extract"
if imp.Name != nil {
name = imp.Name.Name
}
if name == "." {
t.Errorf("%s dot-imports extract; import it by name so its use stays checkable", path)
}
names[name] = true
}
}
if len(names) == 0 {
continue
}
ast.Inspect(f, func(n ast.Node) bool {
if sel, ok := n.(*ast.SelectorExpr); ok {
if id, ok := sel.X.(*ast.Ident); ok && names[id.Name] && sel.Sel.Name == "Text" {
t.Errorf("%s references %s.Text; use extract.TextIn through an os.Root", path, id.Name)
}
}
return true
})
}
}
// PR #156 review round 2: the FIFO fix (open non-blocking, check the handle)
// lives in OpenRegular, and reverting any caller to a plain Root.Open stayed
// green. In the three raw/ reader FILES parsed below (and only those: a
// wrapper in another file is not seen), no Open/OpenFile/ReadFile selector
// may appear outside OpenRegular, called or taken as a method value. This is
// a second layer; the behavioural race tests in internal/api and
// internal/watcher are what catch a rewrite through any API (round 3).
// osOpenAllowed names the functions in guarded files that may call os.Open*
// on a path that is not a raw/ read.
var osOpenAllowed = map[string]bool{
"internal/watcher/watcher.go:appendWatcherLog": true, // ingest log append, not a raw/ read
// Text reads by path; outside this package only TextIn's temp copy reaches
// it (TestNoPackageReferencesExtractTextOutsideExtract enforces that).
"internal/extract/extract.go:Text": true,
}
func TestRawReaderFilesUseNoOpenOrReadFileOutsideOpenRegular(t *testing.T) {
guarded := []string{"internal/api/rawroot.go", "internal/extract/extract.go", "internal/watcher/watcher.go"}
files := parseNonTest(t)
for _, g := range guarded {
var f *ast.File
for p, af := range files {
if strings.HasSuffix(p, g) {
f = af
}
}
if f == nil {
t.Fatalf("guarded file %s not found", g)
}
for _, decl := range f.Decls {
fn, ok := decl.(*ast.FuncDecl)
if !ok || fn.Name.Name == "OpenRegular" {
continue
}
ast.Inspect(fn, func(n ast.Node) bool {
sel, ok := n.(*ast.SelectorExpr)
if !ok {
return true
}
if sel.Sel.Name == "Open" || sel.Sel.Name == "OpenFile" || sel.Sel.Name == "ReadFile" {
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "os" && osOpenAllowed[g+":"+fn.Name.Name] {
return true // a named non-raw write (the ingest log append)
}
t.Errorf("%s: %s calls .%s; raw/ reads must go through OpenRegular", g, fn.Name.Name, sel.Sel.Name)
}
return true
})
}
}
}
-69
View File
@@ -3,10 +3,8 @@ package extract
import ( import (
"fmt" "fmt"
"io"
"os" "os"
"strings" "strings"
"syscall"
) )
// Text reads the file at path and returns its plain-text content. // Text reads the file at path and returns its plain-text content.
@@ -43,70 +41,3 @@ func fileExt(path string) string {
} }
return "" return ""
} }
// TextIn is Text for a file read through an os.Root (hyperguild#153): the
// read cannot leave the root, so a path checked earlier cannot be swapped
// for a symlink pointing elsewhere before it is read. Formats that need an
// external tool (pdf, docmark) are copied to a temp file from the handle
// opened through the root, then extracted from the copy.
func TextIn(r *os.Root, rel string) (string, error) {
fi, err := r.Lstat(rel)
if err != nil {
return "", fmt.Errorf("stat %s: %w", rel, err)
}
if !fi.Mode().IsRegular() {
return "", fmt.Errorf("%s: not a regular file", rel)
}
f, _, err := OpenRegular(r, rel)
if err != nil {
return "", fmt.Errorf("open %s: %w", rel, err)
}
defer func() { _ = f.Close() }()
ext := strings.ToLower(fileExt(rel))
switch ext {
case ".md", ".txt":
b, err := io.ReadAll(f)
if err != nil {
return "", fmt.Errorf("read %s: %w", rel, err)
}
return string(b), nil
case ".pdf", ".docx", ".xlsx", ".pptx", ".png", ".jpg", ".jpeg":
tmp, err := os.CreateTemp("", "ingest-*"+ext)
if err != nil {
return "", fmt.Errorf("temp copy: %w", err)
}
defer func() { _ = os.Remove(tmp.Name()) }()
if _, err := io.Copy(tmp, f); err != nil {
_ = tmp.Close()
return "", fmt.Errorf("copy %s: %w", rel, err)
}
if err := tmp.Close(); err != nil {
return "", fmt.Errorf("copy %s: %w", rel, err)
}
return Text(tmp.Name())
default:
return "", fmt.Errorf("unsupported file extension: %s", ext)
}
}
// OpenRegular opens rel inside r and returns it only if the OPENED handle is
// a regular file. The open is non-blocking, so a FIFO swapped in after an
// Lstat (PR #156 review: it hung open(2) forever) is refused promptly rather
// than stalling the reader. Callers keep their Lstat check, which refuses
// symlinks outright; this closes the race between that check and the open.
func OpenRegular(r *os.Root, rel string) (*os.File, os.FileInfo, error) {
f, err := r.OpenFile(rel, os.O_RDONLY|syscall.O_NONBLOCK, 0)
if err != nil {
return nil, nil, err
}
fi, err := f.Stat()
if err != nil {
_ = f.Close()
return nil, nil, err
}
if !fi.Mode().IsRegular() {
_ = f.Close()
return nil, nil, fmt.Errorf("%s: not a regular file", rel)
}
return f, fi, nil
}
-64
View File
@@ -1,64 +0,0 @@
package extract
import (
"os"
"path/filepath"
"syscall"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// hyperguild#153 item 3: ingest checked a path, then extract.Text reopened it
// by name, so a file swapped for a symlink in between was followed out of
// raw/. TextIn reads through an os.Root, which refuses to leave the root.
func TestTextInReadsInsideTheRootAndRefusesEscapes(t *testing.T) {
root := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(root, "ok.md"), []byte("inside"), 0o644))
outside := filepath.Join(t.TempDir(), "secret.md")
require.NoError(t, os.WriteFile(outside, []byte("SENTINEL-153-TEXTIN"), 0o644))
require.NoError(t, os.Symlink(outside, filepath.Join(root, "leak.md")))
r, err := os.OpenRoot(root)
require.NoError(t, err)
defer func() { _ = r.Close() }()
got, err := TextIn(r, "ok.md")
require.NoError(t, err)
assert.Equal(t, "inside", got)
got, err = TextIn(r, "leak.md")
assert.Error(t, err)
assert.NotContains(t, got, "SENTINEL-153-TEXTIN")
_, err = TextIn(r, "../"+filepath.Base(outside))
assert.Error(t, err)
}
// PR #156 review: a regular file swapped for a FIFO between Lstat and Open
// blocked open(2) forever. OpenRegular opens non-blocking and checks the type
// on the opened handle, so a FIFO is refused promptly instead of hanging.
func TestOpenRegularRefusesAFIFOWithoutBlocking(t *testing.T) {
dir := t.TempDir()
require.NoError(t, syscall.Mkfifo(filepath.Join(dir, "pipe.md"), 0o644))
r, err := os.OpenRoot(dir)
require.NoError(t, err)
defer func() { _ = r.Close() }()
done := make(chan error, 1)
go func() {
f, _, err := OpenRegular(r, "pipe.md")
if f != nil {
_ = f.Close()
}
done <- err
}()
select {
case err := <-done:
assert.Error(t, err)
case <-time.After(2 * time.Second):
t.Fatal("OpenRegular blocked on a FIFO")
}
}
+7 -31
View File
@@ -66,7 +66,7 @@ func Fix(doc []byte) ([]byte, bool, error) {
return doc, false, fmt.Errorf("frontmatter: quoting did not repair the block: %w", err) return doc, false, fmt.Errorf("frontmatter: quoting did not repair the block: %w", err)
} }
_, rest, _ := Split(doc) _, rest, _ := split(doc)
out := []byte(fence + "\n") out := []byte(fence + "\n")
out = append(out, fixed...) out = append(out, fixed...)
out = append(out, []byte(fence)...) out = append(out, []byte(fence)...)
@@ -122,25 +122,14 @@ func needsQuoting(value string) bool {
// block returns the raw YAML between the opening and closing fences. // block returns the raw YAML between the opening and closing fences.
func block(doc []byte) ([]byte, bool) { func block(doc []byte) ([]byte, bool) {
b, _, ok := Split(doc) b, _, ok := split(doc)
return b, ok return b, ok
} }
// Split returns doc's frontmatter block and everything from the closing // split returns the frontmatter block and everything from the closing fence
// fence onward (the closing fence itself excluded, its trailing newline // onward (the closing fence itself excluded, its trailing newline included).
// included), or ok=false if doc has no properly-fenced frontmatter. func split(doc []byte) (block, rest []byte, ok bool) {
// s := string(doc)
// The canonical fence detector for this codebase (hyperguild#138): three
// independent reimplementations of this same logic each grew their own gap
// through separate bug-fixing before being consolidated here —
// internal/mcp's needed a CRLF/BOM guard added after both silently defeated
// its own HasPrefix check, and internal/api's closing-fence match had no
// trailing-newline requirement, so a document ending in "\n---extra" with no
// real newline after the fence was treated as validly closed. Both classes
// are closed here, once, for every caller.
func Split(doc []byte) (block, rest []byte, ok bool) {
s := strings.TrimPrefix(string(doc), "\xef\xbb\xbf")
s = strings.ReplaceAll(s, "\r\n", "\n")
if !strings.HasPrefix(s, fence+"\n") { if !strings.HasPrefix(s, fence+"\n") {
return nil, nil, false return nil, nil, false
} }
@@ -150,18 +139,5 @@ func Split(doc []byte) (block, rest []byte, ok bool) {
if idx < 0 { if idx < 0 {
return nil, nil, false return nil, nil, false
} }
// What follows the closing fence must be either a newline (a real body return []byte(body[:idx+1]), []byte(body[idx+1+len(fence):]), true
// follows) or nothing at all (the fence is the literal end of the
// document — no body, not a bypass attempt). Anything else, e.g.
// "\n---extra", is not a valid close: an earlier version of this check
// accepted it, which is exactly the looser match hyperguild#138 closes.
after := idx + 1 + len(fence)
switch {
case after == len(body):
return []byte(body[:idx+1]), nil, true
case body[after] == '\n':
return []byte(body[:idx+1]), []byte(body[after:]), true
default:
return nil, nil, false
}
} }
@@ -168,9 +168,9 @@ func TestFix_PreservesBodyExactly(t *testing.T) {
require.True(t, changed) require.True(t, changed)
assert.True(t, len(got) > 0) assert.True(t, len(got) > 0)
_, rest, ok := Split([]byte(doc)) _, rest, ok := split([]byte(doc))
require.True(t, ok) require.True(t, ok)
_, gotRest, ok := Split(got) _, gotRest, ok := split(got)
require.True(t, ok) require.True(t, ok)
assert.Equal(t, string(rest), string(gotRest), "the body must not be touched") assert.Equal(t, string(rest), string(gotRest), "the body must not be touched")
} }
@@ -186,55 +186,3 @@ func TestFix_RefusesToReturnAnUnparseableResult(t *testing.T) {
assert.False(t, changed) assert.False(t, changed)
assert.Equal(t, doc, string(got), "an unfixable document must come back byte-identical") assert.Equal(t, doc, string(got), "an unfixable document must come back byte-identical")
} }
// TestSplit_NormalizesCRLFBeforeFenceDetection is hyperguild#138: a note
// whose fence lines use CRLF ("---\r\n") failed strings.HasPrefix(doc,
// "---\n") on the precondition read in internal/mcp's independent
// reimplementation -- encoding alone, no adversarial input, silently
// defeated fence detection there before this consolidation. Split must
// never repeat that.
func TestSplit_NormalizesCRLFBeforeFenceDetection(t *testing.T) {
doc := "---\r\ntitle: 'n'\r\nhalf_life: durable\r\n---\r\n\r\nbody\r\n"
block, rest, ok := Split([]byte(doc))
require.True(t, ok)
assert.Contains(t, string(block), "half_life: durable")
assert.Contains(t, string(rest), "body")
}
// TestSplit_StripsALeadingUTF8BOM is the same mechanism one layer earlier:
// a leading BOM (Notepad's default "UTF-8" save, PowerShell's pre-v6
// Out-File/Set-Content) defeats the same HasPrefix check before CRLF
// normalization even applies.
func TestSplit_StripsALeadingUTF8BOM(t *testing.T) {
doc := "\xef\xbb\xbf---\ntitle: 'n'\n---\n\nbody\n"
block, rest, ok := Split([]byte(doc))
require.True(t, ok)
assert.Contains(t, string(block), "title: 'n'")
assert.Contains(t, string(rest), "body")
}
// TestSplit_RequiresANewlineAfterTheClosingFence closes the looser match
// internal/api's independent reimplementation carried (strings.Index(body,
// "\n"+fence) with no trailing-newline requirement): a document ending in
// "\n---extra" with no newline separating the fence from what follows it is
// not a valid closing fence and must not be treated as one.
func TestSplit_RequiresANewlineAfterTheClosingFence(t *testing.T) {
doc := "---\ntitle: 'n'\n---extra-not-a-real-close"
_, _, ok := Split([]byte(doc))
assert.False(t, ok, "a fence line not followed by a newline is not a valid closing fence")
}
// TestSplit_AcceptsAClosingFenceThatIsTheLastBytesOfTheDocument is a
// regression pin found in review of hyperguild#138: the fix for the
// "\n---extra" bypass above must not also reject the legitimate case of a
// closing fence with nothing after it at all (no trailing newline because
// there's no body, not because a bypass is being attempted) -- this
// package's own private split() accepted that shape before the
// consolidation, and cmd/frontmatterlint's Lint/Fix relied on it.
func TestSplit_AcceptsAClosingFenceThatIsTheLastBytesOfTheDocument(t *testing.T) {
doc := "---\ntitle: 'n'\n---"
block, rest, ok := Split([]byte(doc))
require.True(t, ok, "a closing fence at literal end-of-file, with nothing after it, is valid -- there is no body to require a newline before")
assert.Contains(t, string(block), "title: 'n'")
assert.Equal(t, "", string(rest))
}
+21 -43
View File
@@ -12,14 +12,10 @@ package graphsync
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"io/fs" "os"
"path"
"path/filepath" "path/filepath"
"strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"github.com/mathiasbq/hyperguild/ingestion/internal/graph" "github.com/mathiasbq/hyperguild/ingestion/internal/graph"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphstore" "github.com/mathiasbq/hyperguild/ingestion/internal/graphstore"
) )
@@ -50,25 +46,8 @@ func IndexDoc(ctx context.Context, store Store, brainDir, relPath string) error
if relPath == "" { if relPath == "" {
return nil return nil
} }
// Callers pass caller-supplied paths (brain_tunnel, brain_update): index abs := filepath.Join(brainDir, filepath.FromSlash(relPath))
// only a clean path under wiki/ or knowledge/. content, err := os.ReadFile(abs)
if path.Clean(relPath) != relPath || !filepath.IsLocal(relPath) ||
(!strings.HasPrefix(relPath, "wiki/") && !strings.HasPrefix(relPath, "knowledge/")) {
return fmt.Errorf("index %q: not a clean wiki/ or knowledge/ path", relPath)
}
bfs, err := brainfs.Open(brainDir)
if err != nil {
return fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
return indexDoc(ctx, store, bfs, relPath)
}
// indexDoc reads relPath through the brain root, regular files only
// (hyperguild#157): the brain is a Syncthing folder, and os.ReadFile
// followed a synced symlink into the graph and blocked on a FIFO.
func indexDoc(ctx context.Context, store Store, bfs *brainfs.Brain, relPath string) error {
content, err := bfs.ReadRegular(relPath)
if err != nil { if err != nil {
return fmt.Errorf("read %q: %w", relPath, err) return fmt.Errorf("read %q: %w", relPath, err)
} }
@@ -98,34 +77,33 @@ func BackfillFromBrainDir(ctx context.Context, store Store, brainDir string) (in
if store == nil { if store == nil {
return 0, nil return 0, nil
} }
bfs, err := brainfs.Open(brainDir) roots := []string{"wiki", "knowledge"}
if err != nil { for _, root := range roots {
return 0, fmt.Errorf("open brain: %w", err) base := filepath.Join(brainDir, root)
} if _, err := os.Stat(base); os.IsNotExist(err) {
defer func() { _ = bfs.Close() }()
for _, root := range []string{"wiki", "knowledge"} {
if _, err := bfs.Lstat(root); errors.Is(err, fs.ErrNotExist) {
continue continue
} }
// Any walk error, including a root that is a symlink, aborts. err := filepath.WalkDir(base, func(path string, d os.DirEntry, walkErr error) error {
var walkErr error if walkErr != nil {
err := bfs.WalkRegular(root, func(rel string) error { return walkErr
if path.Ext(rel) != ".md" { }
if d.IsDir() {
return nil return nil
} }
if err := indexDoc(ctx, store, bfs, rel); err != nil { if filepath.Ext(path) != ".md" {
return nil
}
rel, relErr := filepath.Rel(brainDir, path)
if relErr != nil {
return fmt.Errorf("rel %q: %w", path, relErr)
}
rel = filepath.ToSlash(rel)
if err := IndexDoc(ctx, store, brainDir, rel); err != nil {
return fmt.Errorf("index %q: %w", rel, err) return fmt.Errorf("index %q: %w", rel, err)
} }
indexed++ indexed++
return nil return nil
}, func(rel string, err error) {
if walkErr == nil {
walkErr = fmt.Errorf("%s: %w", rel, err)
}
}) })
if err == nil {
err = walkErr
}
if err != nil { if err != nil {
return indexed, fmt.Errorf("walk %s: %w", root, err) return indexed, fmt.Errorf("walk %s: %w", root, err)
} }
@@ -1,92 +0,0 @@
package graphsync
import (
"context"
"fmt"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs/brainfstest"
)
func (f *fakeStore) dump() string {
f.mu.Lock()
defer f.mu.Unlock()
return fmt.Sprintf("%+v %+v", f.upserts, f.replaces)
}
// hyperguild#157 slice 4: IndexDoc and BackfillFromBrainDir read notes with
// os.ReadFile, so a symlinked note put the target's title and wikilinks into
// the graph, and a FIFO hung the read.
func TestIndexDocRefusesSymlinksAndSurvivesSwapRaces(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(facts, "ok.md"), []byte("---\ntitle: zqinside\n---\nbody [[other]]\n"), 0o644))
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(facts, "leak.md")))
store := newFakeStore()
require.NoError(t, IndexDoc(context.Background(), store, brain, "wiki/w/facts/ok.md"))
assert.Contains(t, store.dump(), "zqinside", "positive control: a regular note is indexed")
assert.Error(t, IndexDoc(context.Background(), store, brain, "wiki/w/facts/leak.md"))
// A link to a file elsewhere in the brain: os.Root allows it, only the
// regular-file check refuses it.
inBrain := brainfstest.WriteSentinelAt(t, filepath.Join(brain, "raw", "s.md"))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "s.md"), filepath.Join(facts, "inlink.md")))
assert.Error(t, IndexDoc(context.Background(), store, brain, "wiki/w/facts/inlink.md"))
assert.NotContains(t, store.dump(), brainfstest.Sentinel)
p := filepath.Join(facts, "r.md")
read := func() string {
s := newFakeStore()
_ = IndexDoc(context.Background(), s, brain, "wiki/w/facts/r.md")
return s.dump()
}
brainfstest.AssertNoLeak(t, p, "---\ntitle: zqinside\n---\n", 1500*time.Millisecond, read)
brainfstest.AssertNoLeakFrom(t, p, "---\ntitle: zqinside\n---\n", inBrain, 1500*time.Millisecond, read)
brainfstest.AssertNoHang(t, p, "---\ntitle: zqinside\n---\n", 1500*time.Millisecond, 6*time.Second, func() { _ = read() })
}
func TestBackfillSkipsSymlinkedNotesAndAbortsOnASymlinkedRoot(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(facts, "ok.md"), []byte("---\ntitle: zqinside\n---\n"), 0o644))
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(facts, "leak.md")))
require.NoError(t, os.MkdirAll(filepath.Join(brain, "raw"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(brain, "raw", "r.md"), []byte("---\ntitle: zqrawonly\n---\n"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "..", "raw", "r.md"), filepath.Join(facts, "inlink.md")))
require.NoError(t, os.Symlink(".", filepath.Join(brain, "knowledge")))
store := newFakeStore()
_, err := BackfillFromBrainDir(context.Background(), store, brain)
assert.Error(t, err, "a symlinked root aborts the backfill")
assert.NotContains(t, store.dump(), brainfstest.Sentinel)
assert.NotContains(t, store.dump(), "zqrawonly")
require.NoError(t, os.Remove(filepath.Join(brain, "knowledge")))
store = newFakeStore()
n, err := BackfillFromBrainDir(context.Background(), store, brain)
require.NoError(t, err)
assert.Equal(t, 1, n, "only the regular note is indexed")
assert.Contains(t, store.dump(), "zqinside")
assert.NotContains(t, store.dump(), brainfstest.Sentinel)
}
// IndexDoc gets caller-supplied paths (brain_tunnel, brain_update): only a
// clean path under wiki/ or knowledge/ is indexed.
func TestIndexDocRefusesPathsOutsideWikiAndKnowledgeAndUncleanPaths(t *testing.T) {
brain := t.TempDir()
brainfstest.WriteSentinelAt(t, filepath.Join(brain, "raw", "secret.md"))
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki", "w"), 0o755))
for _, p := range []string{"raw/secret.md", "wiki/w/../../raw/secret.md", "./raw/secret.md", "/raw/secret.md"} {
store := newFakeStore()
assert.Error(t, IndexDoc(context.Background(), store, brain, p), p)
assert.NotContains(t, store.dump(), brainfstest.Sentinel, p)
}
}
-48
View File
@@ -1,48 +0,0 @@
package mcp
import (
"context"
"encoding/json"
"log/slog"
"time"
)
// logToolCall emits one structured record per tools/call (hyperguild#144,
// OWASP MCP08): who called which tool, against which wing/hall, and how it
// ended. Brain content is never copied into the record — only its size — so
// the audit trail does not become a second, unscrubbed copy of the brain.
func logToolCall(ctx context.Context, name string, args json.RawMessage, start time.Time, err error) {
origin, principal := provenanceFrom(ctx)
attrs := []any{
"tool", name,
"principal", principal,
"origin", origin,
"duration_ms", time.Since(start).Milliseconds(),
}
var a struct {
Wing string `json:"wing"`
Hall string `json:"hall"`
Content string `json:"content"`
}
if json.Unmarshal(args, &a) == nil {
if a.Wing != "" {
attrs = append(attrs, "wing", a.Wing)
}
if a.Hall != "" {
attrs = append(attrs, "hall", a.Hall)
}
if a.Content != "" {
attrs = append(attrs, "content_bytes", len(a.Content))
}
}
// The outcome is a class, never the error text: handler errors echo
// argument values (a rejected half_life, a requested path), which would
// make the audit line a copy of what the caller sent (PR #147 review E6).
// The caller already received the full error in the tool response.
outcome := "ok"
if err != nil {
outcome = "error"
}
attrs = append(attrs, "outcome", outcome)
slog.InfoContext(ctx, "mcp tool call", attrs...)
}
-83
View File
@@ -1,83 +0,0 @@
package mcp_test
import (
"bytes"
"encoding/json"
"log/slog"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// captureToolCallLog routes slog to a buffer for fn and returns the decoded
// "mcp tool call" records.
func captureToolCallLog(t *testing.T, fn func()) []map[string]any {
t.Helper()
var buf bytes.Buffer
prev := slog.Default()
slog.SetDefault(slog.New(slog.NewJSONHandler(&buf, nil)))
t.Cleanup(func() { slog.SetDefault(prev) })
fn()
var out []map[string]any
for _, l := range strings.Split(strings.TrimSpace(buf.String()), "\n") {
var rec map[string]any
if json.Unmarshal([]byte(l), &rec) == nil && rec["msg"] == "mcp tool call" {
out = append(out, rec)
}
}
return out
}
// hyperguild#144: brain_query/brain_read left no record of who read what.
func TestToolsCallLogsOneRecordWithToolPrincipalOriginAndScope(t *testing.T) {
srv, _, _, _ := govServer(t)
recs := captureToolCallLog(t, func() {
authCall(t, srv, "Bearer "+capStaticTok, "brain_query",
map[string]any{"query": "anything", "wing": "homelab", "hall": "facts"})
})
require.Len(t, recs, 1)
r := recs[0]
assert.Equal(t, "brain_query", r["tool"])
assert.Equal(t, "local-cli", r["principal"])
assert.Equal(t, "static", r["origin"])
assert.Equal(t, "homelab", r["wing"])
assert.Equal(t, "facts", r["hall"])
assert.Equal(t, "ok", r["outcome"])
assert.Contains(t, r, "duration_ms")
}
func TestToolsCallWithoutDerivablePrincipalLogsOriginUnknown(t *testing.T) {
srv, _, _, _ := govServer(t)
recs := captureToolCallLog(t, func() {
authCall(t, srv, "", "brain_query", map[string]any{"query": "x"})
})
require.Len(t, recs, 1)
assert.Equal(t, "unknown", recs[0]["origin"])
assert.Equal(t, "", recs[0]["principal"])
}
func TestToolsCallLogsErrorOutcome(t *testing.T) {
srv, _, _, _ := govServer(t)
recs := captureToolCallLog(t, func() {
authCall(t, srv, "Bearer "+capStaticTok, "brain_query", map[string]any{})
})
require.Len(t, recs, 1)
assert.Equal(t, "error", recs[0]["outcome"])
}
// The audit line must not become a second copy of brain content.
func TestToolsCallLogOmitsContentButRecordsItsSize(t *testing.T) {
const sentinel = "SENTINEL-BRAIN-CONTENT-91c2"
srv, _, _, _ := govServer(t)
recs := captureToolCallLog(t, func() {
authCall(t, srv, "Bearer "+capStaticTok, "brain_write", map[string]any{
"content": "# t\n\n" + sentinel, "wing": "testing", "hall": "facts", "filename": "audit-sentinel",
})
})
require.Len(t, recs, 1)
raw, _ := json.Marshal(recs[0])
assert.NotContains(t, string(raw), sentinel)
assert.EqualValues(t, len("# t\n\n"+sentinel), recs[0]["content_bytes"])
}
+12 -33
View File
@@ -12,10 +12,10 @@ import (
"github.com/mathiasbq/hyperguild/ingestion/internal/api" "github.com/mathiasbq/hyperguild/ingestion/internal/api"
"github.com/mathiasbq/hyperguild/ingestion/internal/brain" "github.com/mathiasbq/hyperguild/ingestion/internal/brain"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture" "github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/extract"
"github.com/mathiasbq/hyperguild/ingestion/internal/graphsync" "github.com/mathiasbq/hyperguild/ingestion/internal/graphsync"
"github.com/mathiasbq/hyperguild/ingestion/internal/notes" "github.com/mathiasbq/hyperguild/ingestion/internal/notes"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline" "github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
"github.com/mathiasbq/hyperguild/ingestion/internal/provenance"
"github.com/mathiasbq/hyperguild/ingestion/internal/search" "github.com/mathiasbq/hyperguild/ingestion/internal/search"
"github.com/mathiasbq/hyperguild/ingestion/internal/session" "github.com/mathiasbq/hyperguild/ingestion/internal/session"
) )
@@ -255,14 +255,11 @@ func (s *Server) brainWrite(ctx context.Context, args json.RawMessage) (json.Raw
// one implementation (capture uses the same store). The read-after-write // one implementation (capture uses the same store). The read-after-write
// handle {id, path, content_hash} comes back from the store; path is kept // handle {id, path, content_hash} comes back from the store; path is kept
// for backward compatibility. // for backward compatibility.
origin, writtenBy := provenanceFrom(ctx)
ref, err := s.store.Write(ctx, capture.Note{ ref, err := s.store.Write(ctx, capture.Note{
Content: a.Content, Content: a.Content,
Filename: a.Filename, Filename: a.Filename,
Wing: a.Wing, Wing: a.Wing,
Hall: a.Hall, Hall: a.Hall,
Origin: origin,
WrittenBy: writtenBy,
}) })
if err != nil { if err != nil {
return nil, err return nil, err
@@ -307,14 +304,11 @@ func (s *Server) brainUpdate(ctx context.Context, args json.RawMessage) (json.Ra
if a.Path != "" { if a.Path != "" {
slug = a.Path slug = a.Path
} }
origin, writtenBy := provenanceFrom(ctx)
ref, err := s.store.Update(ctx, slug, capture.Note{ ref, err := s.store.Update(ctx, slug, capture.Note{
Content: a.Content, Content: a.Content,
Wing: a.Wing, Wing: a.Wing,
Hall: a.Hall, Hall: a.Hall,
Reason: a.Reason, Reason: a.Reason,
Origin: origin,
WrittenBy: writtenBy,
}) })
if err != nil { if err != nil {
return nil, err return nil, err
@@ -393,10 +387,8 @@ func (s *Server) brainPromote(ctx context.Context, args json.RawMessage) (json.R
if err := json.Unmarshal(args, &a); err != nil { if err := json.Unmarshal(args, &a); err != nil {
return nil, fmt.Errorf("parse args: %w", err) return nil, fmt.Errorf("parse args: %w", err)
} }
origin, writtenBy := provenanceFrom(ctx)
relPath, err := api.PromoteNote(s.brainDir, api.PromoteOptions{ relPath, err := api.PromoteNote(s.brainDir, api.PromoteOptions{
Filename: a.Filename, Wing: a.Wing, Hall: a.Hall, Slug: a.Slug, Filename: a.Filename, Wing: a.Wing, Hall: a.Hall, Slug: a.Slug,
Origin: origin, WrittenBy: writtenBy,
}) })
if err != nil { if err != nil {
return nil, err return nil, err
@@ -499,7 +491,7 @@ func (s *Server) brainIngestRaw(ctx context.Context, args json.RawMessage) (json
if len(a.Pages) == 0 { if len(a.Pages) == 0 {
return nil, fmt.Errorf("pages must be non-empty") return nil, fmt.Errorf("pages must be non-empty")
} }
result, err := pipeline.RunRaw(withCallerStamp(ctx), s.brainDir, a.Source, a.Pages, a.DryRun) result, err := pipeline.RunRaw(s.brainDir, a.Source, a.Pages, a.DryRun)
if err != nil { if err != nil {
return nil, fmt.Errorf("ingest: %w", err) return nil, fmt.Errorf("ingest: %w", err)
} }
@@ -542,13 +534,7 @@ func (s *Server) brainIngest(ctx context.Context, args json.RawMessage) (json.Ra
} }
if a.Path != "" { if a.Path != "" {
// Confined to brain/raw/ with symlinks evaluated (hyperguild#149). text, err := extract.Text(a.Path)
resolved, err := api.ResolveIngestPath(s.brainDir, a.Path)
if err != nil {
return nil, err
}
a.Path = resolved
text, err := api.ReadIngestFile(s.brainDir, a.Path)
if err != nil { if err != nil {
return nil, fmt.Errorf("extract: %w", err) return nil, fmt.Errorf("extract: %w", err)
} }
@@ -604,7 +590,7 @@ func (s *Server) sessionLog(ctx context.Context, args json.RawMessage) (json.Raw
} }
func (s *Server) runIngest(ctx context.Context, content, source string, dryRun bool) (json.RawMessage, error) { func (s *Server) runIngest(ctx context.Context, content, source string, dryRun bool) (json.RawMessage, error) {
result, err := pipeline.Run(withCallerStamp(ctx), s.pipeline, s.brainDir, content, source, dryRun) result, err := pipeline.Run(ctx, s.pipeline, s.brainDir, content, source, dryRun)
if err != nil { if err != nil {
return nil, fmt.Errorf("ingest: %w", err) return nil, fmt.Errorf("ingest: %w", err)
} }
@@ -623,10 +609,3 @@ func (s *Server) runIngest(ctx context.Context, content, source string, dryRun b
} }
return json.Marshal(map[string]any{"pages": pages, "warnings": warnings}) return json.Marshal(map[string]any{"pages": pages, "warnings": warnings})
} }
// withCallerStamp attaches the caller's provenance for the ingest pipeline,
// which stamps every page it writes or merges into (hyperguild#145).
func withCallerStamp(ctx context.Context) context.Context {
origin, writtenBy := provenanceFrom(ctx)
return provenance.WithStamp(ctx, provenance.Stamp{Origin: origin, WrittenBy: writtenBy})
}
@@ -1,24 +0,0 @@
package mcp_test
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/mcp"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
)
// hyperguild#149: brain_ingest's `path` read any file the pod could read and
// ingested it into the brain, where brain_query reads it back out.
func TestBrainIngestRefusesAPathOutsideTheRawInbox(t *testing.T) {
stub := pipeline.Config{Complete: func(context.Context, string, string) (string, error) {
return `[{"title":"Leak","type":"concept","content":"x"}]`, nil
}}
srv := mcp.NewServer(newBrainRepo(t), &stub, nil, nil)
resp := authCall(t, srv, "", "brain_ingest", map[string]any{"path": "/etc/hostname"})
require.NotNil(t, resp["error"], "a path outside brain/raw/ must be refused")
assert.Contains(t, resp["error"].(map[string]any)["message"], "brain/raw/")
}
@@ -1,325 +0,0 @@
package mcp_test
import (
"bytes"
"context"
"encoding/json"
"log/slog"
"os"
"path/filepath"
"regexp"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gopkg.in/yaml.v3"
"github.com/mathiasbq/hyperguild/ingestion/internal/audit"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/capturehttp"
"github.com/mathiasbq/hyperguild/ingestion/internal/classification"
"github.com/mathiasbq/hyperguild/ingestion/internal/mcp"
"github.com/mathiasbq/hyperguild/ingestion/internal/pipeline"
)
// Escapes found by the PR #147 gatekeeper review. Each test drives the real
// mcp.Server as a JWT principal "attacker" and asserts the provenance
// invariant on the file every reader sees.
const jwtAuthz = "Bearer jwt-token"
func jwtServer(t *testing.T) (*mcp.Server, string) {
t.Helper()
dir := newBrainRepo(t)
cfg, err := classification.Load(dir)
require.NoError(t, err)
svc := capture.NewService(brainstore.New(dir), capFakeTracker{}, cfg, audit.NewSlogSink(nil))
srv := mcp.NewServer(dir, nil, nil, nil)
srv.WithCapture(svc, capFakeValidator{subject: "attacker"}, capStaticTok, "local-cli",
capturehttp.NewOriginResolver(nil))
return srv, dir
}
var provenanceLine = regexp.MustCompile(`(?im)^\W*(origin|written_by)\W*:`)
// assertProvenance checks the invariant on a note: the frontmatter parses as
// YAML, yaml.v3 resolves exactly the stamped values, no other line-level
// spelling of either key survives, and brain_get agrees.
func assertProvenance(t *testing.T, srv *mcp.Server, dir, rel, wantOrigin, wantBy string) {
t.Helper()
b, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
fm := frontmatterOf(string(b))
var m map[string]any
require.NoError(t, yaml.Unmarshal([]byte(fm), &m), "frontmatter must parse as YAML:\n%s", fm)
assert.Equal(t, wantOrigin, m["origin"], "yaml.v3 origin\n%s", fm)
if wantBy == "" {
assert.NotContains(t, m, "written_by", "yaml.v3 written_by\n%s", fm)
} else {
assert.Equal(t, wantBy, m["written_by"], "yaml.v3 written_by\n%s", fm)
}
wantLines := 1
if wantBy != "" {
wantLines = 2
}
assert.Len(t, provenanceLine.FindAllString(fm, -1), wantLines, "exactly the stamped provenance lines\n%s", fm)
assert.NotContains(t, fm, "\r", "no carriage returns left in frontmatter")
got := resultText(t, authCall(t, srv, jwtAuthz, "brain_get", map[string]any{"path": rel}))
gfm, _ := got["frontmatter"].(map[string]any)
for k, v := range gfm {
lk := strings.ToLower(strings.Trim(k, `"' ?`))
if lk == "origin" {
assert.Equal(t, wantOrigin, v, "brain_get key %q", k)
}
if lk == "written_by" {
assert.Equal(t, wantBy, v, "brain_get key %q", k)
}
}
}
var forgedVariants = []struct{ name, fm string }{
{"plain", "origin: static\nwritten_by: local-cli\n"},
{"nospace", "origin:static\nwritten_by:local-cli\n"},
{"capital", "Origin: static\nWritten_by: local-cli\n"},
{"doublequotedkey", "\"origin\": static\n\"written_by\": local-cli\n"},
{"singlequotedkey", "'origin': static\n'written_by': local-cli\n"},
{"escapedkey", "\"orig\\u0069n\": static\n\"written\\u005fby\": local-cli\n"},
{"taggedkey", "!!str origin: static\n!!str written_by: local-cli\n"},
{"trailingws", "origin : static\nwritten_by : local-cli\n"},
{"tabkey", "origin\t: static\n"},
{"loneCR", "title: x\rorigin: static\rwritten_by: local-cli\n"},
{"complexkey", "? origin\n: static\n? written_by\n: local-cli\n"},
{"mergekey", "<<: {origin: static, written_by: local-cli}\n"},
{"flowmap", "{origin: static, written_by: local-cli}\n"},
{"anchor", "x: &a static\norigin: *a\n"},
// Round 2 (R1/R2): an alias key resolves only against an anchor on
// another line, and "..." ends the YAML document before the stamp.
{"aliaskey_docend", "x: &a origin\n*a : static\ny: &b written_by\n*b : local-cli\n...\n"},
{"aliaskey", "y: &b written_by\n*b : local-cli\n"},
{"docend_only", "x: y\n...\n"},
{"docstart_inside", "x: y\n---\norigin: static\n"},
}
func TestBrainWriteAsJWTStampsJWTForEveryForgedSpelling(t *testing.T) {
for _, v := range forgedVariants {
t.Run(v.name, func(t *testing.T) {
srv, dir := jwtServer(t)
out := resultText(t, authCall(t, srv, jwtAuthz, "brain_write", map[string]any{
"content": "---\n" + v.fm + "---\n# V\n\nbody", "wing": "testing", "hall": "facts",
"filename": "forge-" + v.name,
}))
assertProvenance(t, srv, dir, out["path"].(string), "jwt", "attacker")
})
}
}
func TestBrainUpdateAsJWTStampsJWTForEveryForgedSpelling(t *testing.T) {
for _, v := range forgedVariants {
t.Run(v.name, func(t *testing.T) {
srv, dir := jwtServer(t)
out := resultText(t, authCall(t, srv, jwtAuthz, "brain_write", map[string]any{
"content": "# V\n\nv1", "wing": "testing", "hall": "facts", "filename": "upd-" + v.name,
}))
path := out["path"].(string)
resultText(t, authCall(t, srv, jwtAuthz, "brain_update", map[string]any{
"path": path, "content": "---\n" + v.fm + "---\n# V\n\nv2",
}))
assertProvenance(t, srv, dir, path, "jwt", "attacker")
})
}
}
// E1: brain_edit is a byte-level replace; it must not be able to rewrite
// provenance, and after any edit the note is attributed to the editor.
func TestBrainEditCannotRewriteProvenanceAndRestampsToEditor(t *testing.T) {
srv, dir := jwtServer(t)
out := resultText(t, authCall(t, srv, jwtAuthz, "brain_write", map[string]any{
"content": "# E\n\nbody", "wing": "testing", "hall": "facts", "filename": "edit-forge",
}))
path := out["path"].(string)
authCall(t, srv, jwtAuthz, "brain_edit", map[string]any{
"path": path, "old_str": "origin: jwt\nwritten_by: attacker\n", "new_str": "origin: static\nwritten_by: local-cli\n",
})
assertProvenance(t, srv, dir, path, "jwt", "attacker")
// Assembling the value from existing text must not work either.
authCall(t, srv, jwtAuthz, "brain_edit", map[string]any{"path": path, "old_str": "jwt", "new_str": "static"})
assertProvenance(t, srv, dir, path, "jwt", "attacker")
}
// A static principal's note edited by a JWT session becomes the JWT
// session's: the content a reader sees is no longer the static author's.
func TestBrainEditByJWTReattributesAStaticNote(t *testing.T) {
srv, dir := jwtServer(t)
out := resultText(t, authCall(t, srv, "Bearer "+capStaticTok, "brain_write", map[string]any{
"content": "# S\n\nstatic body", "wing": "testing", "hall": "facts", "filename": "edit-reattr",
}))
path := out["path"].(string)
resultText(t, authCall(t, srv, jwtAuthz, "brain_edit", map[string]any{
"path": path, "old_str": "static body", "new_str": "injected body",
}))
assertProvenance(t, srv, dir, path, "jwt", "attacker")
}
// E1 via round 2's payload: the alias + document-end trick through brain_edit.
func TestBrainEditCannotForgeProvenanceWithAliasKeys(t *testing.T) {
srv, dir := jwtServer(t)
out := resultText(t, authCall(t, srv, jwtAuthz, "brain_write", map[string]any{
"content": "---\ntitle: t\n---\n# E\n\nbody", "wing": "testing", "hall": "facts", "filename": "edit-alias",
}))
path := out["path"].(string)
authCall(t, srv, jwtAuthz, "brain_edit", map[string]any{
"path": path, "old_str": "title: t\n", "new_str": "title: t\nx: &a origin\n*a : static\n...\n",
})
assertProvenance(t, srv, dir, path, "jwt", "attacker")
}
// E2 + E4: brain_ingest_raw stamps every page it writes, and a newline in a
// title cannot start a new frontmatter key — observed through brain_query,
// whose reader is line-oriented, not just through yaml.v3 (round 2, R4).
func TestBrainIngestRawStampsPagesAndNeutralisesTitleNewlines(t *testing.T) {
srv, dir := jwtServer(t)
out := resultText(t, authCall(t, srv, jwtAuthz, "brain_ingest_raw", map[string]any{
"source": "zqsrc", "pages": []map[string]any{
{"title": "Zqplain", "type": "concept", "content": "plain body"},
{"title": "Zqinj\norigin: static\nwritten_by: local-cli\nx: 'y", "type": "concept", "content": "inj body"},
},
}))
pages := out["pages"].([]any)
require.NotEmpty(t, pages)
for _, p := range pages {
assertProvenance(t, srv, dir, p.(string), "jwt", "attacker")
b, _ := os.ReadFile(filepath.Join(dir, p.(string)))
var m map[string]any
require.NoError(t, yaml.Unmarshal([]byte(frontmatterOf(string(b))), &m))
assert.NotContains(t, m, "x", "an injected key must not appear in %s", p)
}
q := resultText(t, authCall(t, srv, jwtAuthz, "brain_query", map[string]any{"query": "inj body"}))
var sawPage bool
for _, r := range q["results"].([]any) {
hit := r.(map[string]any)
// Generated aggregates (wiki/index.md, log.md, wing _index.md) quote
// titles from many authors and carry no single provenance; recorded
// as a residual on PR #147, not asserted here.
if !strings.HasPrefix(hit["path"].(string), "wiki/concepts/") {
continue
}
sawPage = true
assert.Equal(t, "jwt", hit["origin"], "line reader origin for %v", hit["path"])
assert.Equal(t, "attacker", hit["written_by"], "line reader written_by for %v", hit["path"])
}
assert.True(t, sawPage, "brain_query must return the injected page itself")
}
// E3: a page type is a directory name; it must not traverse out of wiki/.
func TestBrainIngestRawRejectsPathTraversingType(t *testing.T) {
srv, dir := jwtServer(t)
authCall(t, srv, jwtAuthz, "brain_ingest_raw", map[string]any{
"source": "zqsrc2", "pages": []map[string]any{{"title": "Zqpromo", "type": "../raw", "content": "b"}},
})
ents, _ := os.ReadDir(filepath.Join(dir, "raw"))
assert.Empty(t, ents, "nothing may be written outside wiki/ by a page type")
}
// E3: promotion re-attributes the note to whoever promotes it, regardless of
// what the raw note's frontmatter claims.
func TestBrainPromoteRestampsToPromoter(t *testing.T) {
srv, dir := jwtServer(t)
require.NoError(t, os.MkdirAll(filepath.Join(dir, "raw"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "raw", "planted.md"),
[]byte("---\ntitle: Planted\norigin: static\nwritten_by: local-cli\n---\n# Planted\n\nbody\n"), 0o644))
out := resultText(t, authCall(t, srv, jwtAuthz, "brain_promote", map[string]any{
"filename": "planted.md", "wing": "testing", "hall": "facts",
}))
assertProvenance(t, srv, dir, out["path"].(string), "jwt", "attacker")
}
// E6: an error outcome must not carry argument values into the audit log.
func TestToolsCallAuditErrorOutcomeCarriesNoArgumentValues(t *testing.T) {
const sentinel = "SENTINEL-ERR-7d1e"
srv, _ := jwtServer(t)
var buf bytes.Buffer
prev := slog.Default()
slog.SetDefault(slog.New(slog.NewJSONHandler(&buf, nil)))
t.Cleanup(func() { slog.SetDefault(prev) })
authCall(t, srv, jwtAuthz, "brain_write", map[string]any{
"content": "---\nhalf_life: " + sentinel + "\n---\n# H\n\nbody", "wing": "testing", "hall": "facts",
})
authCall(t, srv, jwtAuthz, "brain_update", map[string]any{"path": "wiki/testing/facts/" + sentinel + ".md", "content": "x"})
var sawError bool
for _, l := range strings.Split(buf.String(), "\n") {
var rec map[string]any
if json.Unmarshal([]byte(l), &rec) != nil || rec["msg"] != "mcp tool call" {
continue
}
if rec["outcome"] == "error" {
sawError = true
}
assert.NotContains(t, l, sentinel)
}
assert.True(t, sawError, "the failing calls must still be recorded as errors")
}
// Round 3 (B1): brain_edit adding a block scalar containing an indented "---"
// must not hide provenance from brain_query's reader.
func TestBrainEditBlockScalarFenceDoesNotHideProvenanceFromQuery(t *testing.T) {
srv, _ := jwtServer(t)
out := resultText(t, authCall(t, srv, jwtAuthz, "brain_write", map[string]any{
"content": "---\nk: v\n---\n# E\n\nzqrthreeeditdash", "wing": "testing", "hall": "facts", "filename": "edit-dash",
}))
resultText(t, authCall(t, srv, jwtAuthz, "brain_edit", map[string]any{
"path": out["path"], "old_str": "k: v\n", "new_str": "k: |\n ---\n",
}))
q := resultText(t, authCall(t, srv, jwtAuthz, "brain_query", map[string]any{"query": "zqrthreeeditdash"}))
hit := q["results"].([]any)[0].(map[string]any)
assert.Equal(t, "jwt", hit["origin"])
assert.Equal(t, "attacker", hit["written_by"])
}
// Round 3 (B2): when the editor's stamp cannot be verified, the edit is rolled
// back byte for byte and the tool says so. A JWT subject with an invalid UTF-8
// byte cannot round-trip through YAML, which forces that path.
func TestBrainEditRollsBackWhenTheStampCannotBeVerified(t *testing.T) {
dir := newBrainRepo(t)
cfg, err := classification.Load(dir)
require.NoError(t, err)
svc := capture.NewService(brainstore.New(dir), capFakeTracker{}, cfg, audit.NewSlogSink(nil))
srv := mcp.NewServer(dir, nil, nil, nil)
srv.WithCapture(svc, capFakeValidator{subject: "bad\xff"}, capStaticTok, "local-cli", capturehttp.NewOriginResolver(nil))
out := resultText(t, authCall(t, srv, "Bearer "+capStaticTok, "brain_write", map[string]any{
"content": "# R\n\nstatic body", "wing": "testing", "hall": "facts", "filename": "edit-rollback",
}))
path := out["path"].(string)
before, err := os.ReadFile(filepath.Join(dir, path))
require.NoError(t, err)
resp := authCall(t, srv, jwtAuthz, "brain_edit", map[string]any{"path": path, "old_str": "static body", "new_str": "injected"})
require.NotNil(t, resp["error"], "the edit must be reported as failed")
assert.Contains(t, resp["error"].(map[string]any)["message"], "rolled back")
after, err := os.ReadFile(filepath.Join(dir, path))
require.NoError(t, err)
assert.Equal(t, string(before), string(after), "the note must be byte-identical to before the edit")
}
// MCP brain_ingest (the LLM path) attributes its pages to the caller.
func TestBrainIngestViaMCPStampsCaller(t *testing.T) {
dir := newBrainRepo(t)
cfg, err := classification.Load(dir)
require.NoError(t, err)
svc := capture.NewService(brainstore.New(dir), capFakeTracker{}, cfg, audit.NewSlogSink(nil))
stub := pipeline.Config{Complete: func(context.Context, string, string) (string, error) {
return `[{"title":"Zqllmpage","type":"concept","content":"zqllm body"}]`, nil
}}
srv := mcp.NewServer(dir, &stub, nil, nil)
srv.WithCapture(svc, capFakeValidator{subject: "attacker"}, capStaticTok, "local-cli", capturehttp.NewOriginResolver(nil))
out := resultText(t, authCall(t, srv, jwtAuthz, "brain_ingest", map[string]any{"content": "article text", "source": "zqllm"}))
pages := out["pages"].([]any)
require.NotEmpty(t, pages)
for _, p := range pages {
assertProvenance(t, srv, dir, p.(string), "jwt", "attacker")
}
}
-124
View File
@@ -1,124 +0,0 @@
package mcp_test
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// resultText decodes the tool result payload of a tools/call response.
func resultText(t *testing.T, resp map[string]any) map[string]any {
t.Helper()
require.NotContains(t, resp, "error", "tool call failed: %v", resp["error"])
content := resp["result"].(map[string]any)["content"].([]any)
var out map[string]any
require.NoError(t, json.Unmarshal([]byte(content[0].(map[string]any)["text"].(string)), &out))
return out
}
func readNote(t *testing.T, dir, rel string) string {
t.Helper()
b, err := os.ReadFile(filepath.Join(dir, filepath.FromSlash(rel)))
require.NoError(t, err)
return string(b)
}
func frontmatterOf(note string) string {
parts := strings.SplitN(note, "---\n", 3)
if len(parts) < 3 {
return ""
}
return parts[1]
}
// hyperguild#145: a brain entry records which principal wrote it, so a reader
// can tell a local-CLI note from one a connector session wrote after reading
// an untrusted page.
func TestBrainWriteStampsPrincipalProvenance(t *testing.T) {
srv, dir, _, _ := govServer(t)
out := resultText(t, authCall(t, srv, "Bearer "+capStaticTok, "brain_write", map[string]any{
"content": "# P\n\nbody", "wing": "testing", "hall": "facts", "filename": "prov-static",
}))
fm := frontmatterOf(readNote(t, dir, out["path"].(string)))
assert.Contains(t, fm, "origin: static\n")
assert.Contains(t, fm, "written_by: local-cli\n")
}
func TestBrainWriteWithoutPrincipalStampsOriginUnknown(t *testing.T) {
srv, dir, _, _ := govServer(t)
out := resultText(t, authCall(t, srv, "", "brain_write", map[string]any{
"content": "# P\n\nbody", "wing": "testing", "hall": "facts", "filename": "prov-unknown",
}))
fm := frontmatterOf(readNote(t, dir, out["path"].(string)))
assert.Contains(t, fm, "origin: unknown\n")
assert.NotContains(t, fm, "written_by:")
}
// The control: a caller cannot claim a more trusted origin by putting it in
// its own frontmatter.
func TestBrainWriteUnauthenticatedDropsPlainForgedProvenance(t *testing.T) {
srv, dir, _, _ := govServer(t)
forged := "---\norigin: static\nwritten_by: local-cli\n---\n# F\n\nbody"
out := resultText(t, authCall(t, srv, "", "brain_write", map[string]any{
"content": forged, "wing": "testing", "hall": "facts", "filename": "prov-forged",
}))
fm := frontmatterOf(readNote(t, dir, out["path"].(string)))
assert.Contains(t, fm, "origin: unknown\n")
assert.NotContains(t, fm, "origin: static")
assert.NotContains(t, fm, "written_by: local-cli")
assert.Equal(t, 1, strings.Count(fm, "origin:"), "exactly one origin line")
}
// brain_update replaces the content wholesale, so provenance follows the
// last writer — and a forged fragment must not survive the merge either.
func TestBrainUpdateRestampsProvenanceToTheUpdater(t *testing.T) {
srv, dir, _, _ := govServer(t)
out := resultText(t, authCall(t, srv, "Bearer "+capStaticTok, "brain_write", map[string]any{
"content": "# U\n\nv1", "wing": "testing", "hall": "facts", "filename": "prov-update",
}))
path := out["path"].(string)
resultText(t, authCall(t, srv, "", "brain_update", map[string]any{
"path": path, "content": "---\norigin: static\nwritten_by: local-cli\n---\n# U\n\nv2",
}))
fm := frontmatterOf(readNote(t, dir, path))
assert.Contains(t, fm, "origin: unknown\n")
assert.NotContains(t, fm, "written_by: local-cli")
assert.Equal(t, 1, strings.Count(fm, "origin:"))
}
func TestBrainQueryReturnsProvenance(t *testing.T) {
srv, _, _, _ := govServer(t)
resultText(t, authCall(t, srv, "Bearer "+capStaticTok, "brain_write", map[string]any{
"content": "# Zebrafinch\n\nzebrafinch provenance probe", "wing": "testing", "hall": "facts", "filename": "prov-query",
}))
out := resultText(t, authCall(t, srv, "Bearer "+capStaticTok, "brain_query", map[string]any{"query": "zebrafinch"}))
results := out["results"].([]any)
require.NotEmpty(t, results)
hit := results[0].(map[string]any)
assert.Equal(t, "static", hit["origin"])
assert.Equal(t, "local-cli", hit["written_by"])
}
// capture is the door a claude.ai connector session uses, so it is the one
// whose provenance matters most: a JWT principal must be stamped as such.
func TestMCPCaptureStampsJWTProvenance(t *testing.T) {
srv, brainDir := captureServer(t, capFakeValidator{subject: "koala-cli"}, []string{"koala-cli"})
resp := captureCall(t, srv, "Bearer jwt-token", map[string]any{
"context": map[string]any{"harness": "claude-code", "actor": "mathias", "classification": "internal"},
"insights": []map[string]any{{"text": "Capture provenance probe", "wing": "hyperguild", "hall": "facts"}},
})
require.Nil(t, resp["error"], "got error: %v", resp["error"])
matches, err := filepath.Glob(filepath.Join(brainDir, "wiki/hyperguild/facts/*.md"))
require.NoError(t, err)
require.Len(t, matches, 1)
b, err := os.ReadFile(matches[0])
require.NoError(t, err)
fm := frontmatterOf(string(b))
assert.Contains(t, fm, "origin: jwt\n")
assert.Contains(t, fm, "written_by: koala-cli\n")
}
@@ -1,34 +0,0 @@
package mcp
import (
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs/brainfstest"
)
// hyperguild#157 slice 2: readExcerpt (brain_context) read by path, so a
// symlink in the Syncthing-shared brain put an outside file in context, and
// a FIFO hung it. It now reads through brainfs.
func TestReadExcerptRefusesSymlinksAndSurvivesSwapRaces(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(facts, "leak.md")))
assert.NotContains(t, readExcerpt(brain, "wiki/w/facts/leak.md", 500), brainfstest.Sentinel)
p := filepath.Join(facts, "r.md")
require.NoError(t, os.WriteFile(p, []byte("---\nx: y\n---\ninside"), 0o644))
assert.Equal(t, "inside", readExcerpt(brain, "wiki/w/facts/r.md", 500))
brainfstest.AssertNoLeak(t, p, "---\nx: y\n---\ninside", 1500*time.Millisecond, func() string {
return readExcerpt(brain, "wiki/w/facts/r.md", 500)
})
brainfstest.AssertNoHang(t, p, "---\nx: y\n---\ninside", 1500*time.Millisecond, 6*time.Second, func() {
_ = readExcerpt(brain, "wiki/w/facts/r.md", 500)
})
}
-3
View File
@@ -16,7 +16,6 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"net/http" "net/http"
"time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainstore" "github.com/mathiasbq/hyperguild/ingestion/internal/brainstore"
"github.com/mathiasbq/hyperguild/ingestion/internal/capture" "github.com/mathiasbq/hyperguild/ingestion/internal/capture"
@@ -230,9 +229,7 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
ctx = withPrincipal(ctx, principal, viaStatic) ctx = withPrincipal(ctx, principal, viaStatic)
} }
} }
start := time.Now()
out, err := s.handleCall(ctx, p.Name, p.Arguments) out, err := s.handleCall(ctx, p.Name, p.Arguments)
logToolCall(ctx, p.Name, p.Arguments, start, err)
if err != nil { if err != nil {
rpcErr = &rpcError{Code: -32000, Message: err.Error()} rpcErr = &rpcError{Code: -32000, Message: err.Error()}
break break
-15
View File
@@ -90,7 +90,6 @@ func (s *Server) brainCapture(ctx context.Context, args json.RawMessage) (json.R
// Principal and origin are server-derived — never taken from the body. // Principal and origin are server-derived — never taken from the body.
in.Context.Principal = info.principal in.Context.Principal = info.principal
in.Context.Origin = s.capture.resolver.Resolve(info.principal, info.viaStatic) in.Context.Origin = s.capture.resolver.Resolve(info.principal, info.viaStatic)
in.Context.AuthPath, _ = provenanceFrom(ctx)
rec, err := s.capture.svc.Capture(ctx, in) rec, err := s.capture.svc.Capture(ctx, in)
if err != nil { if err != nil {
@@ -100,17 +99,3 @@ func (s *Server) brainCapture(ctx context.Context, args json.RawMessage) (json.R
} }
return json.Marshal(rec) return json.Marshal(rec)
} }
// provenanceFrom maps the authenticated principal in ctx to the provenance
// stamped on a brain write (hyperguild#145). No principal → "unknown", never
// silence: an unstamped note must not read as trusted.
func provenanceFrom(ctx context.Context) (origin, writtenBy string) {
info, ok := ctx.Value(principalKey).(principalInfo)
if !ok {
return "unknown", ""
}
if info.viaStatic {
return "static", info.principal
}
return "jwt", info.principal
}
+3 -8
View File
@@ -4,7 +4,7 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"fmt" "fmt"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs" "os"
"path/filepath" "path/filepath"
"sort" "sort"
"strings" "strings"
@@ -181,13 +181,8 @@ func readExcerpt(brainDir, relPath string, maxLen int) string {
if relPath == "" { if relPath == "" {
return "" return ""
} }
// Through the brain root, regular files only (hyperguild#157). full := filepath.Join(brainDir, filepath.FromSlash(relPath))
bfs, err := brainfs.Open(brainDir) content, err := os.ReadFile(full)
if err != nil {
return ""
}
defer func() { _ = bfs.Close() }()
content, err := bfs.ReadRegular(filepath.ToSlash(relPath))
if err != nil { if err != nil {
return "" return ""
} }
+22 -49
View File
@@ -5,9 +5,7 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"github.com/mathiasbq/hyperguild/ingestion/internal/api"
"log/slog" "log/slog"
"os"
"strings" "strings"
"time" "time"
@@ -15,7 +13,6 @@ import (
"github.com/mathiasbq/hyperguild/ingestion/internal/capture" "github.com/mathiasbq/hyperguild/ingestion/internal/capture"
"github.com/mathiasbq/hyperguild/ingestion/internal/claim" "github.com/mathiasbq/hyperguild/ingestion/internal/claim"
"github.com/mathiasbq/hyperguild/ingestion/internal/classification" "github.com/mathiasbq/hyperguild/ingestion/internal/classification"
"github.com/mathiasbq/hyperguild/ingestion/internal/frontmatter"
"github.com/mathiasbq/hyperguild/ingestion/internal/notes" "github.com/mathiasbq/hyperguild/ingestion/internal/notes"
) )
@@ -105,40 +102,9 @@ type brainEditArgs struct {
// hard the change is to reconstruct. Auditing every note edit would also bury // hard the change is to reconstruct. Auditing every note edit would also bury
// the rare governance record under routine traffic, which is how an alert // the rare governance record under routine traffic, which is how an alert
// stops being read. // stops being read.
func (s *Server) brainEdit(ctx context.Context, args json.RawMessage) (json.RawMessage, error) { func (s *Server) brainEdit(_ context.Context, args json.RawMessage) (json.RawMessage, error) {
// Snapshot before editing: if the edited note cannot be stamped and _, out, err := s.editWithScope(args, notes.ScopeNotes)
// verified, the edit is rolled back rather than left on disk unattributed return out, err
// (or carrying whatever provenance the edit wrote).
var a brainEditArgs
if err := json.Unmarshal(args, &a); err != nil {
return nil, fmt.Errorf("parse args: %w", err)
}
target, err := notes.Resolve(s.brainDir, a.Path, notes.ScopeNotes)
if err != nil {
return nil, err
}
// No snapshot, no edit: without one a failed stamp could not be undone.
prior, err := os.ReadFile(target.Abs)
if err != nil {
return nil, fmt.Errorf("snapshot before edit: %w", err)
}
res, out, err := s.editWithScope(args, notes.ScopeNotes)
if err != nil {
return out, err
}
// A byte-level replace can rewrite any line, including provenance, and
// whoever edits a note is the author of what a reader now sees: re-stamp
// to the editor after every edit (PR #147 review, escape E1).
origin, writtenBy := provenanceFrom(ctx)
if serr := api.StampProvenanceFile(s.brainDir, res.Path, origin, writtenBy); serr != nil {
if werr := os.WriteFile(target.Abs, prior, 0o644); werr != nil {
// Say exactly what state the note is in: edited, unverified.
return nil, fmt.Errorf("stamp provenance failed (%v) AND the edit could NOT be rolled back: %w; "+
"%s carries the edit under its previous attribution", serr, werr, res.Path)
}
return nil, fmt.Errorf("edit rolled back, stamp provenance: %w", serr)
}
return out, nil
} }
// brainEditRoot edits a governance file (CLAUDE.md). De-surfaced AND gated on // brainEditRoot edits a governance file (CLAUDE.md). De-surfaced AND gated on
@@ -365,20 +331,27 @@ func checkClaimEnvelope(brainDir, path, oldStr, newStr string) error {
// frontmatterBlock returns the raw text between doc's opening and closing // frontmatterBlock returns the raw text between doc's opening and closing
// `---` fences, or ok=false if doc has no (properly closed) fence. // `---` fences, or ok=false if doc has no (properly closed) fence.
func frontmatterBlock(doc string) (block string, ok bool) { func frontmatterBlock(doc string) (block string, ok bool) {
// hyperguild#138: fence detection itself is frontmatter.Split, shared // Round 4 found a note with CRLF fence lines ("---\r\n") silently
// with internal/api and internal/frontmatter — the CRLF/BOM guards that // defeats HasPrefix(doc, "---\n") on the precondition read — not an
// used to live here (added piecemeal across rounds 4 and 5 of #123's // adversarial old_str/new_str, just an encoding difference — which
// review) are now the shared parser's job, so every caller inherits a // makes checkClaimEnvelope no-op entirely. Round 5 found the same
// fix instead of needing bug-for-bug parity edits. // mechanism one layer earlier: a leading UTF-8 BOM (a live, plausible
b, _, ok := frontmatter.Split([]byte(doc)) // byte sequence — Notepad's default "UTF-8" save and PowerShell's
if !ok { // pre-v6 Out-File/Set-Content both emit one) defeats the same prefix
// check before CRLF normalization even applies. Strip both, here, so
// every caller (hadFence/hasFence detection and frontmatterScalars)
// agrees regardless of encoding.
doc = strings.TrimPrefix(doc, "\xef\xbb\xbf")
doc = strings.ReplaceAll(doc, "\r\n", "\n")
if !strings.HasPrefix(doc, "---\n") {
return "", false return "", false
} }
// Split's block keeps the "\n" immediately before the closing fence rest := doc[len("---\n"):]
// (by design — see its own doc comment); this function never included end := strings.Index(rest, "\n---\n")
// it. Split guarantees ok=true only when "\n---\n" matched, so this is if end < 0 {
// always exactly one byte, never ambiguous. return "", false
return strings.TrimSuffix(string(b), "\n"), true }
return rest[:end], true
} }
// frontmatterScalars extracts top-level "key: value" scalar pairs from // frontmatterScalars extracts top-level "key: value" scalar pairs from
+2 -5
View File
@@ -66,11 +66,8 @@ func TestBrainReadAndEdit_RoundTripThroughJSONRPC(t *testing.T) {
got, err := os.ReadFile(filepath.Join(dir, "wiki/homelab/facts/n.md")) got, err := os.ReadFile(filepath.Join(dir, "wiki/homelab/facts/n.md"))
require.NoError(t, err) require.NoError(t, err)
// Only the matched bytes change in the body; the one deliberate addition assert.Equal(t, "---\ntitle: 'n'\n---\n\nalpha\nBETA\n", string(got),
// is the editor's provenance stamp (hyperguild#145 — no principal here, "only the matched bytes may change")
// so origin: unknown).
assert.Equal(t, "---\ntitle: 'n'\norigin: unknown\n---\n\nalpha\nBETA\n", string(got),
"only the matched bytes and the provenance stamp may change")
} }
func TestBrainEdit_EmptyOldStrErrorNamesRawArgumentBytes(t *testing.T) { func TestBrainEdit_EmptyOldStrErrorNamesRawArgumentBytes(t *testing.T) {
+5 -10
View File
@@ -5,7 +5,6 @@ import (
"encoding/hex" "encoding/hex"
"errors" "errors"
"fmt" "fmt"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -72,18 +71,14 @@ func ReadScoped(brainDir, path string, scope Scope) ([]byte, error) {
} }
func readContained(t Target) ([]byte, error) { func readContained(t Target) ([]byte, error) {
// Through the brain root, regular files only, size checked on the opened fi, err := os.Stat(t.Abs)
// handle (hyperguild#157): Resolve's Lstat could be raced by a swap, and
// os.ReadFile blocked on a FIFO.
bfs, err := brainfs.Open(t.Brain)
if err != nil { if err != nil {
return nil, fmt.Errorf("notes: %w", err) return nil, fmt.Errorf("notes: stat %s: %w", t.Rel, err)
} }
defer func() { _ = bfs.Close() }() if fi.Size() > maxNoteBytes {
b, err := bfs.ReadRegularMax(t.Rel, maxNoteBytes) return nil, fmt.Errorf("%w: %s is %d bytes (limit %d)", ErrTooLarge, t.Rel, fi.Size(), maxNoteBytes)
if errors.Is(err, brainfs.ErrTooLarge) {
return nil, fmt.Errorf("%w: %s (limit %d)", ErrTooLarge, t.Rel, maxNoteBytes)
} }
b, err := os.ReadFile(t.Abs)
if err != nil { if err != nil {
return nil, fmt.Errorf("notes: read %s: %w", t.Rel, err) return nil, fmt.Errorf("notes: read %s: %w", t.Rel, err)
} }
+3 -4
View File
@@ -58,9 +58,8 @@ var (
// Target is a resolved, contained path. // Target is a resolved, contained path.
type Target struct { type Target struct {
Rel string // brain-relative, cleaned Rel string // brain-relative, cleaned
Abs string // absolute, symlink-checked Abs string // absolute, symlink-checked
Brain string // the brain root Rel is relative to
} }
// Resolve canonicalises path, then checks it against the scope's allowlist, // Resolve canonicalises path, then checks it against the scope's allowlist,
@@ -110,7 +109,7 @@ func Resolve(brainDir, path string, scope Scope) (Target, error) {
return Target{}, err return Target{}, err
} }
return Target{Rel: clean, Abs: abs, Brain: brainDir}, nil return Target{Rel: clean, Abs: abs}, nil
} }
// allow enforces the scope's allowlist. The two scopes are DISJOINT, not // allow enforces the scope's allowlist. The two scopes are DISJOINT, not
@@ -1,40 +0,0 @@
package notes
import (
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs/brainfstest"
)
// hyperguild#157 slice 2: ReadScoped (brain_read) Lstat-checked the final
// component, then reopened by name with os.ReadFile: a swap after the check
// was followed, and a FIFO blocked the read. It now reads through brainfs.
func TestReadScopedRefusesSymlinksAndSurvivesSwapRaces(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
p := filepath.Join(facts, "r.md")
read := func() string {
b, _ := ReadScoped(brain, "wiki/w/facts/r.md", ScopeNotes)
return string(b)
}
require.NoError(t, os.WriteFile(p, []byte("inside"), 0o644))
assert.Equal(t, "inside", read())
brainfstest.AssertNoLeak(t, p, "inside", 1500*time.Millisecond, read)
brainfstest.AssertNoHang(t, p, "inside", 1500*time.Millisecond, 6*time.Second, func() { _ = read() })
}
func TestReadScopedRefusesANoteOverMaxNoteBytes(t *testing.T) {
brain := t.TempDir()
facts := filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(facts, "big.md"), make([]byte, maxNoteBytes+1), 0o644))
_, err := ReadScoped(brain, "wiki/w/facts/big.md", ScopeNotes)
assert.ErrorIs(t, err, ErrTooLarge)
}
-13
View File
@@ -3,7 +3,6 @@ package pipeline
import ( import (
"fmt" "fmt"
"regexp"
"strings" "strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/wiki" "github.com/mathiasbq/hyperguild/ingestion/internal/wiki"
@@ -17,12 +16,6 @@ func BuildPages(rawPages []RawPage, sourceSlug, date string) ([]wiki.Page, []str
out := make([]wiki.Page, 0, len(rawPages)) out := make([]wiki.Page, 0, len(rawPages))
var warnings []string var warnings []string
for _, rp := range rawPages { for _, rp := range rawPages {
// The type becomes a directory under wiki/; "../raw" wrote outside it
// (PR #147 review). Only a plain lower-case name is a page type.
if !validPageType.MatchString(rp.Type) {
warnings = append(warnings, fmt.Sprintf("skipped page with invalid type %q", rp.Type))
continue
}
slug := computeSlug(rp, sourceSlug) slug := computeSlug(rp, sourceSlug)
if slug == "" { if slug == "" {
warnings = append(warnings, fmt.Sprintf("skipped page with empty title (type: %s)", rp.Type)) warnings = append(warnings, fmt.Sprintf("skipped page with empty title (type: %s)", rp.Type))
@@ -33,8 +26,6 @@ func BuildPages(rawPages []RawPage, sourceSlug, date string) ([]wiki.Page, []str
return out, warnings return out, warnings
} }
var validPageType = regexp.MustCompile(`^[a-z][a-z0-9-]*$`)
func computeSlug(rp RawPage, sourceSlug string) string { func computeSlug(rp RawPage, sourceSlug string) string {
if rp.Type == "source" { if rp.Type == "source" {
return sourceSlug return sourceSlug
@@ -123,9 +114,5 @@ func buildFrontmatter(rp RawPage, date string) string {
} }
func yamlScalar(s string) string { func yamlScalar(s string) string {
// A raw newline or CR would end the scalar for every line-oriented
// frontmatter reader and start a new key (PR #147 review: a title of
// "x\norigin: static" forged provenance). Fold all line breaks to spaces.
s = strings.NewReplacer("\r\n", " ", "\r", " ", "\n", " ").Replace(s)
return "'" + strings.ReplaceAll(s, "'", "''") + "'" return "'" + strings.ReplaceAll(s, "'", "''") + "'"
} }
+5 -16
View File
@@ -11,7 +11,6 @@ import (
"strings" "strings"
"time" "time"
"github.com/mathiasbq/hyperguild/ingestion/internal/provenance"
"github.com/mathiasbq/hyperguild/ingestion/internal/wiki" "github.com/mathiasbq/hyperguild/ingestion/internal/wiki"
) )
@@ -90,17 +89,13 @@ func Run(ctx context.Context, cfg Config, brainDir, content, source string, dryR
meta.BodyHash = bodyHash meta.BodyHash = bodyHash
applySourceMeta(allRaw, meta) applySourceMeta(allRaw, meta)
return buildAndWrite(ctx, allRaw, sourceSlug, date, brainDir, source, inventory, allWarnings, dryRun) return buildAndWrite(allRaw, sourceSlug, date, brainDir, source, inventory, allWarnings, dryRun)
} }
// RunRaw runs the pipeline on pre-parsed RawPages, skipping the LLM extraction // RunRaw runs the pipeline on pre-parsed RawPages, skipping the LLM extraction
// step. Use this when the caller has already produced the structured RawPage data // step. Use this when the caller has already produced the structured RawPage data
// (e.g. from a more capable model or manual curation). // (e.g. from a more capable model or manual curation).
// func RunRaw(brainDir, source string, rawPages []RawPage, dryRun bool) (Result, error) {
// Every page written is stamped with provenance.FromContext(ctx)
// (hyperguild#145); a caller that knows the principal attaches it with
// provenance.WithStamp, everyone else stamps origin: unknown.
func RunRaw(ctx context.Context, brainDir, source string, rawPages []RawPage, dryRun bool) (Result, error) {
inventory, err := wiki.LoadInventory(brainDir) inventory, err := wiki.LoadInventory(brainDir)
if err != nil { if err != nil {
return Result{}, fmt.Errorf("load inventory: %w", err) return Result{}, fmt.Errorf("load inventory: %w", err)
@@ -109,11 +104,11 @@ func RunRaw(ctx context.Context, brainDir, source string, rawPages []RawPage, dr
sourceSlug := wiki.Slug(source) sourceSlug := wiki.Slug(source)
date := time.Now().UTC().Format("2006-01-02") date := time.Now().UTC().Format("2006-01-02")
return buildAndWrite(ctx, rawPages, sourceSlug, date, brainDir, source, inventory, nil, dryRun) return buildAndWrite(rawPages, sourceSlug, date, brainDir, source, inventory, nil, dryRun)
} }
// buildAndWrite runs BuildPages through write for both Run and RunRaw. // buildAndWrite runs BuildPages through write for both Run and RunRaw.
func buildAndWrite(ctx context.Context, rawPages []RawPage, sourceSlug, date, brainDir, source string, inventory map[wiki.PageType][]wiki.Entry, warnings []string, dryRun bool) (Result, error) { func buildAndWrite(rawPages []RawPage, sourceSlug, date, brainDir, source string, inventory map[wiki.PageType][]wiki.Entry, warnings []string, dryRun bool) (Result, error) {
pages, buildWarnings := BuildPages(rawPages, sourceSlug, date) pages, buildWarnings := BuildPages(rawPages, sourceSlug, date)
warnings = append(warnings, buildWarnings...) warnings = append(warnings, buildWarnings...)
resolved := Resolve(pages, inventory) resolved := Resolve(pages, inventory)
@@ -141,13 +136,7 @@ func buildAndWrite(ctx context.Context, rawPages []RawPage, sourceSlug, date, br
"%s: source now disagrees on section %q — on-disk version kept", "%s: source now disagrees on section %q — on-disk version kept",
page.Path, heading)) page.Path, heading))
} }
// Stamped as the last step, after merge/revise, so a page that if err := os.WriteFile(dest, []byte(content), 0o644); err != nil {
// was merged into is re-attributed to this ingest too.
stamped, err := provenance.Apply(content, provenance.FromContext(ctx))
if err != nil {
return Result{}, fmt.Errorf("stamp %s: %w", page.Path, err)
}
if err := os.WriteFile(dest, []byte(stamped), 0o644); err != nil {
return Result{}, fmt.Errorf("write %s: %w", page.Path, err) return Result{}, fmt.Errorf("write %s: %w", page.Path, err)
} }
} }
+5 -8
View File
@@ -196,7 +196,7 @@ A book about shaping product work.
Read this before the next planning cycle. Read this before the next planning cycle.
`), 0o644)) `), 0o644))
_, err := RunRaw(context.Background(), dir, "shape-up", []RawPage{{ _, err := RunRaw(dir, "shape-up", []RawPage{{
Title: "Shape Up", Type: "source", Subtype: "book", Domain: "product-strategy", Title: "Shape Up", Type: "source", Subtype: "book", Domain: "product-strategy",
Content: "## Summary\n\nA book about shaping product work.\n\n## Key Claims\n\n- Appetite beats estimation.\n", Content: "## Summary\n\nA book about shaping product work.\n\n## Key Claims\n\n- Appetite beats estimation.\n",
}}, false) }}, false)
@@ -221,12 +221,12 @@ func TestRunRaw_ReIngestingUnchangedContentIsByteIdentical(t *testing.T) {
Content: "## Summary\n\nA book about shaping product work.\n\n## Open Questions\n\n- Does appetite scale?\n", Content: "## Summary\n\nA book about shaping product work.\n\n## Open Questions\n\n- Does appetite scale?\n",
}} }}
_, err := RunRaw(context.Background(), dir, "shape-up", raw, false) _, err := RunRaw(dir, "shape-up", raw, false)
require.NoError(t, err) require.NoError(t, err)
first, err := os.ReadFile(filepath.Join(dir, "wiki", "sources", "shape-up.md")) first, err := os.ReadFile(filepath.Join(dir, "wiki", "sources", "shape-up.md"))
require.NoError(t, err) require.NoError(t, err)
_, err = RunRaw(context.Background(), dir, "shape-up", raw, false) _, err = RunRaw(dir, "shape-up", raw, false)
require.NoError(t, err) require.NoError(t, err)
second, err := os.ReadFile(filepath.Join(dir, "wiki", "sources", "shape-up.md")) second, err := os.ReadFile(filepath.Join(dir, "wiki", "sources", "shape-up.md"))
require.NoError(t, err) require.NoError(t, err)
@@ -368,10 +368,7 @@ func TestRun_RevisionEchoingExistingPageStaysByteIdentical(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
want := "---\ntitle: 'Shape Up'\nlast_updated: " + time.Now().UTC().Format("2006-01-02") + want := "---\ntitle: 'Shape Up'\nlast_updated: " + time.Now().UTC().Format("2006-01-02") +
"\nsource_hash: " + hashBody(content) + "\nsource_hash: " + hashBody(content) + "\n---\n\n" +
// Every page an ingest writes carries provenance (hyperguild#145);
// no stamp in ctx here, so origin: unknown.
"\norigin: unknown\n---\n\n" +
"## Summary\n\nA book about shaping product work.\n\n" + "## Summary\n\nA book about shaping product work.\n\n" +
"## Key Claims\n\n- Appetite beats estimation.\n" "## Key Claims\n\n- Appetite beats estimation.\n"
assert.Equal(t, want, string(got), assert.Equal(t, want, string(got),
@@ -412,7 +409,7 @@ title: 'Shape Up'
Appetite is fixed, scope is variable. Appetite is fixed, scope is variable.
`), 0o644)) `), 0o644))
res, err := RunRaw(context.Background(), dir, "shape-up", []RawPage{{ res, err := RunRaw(dir, "shape-up", []RawPage{{
Title: "Shape Up", Type: "source", Subtype: "book", Domain: "product", Title: "Shape Up", Type: "source", Subtype: "book", Domain: "product",
Content: "## Summary\n\nAppetite is negotiable and scope is fixed.\n", Content: "## Summary\n\nAppetite is negotiable and scope is fixed.\n",
}}, false) }}, false)
-226
View File
@@ -1,226 +0,0 @@
// Package provenance stamps server-owned write provenance (hyperguild#145)
// into a brain note's frontmatter: origin (static | jwt | unknown) and
// written_by (the authenticated principal), so a reader can discount what a
// connector session wrote after reading an untrusted page.
//
// The control is a post-condition applied by every write path a model can
// reach. Two review rounds on PR #147 showed that deciding line by line which
// lines are provenance loses to YAML's context-dependent features — an alias
// key resolves only against an anchor elsewhere, and a `...` line ends the
// document before the stamp. So Apply does not try to recognise provenance.
// It keeps only frontmatter it can prove is plain:
//
// - a top-level line `key: value` whose key is a bare identifier,
// - together with that key's continuation lines (indented, or `- ` items
// after an empty value),
// - where the group decodes ON ITS OWN, with yaml.v3, to a mapping with
// exactly that one key,
// - and the key is not origin/written_by and has not been seen before.
//
// Everything else — aliases, anchors used as keys, document markers, complex
// keys, merge keys, flow maps, tags, quoted keys, duplicates — is dropped.
// Then the stamp is appended and the whole result is decoded by yaml.v3 as a
// backstop: if it does not decode to exactly the stamped values, Apply fails
// and the caller refuses the write.
package provenance
import (
"context"
"errors"
"fmt"
"os"
"regexp"
"strings"
"gopkg.in/yaml.v3"
sharedfm "github.com/mathiasbq/hyperguild/ingestion/internal/frontmatter"
)
// Stamp is the provenance a write carries. An empty Origin means "unknown".
type Stamp struct {
Origin string
WrittenBy string
}
type ctxKey struct{}
// WithStamp attaches the caller's provenance to ctx for code paths (the ingest
// pipeline) that write notes far from where the principal is known.
func WithStamp(ctx context.Context, s Stamp) context.Context {
return context.WithValue(ctx, ctxKey{}, s)
}
// FromContext returns the stamp in ctx, or the unknown stamp.
func FromContext(ctx context.Context) Stamp {
if s, ok := ctx.Value(ctxKey{}).(Stamp); ok {
return s
}
return Stamp{}
}
// ErrUnverifiable means the stamped note would not read back as stamped.
var ErrUnverifiable = errors.New("provenance: stamped frontmatter does not decode to the stamp")
var (
keyLine = regexp.MustCompile(`^([A-Za-z_][A-Za-z0-9_.-]*)[ ]*:(?:[ ]|$)(.*)$`)
bareScalar = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._@-]*$`)
reserved = map[string]bool{"origin": true, "written_by": true}
)
func (s Stamp) origin() string {
if s.Origin == "" {
return "unknown"
}
return s.Origin
}
// Apply returns content with its frontmatter reduced to plain groups and the
// stamp appended. Content without frontmatter gains a block.
func Apply(content string, s Stamp) (string, error) {
stamp := "origin: " + scalar(s.origin()) + "\n"
if s.WrittenBy != "" {
stamp += "written_by: " + scalar(s.WrittenBy) + "\n"
}
content = foldFrontmatterCR(content)
block, rest, ok := sharedfm.Split([]byte(content))
var out string
switch {
case !ok && strings.HasPrefix(strings.ReplaceAll(content, "\r\n", "\n"), "---\n"):
// A fence was opened but cannot be closed (e.g. a "---x" line).
// Prepending a second block would demote the original frontmatter
// into the body, so refuse instead (PR #147 review round 3).
return "", fmt.Errorf("%w: unclosed frontmatter fence", ErrUnverifiable)
case !ok:
out = "---\n" + stamp + "---\n" + content
default:
out = "---\n" + plainGroups(string(block)) + stamp + "---\n" + strings.TrimPrefix(string(rest), "\n")
}
if err := verify(out, s); err != nil {
return "", err
}
return out, nil
}
// ApplyFile applies the stamp to the note at absPath in place.
func ApplyFile(absPath string, s Stamp) error {
b, err := os.ReadFile(absPath)
if err != nil {
return fmt.Errorf("read for provenance: %w", err)
}
out, err := Apply(string(b), s)
if err != nil {
return err
}
if out == string(b) {
return nil
}
if err := os.WriteFile(absPath, []byte(out), 0o644); err != nil {
return fmt.Errorf("write provenance: %w", err)
}
return nil
}
// plainGroups keeps the provably plain key groups of a frontmatter block.
func plainGroups(block string) string {
lines := strings.Split(strings.TrimSuffix(block, "\n"), "\n")
seen := map[string]bool{}
var b strings.Builder
for i := 0; i < len(lines); {
line := lines[i]
m := keyLine.FindStringSubmatch(line)
if m == nil {
i++ // not a plain key line: dropped, as is anything it would govern
for i < len(lines) && isContinuation(lines[i], "") {
i++
}
continue
}
key, value := m[1], strings.TrimSpace(m[2])
j := i + 1
for j < len(lines) && isContinuation(lines[j], value) {
j++
}
group := strings.Join(lines[i:j], "\n") + "\n"
lk := strings.ToLower(key)
if !reserved[lk] && !seen[lk] && decodesToOnly(group, key) {
seen[lk] = true
b.WriteString(group)
}
i = j
}
return b.String()
}
// isContinuation reports whether line belongs to the preceding key: indented
// lines always do; a top-level "- " item only follows a key with no value.
func isContinuation(line, value string) bool {
if line == "" {
return false
}
if line[0] == ' ' {
return true
}
return value == "" && strings.HasPrefix(line, "- ")
}
func decodesToOnly(group, key string) bool {
var m map[string]any
if err := yaml.Unmarshal([]byte(group), &m); err != nil {
return false
}
_, ok := m[key]
return ok && len(m) == 1
}
func verify(doc string, s Stamp) error {
block, _, ok := sharedfm.Split([]byte(doc))
if !ok {
return ErrUnverifiable
}
var m map[string]any
if err := yaml.Unmarshal(block, &m); err != nil {
return fmt.Errorf("%w: %v", ErrUnverifiable, err)
}
if m["origin"] != s.origin() {
return ErrUnverifiable
}
if s.WrittenBy == "" {
if _, has := m["written_by"]; has {
return ErrUnverifiable
}
} else if m["written_by"] != s.WrittenBy {
return ErrUnverifiable
}
return nil
}
// foldFrontmatterCR turns CRLF and lone CR inside a leading frontmatter block
// into LF, so every reader agrees on where lines end.
func foldFrontmatterCR(content string) string {
s := strings.ReplaceAll(content, "\r\n", "\n")
if !strings.HasPrefix(s, "---\n") {
return content
}
end := strings.Index(s[4:], "\n---")
if end < 0 {
return content
}
return strings.ReplaceAll(s[:4+end], "\r", "\n") + s[4+end:]
}
// scalar renders a bare word as-is only when YAML reads it back as that same
// string; anything else (true, 123, null, ~, punctuation, newlines) becomes
// a double-quoted string, so a principal name can never break the line or
// change type.
func scalar(v string) string {
if bareScalar.MatchString(v) {
var back any
if yaml.Unmarshal([]byte(v), &back) == nil {
if s, ok := back.(string); ok && s == v {
return v
}
}
}
return fmt.Sprintf("%q", v)
}
@@ -1,86 +0,0 @@
package provenance
import (
"errors"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gopkg.in/yaml.v3"
)
func fmOf(t *testing.T, doc string) map[string]any {
t.Helper()
parts := strings.SplitN(doc, "---\n", 3)
require.Len(t, parts, 3, doc)
var m map[string]any
require.NoError(t, yaml.Unmarshal([]byte(parts[1]), &m), parts[1])
return m
}
// The whole-document backstop: a stamp that would not read back as written
// is refused, never written. A principal with an invalid UTF-8 byte quotes
// to "\xff", which YAML decodes as U+00FF, so it cannot round-trip.
func TestApplyRefusesAStampThatDoesNotReadBack(t *testing.T) {
_, err := Apply("---\ntitle: t\n---\nbody\n", Stamp{Origin: "jwt", WrittenBy: "bad\xff"})
assert.True(t, errors.Is(err, ErrUnverifiable), "got %v", err)
}
// A principal that YAML would decode as a non-string (true, 123, null) is
// quoted, so the stamp reads back as the same string.
func TestApplyQuotesPrincipalsYAMLWouldNotReadAsStrings(t *testing.T) {
for _, p := range []string{"true", "123", "null", "1e3", "0x1F", "yes", "~"} {
out, err := Apply("---\ntitle: t\n---\nbody\n", Stamp{Origin: "jwt", WrittenBy: p})
require.NoError(t, err, p)
assert.Equal(t, p, fmOf(t, out)["written_by"], p)
}
}
// A group that does not decode on its own is dropped; its neighbours stay.
func TestApplyDropsGroupsThatDoNotDecodeAlone(t *testing.T) {
out, err := Apply("---\nkeep: 1\ntitle: a: b\nalso: 2\n---\nbody\n", Stamp{Origin: "static", WrittenBy: "local-cli"})
require.NoError(t, err)
m := fmOf(t, out)
assert.Equal(t, 1, m["keep"])
assert.Equal(t, 2, m["also"])
assert.NotContains(t, m, "title")
}
func TestApplyKeepsFirstOfDuplicateKeys(t *testing.T) {
out, err := Apply("---\ntitle: first\ntitle: second\nTitle: third\n---\nbody\n", Stamp{})
require.NoError(t, err)
m := fmOf(t, out)
assert.Equal(t, "first", m["title"])
assert.NotContains(t, m, "Title")
}
func TestApplyFoldsLoneCRSoItCannotHideALine(t *testing.T) {
out, err := Apply("---\ntitle: x\rorigin: static\n---\nbody\n", Stamp{Origin: "jwt", WrittenBy: "a"})
require.NoError(t, err)
assert.NotContains(t, out, "\r")
assert.Equal(t, "jwt", fmOf(t, out)["origin"])
assert.Equal(t, "x", fmOf(t, out)["title"])
}
func TestApplyDropsTabIndentedContinuations(t *testing.T) {
out, err := Apply("---\ntitle: x\n\torigin: static\nk: v\n---\nbody\n", Stamp{Origin: "jwt", WrittenBy: "a"})
require.NoError(t, err)
m := fmOf(t, out)
assert.Equal(t, "jwt", m["origin"])
assert.Equal(t, "v", m["k"])
}
// Content that opens a fence the shared splitter cannot close (e.g. a
// "---x" line) is refused rather than given a second, prepended block that
// demotes the original frontmatter into the body.
func TestApplyRefusesAnUnclosableFrontmatterFence(t *testing.T) {
_, err := Apply("---\ntitle: t\n---x\norigin: static\n---\nbody\n", Stamp{Origin: "jwt", WrittenBy: "a"})
assert.True(t, errors.Is(err, ErrUnverifiable), "got %v", err)
}
func TestApplyAddsFrontmatterToContentWithoutAny(t *testing.T) {
out, err := Apply("# plain\n\norigin: static is prose\n", Stamp{Origin: "static", WrittenBy: "local-cli"})
require.NoError(t, err)
assert.True(t, strings.HasPrefix(out, "---\norigin: static\nwritten_by: local-cli\n---\n# plain"))
}
-174
View File
@@ -1,174 +0,0 @@
package search
import (
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"syscall"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// hyperguild#157 option 1: search walked and read the brain by path, so a
// symlink anywhere in the Syncthing-shared brain dir (wiki/<w>/<h>/leak.md ->
// /any/file) put the target's contents in brain_query results (reproduced in
// the PR #156 review). Search now walks and reads through an os.Root on the
// brain dir and only reads regular files.
const sentinel157 = "zqsentinelonefiftyseven"
func searchBrain(t *testing.T) (brain, facts, outside string) {
t.Helper()
brain = t.TempDir()
facts = filepath.Join(brain, "wiki", "w", "facts")
require.NoError(t, os.MkdirAll(facts, 0o755))
require.NoError(t, os.MkdirAll(filepath.Join(brain, "knowledge"), 0o755))
outside = filepath.Join(t.TempDir(), "secret.md")
require.NoError(t, os.WriteFile(outside, []byte("# secret\n\n"+sentinel157), 0o644))
return brain, facts, outside
}
func leaked(rs []Result) int {
n := 0
for _, r := range rs {
n += strings.Count(r.Excerpt+r.Title, sentinel157)
}
return n
}
func TestQueryDoesNotFollowASymlinkInTheWiki(t *testing.T) {
brain, facts, outside := searchBrain(t)
require.NoError(t, os.Symlink(outside, filepath.Join(facts, "leak.md")))
require.NoError(t, os.WriteFile(filepath.Join(facts, "real.md"), []byte("# real\n\nzqrealnote"), 0o644))
rs, err := Query(brain, QueryOptions{Query: sentinel157, Limit: 10})
require.NoError(t, err)
assert.Zero(t, leaked(rs))
rs, err = Query(brain, QueryOptions{Query: "zqrealnote", Limit: 10})
require.NoError(t, err)
assert.Len(t, rs, 1, "regular notes are still found")
}
func TestQueryDoesNotFollowASymlinkedDirectory(t *testing.T) {
brain, _, outside := searchBrain(t)
require.NoError(t, os.Symlink(filepath.Dir(outside), filepath.Join(brain, "wiki", "linked")))
rs, err := Query(brain, QueryOptions{Query: sentinel157, Limit: 10})
require.NoError(t, err)
assert.Zero(t, leaked(rs))
}
// A search root that is a symlink staying inside the brain (wiki/<w> -> ..)
// was followed by fs.WalkDir, aliasing raw/ and root files into results
// (PR #166 review). The root is now Lstat-checked and skipped.
func TestQueryDoesNotFollowASymlinkedWingRootInsideTheBrain(t *testing.T) {
brain, _, _ := searchBrain(t)
require.NoError(t, os.MkdirAll(filepath.Join(brain, "raw"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(brain, "raw", "r.md"), []byte("# r\n\nzqrawonly"), 0o644))
require.NoError(t, os.Symlink("..", filepath.Join(brain, "wiki", "alias")))
rs, err := Query(brain, QueryOptions{Query: "zqrawonly", Wing: "alias", Limit: 10})
require.NoError(t, err)
assert.Empty(t, rs)
}
// With a hall the search root is wiki/<wing>/<hall>; a symlinked <wing> with
// a real <hall> directory under its target was followed (PR #166 round 2).
func TestQueryDoesNotFollowASymlinkedWingUnderAWingAndHallRoot(t *testing.T) {
brain, _, _ := searchBrain(t)
require.NoError(t, os.MkdirAll(filepath.Join(brain, "raw", "facts"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(brain, "raw", "facts", "r.md"), []byte("# r\n\nzqrawhall"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "raw"), filepath.Join(brain, "wiki", "alias")))
rs, err := Query(brain, QueryOptions{Query: "zqrawhall", Wing: "alias", Hall: "facts", Limit: 10})
require.NoError(t, err)
assert.Empty(t, rs)
}
// swapInto keeps renaming a regular note, and either a symlink to outside or
// a FIFO, into path. Each FIFO has a writer held open (as any local process
// can): a FIFO with no writer reads EOF at once, which would let a reader
// that skips the handle check pass (PR #161 round 2: with a held writer, a
// bypass of readOpened hung Query after 5 iterations).
// startSwap runs swapInto until the test ends and waits for it before
// t.TempDir's cleanup (hyperguild#162).
func startSwap(t *testing.T, path, outside string, fifo bool) {
t.Helper()
var stop atomic.Bool
var wg sync.WaitGroup
wg.Add(1)
go func() { defer wg.Done(); swapInto(path, outside, fifo, &stop) }()
t.Cleanup(func() { stop.Store(true); wg.Wait() })
}
func swapInto(path, outside string, fifo bool, stop *atomic.Bool) {
tmp := path + ".swap"
// Writers stay open until the swapper exits, so a reader parked on a
// FIFO stays parked (closing one would hand the reader EOF and hide the
// hang). Past the fd limit, new FIFOs simply have no writer.
var writers []*os.File
defer func() {
for _, w := range writers {
_ = w.Close()
}
}()
for i := 0; !stop.Load(); i++ {
_ = os.Remove(tmp)
switch {
case i%2 == 0:
_ = os.WriteFile(tmp, []byte("# r\n\nzqinside "+"zqracenote"), 0o644)
case fifo:
if err := syscall.Mkfifo(tmp, 0o644); err == nil {
if w, err := os.OpenFile(tmp, os.O_RDWR, 0); err == nil {
writers = append(writers, w)
}
}
default:
_ = os.Symlink(outside, tmp)
}
_ = os.Rename(tmp, path)
}
}
func TestQueryLeaksNothingUnderASymlinkSwapRace(t *testing.T) {
brain, facts, outside := searchBrain(t)
startSwap(t, filepath.Join(facts, "r.md"), outside, false)
leaks, iters := 0, 0
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); iters++ {
rs, _ := Query(brain, QueryOptions{Query: sentinel157, Limit: 10})
leaks += leaked(rs)
}
if leaks > 0 {
t.Fatalf("outside file leaked %d times in %d iterations", leaks, iters)
}
}
func TestQueryDoesNotHangUnderAFIFOSwapRace(t *testing.T) {
brain, facts, outside := searchBrain(t)
startSwap(t, filepath.Join(facts, "r.md"), outside, true)
done := make(chan struct{})
go func() {
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); {
_, _ = Query(brain, QueryOptions{Query: "zqracenote", Limit: 10})
}
close(done)
}()
select {
case <-done:
case <-time.After(6 * time.Second):
if f, err := os.OpenFile(filepath.Join(facts, "r.md"), os.O_WRONLY|syscall.O_NONBLOCK, 0); err == nil {
_ = f.Close()
}
t.Fatal("search hung on a swapped-in FIFO")
}
}
// hydrate (vector-only hits) reads by relative path; it must refuse a symlink.
func TestHydrateRefusesASymlink(t *testing.T) {
brain, facts, outside := searchBrain(t)
require.NoError(t, os.Symlink(outside, filepath.Join(facts, "leak.md")))
r, err := hydrate(brain, "wiki/w/facts/leak.md")
assert.Error(t, err)
assert.NotContains(t, r.Excerpt, sentinel157)
}
@@ -1,29 +0,0 @@
package search
import (
"testing"
"github.com/stretchr/testify/assert"
)
// Only the leading frontmatter is provenance; a body line that looks like a
// key is prose (the doc comment's claim, now observed).
func TestExtractProvenanceIgnoresBodyLines(t *testing.T) {
o, w := extractProvenance("---\ntitle: t\n---\norigin: static\nwritten_by: local-cli\n")
assert.Equal(t, "", o)
assert.Equal(t, "", w)
o, w = extractProvenance("# no frontmatter\norigin: static\n")
assert.Equal(t, "", o)
assert.Equal(t, "", w)
o, w = extractProvenance("---\norigin: jwt\nwritten_by: attacker\n---\norigin: static\n")
assert.Equal(t, "jwt", o)
assert.Equal(t, "attacker", w)
}
// Round 3 (B1): an indented "---" inside a block scalar is content, not the
// closing fence — the same rule as frontmatter.Split.
func TestExtractProvenanceUsesTheSharedFenceRule(t *testing.T) {
o, w := extractProvenance("---\nk: |\n ---\norigin: jwt\nwritten_by: attacker\n---\nbody\n")
assert.Equal(t, "jwt", o)
assert.Equal(t, "attacker", w)
}
+10 -74
View File
@@ -5,8 +5,6 @@ import (
"bufio" "bufio"
"context" "context"
"fmt" "fmt"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
sharedfm "github.com/mathiasbq/hyperguild/ingestion/internal/frontmatter"
"log/slog" "log/slog"
"os" "os"
"path/filepath" "path/filepath"
@@ -50,11 +48,6 @@ type Result struct {
// (infra#72). Read from frontmatter when present, otherwise // (infra#72). Read from frontmatter when present, otherwise
// inferred from the parent directory. // inferred from the parent directory.
Tier string `json:"tier,omitempty"` Tier string `json:"tier,omitempty"`
// Origin/WrittenBy surface write provenance (hyperguild#145) so a
// reader can discount an entry a connector session wrote. Omitted for
// notes written before provenance existed — never fabricated.
Origin string `json:"origin,omitempty"`
WrittenBy string `json:"written_by,omitempty"`
} }
// tierWeight maps the DIKW tier to a score multiplier applied right // tierWeight maps the DIKW tier to a score multiplier applied right
@@ -120,30 +113,20 @@ func QueryContext(ctx context.Context, brainDir string, opts QueryOptions) ([]Re
return nil, err return nil, err
} }
// Walk and read through an os.Root on the brain dir, regular files only
// (hyperguild#157): a symlink planted in the Syncthing-shared brain must
// not put an outside file in results, nor a FIFO hang the query.
bfs, err := brainfs.Open(brainDir)
if err != nil {
return nil, fmt.Errorf("search: %w", err)
}
defer func() { _ = bfs.Close() }()
var results []Result var results []Result
for _, dir := range roots { for _, dir := range roots {
if _, statErr := os.Stat(dir); os.IsNotExist(statErr) { if _, statErr := os.Stat(dir); os.IsNotExist(statErr) {
continue continue
} }
relDir, relErr := bfs.Rel(dir) err := filepath.WalkDir(dir, func(path string, d os.DirEntry, err error) error {
if relErr != nil { if err != nil {
return nil, fmt.Errorf("search: %w", relErr) slog.Warn("search: skipping path", "path", path, "err", err)
}
err := bfs.WalkRegular(relDir, func(relPath string) error {
if !strings.HasSuffix(relPath, ".md") {
return nil return nil
} }
path := filepath.Join(brainDir, filepath.FromSlash(relPath)) if d.IsDir() || !strings.HasSuffix(path, ".md") {
content, err := bfs.ReadRegular(relPath) return nil
}
content, err := os.ReadFile(path)
if err != nil { if err != nil {
slog.Warn("search: skipping unreadable file", "path", path, "err", err) slog.Warn("search: skipping unreadable file", "path", path, "err", err)
return nil return nil
@@ -163,22 +146,16 @@ func QueryContext(ctx context.Context, brainDir string, opts QueryOptions) ([]Re
rel = filepath.ToSlash(rel) rel = filepath.ToSlash(rel)
wing, hall := extractWingHall(string(content), rel) wing, hall := extractWingHall(string(content), rel)
tier := extractTier(string(content), rel) tier := extractTier(string(content), rel)
origin, writtenBy := extractProvenance(string(content))
results = append(results, Result{ results = append(results, Result{
Path: rel, Path: rel,
Title: extractTitle(string(content), filepath.Base(relPath)), Title: extractTitle(string(content), d.Name()),
Excerpt: excerpt(string(content), 300), Excerpt: excerpt(string(content), 300),
Score: score, Score: score,
Wing: wing, Wing: wing,
Hall: hall, Hall: hall,
Tier: tier, Tier: tier,
Origin: origin,
WrittenBy: writtenBy,
}) })
return nil return nil
}, func(p string, err error) {
slog.Warn("search: skipping path", "path", p, "err", err)
}) })
if err != nil { if err != nil {
return nil, err return nil, err
@@ -288,19 +265,13 @@ func pathInScope(relPath, wing, hall string) bool {
// excerpt, wing, and hall populated. Used for paths that surface only // excerpt, wing, and hall populated. Used for paths that surface only
// via vector search. // via vector search.
func hydrate(brainDir, relPath string) (Result, error) { func hydrate(brainDir, relPath string) (Result, error) {
// Through the brain root, regular files only (hyperguild#157). full := filepath.Join(brainDir, filepath.FromSlash(relPath))
bfs, err := brainfs.Open(brainDir) content, err := os.ReadFile(full)
if err != nil {
return Result{}, err
}
defer func() { _ = bfs.Close() }()
content, err := bfs.ReadRegular(filepath.ToSlash(relPath))
if err != nil { if err != nil {
return Result{}, err return Result{}, err
} }
wing, hall := extractWingHall(string(content), relPath) wing, hall := extractWingHall(string(content), relPath)
tier := extractTier(string(content), relPath) tier := extractTier(string(content), relPath)
origin, writtenBy := extractProvenance(string(content))
return Result{ return Result{
Path: relPath, Path: relPath,
Title: extractTitle(string(content), filepath.Base(relPath)), Title: extractTitle(string(content), filepath.Base(relPath)),
@@ -308,44 +279,9 @@ func hydrate(brainDir, relPath string) (Result, error) {
Wing: wing, Wing: wing,
Hall: hall, Hall: hall,
Tier: tier, Tier: tier,
Origin: origin,
WrittenBy: writtenBy,
}, nil }, nil
} }
// extractProvenance reads origin/written_by from the leading frontmatter
// block only, located with the shared fence rule (frontmatter.Split: a "---"
// at column 0) so an indented "---" inside a block scalar is content, not a
// closing fence (PR #147 review round 3). A body line is never provenance.
func extractProvenance(content string) (origin, writtenBy string) {
block, _, ok := sharedfm.Split([]byte(content))
if !ok {
return "", ""
}
for _, line := range strings.Split(string(block), "\n") {
if line == "" || line[0] == ' ' || line[0] == '\t' {
continue
}
key, val, found := strings.Cut(line, ":")
if !found {
continue
}
v := strings.Trim(strings.TrimSpace(val), `"'`)
switch strings.TrimSpace(key) {
case "origin":
if origin == "" {
origin = v
}
case "written_by":
if writtenBy == "" {
writtenBy = v
}
}
}
return origin, writtenBy
}
// resolveRoots returns the directories to walk for the given wing/hall // resolveRoots returns the directories to walk for the given wing/hall
// filters. Validates hall against the closed vocabulary when set. // filters. Validates hall against the closed vocabulary when set.
func resolveRoots(brainDir, wing, hall string) ([]string, error) { func resolveRoots(brainDir, wing, hall string) ([]string, error) {
-92
View File
@@ -1,92 +0,0 @@
package session
import (
"errors"
"os"
"path/filepath"
"syscall"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// session_log passed the caller's session_id straight into
// filepath.Join(sessionsDir, id+".jsonl") and opened it O_CREATE|O_APPEND:
// path segments in the id created or appended files anywhere the pod could
// write. The id is now a plain name, and the open does not follow a final
// symlink.
func TestAppendRefusesSessionIDsThatAreNotPlainNames(t *testing.T) {
root := t.TempDir()
dir := filepath.Join(root, "sessions")
for _, id := range []string{"../x", "a/b", "/abs", ".", "..", ".hidden", "", "a\\b", "a\x00b"} {
assert.Error(t, Append(dir, id, Entry{SessionID: id}), "%q", id)
}
ents, _ := os.ReadDir(root)
for _, e := range ents {
assert.Equal(t, "sessions", e.Name(), "nothing created beside the sessions dir")
}
for _, id := range []string{"1557b2e1-bf0d-5c4f-bbe3-0754d087e356", "_routing", "2026-05-05-infra.plan", "0"} {
require.NoError(t, Append(dir, id, Entry{SessionID: id}), "positive control: %q", id)
}
}
func TestAppendDoesNotWriteThroughASymlinkedLog(t *testing.T) {
root := t.TempDir()
dir := filepath.Join(root, "sessions")
require.NoError(t, os.MkdirAll(dir, 0o755))
victim := filepath.Join(root, "victim.md")
require.NoError(t, os.WriteFile(victim, []byte("untouched\n"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "victim.md"), filepath.Join(dir, "s.jsonl")))
assert.Error(t, Append(dir, "s", Entry{SessionID: "s"}))
got, _ := os.ReadFile(victim)
assert.Equal(t, "untouched\n", string(got))
}
func TestReadRefusesSessionIDsThatAreNotPlainNames(t *testing.T) {
root := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(root, "x.jsonl"), []byte(`{"session_id":"x"}`+"\n"), 0o644))
_, err := Read(filepath.Join(root, "sessions"), "../x")
assert.Error(t, err)
}
// A name that starts plainly and then climbs out with ".." is exactly what the
// ban on "/" in the id's tail stops: filepath.Join cleans "a/../../x" to a
// path beside the sessions dir without needing sessions/a to exist. The
// assertion is on the refusal itself (ErrInvalidID), not on any error.
func TestAppendRefusesAnIDThatClimbsOutAfterAPlainPrefix(t *testing.T) {
root := t.TempDir()
dir := filepath.Join(root, "sessions")
require.NoError(t, os.MkdirAll(dir, 0o755))
for _, id := range []string{"a/../../escape", "a/../../../escape", "a/../b"} {
err := Append(dir, id, Entry{SessionID: id})
assert.ErrorIs(t, err, ErrInvalidID, "%q", id)
}
_, err := os.Lstat(filepath.Join(root, "escape.jsonl"))
assert.True(t, errors.Is(err, os.ErrNotExist), "nothing created beside the sessions dir")
_, err = os.Lstat(filepath.Join(dir, "b.jsonl"))
assert.True(t, errors.Is(err, os.ErrNotExist), "no in-dir alias created either")
}
// Append does no Lstat type check, so O_NONBLOCK is the only thing that keeps
// a FIFO named <id>.jsonl with no reader from blocking session_log forever.
func TestAppendReturnsPromptlyOnAFIFOLogWithNoReader(t *testing.T) {
dir := filepath.Join(t.TempDir(), "sessions")
require.NoError(t, os.MkdirAll(dir, 0o755))
fifo := filepath.Join(dir, "s.jsonl")
require.NoError(t, syscall.Mkfifo(fifo, 0o644))
done := make(chan error, 1)
go func() { done <- Append(dir, "s", Entry{SessionID: "s"}) }()
select {
case err := <-done:
assert.Error(t, err, "a FIFO with no reader is refused, not written")
case <-time.After(3 * time.Second):
// Unblock the stuck open so the goroutine and TempDir cleanup finish.
if r, err := os.OpenFile(fifo, os.O_RDONLY|syscall.O_NONBLOCK, 0); err == nil {
defer r.Close() //nolint:errcheck
}
<-done
t.Fatal("Append blocked on a FIFO log with no reader")
}
}
+1 -25
View File
@@ -9,8 +9,6 @@ import (
"io/fs" "io/fs"
"os" "os"
"path/filepath" "path/filepath"
"regexp"
"syscall"
"time" "time"
) )
@@ -44,32 +42,13 @@ type Attempt struct {
RunnerOutput string `json:"runner_output,omitempty"` RunnerOutput string `json:"runner_output,omitempty"`
} }
// idPattern is a plain file name: session_log takes the id from the caller,
// and a path segment in it created or appended files anywhere the pod could
// write (hyperguild#169).
var idPattern = regexp.MustCompile(`^[A-Za-z0-9_][A-Za-z0-9._-]{0,127}$`)
// ErrInvalidID is returned when a session id is not a plain file name.
var ErrInvalidID = errors.New("session id is not a plain name")
func checkID(sessionID string) error {
if !idPattern.MatchString(sessionID) {
return fmt.Errorf("%w: %q", ErrInvalidID, sessionID)
}
return nil
}
// Append writes entry as a single JSON line to sessionsDir/{sessionID}.jsonl. // Append writes entry as a single JSON line to sessionsDir/{sessionID}.jsonl.
// The open does not follow a final symlink.
func Append(sessionsDir, sessionID string, entry Entry) error { func Append(sessionsDir, sessionID string, entry Entry) error {
if err := checkID(sessionID); err != nil {
return err
}
if err := os.MkdirAll(sessionsDir, 0o755); err != nil { if err := os.MkdirAll(sessionsDir, 0o755); err != nil {
return fmt.Errorf("create sessions dir: %w", err) return fmt.Errorf("create sessions dir: %w", err)
} }
path := filepath.Join(sessionsDir, sessionID+".jsonl") path := filepath.Join(sessionsDir, sessionID+".jsonl")
f, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0o644) f, err := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o644)
if err != nil { if err != nil {
return fmt.Errorf("open session log: %w", err) return fmt.Errorf("open session log: %w", err)
} }
@@ -91,9 +70,6 @@ func Append(sessionsDir, sessionID string, entry Entry) error {
// Read returns all entries for sessionID. Returns empty slice if no log exists. // Read returns all entries for sessionID. Returns empty slice if no log exists.
func Read(sessionsDir, sessionID string) ([]Entry, error) { func Read(sessionsDir, sessionID string) ([]Entry, error) {
if err := checkID(sessionID); err != nil {
return nil, err
}
path := filepath.Join(sessionsDir, sessionID+".jsonl") path := filepath.Join(sessionsDir, sessionID+".jsonl")
f, err := os.Open(path) f, err := os.Open(path)
if errors.Is(err, fs.ErrNotExist) { if errors.Is(err, fs.ErrNotExist) {
@@ -1,264 +0,0 @@
// Package symlinkguard is the hyperguild#157 tripwire. The brain directory is
// a Syncthing sendreceive folder and Syncthing syncs symlinks; several brain
// readers (search, vectorstore, context, graph) follow a symlink out of the
// brain and would index or return whatever it points to. Until every reader
// reads through an os.Root, a symlink inside the brain, or a directory the
// scan cannot read (which could hide one), is an incident: the server refuses
// to start on one (main), and Watch reports one that appears while it runs.
package symlinkguard
import (
"context"
"errors"
"fmt"
"io/fs"
"log/slog"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"time"
)
// DefaultInterval is used when Watch is given a non-positive interval.
const DefaultInterval = 30 * time.Second
// Alert bounds (PR #160 review): the planter chooses the paths, so neither an
// alert's size nor the queue may grow with what was planted.
const (
maxPathsPerAlert = 10 // further paths are counted, not listed
maxPending = 5 // queued alerts; the rest become one summary
)
// Notifier is the out-of-band alert sink (audit.Notifier satisfies it).
type Notifier interface {
Notify(ctx context.Context, msg string) error
}
// Result is what a scan found: symlinks, and directories it could not read
// (each of which could hide a symlink). Paths are brain-relative.
type Result struct {
Symlinks []string
Unreadable []string
}
// Clean reports whether the scan found nothing to act on.
func (r Result) Clean() bool { return len(r.Symlinks) == 0 && len(r.Unreadable) == 0 }
// String describes the findings for a log line or alert. Paths are quoted, so
// a name containing a newline cannot blur two findings together.
func (r Result) String() string {
q := r.quoted
var parts []string
if len(r.Symlinks) > 0 {
parts = append(parts, fmt.Sprintf("%d symlink(s): %s", len(r.Symlinks), q(r.Symlinks)))
}
if len(r.Unreadable) > 0 {
parts = append(parts, fmt.Sprintf("%d unreadable dir(s): %s", len(r.Unreadable), q(r.Unreadable)))
}
return strings.Join(parts, "; ")
}
// quoted joins at most maxPathsPerAlert paths, each strconv-quoted, and
// counts the rest.
func (Result) quoted(xs []string) string {
shown := xs
if len(shown) > maxPathsPerAlert {
shown = shown[:maxPathsPerAlert]
}
out := make([]string, len(shown))
for i, x := range shown {
out[i] = strconv.Quote(x)
}
s := strings.Join(out, ", ")
if extra := len(xs) - len(shown); extra > 0 {
s += fmt.Sprintf(" and %d more", extra)
}
return s
}
// Scan walks the brain dir and reports every symlink in it and every
// directory it could not read, continuing past both. Only the ROOT .git is
// skipped (it is brain-sync's, and no reader walks it); a nested .git is
// scanned like any other directory. brainDir itself may be a symlink (a
// volume mount); it is resolved first. The error is non-nil when the root
// cannot be resolved (wrapping os.ErrNotExist when it is missing) or cannot be
// listed.
func Scan(brainDir string) (Result, error) {
root, err := filepath.EvalSymlinks(brainDir)
if err != nil {
return Result{}, fmt.Errorf("resolve brain dir: %w", err)
}
var r Result
rootGit := filepath.Join(root, ".git")
walkErr := filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
rel, _ := filepath.Rel(root, p)
rel = filepath.ToSlash(rel)
if err != nil {
if p == root {
return err
}
r.Unreadable = append(r.Unreadable, rel)
if d != nil && d.IsDir() {
return filepath.SkipDir
}
return nil
}
if d.IsDir() && p == rootGit {
return filepath.SkipDir
}
if d.Type()&os.ModeSymlink != 0 {
r.Symlinks = append(r.Symlinks, rel)
}
return nil
})
sort.Strings(r.Symlinks)
sort.Strings(r.Unreadable)
if walkErr != nil {
// Only the root's own error reaches here (deeper errors are recorded
// as Unreadable). A root that cannot be listed cannot be shown clean
// (PR #159 round 2: discarding this reported clean while search
// served a planted symlink).
return r, fmt.Errorf("scan brain dir: %w", walkErr)
}
return r, nil
}
// Watch rescans every interval (DefaultInterval if non-positive) until ctx
// ends. Every symlink found is removed (the link, never its target:
// hyperguild#157 decision), so one is readable for at most one interval.
// Each removal is an alert; a persistent finding (a link it could not remove,
// an unreadable dir, a scan error) alerts once per distinct state. Alerts
// queue until Notify succeeds (see alertQueue): an outage delays them, and
// any that do not fit the bounded queue, or are undeliverable, are counted in
// a summary that is never dropped, so every incident is delivered or counted.
func Watch(ctx context.Context, brainDir string, interval time.Duration, n Notifier) {
if interval <= 0 {
interval = DefaultInterval
}
t := time.NewTicker(interval)
defer t.Stop()
q := &alertQueue{}
reported := ""
for {
select {
case <-ctx.Done():
return
case <-t.C:
}
r, err := Scan(brainDir)
key := ""
if err != nil {
key = "cannot scan the brain dir: " + err.Error()
} else {
removed, failed := removeSymlinks(brainDir, r.Symlinks)
if len(removed) > 0 {
slog.Error("brain symlink tripwire removed symlinks", "count", len(removed))
q.push(fmt.Sprintf("brain symlink tripwire (hyperguild#157): removed %d symlink(s) from the brain dir: %s",
len(removed), Result{}.quoted(removed)))
}
key = Result{Symlinks: failed, Unreadable: r.Unreadable}.String()
}
if key != reported {
reported = key
if key != "" {
slog.Error("brain symlink tripwire", "finding", key)
q.push("brain symlink tripwire (hyperguild#157): " + key +
". Readers may follow symlinks out of the brain; fix them.")
}
}
q.flush(ctx, n)
}
}
// alertQueue holds alerts until Notify accepts them. It is bounded (at most
// maxPending queued) and loses nothing from the record: every overflow and
// every dropped message increments suppressed, sent as a summary that is
// retried every tick and never dropped.
//
// A message is dropped only when the notifier rejects it permanently (an
// error with Permanent() == true, e.g. ntfy 413). Every other failure is an
// outage, and nothing is dropped however long it lasts (PR #160 round 2:
// dropping after N raw failures lost the only incident of a long outage).
// Each flush tries every queued message, so one undeliverable message cannot
// block the others.
type alertQueue struct {
items []string
suppressed int
}
func (q *alertQueue) push(msg string) {
if len(q.items) >= maxPending {
q.suppressed++
return
}
q.items = append(q.items, msg)
}
func (q *alertQueue) flush(ctx context.Context, n Notifier) {
if n == nil {
q.items, q.suppressed = nil, 0
return
}
kept := q.items[:0]
for _, msg := range q.items {
err := n.Notify(ctx, msg)
switch {
case err == nil:
case permanent(err):
slog.Error("symlink tripwire alert rejected permanently; counted in the summary", "err", err)
q.suppressed++
default:
slog.Warn("symlink tripwire notify failed; will retry", "err", err)
kept = append(kept, msg)
}
}
q.items = kept
if q.suppressed > 0 {
msg := fmt.Sprintf("brain symlink tripwire (hyperguild#157): %d further alert(s) were suppressed or undeliverable; check the ingestion logs", q.suppressed)
if err := n.Notify(ctx, msg); err == nil {
q.suppressed = 0
}
}
}
// permanent reports whether err says the message itself can never be sent.
func permanent(err error) bool {
var p interface{ Permanent() bool }
return errors.As(err, &p) && p.Permanent()
}
// removeSymlinks unlinks each brain-relative symlink through an os.Root on
// the brain dir, after re-checking it is still a symlink. Removing a symlink
// never touches its target. It returns what was removed and what was not.
func removeSymlinks(brainDir string, rels []string) (removed, failed []string) {
if len(rels) == 0 {
return nil, nil
}
resolved, err := filepath.EvalSymlinks(brainDir)
if err != nil {
return nil, rels
}
root, err := os.OpenRoot(resolved)
if err != nil {
return nil, rels
}
defer func() { _ = root.Close() }()
for _, rel := range rels {
fi, err := root.Lstat(rel)
if err != nil || fi.Mode()&os.ModeSymlink == 0 {
failed = append(failed, rel)
continue
}
// root.Remove, not os.RemoveAll: if the link is swapped for a
// non-empty directory after the Lstat, Remove refuses rather than
// deleting it recursively, and it cannot act outside the root.
if err := root.Remove(rel); err != nil {
failed = append(failed, rel)
continue
}
removed = append(removed, rel)
}
return removed, failed
}
@@ -1,498 +0,0 @@
package symlinkguard
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// hyperguild#157: the brain dir is a Syncthing sendreceive folder and
// Syncthing syncs symlinks; several readers follow them out of the brain.
// Until every reader goes through an os.Root, a symlink anywhere in the brain,
// or a directory the scan cannot read (which could hide one), is an incident.
func TestScanReportsSymlinksAtEveryDepthIncludingNestedGit(t *testing.T) {
brain := t.TempDir()
outside := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki", "w", "facts", ".git"), 0o755))
require.NoError(t, os.MkdirAll(filepath.Join(brain, ".git", "objects"), 0o755))
require.NoError(t, os.Symlink(filepath.Join(outside, "f"), filepath.Join(brain, "wiki", "w", "facts", "leak.md")))
require.NoError(t, os.Symlink(outside, filepath.Join(brain, "wiki", "dirlink")))
// PR #159 review: only the ROOT .git is skipped; a nested .git is not
// special to the brain readers, which walk into it.
require.NoError(t, os.Symlink(filepath.Join(outside, "g"), filepath.Join(brain, "wiki", "w", "facts", ".git", "leak.md")))
require.NoError(t, os.Symlink(outside, filepath.Join(brain, ".git", "objects", "alt")))
r, err := Scan(brain)
require.NoError(t, err)
assert.ElementsMatch(t, []string{"wiki/w/facts/leak.md", "wiki/dirlink", "wiki/w/facts/.git/leak.md"}, r.Symlinks)
assert.Empty(t, r.Unreadable)
}
// PR #159 review: WalkDir stopped at the first unreadable directory and the
// startup check passed, while search (which keeps walking) served a planted
// symlink. Scan now records the unreadable dir and keeps going.
func TestScanReportsUnreadableDirsAndKeepsWalkingPastThem(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root reads mode-000 directories")
}
brain := t.TempDir()
locked := filepath.Join(brain, "aaa")
require.NoError(t, os.MkdirAll(locked, 0o755))
require.NoError(t, os.Chmod(locked, 0o000))
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "zzz-planted")))
r, err := Scan(brain)
require.NoError(t, err)
assert.Equal(t, []string{"aaa"}, r.Unreadable)
assert.Equal(t, []string{"zzz-planted"}, r.Symlinks, "the walk continues past the unreadable dir")
}
func TestScanResolvesASymlinkedRootAndReportsSymlinkedEntriesInsideIt(t *testing.T) {
real := t.TempDir()
link := filepath.Join(t.TempDir(), "brain")
require.NoError(t, os.Symlink(real, link))
require.NoError(t, os.WriteFile(filepath.Join(real, "n.md"), []byte("x"), 0o644))
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(real, "inner")))
r, err := Scan(link)
require.NoError(t, err)
assert.Equal(t, []string{"inner"}, r.Symlinks)
}
func TestScanOfAMissingRootIsErrNotExist(t *testing.T) {
_, err := Scan(filepath.Join(t.TempDir(), "nope"))
assert.True(t, errors.Is(err, os.ErrNotExist), "got %v", err)
}
type recNotifier struct {
mu sync.Mutex
msgs []string
fail int // fail the first N calls
}
func (r *recNotifier) Notify(_ context.Context, m string) error {
r.mu.Lock()
defer r.mu.Unlock()
r.msgs = append(r.msgs, m)
if r.fail > 0 {
r.fail--
return errors.New("ntfy down")
}
return nil
}
func (r *recNotifier) count() int { r.mu.Lock(); defer r.mu.Unlock(); return len(r.msgs) }
func TestWatchNotifiesOncePerDistinctSetAndAgainForANewSet(t *testing.T) {
brain := t.TempDir()
n := &recNotifier{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, n)
time.Sleep(50 * time.Millisecond)
assert.Zero(t, n.count(), "a clean brain raises nothing")
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "first")))
require.Eventually(t, func() bool { return n.count() == 1 }, 2*time.Second, 10*time.Millisecond)
time.Sleep(60 * time.Millisecond)
assert.Equal(t, 1, n.count(), "the same set is reported once, not every tick")
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "second")))
require.Eventually(t, func() bool { return n.count() == 2 }, 2*time.Second, 10*time.Millisecond)
}
// PR #159 review: last was recorded before Notify, so an alert lost to an
// ntfy outage was never retried while the symlink stayed.
func TestWatchRetriesAnAlertThatFailedToSend(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "planted")))
n := &recNotifier{fail: 2}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, n)
require.Eventually(t, func() bool { return n.count() == 3 }, 2*time.Second, 10*time.Millisecond)
time.Sleep(60 * time.Millisecond)
assert.Equal(t, 3, n.count(), "after one successful send, no more")
}
func TestWatchAlertsOnAnUnreadableDir(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root reads mode-000 directories")
}
brain := t.TempDir()
locked := filepath.Join(brain, "locked")
require.NoError(t, os.MkdirAll(locked, 0o755))
require.NoError(t, os.Chmod(locked, 0o000))
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
n := &recNotifier{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, n)
require.Eventually(t, func() bool { return n.count() == 1 }, 2*time.Second, 10*time.Millisecond)
assert.Contains(t, n.msgs[0], "locked")
}
// PR #159 review: a non-positive interval panicked time.NewTicker inside the
// goroutine and crashlooped the server.
func TestWatchToleratesANonPositiveInterval(t *testing.T) {
brain := t.TempDir()
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { Watch(ctx, brain, 0, nil); close(done) }()
cancel()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("Watch did not return on cancel")
}
}
// PR #159 round 2: a brain root that can be entered but not listed (mode
// 0100) made WalkDir fail at the root; the error was discarded and Scan
// reported clean, while search (which goes straight into wiki/) still served
// a planted symlink. A root that cannot be listed is now an error.
func TestScanOfAnUnlistableRootIsAnError(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root lists mode-0100 directories")
}
brain := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki"), 0o755))
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "wiki", "leak.md")))
require.NoError(t, os.Chmod(brain, 0o100))
t.Cleanup(func() { _ = os.Chmod(brain, 0o755) })
_, err := Scan(brain)
require.Error(t, err)
assert.False(t, errors.Is(err, os.ErrNotExist), "an unlistable root is not a missing root")
}
// The same set that disappears and comes back is reported again.
func TestWatchReportsASetAgainAfterItWasCleared(t *testing.T) {
brain := t.TempDir()
link := filepath.Join(brain, "planted")
n := &recNotifier{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, n)
require.NoError(t, os.Symlink(t.TempDir(), link))
require.Eventually(t, func() bool { return n.count() == 1 }, 2*time.Second, 10*time.Millisecond)
// The tripwire removed it; planting it again is a new event.
require.Eventually(t, func() bool { _, err := os.Lstat(link); return os.IsNotExist(err) }, 2*time.Second, 10*time.Millisecond)
require.NoError(t, os.Symlink(t.TempDir(), link))
require.Eventually(t, func() bool { return n.count() == 2 }, 2*time.Second, 10*time.Millisecond)
}
// hyperguild#157 decision: the runtime tripwire removes a symlink it finds
// (the link, never its target), so it is readable for at most one interval
// instead of until a human acts, and the alert says what was removed.
func TestWatchRemovesAnOutsideFileLinkButNotItsTargetOrRegularFiles(t *testing.T) {
brain := t.TempDir()
outside := filepath.Join(t.TempDir(), "target.md")
require.NoError(t, os.WriteFile(outside, []byte("keep me"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki"), 0o755))
link := filepath.Join(brain, "wiki", "leak.md")
require.NoError(t, os.Symlink(outside, link))
regular := filepath.Join(brain, "wiki", "note.md")
require.NoError(t, os.WriteFile(regular, []byte("note"), 0o644))
n := &recNotifier{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, n)
require.Eventually(t, func() bool {
_, err := os.Lstat(link)
return os.IsNotExist(err)
}, 2*time.Second, 10*time.Millisecond, "the symlink is removed")
require.Eventually(t, func() bool { return n.count() >= 1 }, 2*time.Second, 10*time.Millisecond)
b, err := os.ReadFile(outside)
require.NoError(t, err)
assert.Equal(t, "keep me", string(b), "the target is untouched")
b, err = os.ReadFile(regular)
require.NoError(t, err)
assert.Equal(t, "note", string(b), "regular files are untouched")
n.mu.Lock()
defer n.mu.Unlock()
assert.Contains(t, n.msgs[0], "removed")
assert.Contains(t, n.msgs[0], "wiki/leak.md")
}
func TestDefaultIntervalIsThirtySeconds(t *testing.T) {
assert.Equal(t, 30*time.Second, DefaultInterval)
}
// PR #160 review: the Lstat re-check and the os.Root confinement in
// removeSymlinks had no test. Anything that is not a symlink inside the brain
// is reported failed and nothing is deleted.
func TestRemoveSymlinksRefusesNonSymlinksEscapesAndPathsThroughLinks(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki", "real"), 0o755))
regular := filepath.Join(brain, "wiki", "note.md")
require.NoError(t, os.WriteFile(regular, []byte("note"), 0o644))
inLinked := filepath.Join(brain, "wiki", "real", "x.md")
require.NoError(t, os.WriteFile(inLinked, []byte("x"), 0o644))
require.NoError(t, os.Symlink(filepath.Join(brain, "wiki", "real"), filepath.Join(brain, "wiki", "linkdir")))
outside := filepath.Join(filepath.Dir(brain), "outside-157.md")
require.NoError(t, os.WriteFile(outside, []byte("o"), 0o644))
t.Cleanup(func() { _ = os.Remove(outside) })
removed, failed := removeSymlinks(brain, []string{
"wiki/note.md", "wiki/linkdir/x.md", "../outside-157.md", "wiki/missing.md",
})
assert.Empty(t, removed)
assert.ElementsMatch(t, []string{"wiki/note.md", "wiki/linkdir/x.md", "../outside-157.md", "wiki/missing.md"}, failed)
for _, p := range []string{regular, inLinked, outside} {
_, err := os.Stat(p)
assert.NoError(t, err, "%s must not be deleted", p)
}
}
// Dir targets and in-brain targets survive too, not only an outside file.
func TestWatchRemovesDirAndInBrainLinksButNotTheirTargets(t *testing.T) {
brain := t.TempDir()
outsideDir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(outsideDir, "keep.md"), []byte("keep"), 0o644))
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki"), 0o755))
inBrain := filepath.Join(brain, "wiki", "target.md")
require.NoError(t, os.WriteFile(inBrain, []byte("t"), 0o644))
require.NoError(t, os.Symlink(outsideDir, filepath.Join(brain, "wiki", "dirlink")))
require.NoError(t, os.Symlink(inBrain, filepath.Join(brain, "wiki", "inlink.md")))
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, nil)
require.Eventually(t, func() bool {
r, err := Scan(brain)
return err == nil && len(r.Symlinks) == 0
}, 2*time.Second, 10*time.Millisecond)
for _, p := range []string{filepath.Join(outsideDir, "keep.md"), inBrain} {
_, err := os.Stat(p)
assert.NoError(t, err, "%s must survive", p)
}
}
// PR #160 review: the planter chooses the paths. An alert names at most
// maxPathsPerAlert of them and counts the rest, so its size stays bounded.
func TestWatchAlertNamesBoundedPathsAndCountsTheRest(t *testing.T) {
brain := t.TempDir()
for i := 0; i < 50; i++ {
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, strings.Repeat("n", 100)+strconv.Itoa(i))))
}
n := &recNotifier{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, n)
require.Eventually(t, func() bool { return n.count() >= 1 }, 2*time.Second, 10*time.Millisecond)
n.mu.Lock()
msg := n.msgs[0]
n.mu.Unlock()
assert.Contains(t, msg, "removed 50 symlink(s)")
assert.Contains(t, msg, "and 40 more")
assert.Less(t, len(msg), 2500, "alert size is bounded regardless of how many links were planted")
}
// rejectNotifier refuses every message containing a marker, forever.
type rejectNotifier struct {
recNotifier
reject string
}
// permanentErr is a rejection the notifier says can never succeed (like ntfy 413).
type permanentErr struct{}
func (permanentErr) Error() string { return "413 payload too large" }
func (permanentErr) Permanent() bool { return true }
func (r *rejectNotifier) Notify(ctx context.Context, m string) error {
if strings.Contains(m, r.reject) {
r.mu.Lock()
r.msgs = append(r.msgs, "REJECTED")
r.mu.Unlock()
return permanentErr{}
}
return r.recNotifier.Notify(ctx, m)
}
// PR #160 review: one message the server will never accept blocked every
// later alert forever (head-of-line). It is dropped after maxAttempts.
func TestWatchDropsAMessageThatNeverSendsSoLaterAlertsFlow(t *testing.T) {
brain := t.TempDir()
n := &rejectNotifier{reject: "POISON"}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 10*time.Millisecond, n)
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "POISON")))
time.Sleep(30 * time.Millisecond)
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "later")))
require.Eventually(t, func() bool {
n.mu.Lock()
defer n.mu.Unlock()
for _, m := range n.msgs {
if strings.Contains(m, "later") {
return true
}
}
return false
}, 3*time.Second, 10*time.Millisecond, "a later alert is delivered despite a poison message")
}
// toggleNotifier fails while down is set.
type toggleNotifier struct {
recNotifier
down atomic.Bool
}
func (r *toggleNotifier) Notify(ctx context.Context, m string) error {
if r.down.Load() {
return errors.New("ntfy down")
}
return r.recNotifier.Notify(ctx, m)
}
// PR #160 review: an outage queued one message per removal, unbounded. The
// queue holds at most maxPending messages; the rest become one summary.
func TestWatchQueueIsBoundedDuringAnOutage(t *testing.T) {
brain := t.TempDir()
n := &toggleNotifier{}
n.down.Store(true)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 5*time.Millisecond, n)
for i := 0; i < 30; i++ {
link := filepath.Join(brain, "p"+strconv.Itoa(i))
require.NoError(t, os.Symlink(t.TempDir(), link))
require.Eventually(t, func() bool { _, err := os.Lstat(link); return os.IsNotExist(err) }, 2*time.Second, 2*time.Millisecond)
}
n.down.Store(false)
require.Eventually(t, func() bool { return n.count() >= 1 }, 2*time.Second, 5*time.Millisecond)
time.Sleep(100 * time.Millisecond)
assert.LessOrEqual(t, n.count(), maxPending+1, "queued alerts plus at most one suppression summary")
n.mu.Lock()
defer n.mu.Unlock()
found := false
for _, m := range n.msgs {
if strings.Contains(m, "suppressed") {
found = true
}
}
assert.True(t, found, "suppressed alerts are counted in a summary")
}
// Within the bound, every alert queued during an outage is delivered after
// it, each naming its own removal (PR #160 review: truncating the queue after
// the first send stayed green).
func TestWatchDeliversEveryQueuedAlertAfterAShortOutage(t *testing.T) {
brain := t.TempDir()
n := &toggleNotifier{}
n.down.Store(true)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 5*time.Millisecond, n)
for _, name := range []string{"qa", "qb", "qc"} {
link := filepath.Join(brain, name)
require.NoError(t, os.Symlink(t.TempDir(), link))
require.Eventually(t, func() bool { _, err := os.Lstat(link); return os.IsNotExist(err) }, 2*time.Second, 2*time.Millisecond)
}
n.down.Store(false)
require.Eventually(t, func() bool { return n.count() == 3 }, 2*time.Second, 5*time.Millisecond)
n.mu.Lock()
defer n.mu.Unlock()
for i, name := range []string{"qa", "qb", "qc"} {
assert.Contains(t, n.msgs[i], strconv.Quote(name))
}
}
// PR #160 round 2: dropping after N failures could not tell a poison message
// from an outage, so a 15-minute outage silently lost the only incident. A
// message is only dropped when it keeps failing while others send; through
// an outage nothing is dropped, however long.
func TestWatchLosesNoAlertThroughALongOutage(t *testing.T) {
brain := t.TempDir()
n := &toggleNotifier{}
n.down.Store(true)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 2*time.Millisecond, n)
link := filepath.Join(brain, "only-incident")
require.NoError(t, os.Symlink(t.TempDir(), link))
require.Eventually(t, func() bool { _, err := os.Lstat(link); return os.IsNotExist(err) }, 2*time.Second, 2*time.Millisecond)
time.Sleep(200 * time.Millisecond) // ~100 ticks of outage, far past any attempt limit
n.down.Store(false)
require.Eventually(t, func() bool { return n.count() >= 1 }, 2*time.Second, 5*time.Millisecond)
n.mu.Lock()
defer n.mu.Unlock()
assert.Contains(t, n.msgs[0], strconv.Quote("only-incident"))
}
// Every removal is either delivered or counted in the suppression summary:
// none disappears from the record (round 2: 74 of 300 were in neither).
func TestWatchAccountsForEveryRemovalDuringAnOutage(t *testing.T) {
brain := t.TempDir()
n := &toggleNotifier{}
n.down.Store(true)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 2*time.Millisecond, n)
const plants = 25
for i := 0; i < plants; i++ {
link := filepath.Join(brain, "acct"+strconv.Itoa(i))
require.NoError(t, os.Symlink(t.TempDir(), link))
require.Eventually(t, func() bool { _, err := os.Lstat(link); return os.IsNotExist(err) }, 2*time.Second, 1*time.Millisecond)
}
n.down.Store(false)
require.Eventually(t, func() bool {
n.mu.Lock()
defer n.mu.Unlock()
for _, m := range n.msgs {
if strings.Contains(m, "suppressed") {
return true
}
}
return false
}, 2*time.Second, 5*time.Millisecond)
time.Sleep(50 * time.Millisecond)
n.mu.Lock()
defer n.mu.Unlock()
delivered, counted := 0, 0
for _, m := range n.msgs {
if strings.Contains(m, "removed 1 symlink(s)") {
delivered++
}
if i := strings.Index(m, "): "); i >= 0 && strings.Contains(m, "suppressed") {
_, _ = fmt.Sscanf(m[i+3:], "%d", &counted)
}
}
assert.Equal(t, plants, delivered+counted, "delivered %d + counted %d must equal %d removals", delivered, counted, plants)
}
// A poison message is still dropped (others send while it fails), and the
// drop is counted in the summary.
func TestWatchDropsAPoisonMessageAndCountsIt(t *testing.T) {
brain := t.TempDir()
n := &rejectNotifier{reject: "POISON"}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go Watch(ctx, brain, 5*time.Millisecond, n)
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "POISON")))
time.Sleep(20 * time.Millisecond)
require.NoError(t, os.Symlink(t.TempDir(), filepath.Join(brain, "fine")))
require.Eventually(t, func() bool {
n.mu.Lock()
defer n.mu.Unlock()
for _, m := range n.msgs {
if strings.Contains(m, "1 further alert(s)") {
return true
}
}
return false
}, 3*time.Second, 5*time.Millisecond, "the dropped poison alert is counted in a summary")
}
+17 -32
View File
@@ -2,15 +2,12 @@ package vectorstore
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"io/fs"
"log/slog" "log/slog"
"path" "os"
"path/filepath"
"strings" "strings"
"time" "time"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
) )
// Embedder produces dense vectors. The embed package's Client satisfies // Embedder produces dense vectors. The embed package's Client satisfies
@@ -93,38 +90,31 @@ func Sync(ctx context.Context, brainDir string, store Store, embedder Embedder)
} }
seenParents := make(map[string]struct{}) seenParents := make(map[string]struct{})
// Through the brain root, regular files only (hyperguild#157): the brain
// is a Syncthing folder, and os.ReadFile followed a synced symlink into
// the embeddings and blocked on a FIFO.
bfs, err := brainfs.Open(brainDir)
if err != nil {
return res, fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
for _, sub := range scanDirs { for _, sub := range scanDirs {
if _, err := bfs.Lstat(sub); errors.Is(err, fs.ErrNotExist) { root := filepath.Join(brainDir, sub)
if _, err := os.Stat(root); os.IsNotExist(err) {
continue continue
} }
// Any other walk error, including a scan root that is a symlink,
// aborts the run below. err = filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
var walkErr error if err != nil {
onErr := func(rel string, err error) { return err
if walkErr == nil {
walkErr = fmt.Errorf("%s: %w", rel, err)
} }
} if d.IsDir() || !strings.HasSuffix(path, ".md") || d.Name() == "_index.md" {
err = bfs.WalkRegular(sub, func(relSlash string) error {
if !strings.HasSuffix(relSlash, ".md") || path.Base(relSlash) == "_index.md" {
return nil return nil
} }
rel, err := filepath.Rel(brainDir, path)
if err != nil {
return err
}
relSlash := filepath.ToSlash(rel)
seenParents[relSlash] = struct{}{} seenParents[relSlash] = struct{}{}
if ps, ok := parents[relSlash]; ok { if ps, ok := parents[relSlash]; ok {
// File already has chunks in the store. Re-embed only when // File already has chunks in the store. Re-embed only when
// the file has been edited since the oldest chunk was // the file has been edited since the oldest chunk was
// written. Tolerate clock skew with a sub-second grace. // written. Tolerate clock skew with a sub-second grace.
info, statErr := bfs.Lstat(relSlash) info, statErr := d.Info()
if statErr != nil { if statErr != nil {
res.Errors = append(res.Errors, fmt.Errorf("stat %s: %w", relSlash, statErr)) res.Errors = append(res.Errors, fmt.Errorf("stat %s: %w", relSlash, statErr))
return nil return nil
@@ -142,7 +132,7 @@ func Sync(ctx context.Context, brainDir string, store Store, embedder Embedder)
} }
} }
content, readErr := bfs.ReadRegular(relSlash) content, readErr := os.ReadFile(path)
if readErr != nil { if readErr != nil {
res.Errors = append(res.Errors, fmt.Errorf("read %s: %w", relSlash, readErr)) res.Errors = append(res.Errors, fmt.Errorf("read %s: %w", relSlash, readErr))
return nil return nil
@@ -161,12 +151,7 @@ func Sync(ctx context.Context, brainDir string, store Store, embedder Embedder)
res.Added++ res.Added++
} }
return nil return nil
}, onErr) })
if err == nil {
err = walkErr
}
// Abort before the delete pass: an incomplete walk would leave
// seenParents short and delete chunks of files that still exist.
if err != nil { if err != nil {
return res, fmt.Errorf("walk %s: %w", sub, err) return res, fmt.Errorf("walk %s: %w", sub, err)
} }
@@ -1,126 +0,0 @@
package vectorstore_test
import (
"context"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs/brainfstest"
"github.com/mathiasbq/hyperguild/ingestion/internal/vectorstore"
)
// recordingEmbedder keeps every text it was asked to embed.
type recordingEmbedder struct {
mu sync.Mutex
texts []string
}
func (e *recordingEmbedder) Embed(_ context.Context, text string) ([]float32, error) {
e.mu.Lock()
defer e.mu.Unlock()
e.texts = append(e.texts, text)
return []float32{1}, nil
}
func (e *recordingEmbedder) all() string {
e.mu.Lock()
defer e.mu.Unlock()
return strings.Join(e.texts, "\n")
}
// hyperguild#157 slice 3: Sync walked wiki/ and knowledge/ and read each .md
// with os.ReadFile, so a symlinked note had its target embedded and served by
// semantic search, and a FIFO stalled the sync.
func TestSyncRefusesSymlinkedNotesAndSurvivesSwapRaces(t *testing.T) {
brain := t.TempDir()
writeNote(t, brain, "wiki/w/facts/ok.md", "inside note")
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(brain, "wiki", "w", "facts", "leak.md")))
emb := &recordingEmbedder{}
store := &stubStore{}
_, err := vectorstore.Sync(context.Background(), brain, store, emb)
require.NoError(t, err)
assert.Contains(t, emb.all(), "inside note", "positive control: the regular note is embedded")
assert.NotContains(t, emb.all(), brainfstest.Sentinel)
for p := range store.upserts {
assert.NotContains(t, p, "leak.md")
}
p := filepath.Join(brain, "knowledge", "r.md")
require.NoError(t, os.MkdirAll(filepath.Dir(p), 0o755))
sync := func() string {
e := &recordingEmbedder{}
_, _ = vectorstore.Sync(context.Background(), brain, &stubStore{}, e)
return e.all()
}
brainfstest.AssertNoLeak(t, p, "inside", 1500*time.Millisecond, sync)
brainfstest.AssertNoHang(t, p, "inside", 1500*time.Millisecond, 6*time.Second, func() { _ = sync() })
}
// Chunks embedded from a symlinked note before this fix are deleted: the
// note is no longer seen, so the delete pass drops its rows.
func TestSyncDeletesChunksOfASymlinkedNote(t *testing.T) {
brain := t.TempDir()
writeNote(t, brain, "wiki/w/facts/ok.md", "inside note")
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(brain, "wiki", "w", "facts", "leak.md")))
store := &stubStore{known: map[string]time.Time{
"wiki/w/facts/leak.md#0001": {},
"wiki/w/facts/ok.md#0001": {},
}}
_, err := vectorstore.Sync(context.Background(), brain, store, &recordingEmbedder{})
require.NoError(t, err)
assert.Equal(t, []string{"wiki/w/facts/leak.md#0001"}, store.deletes)
}
// A walk error aborts Sync before the delete pass, so chunks of files the
// walk could not see are kept.
func TestSyncAbortsBeforeDeletingWhenASubdirIsUnreadable(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root ignores directory permissions")
}
brain := t.TempDir()
writeNote(t, brain, "wiki/w/facts/ok.md", "inside note")
locked := filepath.Join(brain, "wiki", "w")
require.NoError(t, os.Chmod(locked, 0o000))
t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
store := &stubStore{known: map[string]time.Time{"wiki/w/facts/ok.md#0001": {}}}
_, err := vectorstore.Sync(context.Background(), brain, store, &recordingEmbedder{})
assert.Error(t, err)
assert.Empty(t, store.deletes)
}
// A scan root that is a symlink inside the brain (knowledge -> .) was
// followed by fs.WalkDir, aliasing raw/, sessions/ and root files into the
// index. Sync now aborts on it, before embedding or deleting anything.
func TestSyncAbortsOnASymlinkedScanRoot(t *testing.T) {
brain := t.TempDir()
writeNote(t, brain, "wiki/w/a.md", "wiki note")
writeNote(t, brain, "sessions/s1.md", "session note")
require.NoError(t, os.Symlink(".", filepath.Join(brain, "knowledge")))
emb := &recordingEmbedder{}
store := &stubStore{known: map[string]time.Time{"wiki/w/a.md#0001": {}}}
_, err := vectorstore.Sync(context.Background(), brain, store, emb)
assert.Error(t, err)
assert.NotContains(t, emb.all(), "session note")
for p := range store.upserts {
assert.False(t, strings.HasPrefix(p, "knowledge/"), "upserted %s", p)
}
assert.Empty(t, store.deletes)
}
// A missing scan root is skipped; the other root still syncs.
func TestSyncSkipsAMissingScanRoot(t *testing.T) {
brain := t.TempDir()
writeNote(t, brain, "wiki/w/a.md", "wiki note")
emb := &recordingEmbedder{}
_, err := vectorstore.Sync(context.Background(), brain, &stubStore{}, emb)
require.NoError(t, err)
assert.Contains(t, emb.all(), "wiki note")
}
+26 -58
View File
@@ -48,25 +48,8 @@ func Start(ctx context.Context, cfg Config) {
func processDir(ctx context.Context, cfg Config, date string) []error { func processDir(ctx context.Context, cfg Config, date string) []error {
rawDir := filepath.Join(cfg.BrainDir, "raw") rawDir := filepath.Join(cfg.BrainDir, "raw")
// Every read goes through an os.Root on raw/ (hyperguild#153): raw/ is
// writable via Syncthing, and a file swapped for a symlink between the walk
// and the read must not lead out of it. A symlinked raw/ is refused.
if fi, lerr := os.Lstat(rawDir); lerr != nil {
if os.IsNotExist(lerr) {
return nil
}
return []error{fmt.Errorf("stat raw dir: %w", lerr)}
} else if fi.Mode()&os.ModeSymlink != 0 || !fi.IsDir() {
return []error{fmt.Errorf("raw dir %s is not a real directory; refusing to ingest from it", rawDir)}
}
root, err := os.OpenRoot(rawDir)
if err != nil {
return []error{fmt.Errorf("open raw root: %w", err)}
}
defer func() { _ = root.Close() }()
var errs []error var errs []error
err = filepath.WalkDir(rawDir, func(path string, d os.DirEntry, err error) error { err := filepath.WalkDir(rawDir, func(path string, d os.DirEntry, err error) error {
if err != nil { if err != nil {
return err return err
} }
@@ -103,10 +86,6 @@ func processDir(ctx context.Context, cfg Config, date string) []error {
if strings.HasPrefix(d.Name(), "tunnel-candidates-") { if strings.HasPrefix(d.Name(), "tunnel-candidates-") {
return nil return nil
} }
// Only regular files are notes; a symlink's target may be anywhere.
if !d.Type().IsRegular() {
return nil
}
// Skip files that have already been processed or permanently failed. // Skip files that have already been processed or permanently failed.
if _, err := os.Stat(path + ".processed"); err == nil { if _, err := os.Stat(path + ".processed"); err == nil {
@@ -116,7 +95,7 @@ func processDir(ctx context.Context, cfg Config, date string) []error {
return nil return nil
} }
if err := processFile(ctx, cfg, root, rawDir, path, date); err != nil { if err := processFile(ctx, cfg, path, date); err != nil {
errs = append(errs, fmt.Errorf("process %s: %w", filepath.Base(path), err)) errs = append(errs, fmt.Errorf("process %s: %w", filepath.Base(path), err))
} }
return nil return nil
@@ -130,15 +109,11 @@ func processDir(ctx context.Context, cfg Config, date string) []error {
// processFile reads a file, calls pipeline.Run, copies it to processed/ or failed/, // processFile reads a file, calls pipeline.Run, copies it to processed/ or failed/,
// and writes a marker file next to the original so the watcher skips it next poll. // and writes a marker file next to the original so the watcher skips it next poll.
// The original file is never deleted, keeping Syncthing-connected vaults (e.g. Obsidian) intact. // The original file is never deleted, keeping Syncthing-connected vaults (e.g. Obsidian) intact.
func processFile(ctx context.Context, cfg Config, root *os.Root, rawDir, path, date string) error { func processFile(ctx context.Context, cfg Config, path, date string) error {
filename := filepath.Base(path) filename := filepath.Base(path)
source := deriveSource(filename) source := deriveSource(filename)
rel, err := filepath.Rel(rawDir, path)
if err != nil {
return fmt.Errorf("relative path: %w", err)
}
content, err := extract.TextIn(root, rel) content, err := extract.Text(path)
if err != nil { if err != nil {
return fmt.Errorf("extract text: %w", err) return fmt.Errorf("extract text: %w", err)
} }
@@ -151,7 +126,7 @@ func processFile(ctx context.Context, cfg Config, root *os.Root, rawDir, path, d
return fmt.Errorf("mkdir failed dir: %w", mkErr) return fmt.Errorf("mkdir failed dir: %w", mkErr)
} }
dest := filepath.Join(failedDir, filename) dest := filepath.Join(failedDir, filename)
if cpErr := copyFromRoot(root, rel, dest); cpErr != nil { if cpErr := copyFile(path, dest); cpErr != nil {
return fmt.Errorf("copy to failed: %w", cpErr) return fmt.Errorf("copy to failed: %w", cpErr)
} }
if mkErr := os.WriteFile(path+".failed", []byte(runErr.Error()), 0o644); mkErr != nil { if mkErr := os.WriteFile(path+".failed", []byte(runErr.Error()), 0o644); mkErr != nil {
@@ -179,7 +154,7 @@ func processFile(ctx context.Context, cfg Config, root *os.Root, rawDir, path, d
base := strings.TrimSuffix(filename, ext) base := strings.TrimSuffix(filename, ext)
dest = filepath.Join(processedDir, base+"-"+time.Now().UTC().Format("150405")+ext) dest = filepath.Join(processedDir, base+"-"+time.Now().UTC().Format("150405")+ext)
} }
if err := copyFromRoot(root, rel, dest); err != nil { if err := copyFile(path, dest); err != nil {
return fmt.Errorf("copy to processed: %w", err) return fmt.Errorf("copy to processed: %w", err)
} }
if err := os.WriteFile(path+".processed", []byte(date), 0o644); err != nil { if err := os.WriteFile(path+".processed", []byte(date), 0o644); err != nil {
@@ -190,6 +165,26 @@ func processFile(ctx context.Context, cfg Config, root *os.Root, rawDir, path, d
return nil return nil
} }
// copyFile copies src to dst, creating dst if it doesn't exist.
func copyFile(src, dst string) error {
in, err := os.Open(src)
if err != nil {
return fmt.Errorf("open src: %w", err)
}
defer in.Close() //nolint:errcheck
out, err := os.Create(dst)
if err != nil {
return fmt.Errorf("create dst: %w", err)
}
if _, err := io.Copy(out, in); err != nil {
out.Close() //nolint:errcheck
return fmt.Errorf("copy: %w", err)
}
return out.Close()
}
// deriveSource turns a filename into a human-readable source name. // deriveSource turns a filename into a human-readable source name.
// "shape-up-book.md" → "Shape Up Book" // "shape-up-book.md" → "Shape Up Book"
func deriveSource(filename string) string { func deriveSource(filename string) string {
@@ -226,30 +221,3 @@ func appendWatcherLog(brainDir, filename string, runErr error, date string) erro
} }
return f.Close() return f.Close()
} }
// copyFromRoot archives a raw/ file read through the root, refusing anything
// that is not a regular file, so a swapped-in symlink cannot copy an outside
// file into raw/processed/ or raw/failed/ (hyperguild#153).
func copyFromRoot(root *os.Root, rel, dst string) error {
fi, err := root.Lstat(rel)
if err != nil {
return err
}
if !fi.Mode().IsRegular() {
return fmt.Errorf("%s: not a regular file", rel)
}
in, _, err := extract.OpenRegular(root, rel)
if err != nil {
return err
}
defer func() { _ = in.Close() }()
out, err := os.Create(dst)
if err != nil {
return err
}
if _, err := io.Copy(out, in); err != nil {
_ = out.Close()
return err
}
return out.Close()
}
-108
View File
@@ -7,10 +7,6 @@ import (
"fmt" "fmt"
"os" "os"
"path/filepath" "path/filepath"
"strings"
"sync"
"sync/atomic"
"syscall"
"testing" "testing"
"time" "time"
@@ -286,107 +282,3 @@ func TestProcessDir_SkipsTunnelCandidateFiles(t *testing.T) {
_, err = os.Stat(filepath.Join(brainDir, "raw", "processed", date, "tunnel-candidates-2026-07-19.md")) _, err = os.Stat(filepath.Join(brainDir, "raw", "processed", date, "tunnel-candidates-2026-07-19.md"))
assert.True(t, os.IsNotExist(err), "tunnel-candidates file should not be copied to processed/") assert.True(t, os.IsNotExist(err), "tunnel-candidates file should not be copied to processed/")
} }
// hyperguild#153: raw/ is writable via Syncthing, and the watcher ingests
// whatever lands there with no API call at all. WalkDir yields symlinked
// files, and extract.Text followed them, so a symlink to any pod-readable
// file would be extracted into the brain. The watcher now ingests only
// regular files, read through an os.Root on raw/.
func TestProcessDir_IngestsTheRealNoteAndSkipsTheSymlinkBesideIt(t *testing.T) {
brainDir := setupBrainDir(t)
outside := filepath.Join(t.TempDir(), "secret.md")
require.NoError(t, os.WriteFile(outside, []byte("SENTINEL-153-WATCHER"), 0o644))
// "a-" sorts before "real": a walk that stopped at the first symlink
// (filepath.SkipAll — one synced link halting all ingestion) would fail
// the call count below (PR #156 review).
require.NoError(t, os.Symlink(outside, filepath.Join(brainDir, "raw", "a-leak.md")))
require.NoError(t, os.WriteFile(filepath.Join(brainDir, "raw", "real.md"), []byte("real note"), 0o644))
var sawSentinel bool
calls := 0
cfg := Config{BrainDir: brainDir, Interval: time.Hour, Pipeline: pipeline.Config{
Complete: func(_ context.Context, _, user string) (string, error) {
calls++
if strings.Contains(user, "SENTINEL-153-WATCHER") {
sawSentinel = true
}
b, _ := json.Marshal([]pipeline.RawPage{{Title: "X", Type: "source", Subtype: "article", Content: "## S\n\nx\n"}})
return string(b), nil
},
Schema: "# Schema",
}}
errs := processDir(context.Background(), cfg, time.Now().UTC().Format("2006-01-02"))
assert.Empty(t, errs)
assert.False(t, sawSentinel, "the symlink target must never reach the pipeline")
assert.Equal(t, 1, calls, "exactly the real note is ingested; the symlink is not a note")
}
// A symlinked raw/ is refused outright: os.OpenRoot would otherwise follow
// it and confine reads to wherever it points.
func TestProcessDir_RefusesASymlinkedRawDir(t *testing.T) {
brainDir := t.TempDir()
elsewhere := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(elsewhere, "x.md"), []byte("SENTINEL-153-RAWLINK"), 0o644))
require.NoError(t, os.Symlink(elsewhere, filepath.Join(brainDir, "raw")))
calls := 0
cfg := Config{BrainDir: brainDir, Interval: time.Hour, Pipeline: pipeline.Config{
Complete: func(context.Context, string, string) (string, error) { calls++; return "[]", nil },
Schema: "# Schema",
}}
errs := processDir(context.Background(), cfg, time.Now().UTC().Format("2006-01-02"))
assert.NotEmpty(t, errs, "a symlinked raw/ must be reported, not silently ingested")
assert.Zero(t, calls)
}
// PR #156: a FIFO swapped into raw/ between the watcher's checks and its
// read or archive copy must not hang the watcher, which runs unattended.
func TestProcessDir_DoesNotHangUnderAFIFOSwapRace(t *testing.T) {
brainDir := setupBrainDir(t)
raw := filepath.Join(brainDir, "raw")
var stop atomic.Bool
var wg sync.WaitGroup
wg.Add(1)
// Awaited before t.TempDir's cleanup (hyperguild#162).
t.Cleanup(func() { stop.Store(true); wg.Wait() })
go func() {
defer wg.Done()
for i := 0; !stop.Load(); i++ {
tmp := filepath.Join(raw, ".swap")
_ = os.Remove(tmp)
if i%2 == 0 {
_ = os.WriteFile(tmp, []byte("x"), 0o644)
} else {
_ = syscall.Mkfifo(tmp, 0o644)
}
_ = os.Rename(tmp, filepath.Join(raw, "r.md"))
}
}()
cfg := Config{BrainDir: brainDir, Interval: time.Hour, Pipeline: pipeline.Config{
Complete: func(context.Context, string, string) (string, error) {
b, _ := json.Marshal([]pipeline.RawPage{{Title: "R", Type: "source", Subtype: "article", Content: "## S\n\nx\n"}})
return string(b), nil
},
Schema: "# Schema",
}}
var iters atomic.Int64
done := make(chan struct{})
go func() {
for deadline := time.Now().Add(1500 * time.Millisecond); time.Now().Before(deadline); {
_ = os.Remove(filepath.Join(raw, "r.md.processed"))
_ = os.Remove(filepath.Join(raw, "r.md.failed"))
_ = processDir(context.Background(), cfg, "2026-10-03")
iters.Add(1)
}
close(done)
}()
select {
case <-done:
stop.Store(true)
case <-time.After(8 * time.Second):
stop.Store(true)
if f, err := os.OpenFile(filepath.Join(raw, "r.md"), os.O_WRONLY|syscall.O_NONBLOCK, 0); err == nil {
_ = f.Close()
}
t.Fatalf("the watcher hung on a swapped-in FIFO after %d iterations", iters.Load())
}
}
+4 -11
View File
@@ -7,7 +7,6 @@ import (
"path/filepath" "path/filepath"
"strings" "strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"github.com/mathiasbq/hyperguild/ingestion/internal/textutil" "github.com/mathiasbq/hyperguild/ingestion/internal/textutil"
) )
@@ -17,11 +16,6 @@ func RebuildIndex(brainDir, date string) error {
if err != nil { if err != nil {
return fmt.Errorf("load inventory: %w", err) return fmt.Errorf("load inventory: %w", err)
} }
bfs, err := brainfs.Open(brainDir)
if err != nil {
return fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
total := len(inv[PageTypeConcept]) + len(inv[PageTypeEntity]) + len(inv[PageTypeSource]) total := len(inv[PageTypeConcept]) + len(inv[PageTypeEntity]) + len(inv[PageTypeSource])
var sb strings.Builder var sb strings.Builder
@@ -40,7 +34,7 @@ func RebuildIndex(brainDir, date string) error {
label := strings.ToUpper(string(pt)[:1]) + string(pt)[1:] label := strings.ToUpper(string(pt)[:1]) + string(pt)[1:]
fmt.Fprintf(&sb, "## %s\n\n", label) fmt.Fprintf(&sb, "## %s\n\n", label)
for _, e := range entries { for _, e := range entries {
summary := pageFirstSentence(bfs, e) summary := pageFirstSentence(brainDir, e)
if summary != "" { if summary != "" {
fmt.Fprintf(&sb, "- [[%s|%s]] — %s\n", e.Slug, e.Title, summary) fmt.Fprintf(&sb, "- [[%s|%s]] — %s\n", e.Slug, e.Title, summary)
} else { } else {
@@ -54,10 +48,9 @@ func RebuildIndex(brainDir, date string) error {
return os.WriteFile(dest, []byte(sb.String()), 0o644) return os.WriteFile(dest, []byte(sb.String()), 0o644)
} }
// pageFirstSentence reads the page through the brain root, regular files func pageFirstSentence(brainDir string, e Entry) string {
// only (hyperguild#157). path := filepath.Join(brainDir, "wiki", string(e.Type), e.Slug+".md")
func pageFirstSentence(bfs *brainfs.Brain, e Entry) string { content, err := os.ReadFile(path)
content, err := bfs.ReadRegular("wiki/" + string(e.Type) + "/" + e.Slug + ".md")
if err != nil { if err != nil {
return "" return ""
} }
+15 -23
View File
@@ -3,45 +3,36 @@ package wiki
import ( import (
"bufio" "bufio"
"bytes"
"errors"
"fmt" "fmt"
"io/fs" "os"
"path/filepath"
"strings" "strings"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
) )
// LoadInventory walks brain/wiki/ and returns all pages grouped by type. // LoadInventory walks brain/wiki/ and returns all pages grouped by type.
// Missing subdirectories are silently skipped. Pages are listed and read // Missing subdirectories are silently skipped.
// through the brain root, regular files only (hyperguild#157); a type
// directory that is a symlink is an error.
func LoadInventory(brainDir string) (map[PageType][]Entry, error) { func LoadInventory(brainDir string) (map[PageType][]Entry, error) {
result := map[PageType][]Entry{ result := map[PageType][]Entry{
PageTypeConcept: {}, PageTypeConcept: {},
PageTypeEntity: {}, PageTypeEntity: {},
PageTypeSource: {}, PageTypeSource: {},
} }
bfs, err := brainfs.Open(brainDir)
if err != nil {
return nil, fmt.Errorf("open brain: %w", err)
}
defer func() { _ = bfs.Close() }()
for pt := range result { for pt := range result {
dir := "wiki/" + string(pt) dir := filepath.Join(brainDir, "wiki", string(pt))
names, err := bfs.ReadDirRegular(dir) entries, err := os.ReadDir(dir)
if errors.Is(err, fs.ErrNotExist) { if os.IsNotExist(err) {
continue continue
} }
if err != nil { if err != nil {
return nil, fmt.Errorf("read dir %s: %w", dir, err) return nil, fmt.Errorf("read dir %s: %w", dir, err)
} }
for _, name := range names { for _, e := range entries {
if !strings.HasSuffix(name, ".md") { if e.IsDir() || !strings.HasSuffix(e.Name(), ".md") {
continue continue
} }
slug := strings.TrimSuffix(name, ".md") slug := strings.TrimSuffix(e.Name(), ".md")
title, aliases := readFrontmatter(bfs, dir+"/"+name, slug) path := filepath.Join(dir, e.Name())
title, aliases := readFrontmatter(path, slug)
result[pt] = append(result[pt], Entry{Slug: slug, Title: title, Aliases: aliases, Type: pt}) result[pt] = append(result[pt], Entry{Slug: slug, Title: title, Aliases: aliases, Type: pt})
} }
} }
@@ -50,14 +41,15 @@ func LoadInventory(brainDir string) (map[PageType][]Entry, error) {
// readFrontmatter extracts title and aliases from YAML frontmatter. // readFrontmatter extracts title and aliases from YAML frontmatter.
// Falls back to slug for title and empty aliases on any error. // Falls back to slug for title and empty aliases on any error.
func readFrontmatter(bfs *brainfs.Brain, rel, fallbackSlug string) (title string, aliases []string) { func readFrontmatter(path, fallbackSlug string) (title string, aliases []string) {
title = fallbackSlug title = fallbackSlug
content, err := bfs.ReadRegular(rel) f, err := os.Open(path)
if err != nil { if err != nil {
return return
} }
defer f.Close() //nolint:errcheck
scanner := bufio.NewScanner(bytes.NewReader(content)) scanner := bufio.NewScanner(f)
inFM := false inFM := false
inAliases := false inAliases := false
for scanner.Scan() { for scanner.Scan() {
-79
View File
@@ -1,79 +0,0 @@
package wiki
import (
"fmt"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs"
"github.com/mathiasbq/hyperguild/ingestion/internal/brainfs/brainfstest"
)
// hyperguild#157 slice 4: LoadInventory listed wiki/<type>/ with os.ReadDir
// and opened each .md by path, so a symlinked page put the target's title and
// aliases into the inventory (and the LLM prompts built from it), and a FIFO
// hung it. RebuildIndex read first sentences the same way.
func TestLoadInventoryRefusesSymlinksAndSurvivesSwapRaces(t *testing.T) {
brain := t.TempDir()
dir := filepath.Join(brain, "wiki", "concepts")
require.NoError(t, os.MkdirAll(dir, 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "ok.md"), []byte("---\ntitle: zqinside\n---\n"), 0o644))
require.NoError(t, os.Symlink(brainfstest.Outside(t), filepath.Join(dir, "leak.md")))
inBrain := brainfstest.WriteSentinelAt(t, filepath.Join(brain, "raw", "s.md"))
require.NoError(t, os.Symlink(filepath.Join("..", "..", "raw", "s.md"), filepath.Join(dir, "inlink.md")))
inv, err := LoadInventory(brain)
require.NoError(t, err)
got := fmt.Sprintf("%+v", inv)
assert.Contains(t, got, "zqinside", "positive control")
assert.NotContains(t, got, brainfstest.Sentinel)
assert.NotContains(t, got, "leak", "a symlinked page is not listed")
assert.NotContains(t, got, "inlink", "a page linked to another brain file is not listed")
p := filepath.Join(dir, "r.md")
read := func() string {
inv, _ := LoadInventory(brain)
return fmt.Sprintf("%+v", inv)
}
brainfstest.AssertNoLeak(t, p, "---\ntitle: zqinside\n---\n", 1500*time.Millisecond, read)
brainfstest.AssertNoLeakFrom(t, p, "---\ntitle: zqinside\n---\n", inBrain, 1500*time.Millisecond, read)
brainfstest.AssertNoHang(t, p, "---\ntitle: zqinside\n---\n", 1500*time.Millisecond, 6*time.Second, func() { _ = read() })
}
func TestLoadInventoryReturnsErrNotRegularForASymlinkedTypeDir(t *testing.T) {
brain := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(brain, "wiki"), 0o755))
require.NoError(t, os.MkdirAll(filepath.Join(brain, "raw"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(brain, "raw", "r.md"), []byte("---\ntitle: zqrawonly\n---\n"), 0o644))
require.NoError(t, os.Symlink(filepath.Join("..", "raw"), filepath.Join(brain, "wiki", "concepts")))
inv, err := LoadInventory(brain)
require.ErrorIs(t, err, brainfs.ErrNotRegular)
assert.NotContains(t, fmt.Sprintf("%+v", inv), "zqrawonly")
}
func TestRebuildIndexDoesNotReadAPageSwappedForASymlink(t *testing.T) {
brain := t.TempDir()
dir := filepath.Join(brain, "wiki", "concepts")
require.NoError(t, os.MkdirAll(dir, 0o755))
p := filepath.Join(dir, "r.md")
require.NoError(t, os.WriteFile(p, []byte("---\ntitle: R\n---\nzqinside sentence.\n"), 0o644))
require.NoError(t, RebuildIndex(brain, "2026-10-04"))
idx, err := os.ReadFile(filepath.Join(brain, "wiki", "index.md"))
require.NoError(t, err)
require.Contains(t, string(idx), "zqinside", "positive control: the summary is read")
read := func() string {
_ = RebuildIndex(brain, "2026-10-04")
b, _ := os.ReadFile(filepath.Join(brain, "wiki", "index.md"))
return string(b)
}
brainfstest.AssertNoLeak(t, p, "---\ntitle: R\n---\nzqinside sentence.\n", 1500*time.Millisecond, read)
inBrain := brainfstest.WriteSentinelAt(t, filepath.Join(brain, "raw", "s.md"))
brainfstest.AssertNoLeakFrom(t, p, "---\ntitle: R\n---\nzqinside sentence.\n", inBrain, 1500*time.Millisecond, read)
brainfstest.AssertNoHang(t, p, "---\ntitle: R\n---\nzqinside sentence.\n", 1500*time.Millisecond, 6*time.Second, func() { _ = read() })
}
-5
View File
@@ -13,7 +13,6 @@ import (
"io" "io"
"log/slog" "log/slog"
"net/http" "net/http"
"os"
"strings" "strings"
) )
@@ -43,10 +42,6 @@ func Query(ctx context.Context, baseURL, query string, limit int) (string, error
return "", nil return "", nil
} }
req.Header.Set("Content-Type", "application/json") req.Header.Set("Content-Type", "application/json")
// The REST API requires a bearer token since hyperguild#149.
if tok := os.Getenv("BRAIN_MCP_TOKEN"); tok != "" {
req.Header.Set("Authorization", "Bearer "+tok)
}
resp, err := http.DefaultClient.Do(req) resp, err := http.DefaultClient.Do(req)
if err != nil { if err != nil {
-15
View File
@@ -65,18 +65,3 @@ func TestQueryUnavailableServerReturnsEmpty(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
assert.Empty(t, result) assert.Empty(t, result)
} }
// hyperguild#149: the REST API requires a bearer token.
func TestQuerySendsBrainMCPToken(t *testing.T) {
var auth string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
auth = r.Header.Get("Authorization")
_, _ = w.Write([]byte(`{"results":[]}`))
}))
defer srv.Close()
t.Setenv("BRAIN_MCP_TOKEN", "lib-test-token")
_, _ = brain.Query(context.Background(), srv.URL, "x", 1)
if auth != "Bearer lib-test-token" {
t.Fatalf("Authorization = %q, want the bearer token", auth)
}
}
+1 -1
View File
@@ -58,7 +58,7 @@ if [ "$FORCE" = false ]; then
while IFS= read -r _match; do while IFS= read -r _match; do
_lineno="${_match%%:*}" _lineno="${_match%%:*}"
_content="${_match#*:}" _content="${_match#*:}"
_token=$(head -1 < <(grep -o '{{[^}]*}}' <<<"$_content")) _token=$(printf '%s' "$_content" | grep -o '{{[^}]*}}' | head -1)
echo " $PROJECT_FILE:$_lineno: unfilled placeholder $_token" >&2 echo " $PROJECT_FILE:$_lineno: unfilled placeholder $_token" >&2
done <<< "$_placeholders" done <<< "$_placeholders"
echo "" >&2 echo "" >&2